I'm attempting to set up an OpsWorks stack for an app. I currently have the app and the infrastructure in the same repo, with the following structure:
proj_name/
infrastructure/
chef-repo/
cookbooks/
proj_name/ # THE COOKBOOK
recipes/
deploy.rb
configure.rb
attributes/
metadata.rb
proj_name/ # THE APP
app/
migrations/
manage.py
I have confirmed that OpsWorks is successfully pulling the repo from Github and installing it in /opt/aws/opsworks. However, when I try to add the proj_name::deploy recipe to the custom recipes section of a custom layer, I get an error message saying that proj_name::deploy could not be found. Looking at the log, I see a line saying INFO: Storing updated cookbooks/proj_name/requirements.txt in the cache. This says to me that OpsWorks is looking in the first proj_name directory (the one containing the app) to find the recipe, not the cookbook named proj_name inside of infrastructure/chef-repo/cookbooks.
Is there any way to tell OpsWorks to look further for the cookbook?
Thanks!
http://docs.aws.amazon.com/opsworks/latest/userguide/workingcookbook-installingcustom-repo.html
If using just cookbooks the answer is no.
If you have a berksfile you could get away with just a top-level berksfile in which you put the path to your cookbooks.
Bottom-line: you will have to place something in the root of the repo.
Related
We are using multiple python deployments into a single GitHub repository with a folder structure. Each directory contains a separate scripts module.
service-1/
deployment-1/
app/
Dockerfile
cloudbuild.yaml
deployment-2/
app/
Dockerfile
cloudbuild.yaml
service-2/
deployment-1/
app/
Dockerfile
cloudbuild.yaml
service-3/
deployment-1/
app/
Dockerfile
cloudbuild.yaml
deployment-2/
app/
Dockerfile
cloudbuild.yaml
.gitignore
README.md
requirements.txt
where deployment-1 will work as a single deployment and deployment-2 as another deployment for each service.
We are planning to manage a single trigger in a pipeline that triggers the build only for the deployment where the latest commit is found.
If anyone can please provide suggestions on how to keep single YAML files & build it better way using the cloud build. So that we don't require to manage multiple triggers.
Sadly, nothing is magic!! The dispatch is either done by configuration (multiple trigger) or by code.
If you want to avoid multiple trigger, you need to code the dispatch:
Detect the code that have change in GIT (could be several service in the same time)
Iterate over the updated folders and run a Cloud Build (so, a new one) for each of them
It's small piece of shell code. Not so difficult but you have to maintain/test/debug it. Is it easier that multiple trigger? It's up to you, according to your team skills in devops area.
I have an app written in Go, which I attempted to deploy to EB.
When trying to access it, I get an Error 502 from nginx, presumably because the app is not running.
Looking at logs, I get a lot of errors like
14:01:29 build.1 | application.go:10:2: cannot find package "github.com/aws/aws-sdk-go/aws" in any of:
14:01:29 build.1 | /opt/elasticbeanstalk/lib/go/src/github.com/aws/aws-sdk-go/aws (from $GOROOT)
14:01:29 build.1 | /var/app/current/src/github.com/aws/aws-sdk-go/aws (from $GOPATH)
Despite the fact, that I have all of my dependencies included in the application bundle under a vendor subdirectory. How come EB does not use vendoring? According to the dashboard, it is running Go 1.9, so vendoring should be supported.
You need to set your GOPATH in your EBS to the root of your project directory, assuming there is a src directory where your vendor directory is located.
For instance, pretend this is your project structure:
app/
src/
vendor/
And pretend that project is located in ~/home, which makes its location ~/home/app.
Then your GOPATH should be set to ~/home/app. Go will attempt to access the dependencies through $GOPATH/src/vendor.
But if this were the kind of structure you were using before, then you would need to have your GOPATH updated during local development as well, so if you aren't already doing that then I imagine you're using a different kind of setup... this solution, however, will work as long as your project is structured as I described above.
My app currently uses a folder called "Documents" that is located in the root of the app. This is where it stores supporting docs, temporary files, uploaded files etc. I'm trying to move my app from Azure to Beanstalk and I don't know how to give permissions to this folder and sub-folders. I think it's supposed to be done using .ebextensions but I don't know how to format the config file. Can someone suggest how this config file should look? This is an ASP.NET app running on Windows/IIS.
Unfortunately, you cannot use .ebextensions to set permissions to files/folders within your deployment directory.
If you look at the event hooks for an elastic beanstalk deployment:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-windows-ec2.html#windows-container-commands
You'll find that commands run before the ec2 app and web server are set up, and
container_commands run after the ec2 app and web server are setup, but before your application version is deployed.
The solution is to use a wpp.targets file to set the necessary ACLs.
The following SO post is most useful
Can Web Deploy's setAcl provider be used on a sub-directory?
Given below is the sample .ebextensions config file to create a directory/file and modify the permissions and add some content to the file
====== .ebextensions/custom_directory.config ======
commands:
create_directory:
command: mkdir C:\inetpub\AspNetCoreWebApps\backgroundtasks\mydirectory
command: cacls C:\inetpub\AspNetCoreWebApps\backgroundtasks\mydirectory /t /e /g username:W
files:
"C:/inetpub/AspNetCoreWebApps/backgroundtasks/mydirectory/mytestfile.txt":
content: |
This is my Sample file created from ebextensions
ebextensions go into the root of the application source code through a directory called .ebextensions. For more information on how to use ebextensions, please go through the documentation here
Place a file 01_fix_permissions.config inside .ebextensions folder.
files:
"/opt/elasticbeanstalk/hooks/appdeploy/pre/49_change_permissions.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
sudo chown -R ec2-user:ec2-user tmp/
Following that you can set your folder permissions as you want.
See this answer on Serverfault.
There are platform hooks that you can use to run scripts at various points during deployment that can get you around the shortcomings of the .ebextension Commands and Platform Commands that Napoli describes.
There seems to be some debate on whether or not this setup is officially supported, but judging by comments made on the AWS github, it seems to be not explicitly prohibited.
I can see where Napoli's answer could be the more standard MS way of doing things, but wpp.targets looks like hot trash IMO.
The general scheme of that answer is to use Commands/Platform commands to copy a script file into the appropriate platform hook directory (/opt/elasticbeanstalk/hooks or C:\Program Files\Amazon\ElasticBeanstalk\hooks\ ) to run at your desired stage of deployment.
I think its worth noting that differences exist between platforms and versions such as Amazon Linux 1 and Linux 2.
I hope this helps someone. It took me a day to gather that info and what's on this page and pick what I liked best.
Edit 11/4 - I would like to note that I saw some inconsistencies with the File .ebextension directive when trying to place scripts drirectly into the platform hook dir's during repeated deployments. Specifically the File directive failed to correctly move the backup copies named .bak/.bak1/etc. I would suggest using a Container Command to copy with overwriting from another directory into the desired hook directory to overcome this issue.
I'm new to ember.js, and node, and npm. I'm having issues running the 'ember new my-project' command. I have a strong suspicion that i've screwed up directory permissions or something by installing something incorrectly but I don't know the best way to go about investigating that.
The environment is as follows: AWS EC2 instance, ubuntu 16.04, nodejs 7.5.0, and npm 4.2.0, bower 1.8.0, ember-cli 2.11.1.
npm, bower, ember were installed with -g and not using sudo as the ember.js docs recommend.
When I run the 'ember new my-project' command I get "Successfully initialized git" and then I get
Error creating new application. Removing generated directory `./my-project`
Command failed: /home/ubuntu/.npm-global/lib/node_modules/ember-cli/node_modules/npm/bin/npm-cli.js install --no-optional --loglevel error
npm WARN deprecated minimatch#2.0.10: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
When I run ~ $ npm ls minimatch to check the minimatch piece of the error, i'm not seeing any modules are requiring 2.0.10 so i'm not sure where that is coming from. I'm also not sure if that is actually what's making the whole thing fail.
I also tried running npm install -g minimatch#3.0.2 but that just put it outside the npm#4.2.0 tree as you can see.
/home/ubuntu
├── minimatch#3.0.2
└─┬ npm#4.2.0
├─┬ fstream-npm#1.2.0
│ └─┬ fstream-ignore#1.0.5
│ └── minimatch#3.0.3
├─┬ glob#7.1.1
│ └── minimatch#3.0.3
├─┬ init-package-json#1.9.4
│ └─┬ glob#6.0.4
│ └── minimatch#3.0.3
├─┬ node-gyp#3.5.0
│ └── minimatch#3.0.3
└─┬ read-package-json#2.0.4
└─┬ glob#6.0.4
└── minimatch#3.0.3
What i've tried so far is basically playing with versions - removing and reinstalling nodejs and npm, using different versions (docs and google have recommended different versions of node and npm for use with ember-cli.. I tried node 4.0.0 and npm 2.x.x and 3.x.x)
I also tried using ember-cli 2.11.0 because that version is working properly for in a local VM for me, and running the command with 'sudo'.
Is there a more detailed error log for npm or ember-cli somewhere I can look at?
Can someone point me in the right direction to figure out what's going on here?
You can run ember new with DEBUG option:
DEBUG=* ember new projectname
this will generate a lot of output, but it should help you figure out where it breaks.
Thanks to #Jure C. for providing the answer to the question I asked, I found an answer to the issue at hand, for anyone who finds this:
From his answer I got to the following: "Running 'ember new my-project --skip-npm ; cd my-project ; npm install' reveals more information... npm install is hanging each time while extracting libraries and being killed - and NOT in the same place every time"
This led me to believe it was a weird system resources and/or timeout issue or some kind, rather than a permissions or code issue.
Things going wrong:
1) There seemed to be a locally (rather than globally) installed version of npm mucking things up. Not sure when I did this but that's a big whoops. It was not super intuitive where to remove this but eventually I googled my way through it and it helped the process go farther.
2) Turns out the poor little AWS instance was just not up to the task!
ulimit -n 2048
may have been overkill but allowed all of the extract processes of npm install to finish before being terminated.
So I am attempting to setup CodeDeploy for my application and I keep getting an error during the BeforeInstall part of the deployment. Below is the error.
Error Code UnknownError
Script Name
Message No such file or directory - /opt/codedeploy-agent/deployment-root/06100f1b-5495-42d9-bd01-f33d59fb5deb/d-NL5K1THE8/deployment-archive/appspec.yml
Log Tail
I assumed this meant the YAML file was in the wrong place. However it is in the root directory of my revision. I have tried using a simple AppSpec file like so instead of a more complex one.
## YAML Template.
---
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu/www
More or less since this is a first deployment I want it to add all files in the revision to the public directory on the web server.
I am tearing my hair out over this and I feel it is a simple issue. I have the IAM policies and roles correct and I have CodeDeploy setup and running on my instance I am trying to deploy to.
It seems to think you had a successful deploy at some point.
Go into /opt/codedeploy-agent/deployment-root/deployment-instructions/ and delete all the files in there. Then it won't look for this last deploy.
I just had this SAME problem and I figured it out! Make sure your AppSpec file has the right EXTENSION! I was using yaml and not yml, now everything works perfectly.
I made it work like this:
I had a couple of failed deployments for various reasons.
The thing is that CD keeps in the EC2 instance and in the path /opt/codedeploy-agent/deployment-root/ a folder named by the ID of the failed deployment [a very long alphanumeric sting] .
Delete this folder and create a new deployment [from the aws UI console] and redeploy the application. This way the appspec.yml file that is in the wrong place will be deleted.
It should now succeed.
Extra Notice:
CD does not rewrite files [that have not been created by it's specific deployment]
CodeDeploy does not deploy in a folder that there is already code[files] as it does not want to interfere with different CD deployments and/or other CI/CD tools [like Jenkins].
It only deploys in a path that has already deploy code with the specific deployment.
You can empty the folder where your deployment want to happen and redeploy your code via CD.
When you login to the host, do you see the appspec.yml file in the directory there? If not are you positive it has been checked in with the rest of your deployed code?
Just encountered this issue too. In my case, the revision zip file extracts into a directory when deployed. Because of that /opt/codedeploy-agent/deployment-root/xxx/xxx/deployment-archive contains the parent directory of my revision files (instead of the actual revision files).
The key is to compress your revision without the parent directory. In mac terminal,
cd your-app-directory-containing-appspec
zip -r app.zip .