I've deployed a stack in Amazon OpsWorks, and I extensively use custom cookbooks to deploy my application. I have a number of instances in my stack that are load-based (they only boot up when needed).
Anytime I make changes to my custom cookbooks, I have to manually update the cookbooks on any running instances (by navigating to Deployments > Run Command). The problem is that any non-booted instances are not updated, and they don't automatically update at their next boot.
I've figured out that I can delete and then recreate all my load-based instances, forcing them to be completely re-setup when they're next needed, but there must be a better way to deploy updated custom cookbooks.
How can I force my offline load-based instances to update their cookbooks at the next boot (even every boot would be fine)?
From this AWS employee response on an Amazon Opsworks forum:
There isn't a way to push updates to stopped instances. We're considering ways to enable this. For now, if you create a new time or load based instance, it will get your updates.
So it would appear that for now, the only way to do what you'd like to do is to delete and recreate each of your load-based instances. This should ensure that the first time they boot up, they receive fresh versions of your custom cookbooks.
You can run Update Custom Cookbooks command from Stack, Run Command window.
As it says: http://docs.aws.amazon.com/opsworks/latest/userguide/workingcookbook-installingcustom-enable-update.html
According to the opsworks documentation:
http://docs.aws.amazon.com/opsworks/latest/userguide/workingcookbook-installingcustom-enable-update-.html
To manually update custom cookbooks
Update your repository with the modified cookbooks. AWS OpsWorks uses the cache URL that you provided when you originally installed the cookbooks, so the cookbook root file name, repository location, and access rights should not change.
For Amazon S3 or HTTP repositories, replace the original .zip file with a new .zip file that has the same name.
For Git or Subversion repositories, edit your stack settings to change the Branch/Revision field to the new version.
On the stack's page, click Run command and select the update custom cookbooks command.
Related
I have a question regarding AWS, have an AMI with windows server installed, IIS installed, and a site up and running.
My AutoScale always maintains two instances created based on this AMI.
However, whenever I need to change something on the site I need to upload a new instance, make the changes, update the AMI and update the auto-scale, which is quite time consuming.
Is there any way to automate this by linking to a Git repository?
This is more like a CI CD work rather than achieved in AWS.
You can schedule a CI CD pipeline to detect any update happens in SCM(GIT) and trigger a build job(Jenkins or similar tool) which will provide an artifact to you. You can deploy the artifact to respective application server using CD tools (ansible/even with jenkins or similar tools) whichever suits your infra. In the deploy script itself you can connect to ec2 service to create a new AMI once deployment is completed.
You need to use set of tools to achieve it SCM webhook/poll, Jenkins, Ansible.
So we are launching an ecommerce store built on magento. We are looking to deploy it on Amazon EC2 instance using RDS as database service and using amazon auto-scaling and elastic load balancer to scale the application when needed.
What I don't understand is this:
I have installed and configured my production magento enviorment on an EC2 instance (database is in RDS). This much is working fine. But now when I want to dynamically scale the number of instances
how will I deploy the code on the dynamically generated instances each time?
Will aws copy the whole instance assign it a new ip and spawn it as a
new instance or will I have to write some code to automate this
process?
Plus will it not be an overhead to pull code from git and deploy every time a new instance is spawned?
A detailed explanation or direction towards some resources on the topic will be greatly appreciated.
You do this in the AutoScalingGroup Launch Configuration. There is a UserData section in the LaunchConfiguration in CloudFormation where you would write a script that is ran when ever the ASG scales up and deploys a new instance.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig.html#cfn-as-launchconfig-userdata
This is the same as the UserData section in an EC2 Instance. You can use LifeCycle hooks that will tell the ASG not to put the EC2 instance into load until everything you want to have configured it set up.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-as-lifecyclehook.html
I linked all CloudFormation pages, but you may be using some other CI/CD tool for deploying your infrastructure, but hopefully that gets you started.
To start, do check AWS CloudFormation. You will be creating templates to design how the infrastructure of your application works ~ infrastructure as code. With these templates in place, you can rollout an update to your infrastructure by pushing changes to your templates and/or to your application code.
In my current project, we have a github repository dedicated for these infrastructure templates and a separate repository for our application code. Create a pipeline for creating AWS resources that would rollout an updated to AWS every time you push to the repository on a specific branch.
Create an infrastructure pipeline
have your first stage of the pipeline to trigger build whenever there's code changes to your infrastructure templates. See AWS CodePipeline and also see AWS CodeBuild. These aren't the only AWS resources you'll be needing but those are probably the main ones, of course aside from this being done in cloudformation template as mentioned earlier.
how will I deploy the code on the dynamically generated instances each time?
Check how containers work, it would be better and will greatly supplement on your learning on how launching new version of application work. To begin, see docker, but feel free to check any resources at your disposal
Continuation with my current project: We do have a separate pipeline dedicated for our application, but will also get triggered after our infrastructure pipeline update. Our application pipeline is designed to build a new version of our application via AWS Codebuild, this will create an image that will become a container ~ from the docker documentation.
we have two triggers or two sources that will trigger an update rollout to our application pipeline, one is when there's changes to infrastructure pipeline and it successfully built and second when there's code changes on our github repository connected via AWS CodeBuild.
Check AWS AutoScaling , this areas covers the dynamic launching of new instances, shutting down instances when needed, replacing unhealthy instances when needed. See also AWS CloudWatch, you can design criteria with it to trigger scaling down/up and/or in/out.
Will aws copy the whole instance assign it a new ip and spawn it as a new instance or will I have to write some code to automate this process?
See AWS ElasticLoadBalancing and also check out more on AWS AutoScaling. On the automation process, if ever you'll push through with CloudFormation, instance and/or containers(depending on your design) will be managed gracefully.
Plus will it not be an overhead to pull code from git and deploy every time a new instance is spawned?
As mentioned, earlier having a pipeline for rolling out new versions of your application via CodeBuild, this will create an image with the new code changes and when everything is ready, it will be deployed ~ becomes a container. The old EC2 instance or the old container( depending on how you want your application be deployed) will be gracefully shut down after a new version of your application is up and running. This will give you zero downtime.
I am trying to set AWS Code deploy with Github to automate my deloyment. the problem i am having is the ec2 instance is already active and has the web app code. the problem is there is also other content on the instance that I don't want under source control.
Code deploy is failing with the following error:
The deployment failed because a specified file already exists at this location
this is because the source code is on the instance and was not added as a revision.
So my question is there any way to make the code deploy recognise the current files on the instance as an inital revision?
This is possible! This is an option during a deployment: "Content options:
Choose the action for AWS CodeDeploy to take during a deployment when a file on a target instance has the same name as a file in the application revision, for the same target location."
You can choose fail, overwrite and retain. Retain is probably the best choice in your case.
You can find more information
docs.aws.amazon.com (strangely under 'rollback')
cli docs.aws.amazon.com (see --file-exists-behavior (string))
This is my sample appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /var/www/wordpress/
overwrite: true
file_exists_behavior: OVERWRITE
And this is Updated AWS User Guide for AppSpec 'files' section (EC2/On-Premises deployments only). In there you will find three options, DISALLOW,OVERWRITE and RETAIN.
You can simply delete the entire contents of your web root and allow CodeDeploy to update it. Any type of deployment will replace the entire contents.
If you're feeling unsure then copy the webroot to another folder, and in the very unlikely event of a failure you can copy it all back.
Deployments often fail when a file has been modified on the disk, I assume CodeDeploy is checking the modified time of the file.
If you just want the error to go away, you can use one of the earlier Lifecycle Hook scripts (BeforeInstall maybe) to delete that folder before the source gets re-added in a later Hook. If the source doesn't get re-added in a later Hook, add it yourself.
I was getting this error even on new instances, so CodeDeploy must be caching that folder somehow, then getting upset that it did so.
The issue is that CodeDeploy maintains a cleanup file that it updates after every deployment. These are the files it will delete before installing the new ones. You could of course create this yourself, but that's hard. The point of CodeDeploy is to make things easy. This is what I did (Note: My app has a lot of files and I was about to find something other than CodeDeploy that actually made transitioning to it easy instead of painful and stupid, and then I formulated this approach; if you have a small number of files, it might be easier to create a BeforeInstall script or delete them manually):
create a free-tier EC2 instance (t2.micro). If your app is giant, you may need more disk space, so adjust as necessary. THIS WON'T BE RUNNING YOUR APP!
Create a new deployment group pointing to that new instance.
Create a new YAML that doesn't actually do any BeforeInstall or AfterInstall tasks. You are not trying to run your app. You are literally moving files over. This is so that you can grab that cleanup file.
After deployment, ssh into the ec2 instance (remember to allow SSH in the Security Group!) and go to /opt/codedeploy-agent/deployment-root/deployment-instructions/
You will see {deployment-group-identifier}-cleanup. scp this to the ec2 instance on which you actually want to run your app. Or, FileZilla it out onto your main machine and then onto the main ec2 instance. That was the easiest choice for me, because my local machine already had all the right keys.
SSH into the main ec2 instance and move the cleanup file into the /opt/codedeploy-agent/deployment-root/deployment-instructions/ if it isn't already there. Note the failed {deployment-group-identifier} and replace the cleanup file's {deployment-group-identifier} with the failed identifier.
Rerun the deployment.
8.???
Profit.
I will provide more in-depth instructions for each of these steps (except for 8 and 9... which will go away) when I have time.
I had same issue when working in CodeDeploy with EC2 Instances.
I removed EC2 Instances from AWS CodeDeploy Deployment Group. I added new EC2 instances to the deployment group.
Otherwise, Replace new deployment group with an existing deployment group.
I know this has been partially answered in a bunch of places, but the answers are so.. all over the map, dated and not well explained. I'm looking the best practice as of February 2016.
The setup:
A PHP-based RESTful application service that lives in an EC2 instance. The EC2 instance uses S3 for uploaded user data (image files), and RDS MySql for its DB (these two points aren't particularly important.)
We develop in PHPStorm, and our source control is GitHub. When we deploy, we just use PHPStorm's built-in SFTP deployment to upload files directly to the EC2 instance (we have one instance for our Staging environment, and another for our Production environment). I deploy to Staging very often. Could be 20 times a day. I just click on a file in PHPStorm and say 'deploy to Staging', which does the SFTP transfer. Or, I might just click on the entire project and click 'deploy to Staging' - certain folders and files are excluded from the upload, which is part of PHPStorm's deployment configuration.
Recently, I put our EC2 instance behind a Load Balancer. I did this so that I can take advantage of Amazon's free SSL offering via the Certificate Manager, which does not support individual EC2 instances.
So, right now, there's a Load Balancer with only a single EC2 instance behind it. I maintain an Elastic IP pointing to the EC2 instance so that I can access it directly (see my current deployment method above).
Question:
I have not yet had the guts to create additional (clone) EC2 instances behind my Load Balancer, because I'm not sure how I should be deploying to them. A few ideas came to mind, but they're all pretty hacky.
Given the scenario above, what is currently the smoothest and best way to A) quickly deploy a codebase to a set of EC2 instances behind a Load Balancer, and B) actually 'clone' my current EC2 instance to create additional instances.
I haven't been able to really paint a clear picture of the above in my head yet, despite the fact that I've gone over a few (highly technical) suggestions.
Thanks!
You need to treat your EC2 instance as 100% dispensable. Meaning, that it can be terminated at any time and you should not care. A replacement EC2 instance would start and take over the work.
There are 3 ways to accomplish this:
Method 1: Each deployment creates a new AMI image.
When you deploy you app, you deploy it to a worker EC2 instance whose sole purpose is for "setup" of your app. Once the new version is deployed, you create a fresh AMI image from the EC2 instance and update your Auto Scaling launch configuration with the new AMI image. The old EC2 instances are terminated and replaced with the new code.
New EC2 instances have the recent code already on them so they're ready to be added to the load balancer.
Method 2: Each deployment is done to off-instance storage (like Amazon S3).
The EC2 instances will download the recent code from Amazon S3 and install it on boot.
So to put the new code in action, you terminate the old instances and new ones are launched to replace them which start using the new code.
This could be done in a rolling-update fashion, or as a blue/green deployment.
Method 3: Similar to method 2, but this time the instances have some smarts and can be signaled to download and install the code.
This way, you don't need to terminate instances: the existing instances are told to update from S3 and they do so on their own.
Some tools that may help include:
Chef
Ansible
CloudFormation
Update:
Methods 2 & 3 both start with a "basic" AMI which is configured to pull the webpage assets from S3. This AMI is not changed from version-to-version of your website.
For example, the AMI can have Apache and PHP already installed and on boot it pulls the .php website assets from S3 and puts them in /var/www/html.
CloudFormation works well for this. In addition, for method 3, you can use cfn-hup to wait for update signals. When signaled, it'll pull updated assets from S3.
Another possibility is using Elastic Beanstalk which could be used to manage all of this for you.
Update:
To have your AMI image pull from Git, try the following:
Setup an EC2 instance with everything installed that you need to have installed for your web app
Install Git and setup a local repo ready to Git pull.
Shutdown and create an AMI of your instance.
When you deploy, you do the following:
Git push to GitHub
Launch a new EC2 instance, based on your AMI image.
As part of the User Data (specified during the EC2 instance launch), specify something like the following:
#!/bin/sh
cd /git/app
git pull
; copy files from repo to web folder
composer install
When done like this, that user data acts as a script which will run on first boot.
I have configured an aws asg using ansible to provision new instances and then install the codedeploy agent via "user_data" script in a similar fashion as suggested in this question:
Can I use AWS code Deploy for pulling application code while autoscaling?
CodeDeploy works fine and I can install my application onto the asg once it has been created. When new instances are triggered in the ASG via one of my rules (e.g. high cpu usage), the codedeploy agent is installed correctly. The problem is, CodeDeploy does not install the application on these new instances. I suspect it is trying to run before the user_data script has finished. Has anyone else encountered this problem? Or know how to get CodeDeploy to automatically deploy the application to new instances which are spawned as part of the ASG?
AutoScaling tells CodeDeploy to start the deployment before the user data is started. To get around this CodeDeploy gives the instance up to an hour to start polling for commands for the first lifecycle event instead of 5 minutes.
Since you are having problems with automatic deployments but not manual ones and assuming that you didn't make any manual changes to your instances you forgot about, there is most likely a dependency specific to your deployment that's not available yet at the time the instance launches.
Try listing out all the things that your deployment needs to succeed and make sure that each of those is available before you install the host agent. If you can log onto the instance fast enough (before AutoScaling terminates the instance), you can try and grab the host agent logs and your application's logs to find out where the deployment is failing.
If you think the host agent is failing to install entirely, make sure you have Ruby2.0 installed. It should be there by default on AmazonLinux, but Ubuntu and RHEL need to have it installed as part of the user data before you can install the host agent. There is an installer log in /tmp that you can check for problems in the initial install (again you have to be quick to grab the log before the instance terminates).