I have used Cloudformation template to create my EC2 instance. In this instance there is a file
in home directory that I copy from S3 while creating stack.
I have that file stored locally as well. Now, I modify that file locally and want to copy it to S3
and from S3 to EC2 instance.
I want to automate this process through Cloudformation. So that, whenever I modify this file locally,
after doing update stack, it uploads the modified file to S3 and from S3 to my EC2 instance.
Can anyone please help how this can be achieved?
One thing that comes to mind (bearing in mind the application specific nature of what you are trying to do) is using ECS instead of just EC2.
Note: This may be overkill, but it would work. Also if updates were extremely frequent this would be a major pain so just uploading the file to S3 with a script alongside update-stack (if the update-stack is even necessary) and then polling for changes to that S3 file in your EC2 application would be fine.
Anyway, this is a pattern we use when we are doing something like training a model with new data, which we then wish to deploy to AWS, replacing an application with an older version of the model.
You build the Docker image locally and your special file gets included inside the container. You push the Docker image to DockerHub or AWS ECS Registry or wherever. You update the cloudformation template ECS configuration to use the tag of this new Docker image and update the stack. Then ECS pulls this new image and the new Docker container(s) take the place of the old one(s) and it will have your special file inside it.
I know of at least one way: set up your EC2 in an AutoScaling Group (ASG). Then, on creation, use cfn-init on your UserData and have it fetch the file under sources. Use both Creation and Update policies. On update, have the WillReplace attribute set to true under AutoScalingReplacingUpdate. When you update, CloudFormation will create a new ASG with your fresh new file copy. If you signal success on your update, it will remove the previous instance and ASG, giving you an immutable infrastructure setup. Put everything behind a load balancer, and you got yourself a highly available blue/green deployment too.
Related
I'm new to AWS and a little perplexed as to the situation with var/www/html folder in an EC2 instance in which Apache has been installed.
After setting up an Elastic Beanstalk service and uploading the files, I see that these are stored in the regular var/www/html folder of the instance.
From reading AWS documents, it seems that instances may be deleted and re-provisioned, which is why use of an S3 bucket, EFS or EBS is recommended.
Why, then, is source code stored in the EC2 instance when using an apache server? Won't these files potentially be deleted with the instance?
If you manually uploaded some files and data to /var/www/html then off course they will be wiped out when AWS is going to replace/delete the instance, e.g. due to autoscalling.
All files that you use on EB should be part of your deployment package, and all files that, e.g. your users upload, should be stored outside of Eb, e.g. on S3.
Even if the instance is terminated for some reason, since the source code is part of your deployment package on Beanstalk it can provision another instance to replace with the exact same application and configurations. Basically you are losing this data, but it doesn't matter.
The data loss concern is for anything you do that is not part of the automated provisioning/deployment, ie any manual configuration changes or any data your application may write to ephemeral storage. This is what you would need a more persistent storage option for.
Seems that, when the app is first deployed, all files are uploaded to an S3 bucket, from where these are copied into the relevant directory of each new instance. In the event new instances have to be created (such as for auto-scaling) or replaced, these instances also pull the code from the S3 bucket. This is also how the app is re-deployed - the bucket is updated and each instance makes a new copy of the code.
Apologies if this is stating the obvious to some people, but I had seen a few similar queries.
I have created ELB (load Balancing) and Auto Scaling Group (ASG) after instance coming 'this is my 1st instance working fine after I have login via putty and changed the "index.html" file updated index.html ("This is my 2nd Instance updated') file then deleted instance, new instance coming old data only coming -- Here how to recover new data I mean updated instance,
I want "This is my 2nd Instance updated"
I'm not sure I followed but I think what you're saying is:
You manually SSH to the instance and update index.html, then AutoScaling launches a new instance and it doesn't have those changes. Is that correct?
AutoScaling has no way of knowing what you did inside the instance. AWS can't peak inside your instances and look at your data, that would be a pretty big privacy breach.
When you make updates you need to modify the launch template or launch configuration in some way to make sure the new instances have the updates. Either:
Make a new AMI with the changes
Put the changes in a userdata script
Alternatively, you could have some sort of automation help with this such as:
Have a userdata script which downloads the newest files from an S3 bucket. Just make sure to update the website files in this bucket whenever you make changes
Use some sort of CI/CD pipeline tool such as AWS CodeDeploy. This will automatically push changes to your existing instances, and make sure new instances are launched with the newest code.
So we are launching an ecommerce store built on magento. We are looking to deploy it on Amazon EC2 instance using RDS as database service and using amazon auto-scaling and elastic load balancer to scale the application when needed.
What I don't understand is this:
I have installed and configured my production magento enviorment on an EC2 instance (database is in RDS). This much is working fine. But now when I want to dynamically scale the number of instances
how will I deploy the code on the dynamically generated instances each time?
Will aws copy the whole instance assign it a new ip and spawn it as a
new instance or will I have to write some code to automate this
process?
Plus will it not be an overhead to pull code from git and deploy every time a new instance is spawned?
A detailed explanation or direction towards some resources on the topic will be greatly appreciated.
You do this in the AutoScalingGroup Launch Configuration. There is a UserData section in the LaunchConfiguration in CloudFormation where you would write a script that is ran when ever the ASG scales up and deploys a new instance.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig.html#cfn-as-launchconfig-userdata
This is the same as the UserData section in an EC2 Instance. You can use LifeCycle hooks that will tell the ASG not to put the EC2 instance into load until everything you want to have configured it set up.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-as-lifecyclehook.html
I linked all CloudFormation pages, but you may be using some other CI/CD tool for deploying your infrastructure, but hopefully that gets you started.
To start, do check AWS CloudFormation. You will be creating templates to design how the infrastructure of your application works ~ infrastructure as code. With these templates in place, you can rollout an update to your infrastructure by pushing changes to your templates and/or to your application code.
In my current project, we have a github repository dedicated for these infrastructure templates and a separate repository for our application code. Create a pipeline for creating AWS resources that would rollout an updated to AWS every time you push to the repository on a specific branch.
Create an infrastructure pipeline
have your first stage of the pipeline to trigger build whenever there's code changes to your infrastructure templates. See AWS CodePipeline and also see AWS CodeBuild. These aren't the only AWS resources you'll be needing but those are probably the main ones, of course aside from this being done in cloudformation template as mentioned earlier.
how will I deploy the code on the dynamically generated instances each time?
Check how containers work, it would be better and will greatly supplement on your learning on how launching new version of application work. To begin, see docker, but feel free to check any resources at your disposal
Continuation with my current project: We do have a separate pipeline dedicated for our application, but will also get triggered after our infrastructure pipeline update. Our application pipeline is designed to build a new version of our application via AWS Codebuild, this will create an image that will become a container ~ from the docker documentation.
we have two triggers or two sources that will trigger an update rollout to our application pipeline, one is when there's changes to infrastructure pipeline and it successfully built and second when there's code changes on our github repository connected via AWS CodeBuild.
Check AWS AutoScaling , this areas covers the dynamic launching of new instances, shutting down instances when needed, replacing unhealthy instances when needed. See also AWS CloudWatch, you can design criteria with it to trigger scaling down/up and/or in/out.
Will aws copy the whole instance assign it a new ip and spawn it as a new instance or will I have to write some code to automate this process?
See AWS ElasticLoadBalancing and also check out more on AWS AutoScaling. On the automation process, if ever you'll push through with CloudFormation, instance and/or containers(depending on your design) will be managed gracefully.
Plus will it not be an overhead to pull code from git and deploy every time a new instance is spawned?
As mentioned, earlier having a pipeline for rolling out new versions of your application via CodeBuild, this will create an image with the new code changes and when everything is ready, it will be deployed ~ becomes a container. The old EC2 instance or the old container( depending on how you want your application be deployed) will be gracefully shut down after a new version of your application is up and running. This will give you zero downtime.
I am trying to set AWS Code deploy with Github to automate my deloyment. the problem i am having is the ec2 instance is already active and has the web app code. the problem is there is also other content on the instance that I don't want under source control.
Code deploy is failing with the following error:
The deployment failed because a specified file already exists at this location
this is because the source code is on the instance and was not added as a revision.
So my question is there any way to make the code deploy recognise the current files on the instance as an inital revision?
This is possible! This is an option during a deployment: "Content options:
Choose the action for AWS CodeDeploy to take during a deployment when a file on a target instance has the same name as a file in the application revision, for the same target location."
You can choose fail, overwrite and retain. Retain is probably the best choice in your case.
You can find more information
docs.aws.amazon.com (strangely under 'rollback')
cli docs.aws.amazon.com (see --file-exists-behavior (string))
This is my sample appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /var/www/wordpress/
overwrite: true
file_exists_behavior: OVERWRITE
And this is Updated AWS User Guide for AppSpec 'files' section (EC2/On-Premises deployments only). In there you will find three options, DISALLOW,OVERWRITE and RETAIN.
You can simply delete the entire contents of your web root and allow CodeDeploy to update it. Any type of deployment will replace the entire contents.
If you're feeling unsure then copy the webroot to another folder, and in the very unlikely event of a failure you can copy it all back.
Deployments often fail when a file has been modified on the disk, I assume CodeDeploy is checking the modified time of the file.
If you just want the error to go away, you can use one of the earlier Lifecycle Hook scripts (BeforeInstall maybe) to delete that folder before the source gets re-added in a later Hook. If the source doesn't get re-added in a later Hook, add it yourself.
I was getting this error even on new instances, so CodeDeploy must be caching that folder somehow, then getting upset that it did so.
The issue is that CodeDeploy maintains a cleanup file that it updates after every deployment. These are the files it will delete before installing the new ones. You could of course create this yourself, but that's hard. The point of CodeDeploy is to make things easy. This is what I did (Note: My app has a lot of files and I was about to find something other than CodeDeploy that actually made transitioning to it easy instead of painful and stupid, and then I formulated this approach; if you have a small number of files, it might be easier to create a BeforeInstall script or delete them manually):
create a free-tier EC2 instance (t2.micro). If your app is giant, you may need more disk space, so adjust as necessary. THIS WON'T BE RUNNING YOUR APP!
Create a new deployment group pointing to that new instance.
Create a new YAML that doesn't actually do any BeforeInstall or AfterInstall tasks. You are not trying to run your app. You are literally moving files over. This is so that you can grab that cleanup file.
After deployment, ssh into the ec2 instance (remember to allow SSH in the Security Group!) and go to /opt/codedeploy-agent/deployment-root/deployment-instructions/
You will see {deployment-group-identifier}-cleanup. scp this to the ec2 instance on which you actually want to run your app. Or, FileZilla it out onto your main machine and then onto the main ec2 instance. That was the easiest choice for me, because my local machine already had all the right keys.
SSH into the main ec2 instance and move the cleanup file into the /opt/codedeploy-agent/deployment-root/deployment-instructions/ if it isn't already there. Note the failed {deployment-group-identifier} and replace the cleanup file's {deployment-group-identifier} with the failed identifier.
Rerun the deployment.
8.???
Profit.
I will provide more in-depth instructions for each of these steps (except for 8 and 9... which will go away) when I have time.
I had same issue when working in CodeDeploy with EC2 Instances.
I removed EC2 Instances from AWS CodeDeploy Deployment Group. I added new EC2 instances to the deployment group.
Otherwise, Replace new deployment group with an existing deployment group.
I know this has been partially answered in a bunch of places, but the answers are so.. all over the map, dated and not well explained. I'm looking the best practice as of February 2016.
The setup:
A PHP-based RESTful application service that lives in an EC2 instance. The EC2 instance uses S3 for uploaded user data (image files), and RDS MySql for its DB (these two points aren't particularly important.)
We develop in PHPStorm, and our source control is GitHub. When we deploy, we just use PHPStorm's built-in SFTP deployment to upload files directly to the EC2 instance (we have one instance for our Staging environment, and another for our Production environment). I deploy to Staging very often. Could be 20 times a day. I just click on a file in PHPStorm and say 'deploy to Staging', which does the SFTP transfer. Or, I might just click on the entire project and click 'deploy to Staging' - certain folders and files are excluded from the upload, which is part of PHPStorm's deployment configuration.
Recently, I put our EC2 instance behind a Load Balancer. I did this so that I can take advantage of Amazon's free SSL offering via the Certificate Manager, which does not support individual EC2 instances.
So, right now, there's a Load Balancer with only a single EC2 instance behind it. I maintain an Elastic IP pointing to the EC2 instance so that I can access it directly (see my current deployment method above).
Question:
I have not yet had the guts to create additional (clone) EC2 instances behind my Load Balancer, because I'm not sure how I should be deploying to them. A few ideas came to mind, but they're all pretty hacky.
Given the scenario above, what is currently the smoothest and best way to A) quickly deploy a codebase to a set of EC2 instances behind a Load Balancer, and B) actually 'clone' my current EC2 instance to create additional instances.
I haven't been able to really paint a clear picture of the above in my head yet, despite the fact that I've gone over a few (highly technical) suggestions.
Thanks!
You need to treat your EC2 instance as 100% dispensable. Meaning, that it can be terminated at any time and you should not care. A replacement EC2 instance would start and take over the work.
There are 3 ways to accomplish this:
Method 1: Each deployment creates a new AMI image.
When you deploy you app, you deploy it to a worker EC2 instance whose sole purpose is for "setup" of your app. Once the new version is deployed, you create a fresh AMI image from the EC2 instance and update your Auto Scaling launch configuration with the new AMI image. The old EC2 instances are terminated and replaced with the new code.
New EC2 instances have the recent code already on them so they're ready to be added to the load balancer.
Method 2: Each deployment is done to off-instance storage (like Amazon S3).
The EC2 instances will download the recent code from Amazon S3 and install it on boot.
So to put the new code in action, you terminate the old instances and new ones are launched to replace them which start using the new code.
This could be done in a rolling-update fashion, or as a blue/green deployment.
Method 3: Similar to method 2, but this time the instances have some smarts and can be signaled to download and install the code.
This way, you don't need to terminate instances: the existing instances are told to update from S3 and they do so on their own.
Some tools that may help include:
Chef
Ansible
CloudFormation
Update:
Methods 2 & 3 both start with a "basic" AMI which is configured to pull the webpage assets from S3. This AMI is not changed from version-to-version of your website.
For example, the AMI can have Apache and PHP already installed and on boot it pulls the .php website assets from S3 and puts them in /var/www/html.
CloudFormation works well for this. In addition, for method 3, you can use cfn-hup to wait for update signals. When signaled, it'll pull updated assets from S3.
Another possibility is using Elastic Beanstalk which could be used to manage all of this for you.
Update:
To have your AMI image pull from Git, try the following:
Setup an EC2 instance with everything installed that you need to have installed for your web app
Install Git and setup a local repo ready to Git pull.
Shutdown and create an AMI of your instance.
When you deploy, you do the following:
Git push to GitHub
Launch a new EC2 instance, based on your AMI image.
As part of the User Data (specified during the EC2 instance launch), specify something like the following:
#!/bin/sh
cd /git/app
git pull
; copy files from repo to web folder
composer install
When done like this, that user data acts as a script which will run on first boot.