So we are launching an ecommerce store built on magento. We are looking to deploy it on Amazon EC2 instance using RDS as database service and using amazon auto-scaling and elastic load balancer to scale the application when needed.
What I don't understand is this:
I have installed and configured my production magento enviorment on an EC2 instance (database is in RDS). This much is working fine. But now when I want to dynamically scale the number of instances
how will I deploy the code on the dynamically generated instances each time?
Will aws copy the whole instance assign it a new ip and spawn it as a
new instance or will I have to write some code to automate this
process?
Plus will it not be an overhead to pull code from git and deploy every time a new instance is spawned?
A detailed explanation or direction towards some resources on the topic will be greatly appreciated.
You do this in the AutoScalingGroup Launch Configuration. There is a UserData section in the LaunchConfiguration in CloudFormation where you would write a script that is ran when ever the ASG scales up and deploys a new instance.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig.html#cfn-as-launchconfig-userdata
This is the same as the UserData section in an EC2 Instance. You can use LifeCycle hooks that will tell the ASG not to put the EC2 instance into load until everything you want to have configured it set up.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-as-lifecyclehook.html
I linked all CloudFormation pages, but you may be using some other CI/CD tool for deploying your infrastructure, but hopefully that gets you started.
To start, do check AWS CloudFormation. You will be creating templates to design how the infrastructure of your application works ~ infrastructure as code. With these templates in place, you can rollout an update to your infrastructure by pushing changes to your templates and/or to your application code.
In my current project, we have a github repository dedicated for these infrastructure templates and a separate repository for our application code. Create a pipeline for creating AWS resources that would rollout an updated to AWS every time you push to the repository on a specific branch.
Create an infrastructure pipeline
have your first stage of the pipeline to trigger build whenever there's code changes to your infrastructure templates. See AWS CodePipeline and also see AWS CodeBuild. These aren't the only AWS resources you'll be needing but those are probably the main ones, of course aside from this being done in cloudformation template as mentioned earlier.
how will I deploy the code on the dynamically generated instances each time?
Check how containers work, it would be better and will greatly supplement on your learning on how launching new version of application work. To begin, see docker, but feel free to check any resources at your disposal
Continuation with my current project: We do have a separate pipeline dedicated for our application, but will also get triggered after our infrastructure pipeline update. Our application pipeline is designed to build a new version of our application via AWS Codebuild, this will create an image that will become a container ~ from the docker documentation.
we have two triggers or two sources that will trigger an update rollout to our application pipeline, one is when there's changes to infrastructure pipeline and it successfully built and second when there's code changes on our github repository connected via AWS CodeBuild.
Check AWS AutoScaling , this areas covers the dynamic launching of new instances, shutting down instances when needed, replacing unhealthy instances when needed. See also AWS CloudWatch, you can design criteria with it to trigger scaling down/up and/or in/out.
Will aws copy the whole instance assign it a new ip and spawn it as a new instance or will I have to write some code to automate this process?
See AWS ElasticLoadBalancing and also check out more on AWS AutoScaling. On the automation process, if ever you'll push through with CloudFormation, instance and/or containers(depending on your design) will be managed gracefully.
Plus will it not be an overhead to pull code from git and deploy every time a new instance is spawned?
As mentioned, earlier having a pipeline for rolling out new versions of your application via CodeBuild, this will create an image with the new code changes and when everything is ready, it will be deployed ~ becomes a container. The old EC2 instance or the old container( depending on how you want your application be deployed) will be gracefully shut down after a new version of your application is up and running. This will give you zero downtime.
Related
I have a service that needs to run on our own EC2 instances, since it requires some support from the kernel. My previous experience is all with containers in AWS. The application itself is distributed as a single JAR file and I'm looking for advice for how I should automate deployments. The architecture is:
An ALB in front of the ASG.
EC2 instance running a single Java application.
Any open sockets are open for an hour tops and to not cause any trouble, we have to drain the connections to the EC2 instances before performing an update, so a hard requirement is for the ALB to stop opening new connections for an hour before updating the software. The application is mission critical and ECS has had some issues last year, so I want to minimize the AWS services I depend on. While I could do what I want on my own ECS cluster with custom AMIs, I don't want to do it, since I will run a single instance of the app per host and don't need the extra layer.
My question: What is the simplest method to achieve this using CodePipeline? My understanding is that I need to use a CodeDeploy deployment step to push something to bare EC2 instances. How does draining with an ALB work in this case? We're using CloudFormation for the deployment.
You need to use codedeploy. You can find tutorial on AWS codedeploy documentation.
Codedeploy deployment lifecycle hooks for EC2.
https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-hooks.html#appspec-hooks-server
AWS CodeDeploy's model defines an Application, which is a long-lived high-level object and represents software that needs to be deployed somewhere. An application can have many Deployment Groups, which represent targets (e.g. particular EC2 servers that have a particular combination of tags). A deployment is the release of one particular revision of software onto a deployment group defined within an application.
It is possible to get feedback on the progress of CodeDeploy via CloudWatch events. Given that EC2 servers can be up or down at the time of a deployment, and given that the tags on EC2 servers may vary over time, is there a way of determining from a CloudWatch CodeDeploy event the exact set of EC2 servers that were targeted by a particular deployment?
Specifically:
If a server is down at the time a deployment is launched, will it be targeted for release when it comes back up?
If I add a new server with identical tags to the first one after I have done the deployment, or I change the tags on the first server, will the CloudWatch event associated with my CodeDeploy event contain details of exactly which servers were targeted for deployment at the time, even if their current state means that they would not be targeted for deployment if I were to re-release the same deployment?
I tested few scenarios using a simple CodeDeploy setup. Deployment group was identified based on instance tags only (no ASG). My observations are as follows:
Server down at the time a deployment is launched
I simulated this scenario by having a stopped instance. The deployment hanged on the stopped instance. It would probably timeout if I let it hang for long. Once the instance was re-started, the deployment continued.
New instances started with the same tag
CodeDeploy did not detect them automatically. Had to redeploy the last deployment so that the new instances get detected and run the up-to-date application version.
Changing a tag of an instance
The instance with changed tag is not included in a new deployment. Thus you end up with one instance running an old version of your application, while the rest run the new version.
Deployment id and list-deployment-targets AWS CLI
The list-deployment-targets prints out IDs of instances for which the deployment happened at the time of deployment. When you redeploy (deployment id does not change in this case), the list will include instances for redeployment. Original list of instances is lost.
Note
Deployments to ASG will behave differently, since CodeDeploy integrates with ASG through its lifestyle hooks.
Hope this helps.
I have an environment in AWS where EC2 instances are in autoscaling mode, i.e. new instances spin up as per the load on deployed instances.
Now, if I want to integrate this environment with Jenkins, how can I push my codes from Github to these EC2 instances, where my application is deployed. And with every change in my code version, Github should invoke EC2 instances to have the same versions deployed, and also every new instances should be created with this updated version of code, i.e. every autoscaled instances must have the same code version running. Please help.
I assume you have an executable version of your latest code on a deploy server. You can do this by forcing Jenkins to deploy your code when a new commit is made on a specific branch in GitHub. Then all you need is an AMI for your Auto Scaling Group that has a job/task which runs let's say every 5 minutes (based on how long one single task takes). This job/task fetches (copies) the code from the deploy server and then starts the application. As an example, in Windows Task Scheduler you can add two actions to a task: one for updating (e.g. a simple robocopy) the code and one for running the app.
I know this has been partially answered in a bunch of places, but the answers are so.. all over the map, dated and not well explained. I'm looking the best practice as of February 2016.
The setup:
A PHP-based RESTful application service that lives in an EC2 instance. The EC2 instance uses S3 for uploaded user data (image files), and RDS MySql for its DB (these two points aren't particularly important.)
We develop in PHPStorm, and our source control is GitHub. When we deploy, we just use PHPStorm's built-in SFTP deployment to upload files directly to the EC2 instance (we have one instance for our Staging environment, and another for our Production environment). I deploy to Staging very often. Could be 20 times a day. I just click on a file in PHPStorm and say 'deploy to Staging', which does the SFTP transfer. Or, I might just click on the entire project and click 'deploy to Staging' - certain folders and files are excluded from the upload, which is part of PHPStorm's deployment configuration.
Recently, I put our EC2 instance behind a Load Balancer. I did this so that I can take advantage of Amazon's free SSL offering via the Certificate Manager, which does not support individual EC2 instances.
So, right now, there's a Load Balancer with only a single EC2 instance behind it. I maintain an Elastic IP pointing to the EC2 instance so that I can access it directly (see my current deployment method above).
Question:
I have not yet had the guts to create additional (clone) EC2 instances behind my Load Balancer, because I'm not sure how I should be deploying to them. A few ideas came to mind, but they're all pretty hacky.
Given the scenario above, what is currently the smoothest and best way to A) quickly deploy a codebase to a set of EC2 instances behind a Load Balancer, and B) actually 'clone' my current EC2 instance to create additional instances.
I haven't been able to really paint a clear picture of the above in my head yet, despite the fact that I've gone over a few (highly technical) suggestions.
Thanks!
You need to treat your EC2 instance as 100% dispensable. Meaning, that it can be terminated at any time and you should not care. A replacement EC2 instance would start and take over the work.
There are 3 ways to accomplish this:
Method 1: Each deployment creates a new AMI image.
When you deploy you app, you deploy it to a worker EC2 instance whose sole purpose is for "setup" of your app. Once the new version is deployed, you create a fresh AMI image from the EC2 instance and update your Auto Scaling launch configuration with the new AMI image. The old EC2 instances are terminated and replaced with the new code.
New EC2 instances have the recent code already on them so they're ready to be added to the load balancer.
Method 2: Each deployment is done to off-instance storage (like Amazon S3).
The EC2 instances will download the recent code from Amazon S3 and install it on boot.
So to put the new code in action, you terminate the old instances and new ones are launched to replace them which start using the new code.
This could be done in a rolling-update fashion, or as a blue/green deployment.
Method 3: Similar to method 2, but this time the instances have some smarts and can be signaled to download and install the code.
This way, you don't need to terminate instances: the existing instances are told to update from S3 and they do so on their own.
Some tools that may help include:
Chef
Ansible
CloudFormation
Update:
Methods 2 & 3 both start with a "basic" AMI which is configured to pull the webpage assets from S3. This AMI is not changed from version-to-version of your website.
For example, the AMI can have Apache and PHP already installed and on boot it pulls the .php website assets from S3 and puts them in /var/www/html.
CloudFormation works well for this. In addition, for method 3, you can use cfn-hup to wait for update signals. When signaled, it'll pull updated assets from S3.
Another possibility is using Elastic Beanstalk which could be used to manage all of this for you.
Update:
To have your AMI image pull from Git, try the following:
Setup an EC2 instance with everything installed that you need to have installed for your web app
Install Git and setup a local repo ready to Git pull.
Shutdown and create an AMI of your instance.
When you deploy, you do the following:
Git push to GitHub
Launch a new EC2 instance, based on your AMI image.
As part of the User Data (specified during the EC2 instance launch), specify something like the following:
#!/bin/sh
cd /git/app
git pull
; copy files from repo to web folder
composer install
When done like this, that user data acts as a script which will run on first boot.
We started our wordpress blog some time ago with only one single EC2 Instance and a Multi-AZ RDS Database.
The traffic increased with heavy ups and downs (up to 1.500 user per minute), so we decided to use EC2 Auto Scaling. Here is our problem: Every time we changed some code, we have to create a new AMI for the Auto Scaling Group and terminate all instance so new instances will start with the new AMI Data.
Is there a easy way to synchronize all instance automatically, when changing some code on one of them? Perhaps Opsworks could to that but I haven't experience with this. I already searched a couple of days for a tutorial, but could not find anything helpful.
You could configure your AMI to download the latest code on startup, so that you don't have to constantly update the AMI.
Or you could just use Elastic Beanstalk and let it manage all this stuff for you.
If you want an easy way to deploy changes to instances in your autoscaling group, I would recommend using Code Deploy.
Code Deploy integrates nicely with Autoscaling. If a scaling up event occurs, it will start a deployment to the newly launched instance and won't bring that instance into service in the AutoScaling group until the deployment has finished.
The deployments can be as simple as changing the code or else they can involve more thanks to Code Deploys deployment hooks.
Also you can have Code Deploy grab your code from S3, Github or CodeCommit.
Code Deploy is pretty easy to set up and the documentation is great:
Docs AutoScaling Integreation