I have completely understood the concept of Auto-Scaling in AWS. My only question is, what AMI will the launch configuration use while in production environment?
According to my understanding Image of existing instance should be used. Lets say I have used an image of existing instance.
What if there are any changes in existing instance in future? In this scenario we have to update the AMI.
Is there any process to automate this process?
When you create new AMI and set it in a new launch configuration (LC; LC can't be edited) or new version of a launch template (LT), then you will have to update the ASG configuration with the new LC/LT.
However, ASG by default will not update existing instances with new LC/LT. Only new instance that ASG launches will have the new LC/LT, and subsequently, the new AMI. Therefore, you will end up with ASG in which part of instances is running old AMI, and the other part is running new AMI.
You can deal with this in two commonly used ways:
Create your LC/LT and ASG using CloudFormation and specify UpdatePolicy. The update policy will be triggered when LC/LT changes, and existing instances in ASG will be updated based on the rules you specify in the policy.
Perform blue/green deployment of your ASG. How to perform the deployment is described and explained in details in an excellent AWS white paper:
Blue/Green Deployments on AWS
Auto scaling uses AMIs which are a point in time snapshot of your instance. Any changes made thereafter will not be applied to the AMI.
If you want any change to your base image you will need to recreate an image and roll it out across your Launch Configuration/Launch Template again.
There are many tools people use to provision the configuration of instances for AMIs such as Ansible, Chef and Puppet.
AWS also launched an automation tool for building images last year, the EC2 Image Builder
For some additional reading take a look at the golden ami pipeline.
Related
So we are launching an ecommerce store built on magento. We are looking to deploy it on Amazon EC2 instance using RDS as database service and using amazon auto-scaling and elastic load balancer to scale the application when needed.
What I don't understand is this:
I have installed and configured my production magento enviorment on an EC2 instance (database is in RDS). This much is working fine. But now when I want to dynamically scale the number of instances
how will I deploy the code on the dynamically generated instances each time?
Will aws copy the whole instance assign it a new ip and spawn it as a
new instance or will I have to write some code to automate this
process?
Plus will it not be an overhead to pull code from git and deploy every time a new instance is spawned?
A detailed explanation or direction towards some resources on the topic will be greatly appreciated.
You do this in the AutoScalingGroup Launch Configuration. There is a UserData section in the LaunchConfiguration in CloudFormation where you would write a script that is ran when ever the ASG scales up and deploys a new instance.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig.html#cfn-as-launchconfig-userdata
This is the same as the UserData section in an EC2 Instance. You can use LifeCycle hooks that will tell the ASG not to put the EC2 instance into load until everything you want to have configured it set up.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-as-lifecyclehook.html
I linked all CloudFormation pages, but you may be using some other CI/CD tool for deploying your infrastructure, but hopefully that gets you started.
To start, do check AWS CloudFormation. You will be creating templates to design how the infrastructure of your application works ~ infrastructure as code. With these templates in place, you can rollout an update to your infrastructure by pushing changes to your templates and/or to your application code.
In my current project, we have a github repository dedicated for these infrastructure templates and a separate repository for our application code. Create a pipeline for creating AWS resources that would rollout an updated to AWS every time you push to the repository on a specific branch.
Create an infrastructure pipeline
have your first stage of the pipeline to trigger build whenever there's code changes to your infrastructure templates. See AWS CodePipeline and also see AWS CodeBuild. These aren't the only AWS resources you'll be needing but those are probably the main ones, of course aside from this being done in cloudformation template as mentioned earlier.
how will I deploy the code on the dynamically generated instances each time?
Check how containers work, it would be better and will greatly supplement on your learning on how launching new version of application work. To begin, see docker, but feel free to check any resources at your disposal
Continuation with my current project: We do have a separate pipeline dedicated for our application, but will also get triggered after our infrastructure pipeline update. Our application pipeline is designed to build a new version of our application via AWS Codebuild, this will create an image that will become a container ~ from the docker documentation.
we have two triggers or two sources that will trigger an update rollout to our application pipeline, one is when there's changes to infrastructure pipeline and it successfully built and second when there's code changes on our github repository connected via AWS CodeBuild.
Check AWS AutoScaling , this areas covers the dynamic launching of new instances, shutting down instances when needed, replacing unhealthy instances when needed. See also AWS CloudWatch, you can design criteria with it to trigger scaling down/up and/or in/out.
Will aws copy the whole instance assign it a new ip and spawn it as a new instance or will I have to write some code to automate this process?
See AWS ElasticLoadBalancing and also check out more on AWS AutoScaling. On the automation process, if ever you'll push through with CloudFormation, instance and/or containers(depending on your design) will be managed gracefully.
Plus will it not be an overhead to pull code from git and deploy every time a new instance is spawned?
As mentioned, earlier having a pipeline for rolling out new versions of your application via CodeBuild, this will create an image with the new code changes and when everything is ready, it will be deployed ~ becomes a container. The old EC2 instance or the old container( depending on how you want your application be deployed) will be gracefully shut down after a new version of your application is up and running. This will give you zero downtime.
I have setup a process for installing and setting up my application stack on a ubuntu base image and then creating an AMI from the machine. All of this is automated using packer. Now once I have the ami-id once packer is done I manually create a new launch configuration and update my ASG configuration and then schedule scale up and scale down action which gets rid of the old instances.
So what I am looking for is:
Please suggest a better way to update my application stack whenever there is a new update for any software version (e.g ngnix ruby etc)
How can I automate the roll out process so the new ami is picked up and old instance should degrade.
There is a good strategy for this in Faster Auto Scaling in AWS CloudFormation Stacks with Lambda-backed Custom Resources
To orchestrate this process, you bootstrap a reference instance with a
user data script, use wait conditions to trigger an AMI capture, and
finally create an Auto Scaling group launch configuration that
references the newly created AMI. The reference instance that is used
to capture the AMI can then be terminated, or it can be repurposed for
administrative access or for performing scheduled tasks.
The process does not use Packer and does not require a dedicated server for creating the AMI, and instead uses a Lambda-backed custom resource.
Second Option
As you already have your AMI creation in Packer working, you should consider using Lambda to copy your existing Launch Configuration with the updated AMI. You can see a good approach to this from Patch an AMI and Update an Auto Scaling Group:
The following example builds on the Simplify AMI Patching Using
Automation, Lambda, and Parameter Store example by adding a step that
updates an Auto Scaling group with the newly-patched AMI. This
approach ensures that new images are automatically made available to
different computing environments that use Auto Scaling groups.
The final step of the Automation workflow in this example uses an AWS
Lambda function to copy an existing launch configuration and set the
AMI ID to the newly-patched AMI. The Auto Scaling group is then
updated with the new launch configuration. In this type of Auto
Scaling scenario, users could terminate existing instances in the Auto
Scaling group to force a new instance to launch that uses the new
image. Or, users could wait and allow scale-in or scale-out events to
naturally launch newer instances.
I have created Auto scaling group in AWS using a customized AMI. Now to rollout my new code I need to either update all instances running but then if a new instance comes up it won't be updated. So, I need a way to update AMI. One way could be creating new AMI and Autoscaling group.
Thanks in advance.
This is one way to go about it:
Spin up a stand-alone instance using the AMI
Make changes
Stop instance
Create new AMI from this instance
Create a new Launch Configuration that uses the new AMI
Update the Autoscaling Group to use the new Launch Configuration
Slowly terminate the old instances in the Autoscaling Group, and let them be automatically replaced with instances using the new AMI
Of course all this is a pain to deal with manually every time you need to make a change. Elastic Beanstalk and CloudFormat both provide mechanisms to deal with this in a more automated way.
If you are just changing the code you are deploying to your servers, then there are other ways to handle this, such as using AWS CodeDeploy. You could also update the running servers in some automated or manual fashion, and configure the AMI such that any new instances that are created will go get the latest code on startup.
We use AWS cloudformation service to initialize our stack, and set up the auto scaling service to bring up new app servers when load is rising.
My understanding is that Auto Scaling can only start predefined AMI as new instances. These instances could be different from other running instances, because we may have updated packages/source code deployed on those instances.
How can I bring the new instances up-to-date?
Should I update the AMIs everytime I deploy something new to the running instances? Or is there anyway to trigger auto-deployment on new instances (Opsworks) when auto scaling?
I am new to AWS, so pardon me if my question is rudimentary.
There are multiple ways of doing this. My preferred approach is never to touch the servers directly, but instead create a new AMI whenever I deploy a new version of the software.
To do this, use the AutoScalingRollingUpdate property for the auto-scaling group. When you then change ImageId for the launch configuration, AWS will automatically replace your old servers with new ones as a rolling upgrade.
I have a simple deploy script that creates a new AMI, replaces ImageId in the template, and then does a stack update - AWS takes care of the rest.
When creating EC2 instances from Beanstalk, it automatically creates a AutoScaling Group and Launch Configuration based on the specified environment selections. Creating the instance from base AMI is done using a custom code call user data which includes the shell script to create folders and install relevant software.
You can add a new shell scripts or commands there to do your custom work before starting a new instance. This way it is much simpler. e.g. you can run yum update before starting a instance
To find user data section
Go to EC2 Console -> Go to launch configurations section (on left) -> Select the correct launch configuration and copy it -> Click view user data -> Add your scripts and commands as required -> Modify the relevant Auto Scaling group to point to the new launch configuration
I have an instance in AWS that I set up my entire environment (I'll call it my ready instance) on and is running perfectly. I then created a load balancer (ELB) with an autoscaling policy.
When I created a load balancer with an autoscaling policy (min of 2 instances), 2 instances sprung up. The instances were empty, however. For the launch configuration, I specified my ready instance AMI. Isn't this supposed to tell the autoscaling policy which instance to clone? In this case, shouldn't my ready instance be cloned into them and they should have the same content?
Instances are not created based on a clone of a running instance, but rather just the disk image stored in the AMI. It might be a case where you need to create a new AMI from your running instance and use that AMI as the basis for your autoscaling group.
once upon a time even I was this phase of my life.Basically it will just the boot the AMI which you have specified in your as configuration. If your AMI has got old code, then it will boot and serve your client from the out-dated code. Do solve this you can automate the code management process, all you have to do is boot the new ami with a user-data script which will perfom certain actions during the boot. Using user-data script you can automate this process. Autoscaling configuration also have a provision to accept the user-script data.