In my auto scaling group I want to copy my existing AMI and copy it to bring up a new instance.
I can easily use a script to create the new instance but the developers change the code without committing to git hub so the new instance will not have the changes.So bootstrap or user script is out of the windows.
Creating a snapshot and making an instance manually is been ruled out.
I there any way using AWS CLI or SDK so that I can copy an existing running AMI and bring it up in auto scaling group.
Related
I have completely understood the concept of Auto-Scaling in AWS. My only question is, what AMI will the launch configuration use while in production environment?
According to my understanding Image of existing instance should be used. Lets say I have used an image of existing instance.
What if there are any changes in existing instance in future? In this scenario we have to update the AMI.
Is there any process to automate this process?
When you create new AMI and set it in a new launch configuration (LC; LC can't be edited) or new version of a launch template (LT), then you will have to update the ASG configuration with the new LC/LT.
However, ASG by default will not update existing instances with new LC/LT. Only new instance that ASG launches will have the new LC/LT, and subsequently, the new AMI. Therefore, you will end up with ASG in which part of instances is running old AMI, and the other part is running new AMI.
You can deal with this in two commonly used ways:
Create your LC/LT and ASG using CloudFormation and specify UpdatePolicy. The update policy will be triggered when LC/LT changes, and existing instances in ASG will be updated based on the rules you specify in the policy.
Perform blue/green deployment of your ASG. How to perform the deployment is described and explained in details in an excellent AWS white paper:
Blue/Green Deployments on AWS
Auto scaling uses AMIs which are a point in time snapshot of your instance. Any changes made thereafter will not be applied to the AMI.
If you want any change to your base image you will need to recreate an image and roll it out across your Launch Configuration/Launch Template again.
There are many tools people use to provision the configuration of instances for AMIs such as Ansible, Chef and Puppet.
AWS also launched an automation tool for building images last year, the EC2 Image Builder
For some additional reading take a look at the golden ami pipeline.
In AWS cloud formation, i use the cloud former tool. I can use that tool to create a cloud formation template from existing resources. And then use the template to create a stack. I tested with that tool. It can work, (as in it can create instances with same memory size, with same volume size, same VPC settings, and auto start the instances). But there is no files in the volume.
Do i have to create a snapshot of the existing volume, create a new volume from the snapshot, attach it to the newly created instance, and copy the files manually ?
Or is there any better way ?
Do i have to create a snapshot of the existing volume, create a new volume from the snapshot, attach it to the newly created instance, and copy the files manually ?
Cloudformation is provisioning resources, but is not responsible for provisioning the contents of those resources - that you have to do yourself.
You can leverage the EC2 Userdata to manually pull files from S3 or other repos as the instance boots.
Or is there any better way ?
If you want to share data between applications, EFS is always an option. In your case, though, using Userdata might be effective.
If you wish to launch new EC2 instances with software automatically loaded, there are basically two choices:
Use a pre-configured AMI, or
Use a startup script to load the software
Pre-configured AMI
An Amazon Machine Image (AMI) is a copy of a disk. When a new EC2 instance is launched, an AMI is selected and the boot disk (and optionally other disks) are automatically pre-loaded with the contents of the AMI.
A common practice is to boot an EC2 instance and configure it as desired. Then, create an AMI. Thereafter, when a new EC2 instance is required for the application, launch it using the pre-built AMI.
There are also tools available to automate the building of an AMI, such as Netflix Aminator and Packer.
Benefits: New machine boots quickly, fully-configured.
Issues: Need to create a new AMI whenever you update your software.
Use a startup script to load the software
When an Amazon EC2 instance is launched from a standard Amazon-provided AMI (eg Amazon Linux, Microsoft Windows), software on the AMI automatically looks at the User Data passed to an EC2 instance. If the User Data contains a startup script, the script will be executed -- but only the first time that an instance is launched. This is an excellent way to install software on the instance.
You are responsible for writing the script. The script should install whatever tools, software and data you want on the instance.
Benefits: Updating your software? Just launch a new instance and the script will install the latest version of your software (assuming you have written the script to always point to the latest version).
Issues: It takes longer to launch the new instance, since the software is being installed.
I have setup a process for installing and setting up my application stack on a ubuntu base image and then creating an AMI from the machine. All of this is automated using packer. Now once I have the ami-id once packer is done I manually create a new launch configuration and update my ASG configuration and then schedule scale up and scale down action which gets rid of the old instances.
So what I am looking for is:
Please suggest a better way to update my application stack whenever there is a new update for any software version (e.g ngnix ruby etc)
How can I automate the roll out process so the new ami is picked up and old instance should degrade.
There is a good strategy for this in Faster Auto Scaling in AWS CloudFormation Stacks with Lambda-backed Custom Resources
To orchestrate this process, you bootstrap a reference instance with a
user data script, use wait conditions to trigger an AMI capture, and
finally create an Auto Scaling group launch configuration that
references the newly created AMI. The reference instance that is used
to capture the AMI can then be terminated, or it can be repurposed for
administrative access or for performing scheduled tasks.
The process does not use Packer and does not require a dedicated server for creating the AMI, and instead uses a Lambda-backed custom resource.
Second Option
As you already have your AMI creation in Packer working, you should consider using Lambda to copy your existing Launch Configuration with the updated AMI. You can see a good approach to this from Patch an AMI and Update an Auto Scaling Group:
The following example builds on the Simplify AMI Patching Using
Automation, Lambda, and Parameter Store example by adding a step that
updates an Auto Scaling group with the newly-patched AMI. This
approach ensures that new images are automatically made available to
different computing environments that use Auto Scaling groups.
The final step of the Automation workflow in this example uses an AWS
Lambda function to copy an existing launch configuration and set the
AMI ID to the newly-patched AMI. The Auto Scaling group is then
updated with the new launch configuration. In this type of Auto
Scaling scenario, users could terminate existing instances in the Auto
Scaling group to force a new instance to launch that uses the new
image. Or, users could wait and allow scale-in or scale-out events to
naturally launch newer instances.
I have created Auto scaling group in AWS using a customized AMI. Now to rollout my new code I need to either update all instances running but then if a new instance comes up it won't be updated. So, I need a way to update AMI. One way could be creating new AMI and Autoscaling group.
Thanks in advance.
This is one way to go about it:
Spin up a stand-alone instance using the AMI
Make changes
Stop instance
Create new AMI from this instance
Create a new Launch Configuration that uses the new AMI
Update the Autoscaling Group to use the new Launch Configuration
Slowly terminate the old instances in the Autoscaling Group, and let them be automatically replaced with instances using the new AMI
Of course all this is a pain to deal with manually every time you need to make a change. Elastic Beanstalk and CloudFormat both provide mechanisms to deal with this in a more automated way.
If you are just changing the code you are deploying to your servers, then there are other ways to handle this, such as using AWS CodeDeploy. You could also update the running servers in some automated or manual fashion, and configure the AMI such that any new instances that are created will go get the latest code on startup.
We use AWS cloudformation service to initialize our stack, and set up the auto scaling service to bring up new app servers when load is rising.
My understanding is that Auto Scaling can only start predefined AMI as new instances. These instances could be different from other running instances, because we may have updated packages/source code deployed on those instances.
How can I bring the new instances up-to-date?
Should I update the AMIs everytime I deploy something new to the running instances? Or is there anyway to trigger auto-deployment on new instances (Opsworks) when auto scaling?
I am new to AWS, so pardon me if my question is rudimentary.
There are multiple ways of doing this. My preferred approach is never to touch the servers directly, but instead create a new AMI whenever I deploy a new version of the software.
To do this, use the AutoScalingRollingUpdate property for the auto-scaling group. When you then change ImageId for the launch configuration, AWS will automatically replace your old servers with new ones as a rolling upgrade.
I have a simple deploy script that creates a new AMI, replaces ImageId in the template, and then does a stack update - AWS takes care of the rest.
When creating EC2 instances from Beanstalk, it automatically creates a AutoScaling Group and Launch Configuration based on the specified environment selections. Creating the instance from base AMI is done using a custom code call user data which includes the shell script to create folders and install relevant software.
You can add a new shell scripts or commands there to do your custom work before starting a new instance. This way it is much simpler. e.g. you can run yum update before starting a instance
To find user data section
Go to EC2 Console -> Go to launch configurations section (on left) -> Select the correct launch configuration and copy it -> Click view user data -> Add your scripts and commands as required -> Modify the relevant Auto Scaling group to point to the new launch configuration