Best DevOps use of AWS Auto-scaling Groups? - amazon-web-services

Question

I've been working on a DevOps pipeline for an application hosted on AWS. I want to make an improvement to my current setup, but I'm not sure the best way to go about doing it. My current set up is as follows:
ASG behind ELB
Desired capacity: 1
Min capacity: 1
Max capacity: 1
Code deployment process:
move deployable to S3
terminate instance in ASG
new instance is automatically provisioned
new instance pulls down deployable in user data
The problem with this setup is that the environment is down from when the instance is terminated to when the new instance has been completely provisioned.
I've been thinking about ways that I can improve this process to eliminate the downtime, and I've come up with two possible solutions:
SOLUTION #1:
ASG behind ELB
Desired capacity: 1
Min capacity: 1
Max capacity: 2
Code deployment process:
move deployable to S3
launch new instance into ASG
new instance pulls down deployable in user data
terminate instance with old deployable
With this solution, there is always at least one instance capable of serving requests in the ASG. The problem is, ASGs don't seem to support a simple operation of manually calling on it to spin up a new instance. (They only launch new instances when the scaling policies call for it.) You can attach existing instances to the group, but this causes the desired capacity value to increase, which I don't want.
SOLUTION #2:
ASG behind ELB
Desired capacity: 2
Min capacity: 2
Max capacity: 2
Code deployment process:
move deployable to S3
terminate instance-A
new instance-A is automatically provisioned
instance-A pulls down new deployable by user data script
terminate instance-B
new instance-B is automatically provisioned
instance-B pulls down new deployable by user data script
Just as with the previous solution, there is always at least one instance available to serve requests. The problem is, there are usually two instances, even when only one is needed. Additionally, the code deployment process seems needlessly complicated.
So which is better: solution #1, solution #2, or some other solution I haven't thought of yet? Also a quick disclaimer: I understand that I'm using ASGs for something other than their intended purpose, but it seemed the best way to implement automated code deployments along AWS's "EC2 instances are cattle" philosophy.

Answer 1

The term you are looking for is "zero-downtime deployment."
The problem is, ASGs don't seem to support a simple operation of manually calling on it to spin up a new instance. (They only launch new instances when the scaling policies call for it.) You can attach existing instances to the group, but this causes the desired capacity value to increase, which I don't want.
If you change desired capacity yourself (e.g. via an API call), the Auto Scaling Group will automatically launch an extra instance for you. For example, here is a simple way to implement zero-downtime deployment for your Auto Scaling Group (ASG):
Run the ASG behind an Elastic Load Balancer (ELB).
Initially, the desired capacity is 1, so you have just one EC2 Instance in the ASG.
To deploy new code, you first create a new launch configuration with the new code (e.g. new AMI or new User Data).
Next, you change the desired capacity from 1 to 2. The ASG will automatically launch a new EC2 Instance with the new launch configuration.
Once the new EC2 Instance is up and running and registered in your ELB, you change the desired capacity from 2 back to 1, and the ASG will automatically terminate the older EC2 Instance.
You can implement this manually or use existing tools to do it for you, such as:
Define your ASG using CloudFormation and specify an UpdatePolicy that does a zero-downtime rolling deployment.
Define your ASG using Terraform and use the create_before_destroy lifecycle property to do a zero-downtime (sort-of) blue-green deployment as described here.
Define your ASG using Ansible and use the serial keyword to do rolling upgrades.
Use the aws-ha-release script.
You can learn more about the trade-offs between tools like Terraform, CloudFormation, Ansible, Chef, and Puppet here.

Answer 2

Even though this is a DevOps pipeline and not a production environment, what you are describing sounds like a blue/green deployment scenario in which you want to be able to switch between environments without downtime. I think the best answer is largely specific to your requirements (which we don't 100% know), but a guide like The DOs and DON'Ts of Blue/Green Deployment will be beneficial in finding the best way to achieve your goals, whether it is #1, #2, or something else.