I'm using Codedeploy to push to my ec2 instances within an auto scaling group. At times, that auto scaling group doesn't have any existing instances running. When I deploy in that situation, codedeploy ALWAYS fails, even though I've set the minimum healthy hosts to 0 instances.
Is there anyway I can get code deploy to say "success" when there are 0 instances?
It appears when codedeploy fails, it doesn't update the revision. This is a real pain in my situation.
You need to have at least a single instance in your deployment group for the deployment to succeed. After you hook the Autoscaling group (containing at least 1 instance) with CodeDeploy, you should do a successful deployment to update the target revision of the deployment group. After this, any new instance scale up should automatically pick up the target revision.
You could also set the :min property of your autoscaling group to 1 to always keep a single instance in it.
I know it's been over two years, but I faced this same issue. My workaround was creating my own Lifecycle Hook for my Auto Scaling Group and an SNS + Lambda for deploying my revisions.
The catch is, you should first Register a Revision for the application without deploying it. As soon as a new instance is created by the Auto Scaling Group the hook will send an SNS message to the Lambda and then you can (based on the message received + Environment variables) look for revisions (already created) and deploy them to the new instances.
I've linked all this by using CloudFormation - which I extremely recommend for this workaround and all other AWS related services.
Related
I'm trying to use CodeDeploy and ECS with an Application Load Balancer, and Auto Scaling with a strategy on the number of request of this ALB.
I'm using this URL as some sort of tutorial, but i don't really understand how CodeDeploy will integrate with ECS and other stuffs.
First, as i can see, i need two target groups on my ALB. But i only have one (that is, instance(s) that are / will be created by Auto Scaling)
So what do i need to do ? Does creating an empty target group, and telling CodeDeploy to use both work ?
What will it do, deploy instances in this target group and redirect a part of or the whole trafic to it once working ? As stated in the same link
"During deployment, CodeDeploy installs your update into a new, replacement task set."
So it seems to create new tasks, but on what instance then ?
So what do i need to do ? Does creating an empty target group, and telling CodeDeploy to use both work ?
It can't be empty. Your new TG can be same as your first one. So at this stage you will have to TGs which are same (except thier name).
The second TG is specified when you create your ECS deployment group in CodeDeploy.
What will it do, deploy instances in this target group and redirect a part of or the whole traffic to it once working ? As stated in the same link
These will be same instances as the one running your current ecs task.
So it seems to create new tasks, but on what instance then ?
These will be same instances as those that run your current task.
I have tried setting up a Blue/Green deployment by copying AutoScalingGroup, however this leaves the CloudFormation stack detached from its original resources as CodeDeploy creates a new copy and deletes the original. I understand from another post (https://forums.aws.amazon.com/thread.jspa?messageID=861085) that AWS are developing improvements for this, however for now I am trying the following workaround. Any ideas would be really helpful.
CloudFormation creates the following:
Elastic Load Balancer
Target Group
AutoScalingGroup One (with LaunchConfiguration)
AutoScalingGroup Two (same as one but has no instances)
DeploymentGroup (with In-Place DeploymentStyle) which deploys a revision to AutoScalingGroup One
After CloudFormation finishes, I do the following manually in the console:
I update the created Deployment Group to be of Deployment Style Blue/Green and set its original environment to be AutoScalingGroup One.
I add an instance to AutoScalingGroup Two
I create a deployment in CodeDeploy. However, this does not work as when a new instance is attached to AutoScalingGroup Two, it gets added to the TargetGroup immediately and does not pass health checks.
Any ideas on how to implement a set of resources with CloudFormation that will make blue green deployments simple, i.e. one click in CodeDeploy and CloudFormation resources still remaining intact?
With regard to the initial issue you are describing, did you experiment with the Health Check Grace Period? That should prevent the problems you describe with the failing health check when the instance hits the target group.
An alternative approach (which has plenty of its own downsides) is to adapt the CloudFormation template to compensate for the behavior when CodeDeploy replaces the ASG in a Blue-Green deployment.
Within the ASG template, create a "yes/no" parameter called
"ManageAutoScalingGroup". Create the ASG conditionally on the value
of this parameter being "yes". Set a deletion policy on the ASG of
retain so that CloudFormation will leave the group in place when the
parameter is changed to "no".
Spin up the group with a default "yes"
on this parameter.
Once the instances are healthy, and CodeDeploy has completed an initial in-place deployment, you can change the DeploymentGroup to use Blue-Green where CodeDeploy will replace your ASG.
Be sure to update the ASG and change ManageAutoScalingGroup to "no". CloudFormation will delete the reference from your stack, but it will leave the resource in place.
This will give you the one-click deployments you desire through CodeDeploy, but be aware that it comes with some costs:
CodeDeploy will not copy the TargetGroup parameter of your Auto Scaling Group (as described by others in https://forums.aws.amazon.com/thread.jspa?threadID=249406&tstart=0). You should be able to work around this with a clever use of CloudWatch event rules and SSM Automation to mark the instance unhealthy when the ALB changes its status.
The copies that CodeDeploy produces seem to be fairly unreliable. At least once, I've seen my LaunchTemplate version reset to an incorrect value. I've also run into scenarios where the deployment group lost track of which ASG it was supposed to track.
Continuing to apply changes from your template to the ASG is a hassle. The process to "refresh" the group is: 1) Revert the parameter described earlier such that CloudFormation will generate a new group. 2) Modify the deployment group to target this group and complete an in-place deployment. 3) Modify the deployment group to restore Blue-Green deployments and update your stack accordingly.
I'm not too impressed with CodeDeploy in this department. I'd love to see them work in the same fashion as an ASG that is set to replace itself on application of a new LaunchTemplate version. If you are feeling a bit ambitious, you could mimic this behavior by leveraging Step Functions with ASG instance lifecycle hooks. This is a solution that I'm considering once I have the time.
I'm looking to see if I can create an instance and deploy applications to athis instance dynamically via the API. I only want these instances to be created when my application needs them, or I request for them to be created.
I have two applications that I need to deploy to each created instance which require some set up and installation of dependencies prior to their launch. When I am finished with this application, I want to terminate the instance.
Am I able to do this? If so, could someone please point me to the right section of the documentation. I have searched on the documentation and found some information about creating images but I am unsure as to what exactly I will need to achieve this task.
Yes. Using an Autoscaling Group, you can create a launch configuration that will launch you instances. Using CodeDeploy, you would link your deployment group to the auto-scaling group.
See Integrating AWS CodeDeploy with Auto Scaling
AWS CodeDeploy supports Auto Scaling, an AWS service that can launch
Amazon EC2 instances automatically according to conditions you define.
These conditions can include limits exceeded in a specified time
interval for CPU utilization, disk reads or writes, or inbound or
outbound network traffic. Auto Scaling terminates the instances when
they are no longer needed. For more information, see What Is Auto
Scaling?.
Assuming you set your desired/minimum instances to 0, then the default state of the ASG will be to have no instances.
When you application needs an instance spun up, it would simply change the desired instance value to 1. When your application is completed with the instance, it would set your desired count to 0 again, thereby terminating that instance.
To develop this setup, start by running your instance normally (manually) and get the application deployment working. When that works, then create your auto scaling group. Finally, update your deployment group to refer to the ASG so that your code is deployed when you have scaling events.
You can create new Launch Configuration (updating AMI or whatever) and attach this with an existing Autoscaling Group. Per AWS Docs: After you change the launch configuration for an Auto Scaling group, any new instances are launched using the new configuration options, but existing instances are not affected.
How do you force this? Meaning relaunch all new instances now (with the new AMI). Do I have to delete the existing Autoscaling Group and create a new Autoscaling Group (with new Config)? Or I simple delete existing instances (one by one manually) and then ASG relaunch with new AMI. Any best practices/gotchas?
CloudFormation has the RollingUpdate flag (not sure of this outside of CF)
Thanks
AWS has some OOTB solutions for this, CloudFormation (like you say), Elastic Beanstalk (built on top of CF), and CodeDeploy blue-green deployments (I've not tried this).
Personally for our SQS polling ASG, we do a "cold deploy" i.e. only "deploy" when there are no messages to process (and hence, due a scaling policy, no instances). It's been really effective.
A deploy can be done safely whilst there are messages, provided that you set scale-in-protection on the instance during message processing (and remove it and wait briefly before polling):
set desired-capacity to 0
wait a bit (for there to be no instances running)
set desired-capacity back to N.
Note: you can do this all in the console.
You can code a solution yourself that does this... but I probably wouldn't.
Be careful:
simple delete existing instances (one by one manually)
Whether you can do this, or depends on whether the instances are still handling requests/processing (usually you can't simply terminate an instance without dropping service).
I recommend Elastic Beanstalk which gives a rolling update feature for free and is very easy to get started. I've not tried the CodeDeploy blue-green but it looks interesting. If you want more advanced behavior (or are already using it) look into Cloud Formation... do not code your own solution for rolling deployments: just use CloudFormation.
if your issue is with "in flight" requests simply enable connection draining or increase de-registration delay of the ELB or "target groups" attached with the ASG. You can set a value up to one hour.
When you enable connection draining, you can specify a maximum time for the load balancer to keep connections alive before reporting the instance as de-registered. The maximum timeout value can be set between 1 and 3,600 seconds (the default is 300 seconds). When the maximum time limit is reached, the load balancer forcibly closes connections to the de-registering instance.
Then you can detached old instances.
If you detach an instance from an Auto Scaling group that has an attached load balancer, the instance is deregistered from the load balancer. If you detach an instance from an Auto Scaling group that has an attached target group, the instance is deregistered from the target group. If connection draining is enabled for your load balancer, Auto Scaling waits for in-flight requests to complete.
If you don't want to do any manual scaling I guess the best approach is to changing the termination policy to OldestInstance and leave the ASG as it is. When the scale-in activity happens ASG will automatically terminate the old instances.(in your case old launch config instances)
OldestInstance. Auto Scaling terminates the oldest instance in the group. This option is useful when you're upgrading the instances in the Auto Scaling group to a new EC2 instance type. You can gradually replace instances of the old type with instances of the new type.
Here's what I have in AWS:
Application ELB
Auto Scaling Group with 2 instances in different regions (Windows IIS servers)
Launch Config pointing to AMI_A
all associated back end stuff configured (VPC, subnets, security groups, ect)
Everything works. However, when I need to make an update or change to the servers, I am currently manually creating a new AMI_B, creating a new LaunchConfig using AMI_B, updating the AutoScalingGroup to use the new LaunchConfig, increasing min number of instances to 4, waiting for them to become available, then decreasing the number back to 2 to kill off the old instances.
I'd really love to automate this process. Amazon gave me some links to CLI stuff, and I'm able to script the AMI creation, create the LaunchConfig, and update the AutoScalingGroup...but I don't see an easy way to script spinning up the new instances.
After some searching, I found some CloudFormation templates that look like they'd do what I want, but most do more, and it's a bit confusing to me.
Should I be exploring CloudFormation? Is there a simple guide I can follow to get started? Or should I stay with the scripting I have started?
PS - sorry if this is a repeated question. Things change frequently at AWS, so sometimes the older responses may not be the current best answers.
You have a number of options to automate the process of updating the instances in an Auto Scaling Group to a new or updated Launch Configuration:
CloudFormation
If you do want to use CloudFormation to manage updates to your Auto Scaling Group's instances, refer to the UpdatePolicy attribute of the AWS::AutoScaling::AutoScalingGroup Resource for documentation, and the "What are some recommended best practices for performing Auto Scaling group rolling updates?" page in the AWS Knowledge Center for more advice.
If you'd also like to script the creation/update of your AMI within a CloudFormation resource, see my answer to the question, "Create AMI image as part of a cloudformation stack".
Note, however, that CloudFormation is not a simple tool- it's a complex, relatively low-level service for orchestrating AWS resources, and migrating your existing scripts to it will likely take some time investment due to its steep learning curve.
Elastic Beanstalk
If simplicity is most important, then I'd suggest you evaluate Elastic Beanstalk, which also supports both rolling and immutable updates during deployments, in a more fully managed, console-oriented, platform-as-a-service environment. Refer to my answer to the question, "What is the difference between Elastic Beanstalk and CloudFormation for a .NET project?" for further comparisons between CloudFormation and Elastic Beanstalk.
CodeDeploy
If you want a solution for updating instances in an auto-scaling group that you can plug into existing scripts, AWS CodeDeploy might be worth looking into. You install an agent on your instances, then trigger deployments through the API/CLI/Console and it manages deploying application updates to your fleet of instances. See Deploy an Application to an Auto Scaling Group Using AWS CodeDeploy for a complete tutorial. While CodeDeploy supports 'in-place' deployments and 'blue-green' deployments (see Working With Deployments for details), I think this service assumes an approach of swapping out S3-hosted application packages onto a static base AMI rather than replacing AMIs on each deployment. So it might not be the best fit for your AMI-swapping use case, but perhaps worth looking into anyway.
You want a custom Termination policy on the Auto Scaling Group.
OldestLaunchConfiguration. Auto Scaling terminates instances that have the oldest launch configuration. This policy is useful when you're updating a group and phasing out the instances from a previous configuration.
To customize a termination policy using the console
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
On the navigation pane, choose Auto Scaling Groups.
Select the Auto Scaling group.
For Actions, choose Edit.
On the Details tab, locate Termination Policies. Choose one or more
termination policies. If you choose multiple policies, list them in
the order that you would like them to apply. If you use the Default
policy, make it the last one in the list.
Choose Save.
On the CLI
aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg --termination-policies "OldestLaunchConfiguration"
https://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html
We use Ansible's ec2_asg module for that purpose. There are replace_all_instances and replace_batch_size settings for that purpose. Per documentation:
In a rolling fashion, replace all instances that used the old launch configuration with one from the new launch configuration.
It increases the ASG size by C(replace_batch_size), waits for the new instances to be up and running.
After that, it terminates a batch of old instances, waits for the replacements, and repeats, until all old instances are replaced.
Once that's done the ASG size is reduced back to the expected size.
If you provide target_group_arns, module will check for health of instances in target groups before going to next batch.
Edit: in order to maintain desired number of instances, we first set min to desired.