How can I make sure ASG is scaling EC2 instances in a correct Zone sequence, i.e when I scale ASG from 3 instances to 5 instances, it needs to have 2 nodes in Zone-A, 2 in Zone-B and 1 in Zone-C. But in our case, it ends up in 2 nodes in Zone-A, 1 node in Zone-B and 2 nodes in Zone-C.
AWS ASG launches new instances in all Availability zones you enabled for that particular ASG. This is an extract from the official documentation.
Amazon EC2 Auto Scaling attempts to distribute instances evenly between the Availability Zones that are enabled for your Auto Scaling group. Amazon EC2 Auto Scaling does this by attempting to launch new instances in the Availability Zone with the fewest instances. If the attempt fails, however, Amazon EC2 Auto Scaling attempts to launch the instances in another Availability Zone until it succeeds
If you increase the desired capacity to say 9 (and you have 3 AZ's), you'll see there's a high chance there will be 3 instances on each AZ.
There is no way to control which AZ the AutoScaling Group will launch instances in.
The only work around I can think of is that you could make 1 ASG per AZ and then control the desired on your own via a script instead of using a scaling policy. I would recommend trying to make sure your application is as ephemeral as possible without zonal dependencies so that instances can be added in any zone
Related
There is set of rules to terminate instance for Auto Scaling when we have multiple AZ.. Same way if we wanted scale up if we have multiple available zones, where exactly instances will be created .. is there any hierarchy?
According to aws docs, if you have multiple availabilty zones for an autoscaling group, aws try to distribute the instance in evenly manner. So if your desired capacity is 8 and there are 4 instances in az-1 and 3 in az-2, the remaining one instance will be created in az-2.
When one Availability Zone becomes unhealthy or unavailable, Amazon EC2 Auto Scaling launches new instances in an unaffected Availability Zone. When the unhealthy Availability Zone returns to a healthy state, Amazon EC2 Auto Scaling automatically redistributes the application instances evenly across all the Availability Zones for your Auto Scaling group. Amazon EC2 Auto Scaling does this by attempting to launch new instances in the Availability Zone with the fewest instances. If the attempt fails, however, Amazon EC2 Auto Scaling attempts to launch in other Availability Zones until it succeeds.
You can read more about this here.
The definition of the vpc_zone_identifier parameter is a list of subnet IDs to launch resources in. Subnets automatically determine which availability zones the group will reside.
So suppose I list eu-west-1a and eu-west-1c for that parameter and a desired capacity of 3.
Is my ASG going to deploy my desired capacity randomly across the AZs (e.g. 2 + 1) or it will deploy 3 per AZ?
There will be only 3 instances distributed across the two AZs if the selected AZs have enough capacity. AWS tries to prioritize high-availability, so it will try to place the instances evenly across the AZs (2+1 in your case). Exact details are:
Amazon EC2 Auto Scaling attempts to distribute instances evenly between the Availability Zones that are enabled for your Auto Scaling group. Amazon EC2 Auto Scaling does this by attempting to launch new instances in the Availability Zone with the fewest instances. If the attempt fails, however, Amazon EC2 Auto Scaling attempts to launch the instances in another Availability Zone until it succeeds. For Auto Scaling groups in a VPC, if there are multiple subnets in an Availability Zone, Amazon EC2 Auto Scaling selects a subnet from the Availability Zone at random.
My AWS solution spans over 3 availability zones. In my backend the user is able to trigger a heavy compute job with beefy px instances. Therefore I wrote a CFN template, which provision all resorucess to execute the compute job (secret store, IAM Role, EC2 instance, log group). However when I try to create the template, it returns with a 500 and states that no capacity for my instance type is available for the availability zone i choose. My template provides a subnet for the EC2 instance and an availability zone for the attached volume. In the end I don't care in which availability zone the ec2 is provisioned as long it is in one of my subnets. Does someone know a way to provision an EC2 instance and it's volume (with cloudforamtion) by not specifically choosing one availability zone, but rather provide a range of subnets/availability zones ?
TLDR:
Does someone know a way to provision an EC2 instance and it's volume (with cloudforamtion) by not specifically choosing one availability zone, but rather provide a range of subnets/availability zones ?
I have an Auto Scaling Group and I want to stop that instance from Auto Scaling Group rather than terminating, Is it possible to do so?
No. From the official definition:
Auto Scaling is a web service designed to launch or terminate Amazon EC2 instances automatically based on user-defined policies, schedules, and health checks.
When scaling-out, new instances are launched into the Auto Scaling group.
When scaling-in, instances are terminated.
Auto Scaling does not start/start instances.
Some benefits of this approach are:
Instances can be launched in different Availability Zones in case there is a failure in a particular AZ
Failed instances can be easily replaced
There is no limit to the number of instances that could be launched (compared to running out of 'stopped' instances)
Launch Configurations can be updated, so any newly-launched instances will use the new configuration (as opposed to recycling old instances)
I am looking for how to specify the zone I want to deploy to in a single instance deployment, with autoscaling, while also having automatic failover to another zone -- Do any options exist to achieve this?
More context
Due to how reserved instances are linked to a single availability zone (AZ), we find it to be a good strategy (from an "ease of management"/simplicity perspective), when buying reserved instances for our dev environment, to buy them all in a single zone and then launch all dev instances in that single zone. (In production, we buy across zones and run with autoscale groups that specify to deploy across all zones).
I am looking for how to:
Specify the AZ that I want an instance to be deployed to, so that I can leverage the reserved instances that are tied to a single (and consistent) AZ.
while also having
The ability to failover to an alternate zone if the primary zone fails (yes, you will pay more money until you move the reserved instances, but presumably the failover is temporary e.g. 8 hours, and you can fail back once the zone is back online).
The issue is that I can see how you can achieve 1 or 2, but not 1 and 2 at the same time.
To achieve 1, I would specify a single subnet (and therefore AZ) to deploy to, as part of the autoscale group config.
To achieve 2, I would specify more than one subnet in different AZs, while keeping the min/max/capacity setting at 1. If the AZ that the instance non-deterministically got deployed to fails, the autoscale group will spin up an instance in the other AZ.
One cannot do 1 and 2 together to achieve a preference for which zone an autoscale group of min/max/capacity of 1 gets deployed to while also having automatic failover if the zone the server is in fails; they are competing solutions.
This solution uses all AWS mechanisms to achieve the desired effect:
Launch the instance into the preferred zone by specifying that zone's subnet in the 1st autoscale group's config; this group's min/max/capacity is set to 1/1/1.
Create a second autoscale group with the same launch config as the 1st, but this other autoscale group is set to a min/max/desired of 0/1/0; this group should be configured with the subnets in every available zone in the region except the one specified in the 1st autoscale group.
Associate the 2nd autoscale group with the same ELB that is associated with the 1st autoscale group.
Set up a CloudWatch alarm that triggers on the unhealthy host alarm for #1's autoscale group; have the alarm change the #2 autoscale group's to a min/max/desired of 1/1/1. (As well as send out a notification so that you know this happened).
If you don't expect to get unhealthy host alarms except in the cases where there is an actual host failure or if the AZ goes down -- which is true in our case -- this is a workable solution.
As you have already figured out, (as of mid-2015) that's not possible. Auto-scaling doesn't have the concept of failover, strictly speaking. It expects you to provide more than one AZ and machines enough in each one if you want to have high availability. If you don't, then you aren't going to get it.
The only possible workaround I can imagine for this is setting up a watchdog yourself which changes the auto-scaling group's subnet once an AZ becomes unavailable. Not so hard to do, but no so reliable as well.