I was reading through this AWS DOC https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-step-scaling-policies.html around Application Auto Scaling Step policies as target auto scaling policies won't work for my use case.
Something that's not clear to me is whether if I define a Step that adds 1 capacity to, say, number of tasks of an ECS Service, that tracks an alarm threshold X (measured in percentage) and, even after the scale out action the X percentage keeps relatively still, if that will continue increasing the number of tasks (after the cooldown period).
Eg.:
T0
number of tasks = 10
metric X = 60%
with a step scaling policy that scales-out when X >= 70%
T1
metric X goes up to 80%
scale out action is triggered
T2
number of tasks is now 11
the new task reduces the burden and metric X is reduced to 75%
then here comes the question; will that step scale policy trigger another scale out (given that the metric X is still > 70%)?
Yes, every minute that a cloudwatch alarm stays in alarm state it will trigger the AutoScaling action. So as long as the alarm triggers the policy after the cooldown ends, it would scale again (remembering that the cooldown doesn't start until the first scaling action /finishes/).
You can also define multiple steps, and if a larger one is triggered while the first is still in progress, the difference will happen to allow another scale out. For example:
T0 step policy triggers +1
T1 (1 minute later while the +1 is in progress): step policy is triggered at the +3 step. 2 more tasks would be added now.
As a side note, why doesn't target tracking work for you, what metric are you using? You can define a custom metric specification if creating the policy outside the CLI to use non-predefined metrics
Related
What is the difference between AWS ASG cooldown period (which I can edit when I want to update my autoscaling group) and the warmup period in the scaling policy?
cooldowns prevent runaway scaling events. If your system is running high on CPU and your auto scaling rule adds an instance, it is going to take 5 minutes or so before the instance is fully spun up and helping with the load. Without a cooldown, the rule would keep firing and might add 4 or 5 instances before the CPU metrics came down, resulting in wasteful over-provisioning. Or in the scale down case, overshoot and result in under-provisioning.
with a cooldown period in place, the Auto Scaling group launches an
instance and then suspends scaling activities due to simple scaling
policies or manual scaling until the specified time elapses. (The
default is 300 seconds.) This gives newly launched instances time to
start handling application traffic. After the cooldown period expires,
any suspended scaling actions resume. If the CloudWatch alarm fires
again, the Auto Scaling group launches another instance, and the
cooldown period takes effect again. If, however, the additional
instance was enough to bring the CPU utilization back down, then the
group remains at its current size.
Cooldown
Instance Warmup
Warm-up value for Instances allows you to control the time until a newly launched instance can contribute to the CloudWatch metrics, so when warm-up time has expired, an instance is considered to be a part Auto Scaling group and will receive traffic.
With step scaling policies, you can specify the number of seconds that
it takes for a newly launched instance to warm up. Until its specified
warm-up time has expired, an instance is not counted toward the
aggregated metrics of the Auto Scaling group. While scaling out, AWS
also does not consider instances that are warming up as part of the
current capacity of the group. Therefore, multiple alarm breaches that
fall in the range of the same step adjustment result in a single
scaling activity. This ensures that we don't add more instances than
you need.
as-scaling-target-tracking
I determined two key differences and one observation of warm-up:
warm-up applies only to scale out
cooldown applies to both scale in and scale out
warm-up applies to all scaling policies
cooldown applies only to simple scaling policies
warm-up of a step-scaling policy only inhibits scale out of the number of instances it follows on from
Scaling according to the first step, from 10->11 will trigger a warm up period
During this period metrics within the first range can be ignored entirely
A metric in a second step, say adding 3 instances, will only trigger adding 3-1 = 2 instances instead, whilst warm-up is in effect
I was looking at the AWS EC2 Auto-Scaling with Cloud Watch feature.
What I cannot glean is if the auto scaling can be applied only once or repeatedly? I think only once though.
E.g. scale from 2 to 4 max if, say, 60% cpu reached.
Then, what if, having gone from 2 to 4, and then 60% cpu reached again according to Cloud Watch target rules, can we reapply autoscaling again?
I saw a "step" option, but that seems to work with different target values. Could the step have repeated 60% configuration or does it need 60, 70% and so on? This aspect is not entirely clear.
Or is such an iterative approach simply not possible? Implying one needs to re-baseline the initial allocations? I think not based on the step approach.
Autoscaling will be triggered by CloudWatch alarms when your alarm will breach your threshold.
So if your policy is to add 50% capacity when average CPU across the fleet is above 60% for more than 15min, the alarm will trigger a change in your scaling group, increasing the value of your "desired capacity" property, let's say going from 2 to 3 instances. Autoscaling will then react and bring the fleet to "desired capacity"
If, later, the same alarm is still on, a new change will be triggered to your scaling group, bringing desired capacity to 4. Then AutoScaling will create EC2 instances to bring your fleet to the new desired capacity.
And so on.
Of course, you do not want to increase the fleet size above your budget. So you can define a MAX fleet size, that AutoScaling will never go above. If your max fleet size is 3, the second alarm in my example will leave desired capacity as it is (3) because new desired capacity > max capacity. This will end up the scaling process.
To go back to normal, you must also create scale in policies, in addition of your scale out policies : i.e. when the average CPU on my fleet is below 15% for 1h, let's remove an instance. Your CloudWatch alarm will trigger, causing a change in the desired capacity of your fleet and AutoScaling will adjust (kill instances) to go to the new desired capacity value.
Of course, you do not want AutoScaling to kill the last instance from your fleet (going from desired capacity of 1 to desired capacity of 0), so, similarly to a MAX Capacity value, you also define a MIN capacity, i.e. the minimum number of instances to keep in your fleet, whatever alarm are triggering and trying to change the desired capacity.
I have created the following two alerts for an autoscaling group:
Scale up 1 instance if "CPUUtilization >= 75%" changes to state ALARM
Scale down 1 instance if "CPUUtilization >= 30%" changes to state OK
I have chosen to trigger the scale down event on OK to not have a constant ALARM in Cloudwatch if load is below 30%. On the other hand that is exactly the issue. When an upscale happens leaving the group with an average load between 30% and 75% the state is set to ALARM.
Is there any way to configure Cloudwatch to trigger scale up and down events properly without being left with an ALARM state after the scaling happened?
The "Scale down" operation should be set to "CPUUtilization < 75%" (state OK), and don't worry about leaving the group without machines, though you have a min number of instances and it does not go beyond that.
I am planning on using AWS Autoscaling to scale my EC2 services, I have 4 policies that need to control my instance behavior, 2 for scale out and 2 for scale in. My question is what order will they be evaluated in? Scale out first then scale in? or vice-versa? Random? or something else?
Thank you,
Policies are not evaluated in an order. Each policy is compared against the metrics that policy is set up to measure, and takes actions based on the results.
For example, perhaps you have the following four policies:
Add 1 instance when an SQS queue depth is > 1000 messages
Remove 1 instance when the same SQS queue depth is < 200 messages
Add 1 instance when the average CPU of all instances in the autoscaling group is > 80%
Remove 1 instance when the average CPU of all instances in the autoscaling group is < 30%
As you can see, ordering doesn't make sense in this context. The appropriate action(s) will be executed whenever the conditions are met.
Note that without planning and testing you can encounter loops of instances that constantly cycle up and down. Drawing from the previous example, imagine that a new instance is launched because there are > 1000 messages in the queue. But the CPU usage is only 20% for all the instances, so then the 4th policy fires to remove an instance. Thus all the policies should be considered in concert.
I have an AWS autoscale group. Is it possible to set an alarm for a percentage increase of CPU? For example, if CPU increases 40% over 1min, trigger the alarm? Thus if CPU is at 0% at 12:51 and 40% at 12:52 the alarm will be triggered.
You can set alarms to check CPU average is greater than predefined value for a period of time.
But to check if it increased with precentage, you might need to implement a custom metrics as mentioned below AWS Blog
Yes You can Set Custom Alarm for CPU Load average Increase for AWS Auto scaling. Just follow the steps below.
You can find this in Auto Scaling Groups select a group for which you want to set alam, further goto Scaling Policies click on Add policy then next to Execute policy when: click on Create new alarm, set alarm for your desired aws auto scaling instance according to your need and set that alarm in Execute policy when: and in Take the action: you can select Add, Remove or Set to options according to your needs and you are done. When auto scaling group reaches to provided threshold which is given in Scaling Policies it will execute policy which you set. For reference you can check AWS Docs