My primary requirement is as follows:
When CPU consumption on an instance exceeds 50 % then adjust capacity of autoscaling group to 5 instances, when CPU consumption exceeds 80% then adjust capacity to 10 instances.
However if I use cloudwatch alarms to set capacity I can imagine the following race condition:
5 instances exist
CPU consumption exceeds 80 %
Alarm is triggered
Capacity is changed to 19 instances
CPU consumption drops below 50 %
Eventually CPU consumption again exceeds 50% but now capacity will be changed to 5 instances (which is something I don't want to happen)
So what I would ideally like to happen is that in response to alarm triggers I would like to ensure that capacity is altleast the corresponding threshold.
I am aware that this can be done by manually setting the capacity through AWS SDK - which could be triggered in response to lifecycle events monitored by a supervisor, but is there a better approach, preferably one that does not require setting up additional supervisors or webhooks for alarms ?
A general approach is to fine grain the scaling actions:
Do not jump that big:
if the ASG avg CPU is over 70% > Add an instance
if the ASG avg CPU is over 90% > Add "n" instances
if the ASG avg CPU is under 40% > remove an instance
if the ASG avg CPU is under 10% > remove "n" instance
All of these values are the last 5 mins AVG. So if you have a really fast pike, you need more aggressive scaling. So in half an hour you can easily add 6 servers or even more.
Also scaling works better with higher numbers. So if your system needs only 1-3 instances, it may make sense to decrease the instance size so you can have 2-6 instances. It give some extra flexibility to your system.
But again, the question is, what is your expected load? Big pikes or an expected up and down during the day?
I would suggest looking into an AWS lambda function, triggered by an SNS message from cloudwatch - it should give you free reign to put as much logic into the scaling decision as you want.
Good Luck!
Related
I would like to get all the AWS EC2 instances list which are idle and underutilized. I need to filter them based on CPU utilization used less than 2% and network I/O less than 5Mb for last 30 days.
Can you please provide me with the commands or any scripts to get the list or guide me to get that achieved.
I need this to get that list and terminate those instances for cost management.
You can create a CloudWatch alarm that is triggered when the average
CPU utilization percentage has been lower than 2 percent for 24 hours,
signaling that it is idle and no longer in use. You can adjust the
threshold, duration, and period to suit your needs, plus you can add
an SNS notification so that you will receive an email when the alarm
is triggered.
Kindly refer to this documentation to create an alarm to terminate an idle instance using the Amazon CloudWatch console.
I have setup a AWS CloudWatch alarm: CPU utilization > 90 %: https://www.screencast.com/t/BPs3hlY2hEZ
I have added the alarm / CPU % utilization metric to a dashboard: https://content.screencast.com/users/MartinBakDK/folders/Jing/media/5c01c414-95d7-4a20-ab7d-e0a6c9debc01/2020-05-20_2158.png
I do know - that the metric shows a 5 min average - But for more than 1 hour now the actual CPU usage of my EC2 instance has been 100% (So it should show 100% as well):
https://www.screencast.com/t/BvITivn0ff
Because CloudWatch only reg. 20% and not the actual 100% => CloudWatch is useless as a monitoring-system.
Can this really be true? Please tell me what is going on and how AWS can provide such a "service".
Look at CpuCreditBalance metric. If this is at 0 your CPU will be capped at a fixed percentage (this is why you see the straight line).
Your host sees itself as 100% because it cannot use anymore CPU.
All T instances are burstable, so once they’re depleted the CPU is capped for performance. You can either change instance type or enable unlimited credits (there will be additional cost).
Further reading: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances-monitoring-cpu-credits.html
I was looking at the AWS EC2 Auto-Scaling with Cloud Watch feature.
What I cannot glean is if the auto scaling can be applied only once or repeatedly? I think only once though.
E.g. scale from 2 to 4 max if, say, 60% cpu reached.
Then, what if, having gone from 2 to 4, and then 60% cpu reached again according to Cloud Watch target rules, can we reapply autoscaling again?
I saw a "step" option, but that seems to work with different target values. Could the step have repeated 60% configuration or does it need 60, 70% and so on? This aspect is not entirely clear.
Or is such an iterative approach simply not possible? Implying one needs to re-baseline the initial allocations? I think not based on the step approach.
Autoscaling will be triggered by CloudWatch alarms when your alarm will breach your threshold.
So if your policy is to add 50% capacity when average CPU across the fleet is above 60% for more than 15min, the alarm will trigger a change in your scaling group, increasing the value of your "desired capacity" property, let's say going from 2 to 3 instances. Autoscaling will then react and bring the fleet to "desired capacity"
If, later, the same alarm is still on, a new change will be triggered to your scaling group, bringing desired capacity to 4. Then AutoScaling will create EC2 instances to bring your fleet to the new desired capacity.
And so on.
Of course, you do not want to increase the fleet size above your budget. So you can define a MAX fleet size, that AutoScaling will never go above. If your max fleet size is 3, the second alarm in my example will leave desired capacity as it is (3) because new desired capacity > max capacity. This will end up the scaling process.
To go back to normal, you must also create scale in policies, in addition of your scale out policies : i.e. when the average CPU on my fleet is below 15% for 1h, let's remove an instance. Your CloudWatch alarm will trigger, causing a change in the desired capacity of your fleet and AutoScaling will adjust (kill instances) to go to the new desired capacity value.
Of course, you do not want AutoScaling to kill the last instance from your fleet (going from desired capacity of 1 to desired capacity of 0), so, similarly to a MAX Capacity value, you also define a MIN capacity, i.e. the minimum number of instances to keep in your fleet, whatever alarm are triggering and trying to change the desired capacity.
Our website is hosted on AWS in a t2.small instance. User-facing sign-up is currently timing out.
Initially, I was getting a loadbalancer latency alarm notification for this instance, so I increased the limit, which seemed to work as a temporary solution.
However, once I increased the limit, I started getting 2 other alarm notifications, which were as follows:
1) production-remove-capacity-alarm
Description: None
Threshold: CPUUtilization <= 40 for 3 datapoints within 15 minutes
2) AWSEBCloudwatchAlarmLow
Description: ElasticBeanstalk Default Scale Down alarm
Threshold: NetworkOut < 2,000,000 for 1 datapoints within 5 minutes
It seems to me that I should simply change the alarm notifications so that I'm no longer alerted to #2, as I don't see how this is interfering with anything, but please correct me if I seem to be missing something.
Regarding #1, does it seem likely that somehow adjusting CPU Utilization in AWS will solve the timeout issue with website sign-up?
And if so, what specifically ought to be done?
Everything is okay. Don't panic.
The first priority is that your application operates correctly. Hopefully your adjustment to the instance type) satisfactorily fixed this (but it is still worth watching).
The above two alarms are basically saying:
CPU is under 40%
There's not a lot of network traffic
These alarms can be used to scale-in instances (reduce the number of instances) so that you are not paying for excess capacity. There would be similar alarms that let you scale-out (add additional instances).
ALARM simply means the check is True. That is, the condition has been satisfied. It does not necessarily indicate a problem.
I'm going to presume that you currently have only one instance running. If so, you can ignore those alarms (and Auto Scaling will ignore them too) because you are already at the minimum capacity.
If Auto Scaling has been configured to scale-out to more instances, these alarms would later scale-in to save you money. They're probably a bit trigger-happy, only looking at 15 minutes CPU and 5 minutes of network traffic — it would normally be better to wait for a longer period before deciding to remove capacity.
Bottom line: If your application is running correctly and you are only operating a single instance, there's nothing to worry about. It's all working as expected.
I have an auto scale group with triggers as follows:
Average CPU Utliziation > 90% scale up 1 instance
Average CPU Utilization < 25% scale down 1 instance
The metric is being calculated every 2 minutes and the breach limit is 10 minutes.
The problem I am experiencing is that the triggers are being triggered constantly it seems. The instances are being created and destroyed every 10 minutes. I have been monitoring the CPU Utilization and it never surpasses the scale up threshold. The maximium it hits is around 80% and this only happened 1 time, most of the time it is in the 20 to 25% range. I only have 1 instance running normally, but eveyr 10 minutes ELB will create a new instance, and soon after it will terminate it.
Any thing I am doing wrong here? Am I not understanding how the average CPU Utilization works?
The new EC2 instances are being created by Auto-Scaling (not Load Balancer).
There is a "Scaling History" tab in the Auto Scaling group that might provide some hints as to what is triggering the scale-out policy.
Check whether "Detailed Monitoring" is enabled on the Auto Scaling group and/or Launch Configuration -- this will cause metrics (eg CPU) to be collected every 1 minute instead of the default 5 minutes.
Check the setting on your CloudWatch chart to match the metric collection interval -- if metrics are being collected every minute, set the CloudWatch chart to 1-minute also. Otherwise, you might be viewing metrics at a lower "resolution" than the alarm itself.
Worst case, increase the timing settings for the Alarm, such as "Above 90% for 2 consecutive periods" rather than just one period.