Is it possible to set a CloudWatch alarm for when we are approaching the limit of EC2 instances currently allowed on our account?
For instance, if limit for EC2 instances is currently 250, when instance number 240 is provisioned, I want an alarm to trigger.
If you have an auto scaling group which launches new instances and you want to control it, you can use GroupInServiceInstances which gives you the number of instances running as part of the ASG. Read more here.
Yes, you could do this with a Lambda function, a CloudWatch Metric and a CloudWatch alarm.
Your alarm would be configured to alarm on the metric, if it exceeds some threshold (the threshold being your instance limit).
Your Lambda function, would run on a schedule e.g. every 5 mins, and would do the following:
Use the ec2:DescribeAccountAttributes API to get the account instance limit and cloudwatch:DescribeAlarms to get the current threshold of the alarm. If they differ, the alarm threshold should be updated the the instance limit via the cloudwatch:PutMetricAlarm API.
Use the ec2:DescribeInstances API and count the number of instances that are running and publish the value to a custom CloudWatch metric with the cloudwatch:PutMetricData API.
If the value published to the metric exceeds the threshold of the alarm, it will fire. The lambda function will keep the alarm threshold configured to the limit of instances and will publish datapoints to the metric based on the number of instances currently running.
Related
I'm setting up an AWS EC2 Autoscaling Group (ASG) and it's using TargetTrackingScaling that is tracking a custom metric. This metric is published by each instance in the ASG every 30 seconds.
It's working fine, but I'd like the scale-out action to happen more quickly. The source of this looks to be due to the ASG Alarm that gets auto generated. To me it looks like it's waiting for at least 3 datapoints over 3 minutes before ringing the alarm.
Is there a way I can configure the ASG/Scaling policy such that the alarm only needs 1 datapoint (or less time) before deciding to ring the alarm? Or if that's not possible, can I create a custom alarm and use that instead of the alarm that the ASG auto generated.
I'm using the AutoScalingGroup to launch a group of EC2 instances. These instances are acting as workers which are continuously listening to SQS for any new request.
Requirement:
Do upscale on something like throughput (i.e Total number of messages present in SQS by total number instances).
And I want to downscale whenever any instance which is part of ASG is sitting idle (CPUIdle) for let's say more than 15 mins.
Note: I am not looking for any metric which applies as whole to a particular ASG (eg: Average CPU).
One way of doing that could be defining the custom metric and allowing it to trigger a cloudwatch alarm to do that.
Is there a better way to accomplish this?
If you are defining the scaling policy at instance level, then you defeating the entire purposes of ASG. If you need to scale based on changing conditions, such as the queue size, then you can configure ASG based on the conditions specified here
https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-using-sqs-queue.html
A custom metric to send to Amazon CloudWatch that measures the number of messages in the queue per EC2 instance in the Auto Scaling group.
A target tracking policy that configures your Auto Scaling group to scale based on the >custom metric and a set target value. CloudWatch alarms invoke the scaling policy.
If you know a specific time window when the queue size goes up or down, you can also scale based on schedule.
You can always start with a very low instance count in ASG and set the desired capacity as such (say 1) and scale up based on queue, so you can continue using ASG policies.
How to setup cloudwatch alarm for Autoscaling group when its scaled upto a particular number of Instances?
Suppose, the maximum number of instances is 6. I need to get notified when the no: of instances crossed 3.
There is a metric called GroupTotalIsntances accessible via CloudWatch Console > Metrics > Auto Scaling > Group Metrics which gives you the total number of instances in your ASG. Create a new alarm and use this metric. Read more here.
What is the best way to check the EC2 instance uptime and possibly send alerts if uptime for instance is more then N hours? How can it be organized with default AWS tools such as CloudWatch, Lambda ?
Here's another option which can be done just in CloudWatch.
Create an alarm for your EC2 instance with something like CPUUtilization - you will always get a value for this when the instance is running.
Set the alarm to >= 0; this will ensure that whenever the instance is running, it matches.
Set the period and consecutive periods to match the required alert uptime, for example for 24 hours you could set the period to 1 hour and the consecutive periods to 24.
Set an action to send a notification when the alarm is in ALARM state.
Now, when the instance has been on less than the set time, the alarm will be in INSUFFICIENT DATA state. Once it has been on for the uptime, it will go to ALARM state and the notification will be sent.
One option is to use AWS CLI and get the launch time. From that calculate the uptime and send it to Cloudwatch:
aws ec2 describe-instances --instance-ids i-00123458ca3fa2c4f --query 'Reservations[*].Instances[*].LaunchTime' --output text
Output
2016-05-20T19:23:47.000Z
Another option is to periodically run a cronjob script that:
calls uptime -p command
converts the output to hours
sends the result to Cloudwatch with dimension Count
After adding the cronjob:
add a Cloudwatch alarm that sends an alert when this value exceeds a threshold or if there is INSUFFICIENT DATA
INSUFFICIENT DATA means the machine is not up
I would recommend looking into an "AWS" native way of doing this.
If it is basically sending OS level metrics (e.g. Free Memory, Uptime, Disk Usage etc...) to Cloudwatch then this can be achieved by following the guide:
This installs the Cloudwatch Logs Agent on your EC2 instances.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html
The great thing about this is you then get the metrics show up in Cloudwatch logs (see attached picture which shows the CW Logs interface in AWS Console.).
Use case:
Every day morning the SQS will be populated (only one time, and the number of messages can vary drastically), I want to spawn new instances as per the number of messages in the queue.
eg: For 200000 messages 4 Instances, 400000 8 instances.
Is there a way by which we can achieve this?
You can set up a cron-job on your server or a time-triggered Lambda to query SQS to find out the number of visible messages in the queue. If you're using the AWS CLI you would run aws sqs get-queue-attributes and read the ApproximateNumberOfMessages response field to get the number of items in the queue. You would then use that number to calculate the number of instances and either call aws ec2 run-instances --count 4 plus the rest of the parameters. Once everything is done you would terminate the instances.
Another way to do this would be to utilize auto-scaling and alarms. You can set up an ScaleOut policy that adds 1 server to your AutoScaling Group and trigger that policy with a CloudWatch alarm on SQS ApproximateNumberOfMessages >= some threshold. This option wouldn't wait for morning to process the queues, you'd have it running all the time. You could also have a ScaleIn policy to reduce the Desired Capacity (# of servers) in your AutoScaling Group when ApproximateNumberOfMessages <= some threshold.