I have a business case when an EC2 instance runs out of space, we need to spawn new EBS volume, attach it to EC2 instance and format it.
I have created one cron job which keeps sending disk usage to cloud watch and trying to create one alarm this custom metric.
Now I am not able to find out any information regarding how to spawn an EBS volume when this alarm triggers.
So I would like to know if it is it possible to spawn EBS volume when cloudwatch alarm triggers? If yes, please give some steps or point to the document where I can find this information.
As if now all I have found out is that we can either spawn new instances or send some emails whenever alarm triggers.
You can fire an notification to an SNS topic when the CloudWatch alarm fires, and have a SQS queue as a subscriber to that topic. Then, an EC2 instance consuming that SQS queue can perform the desired change using the AWS CLI or SDKs.
Related
I have been trying to create cloudwatch alarm when instance gets stopped but couldn't find direct way. From event subscriptions i can send notification when instance gets stopped. Is there any way cloud watch alarm can be triggered for the same.
Amazon EventBridge can be configured to trigger an event when a state change occurs on an EC2 instance. Use:
Event source: EC2
Event type: EC2 Instance State-change Notification
You can setup a target for the event to be a Lambda
The Lambda can API_PutMetricData for a metric you create. You can setup a CloudWatch Alarm on this metric
This tutorial shows you how to setup the EventBridge rule and the Lambda.
Hope you all doing well, I am new with SQS and Cloudwatch and I need to create a Cloudwatch that would monitor SQS and would trigger Lambda with an event every time a message enters and every time a message is left.
On another note, the lambda function should scale up and down ASG service. So if anyone has a cookbook regarding those issues it would be very helpful
thank you so much!
It appears that your requirement is to scale Amazon EC2 instances when messages are waiting to be processed in an Amazon SQS queue.
The correct architecture for this would be to configure the Auto Scaling group to use a scaling policy based on the metric ApproximateNumberOfMessagesVisible. This is a metric that Amazon SQS queues send to Amazon CloudWatch Metrics. There is no need to use an AWS Lambda function.
For reference, see:
Scaling based on Amazon SQS - Amazon EC2 Auto Scaling
Rapid Auto Scaling with Amazon SQS | AWS News Blog
I am trying to send an SNS notification when a backup in the backup vault fails twice consecutively. Is there a CloudWatch alarm or any other way to do this in CloudFormation?
You can use CloudWatch metrics for this purpose and then setup alarms based on the thresholds that you need.
You can find the list of Metrics that are emitted to CloudWatch in this document: https://docs.aws.amazon.com/aws-backup/latest/devguide/cloudwatch.html
For instance you can setup an alarm on NumberOfBackupJobsFailed metric.
How to write a cloudwatch alarm for EMR using CLI command??
My requirement is to terminate cluster which are idle for more than 2hours. I need to do this using aws CLI command.
From Monitor Metrics with CloudWatch - Amazon EMR:
The IsIdle metric tracks whether a cluster is live, but not currently running tasks. You can set an alarm to fire when the cluster has been idle for a given period of time, such as thirty minutes.
Therefore, you can configure an alarm on this metric. However, the alarm itself is not able to terminate an Amazon EMR cluster. You would need an additional component, such as an AWS Lambda function, to actually terminate the cluster.
The components would be:
An Amazon CloudWatch IsIdle metric (automatically provided)
An Alarm on the metric that triggers when the cluster is idle for longer than the desired period
Configure the Alarm to send a message to an Amazon SNS topic
Create an AWS Lambda function and subscribe the function to the SNS topic
Code the Lambda function to terminate the Amazon EMR cluster
There is a more complex version of this auto-shutdown process documented at: Optimize Amazon EMR costs with idle checks and automatic resource termination using advanced Amazon CloudWatch metrics and AWS Lambda | AWS Big Data Blog
What is the best way to check the EC2 instance uptime and possibly send alerts if uptime for instance is more then N hours? How can it be organized with default AWS tools such as CloudWatch, Lambda ?
Here's another option which can be done just in CloudWatch.
Create an alarm for your EC2 instance with something like CPUUtilization - you will always get a value for this when the instance is running.
Set the alarm to >= 0; this will ensure that whenever the instance is running, it matches.
Set the period and consecutive periods to match the required alert uptime, for example for 24 hours you could set the period to 1 hour and the consecutive periods to 24.
Set an action to send a notification when the alarm is in ALARM state.
Now, when the instance has been on less than the set time, the alarm will be in INSUFFICIENT DATA state. Once it has been on for the uptime, it will go to ALARM state and the notification will be sent.
One option is to use AWS CLI and get the launch time. From that calculate the uptime and send it to Cloudwatch:
aws ec2 describe-instances --instance-ids i-00123458ca3fa2c4f --query 'Reservations[*].Instances[*].LaunchTime' --output text
Output
2016-05-20T19:23:47.000Z
Another option is to periodically run a cronjob script that:
calls uptime -p command
converts the output to hours
sends the result to Cloudwatch with dimension Count
After adding the cronjob:
add a Cloudwatch alarm that sends an alert when this value exceeds a threshold or if there is INSUFFICIENT DATA
INSUFFICIENT DATA means the machine is not up
I would recommend looking into an "AWS" native way of doing this.
If it is basically sending OS level metrics (e.g. Free Memory, Uptime, Disk Usage etc...) to Cloudwatch then this can be achieved by following the guide:
This installs the Cloudwatch Logs Agent on your EC2 instances.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html
The great thing about this is you then get the metrics show up in Cloudwatch logs (see attached picture which shows the CW Logs interface in AWS Console.).