Hope you all doing well, I am new with SQS and Cloudwatch and I need to create a Cloudwatch that would monitor SQS and would trigger Lambda with an event every time a message enters and every time a message is left.
On another note, the lambda function should scale up and down ASG service. So if anyone has a cookbook regarding those issues it would be very helpful
thank you so much!
It appears that your requirement is to scale Amazon EC2 instances when messages are waiting to be processed in an Amazon SQS queue.
The correct architecture for this would be to configure the Auto Scaling group to use a scaling policy based on the metric ApproximateNumberOfMessagesVisible. This is a metric that Amazon SQS queues send to Amazon CloudWatch Metrics. There is no need to use an AWS Lambda function.
For reference, see:
Scaling based on Amazon SQS - Amazon EC2 Auto Scaling
Rapid Auto Scaling with Amazon SQS | AWS News Blog
Related
Here are my numbers from AWS usage even though I am not using SQS at all in my application.
I have a lambda service (it's not even in production yet) and it talks to external API + My frontend via API Gateway.
How come my SQS usage is so high?
I don't even have any aps in sqs [I used to when I did some tutorials with sqs, but that was a simple feature]. And my guess that S3 is si high because of the SAM deploy.
How can I check what's causing this traffic?
You presumably do have, or did have, some SQS queues. See Monitoring Amazon SQS queues using CloudWatch.
Amazon SQS and Amazon CloudWatch are integrated so you can use CloudWatch to view and analyze metrics for your Amazon SQS queues. You can view and analyze your queues' metrics from the Amazon SQS console, the CloudWatch console, using the AWS CLI, or using the CloudWatch API. You can also set CloudWatch alarms for Amazon SQS metrics.
Also, see Why are my Amazon SQS charges higher than expected?
I am trying to send an SNS notification when a backup in the backup vault fails twice consecutively. Is there a CloudWatch alarm or any other way to do this in CloudFormation?
You can use CloudWatch metrics for this purpose and then setup alarms based on the thresholds that you need.
You can find the list of Metrics that are emitted to CloudWatch in this document: https://docs.aws.amazon.com/aws-backup/latest/devguide/cloudwatch.html
For instance you can setup an alarm on NumberOfBackupJobsFailed metric.
I am using various AWS services (Lambda, Glue, S3, Redshift, EC2) for ETL processing. I am trying to create a 'log ETL flow' to have monitoring and notifications sent out (email or otherwise) when a step fails in the process.
I have checked that each service I am using has metrics being logged in CloudWatch. I am now trying to figure out a smart way of processing this data in order to send out notifications when a step fails and/or have a central monitoring of the entire flow.
Are there any best practices or examples of this setup?
It seems to be the perfect case for the CloudWatch Alarms.
You can create a CloudWatch alarm that watches a single CloudWatch metric or the result of a math expression based on CloudWatch metrics. The alarm performs one or more actions based on the value of the metric or expression relative to a threshold over a number of time periods. The action can be an Amazon EC2 action, an Amazon EC2 Auto Scaling action, or a notification sent to an Amazon SNS topic.
You can create a chain CloudWatch Alarm -> SNS
You can either use SNS to notify users via SMS or Push Notifications.
Or you can do one step more SNS -> SES to deliver emails.
How to write a cloudwatch alarm for EMR using CLI command??
My requirement is to terminate cluster which are idle for more than 2hours. I need to do this using aws CLI command.
From Monitor Metrics with CloudWatch - Amazon EMR:
The IsIdle metric tracks whether a cluster is live, but not currently running tasks. You can set an alarm to fire when the cluster has been idle for a given period of time, such as thirty minutes.
Therefore, you can configure an alarm on this metric. However, the alarm itself is not able to terminate an Amazon EMR cluster. You would need an additional component, such as an AWS Lambda function, to actually terminate the cluster.
The components would be:
An Amazon CloudWatch IsIdle metric (automatically provided)
An Alarm on the metric that triggers when the cluster is idle for longer than the desired period
Configure the Alarm to send a message to an Amazon SNS topic
Create an AWS Lambda function and subscribe the function to the SNS topic
Code the Lambda function to terminate the Amazon EMR cluster
There is a more complex version of this auto-shutdown process documented at: Optimize Amazon EMR costs with idle checks and automatic resource termination using advanced Amazon CloudWatch metrics and AWS Lambda | AWS Big Data Blog
I have a business case when an EC2 instance runs out of space, we need to spawn new EBS volume, attach it to EC2 instance and format it.
I have created one cron job which keeps sending disk usage to cloud watch and trying to create one alarm this custom metric.
Now I am not able to find out any information regarding how to spawn an EBS volume when this alarm triggers.
So I would like to know if it is it possible to spawn EBS volume when cloudwatch alarm triggers? If yes, please give some steps or point to the document where I can find this information.
As if now all I have found out is that we can either spawn new instances or send some emails whenever alarm triggers.
You can fire an notification to an SNS topic when the CloudWatch alarm fires, and have a SQS queue as a subscriber to that topic. Then, an EC2 instance consuming that SQS queue can perform the desired change using the AWS CLI or SDKs.