Does AWS expose a API/method if i can pull out number of message sent to SQS queue for a given time frame(for example a week).I am trying to build a tool to monitor cloud infra for an internal project in java.This is one of the requirement to display a graph with number of message pushed to SQS for a week. I also know we can check this in AWS console.
You can retrieve queue metrics from Amazon CloudWatch. For example:
NumberOfMessagesSent
NumberOfMessagesReceived
See: Amazon SQS Metrics and Dimensions - Amazon CloudWatch
Related
Hope you all doing well, I am new with SQS and Cloudwatch and I need to create a Cloudwatch that would monitor SQS and would trigger Lambda with an event every time a message enters and every time a message is left.
On another note, the lambda function should scale up and down ASG service. So if anyone has a cookbook regarding those issues it would be very helpful
thank you so much!
It appears that your requirement is to scale Amazon EC2 instances when messages are waiting to be processed in an Amazon SQS queue.
The correct architecture for this would be to configure the Auto Scaling group to use a scaling policy based on the metric ApproximateNumberOfMessagesVisible. This is a metric that Amazon SQS queues send to Amazon CloudWatch Metrics. There is no need to use an AWS Lambda function.
For reference, see:
Scaling based on Amazon SQS - Amazon EC2 Auto Scaling
Rapid Auto Scaling with Amazon SQS | AWS News Blog
Here are my numbers from AWS usage even though I am not using SQS at all in my application.
I have a lambda service (it's not even in production yet) and it talks to external API + My frontend via API Gateway.
How come my SQS usage is so high?
I don't even have any aps in sqs [I used to when I did some tutorials with sqs, but that was a simple feature]. And my guess that S3 is si high because of the SAM deploy.
How can I check what's causing this traffic?
You presumably do have, or did have, some SQS queues. See Monitoring Amazon SQS queues using CloudWatch.
Amazon SQS and Amazon CloudWatch are integrated so you can use CloudWatch to view and analyze metrics for your Amazon SQS queues. You can view and analyze your queues' metrics from the Amazon SQS console, the CloudWatch console, using the AWS CLI, or using the CloudWatch API. You can also set CloudWatch alarms for Amazon SQS metrics.
Also, see Why are my Amazon SQS charges higher than expected?
I want to create CloudWatch Rule that would be triggered upon creation of Log Event. For that reason as an event pattern I selected CloudWatch Logs service but when I try to generate some Cloud Watch logs the rule is not getting triggered. I can not find any example of using aws.logs as a source for an event and hence my question if I'm doing something wrong.
This is because the only events for logs available are AWS API Call via CloudTrail. CloudWatch Logs does not generate CloudWatch events on receiving new log entries.
For the Logs API call events to work, you need to setup CloudTrial trial.
However, if you want to trigger your lambda function based on log entries, I can recommend using subscription filters for lambda:
You can use subscriptions to get access to a real-time feed of log events from CloudWatch Logs and have it delivered to other services such as a Amazon Kinesis stream, Amazon Kinesis Data Firehose stream, or AWS Lambda for custom processing, analysis, or loading to other systems.
I am using various AWS services (Lambda, Glue, S3, Redshift, EC2) for ETL processing. I am trying to create a 'log ETL flow' to have monitoring and notifications sent out (email or otherwise) when a step fails in the process.
I have checked that each service I am using has metrics being logged in CloudWatch. I am now trying to figure out a smart way of processing this data in order to send out notifications when a step fails and/or have a central monitoring of the entire flow.
Are there any best practices or examples of this setup?
It seems to be the perfect case for the CloudWatch Alarms.
You can create a CloudWatch alarm that watches a single CloudWatch metric or the result of a math expression based on CloudWatch metrics. The alarm performs one or more actions based on the value of the metric or expression relative to a threshold over a number of time periods. The action can be an Amazon EC2 action, an Amazon EC2 Auto Scaling action, or a notification sent to an Amazon SNS topic.
You can create a chain CloudWatch Alarm -> SNS
You can either use SNS to notify users via SMS or Push Notifications.
Or you can do one step more SNS -> SES to deliver emails.
I want to set up CloudWatch alarm on the size of specific queue, but I can only see metrics for one queue in the list. How can I see metrics for all queues?
SQS Only generates metrics for queues which are "active". Try making some requests against the queue (ReceiveMessage for example) and then see if you start getting cloudwatch metrics.