Use Case
We have 1 sqs that is being consumed by a lambda and we would like to know if all messages in the sqs have been consumed or not after a certain timestamp.
For example, Some other system will start sending messages to the SQS start from 6AM and it will take 4 hours for the lambda to process them all. We would like to know at 10AM, have all messages in the queue been consumed or not.
And we only need to detect the depth of the sqs once every day.
Questions.
Is there a easy way to setup the alarm in cloudwatch to achieve our use case?
There are many potential solutions such as use cloudwatch rule to trigger a lambda and detect the size of queue and send metric. We can alarm on the metric, but this seems to be heavy lifting
Any feedbacks are welcomed!
thanks
Continual checking
You can create an Amazon CloudWatch alarm on the ApproximateNumberOfMessagesVisible metric for the queue.
For example, if you wish to be notified when the queue has not been empty over the past hour, you can create an alarm when the Min of ApproximateNumberOfMessagesVisible > 0 for 60 minutes. This is saying that the smallest number of messages in the queue for the past hour was above zero.
Checking at a specific time
If you want to check the queue length at a particular time, you will need to use Amazon CloudWatch Events to trigger an AWS Lambda function at a given time.
The Lambda function can call get_queue_attributes() to retrieve ApproximateNumberOfMessages. If this is bigger than desired, the Lambda function could send a message to an Amazon SNS topic. Users can subscribe to the topic to receive an email or SMS notification.
Related
Context:
I have hundreds of SQS queues. The producers for these queues are S3 bucket notifications that automatically trigger on "S3 upload events" to a specific folder and put the triggering events in a corresponding SQS queue.
I am trying to monitor each of these SQS queues via the "Number of Messages Sent" metric in AWS Cloudwatch. My goal is to see if any of the SQS queues show "Number of Messages Sent" as 0 for an extended duration of time.
Problem: I'm unsure of how to avoid the tedium of creating hundreds of Cloudwatch Alarms (1 per SQS queue) to achieve this.
Attempted Solution:
I tried to bundle the metrics into one Alarm via the metric builder, and then use the MIN() function to see if any one of the many SQS queues had Messages Sent == 0 for an extended period of time, but faced the issue of not being able to "create an alarm on an expression that depends on more than 10 metrics."
What would be the best way to get around this issue or otherwise monitor the queues based on a given metric?
I got an SQS FIFO queue, I want to know if there is a way to trigger an AWS lambda once the queue is not empty.
For example, if my queue is empty and a new message enters trigger lambda, but if the queue is already containing at least one message and a new message enters no lambda will be triggered.
Is it possible?
There is an Amazon CloudWatch metric called ApproximateNumberOfMessagesVisible that shows the number of messages in the queue. The documentation says that "For FIFO queues, the result is exact."
You could create a CloudWatch Alarm that triggers when the number of messages drops to zero for a period of time. The Alarm can send a message to an Amazon SNS topic. If you subscribe your AWS Lambda function to this topic, it will be triggered when the queue is empty for the specified duration (eg over a period of 5 minutes). It will only be triggered when the alarm enters the 'Alarm' state and it will not trigger again until the alarm exits the state and enters the state again.
Important: When configuring the alarm, go to the Additional configuration and set Missing data treatment to "Treat missing data as bad (breaching threshold)". This is required because the SQS queue will not send metrics if the queue is empty. (Many queues are empty, so this saves a lot of metric storage!)
Unusual pattern.
You could perhaps set the Lambda function concurrency to 1, meaning that only one invocation can happen concurrently, and then have your Lambda function kick off your workflow and then remove the actual SQS event trigger that caused the Lambda to be invoked in the first place. That should prevent further invocations. Add the SQS event trigger back when you're done to get ready for the next batch of messages.
You may set a concurrent execution limit to 1 to make sure only 1 lambda instance reads the queue. But I'm not sure this is something you may want to do. Lambda can read 10 messages at most on single execution and if your queue gets too many incoming messages then your message consumption process may take too much time.
I have a lambda function is responsible for checking the server status. It needs to be called when SQS receives new messages and It is not allowed to change anything in SQS. I tried using SQS Lambda trigger but it will push the message into lambda function => that changed SQS queue.
I am looking the way to handle this problem. I try to use CloudWatch to handle this but I don't know is this possible or not? How Cloudwatch can trigger Lambda functions when SQS receives new messages?
Thanks in advance.
This will be difficult because, if the message is consumed quickly, it might not have an impact on Amazon CloudWatch metrics. You'll need to experiment to see whether this is the case. For example, set a metric for the maximum number of messages received in a 1-minute time period and try to trigger a CloudWatch Alarm when it is greater than zero.
Alternatively, have the system that sends the SQS message send it to Amazon SNS instead. Then, both the SQS queue and the Lambda function can subscribe to the SNS topic and both can be activated.
In fact, I know somebody who always uses SNS in front of SQS "just in case" this type of thing is necessary.
We are evaluating SNS for our messaging requirements to integrate multiple applications. we have a single producer that publishes messages to multiple topics on SNS. Each topic has 2-5 subscribers. In event of subscriber failures (down for maintenance) I have a few questions on the recommended strategy of using SQS queues per consumer
Is it possible to configure SNS to push to SQS only in event of failure in delivering the message to a subscriber? Dumping all the messages in SQS queue creates a problem for the consumer to analyze all messages in the queue when it restarts.
In event of subscriber failure, it can read messages from SQS queue on restart but how would it know that it missed messages from SNS when it was overloaded?
Any suggestions on handling subscriber failures are welcome.
Thanks!
No, it is not possible to "configure SNS to push to SQS only in event of failure".
Rather than trying to recover a message after a failure, you can configure the Amazon SNS retry policies.
From Setting Amazon SNS Delivery Retry Policies for HTTP/HTTPS Endpoints:
You can use delivery policies to control not only the total number of retries, but also the time delay between each retry. You can specify up to 100 total retries distributed among four discrete phases. The maximum lifetime of a message in the system is one hour. This one hour limit cannot be extended by a delivery policy.
So, you don't need to worry as long as the destination is back online within an hour.
If it is likely to be offline for more than an hour, you will need to find a way to store and "replay" the messages, possibly by inspecting CloudWatch Logs.
Or, here's another idea...
Push initially to SQS. Have an AWS Lambda function triggered by SQS. The Lambda function can do the 'push' that would normally be done by SNS. If it fails, then the standard SQS invisibility process will retry it later, eventually going to a Dead Letter Queue.
I have a AWS Lambda function which is triggered by SQS. This function is triggered approximately 100 times daily, but request count to the SQS queue is approximately 20.000 times daily. I don't understand why the number of requests made to the SQS is too high. My expectation is that the number of requests made to the SQS should be same with the Lambda invocation.
I have only one Lambda function and one SQS queue in my account.
Can be related with polling of SQS queue? I tried to change the polling interval of SQS from the queue configuration but nothing changed. Another possibility is to change polling interval from Lambda function configuration. However, I cannot find any related parameter.
Briefy, I want to reduce number of SQS request, how can i do that while invoking Lmabda function with SQS?
When using SQS as an event source for AWS Lambda, AWS Lambda regularly polls the configured SQS queue to fetch new messages. While the official documentation isn't clear really about that, the blog post announcing that feature goes into the details:
When an SQS event source mapping is initially created and enabled, or when messages first appear after a period with no traffic, then the Lambda service will begin polling the SQS queue using five parallel long-polling connections.
According to the AWS documentation, the default duration for a long poll from AWS Lambda to SQS is 20 seconds.
That results in five requests to SQS every 20 seconds for AWS Lambda functions without significant load, which sums up to the ~21600 per day, which is close to the 20000 you're experiencing.
While increasing the long poll duration seems like an easy way to decrease the number of requests, that's not possible, as the 20 seconds AWS Lambda is using by default is already the maximum possible duration for an SQS queue. I'm afraid there is no easy way to decrease the requests to SQS, when using it as event source for AWS Lambda. Instead depending it could be worth evaluating if another event source, like SNS, would fit your use case as well.
Here is how we originally implemented when there is no SQS trigger.
Create a SNS trigger with the SQS Cloudwatch Metric
ApproximateNumberOfMessagesVisible > 0
Trigger a Lambda from SNS, Read Messages from SQS and deliver it to whichever the lambda needs the message.
Alternatively, you can use Kinesis to deliver it to Lambda.
SQS --> Cloudwatch (Trigger Lambda) --> Lambda(Reads Messages) -->
Kinesis (Set Batch Size) --> Lambda (Handle Actual Message)
You can also use Kinesis directly but there is no delayed delivery.
Hope it helps.