I'm trying send individual SQS messages with a delay. Using https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-send-message-with-timer.html to test message delays, I'm not seeing any delay.
If I set a delay on the queue, that delay is honoured, however not for individual messages. I'm doing the following (all from the AWS console):
Open the SQS console and right click on my test queue - verify that there the delay value for the queue is 0.
Right click on the test queue and click 'Send a Message'
Add a message body
Set 'Delay delivery of this message by' to 30 seconds
Click Send Message.
Receive a confirmation stating 'Your message has been sent and will be ready to be received in 30 seconds.'
However when I check CloudWatch I can see that the message was delivered almost immediately. I'm I misunderstanding something, or have I missed something in the configuration?
My testing shows that messages sent to an SQS queue which a Lambda function is using as a trigger will immediately trigger the Lambda function, even if a Delay setting is provided.
See: amazon sqs - How do I return a message back to SQS from lambda trigger - Stack Overflow
Related
I have a pretty standard setup of feeding SQS to Lambda. The lambda reads the message and makes a web request to a defined endpoint.
If I encounter an exception during processing of the SQS message that is due to the form of the message then I put the message on a dead letter queue.
If I encounter an error with the web request, I put the message back on the feeding queue to make the HTTP request at a later time.
This seems to work fine, but we just ran into an issue where an HTTP endpoint was down for 4 days and the feeding queue dropped the message. I imagine this has something to do with the retention period setting of the queue.
Questions
Is there a way to know, in the lambda, how many times a message has been replayed?
How did the feeder queue know that the message that was re-enqueued was the same as the one that was originally put on the queue?
I'm currently not explicitly deleting a message off the queue. Not having that, hasn't seemed to cause any issues, no re-processing of messages or anything. Should I be explicitly deleting them?
The normal process would be:
The AWS Lambda function is triggered, with the message(s) passed via the event parameter
If the Lambda function successfully processes the message(s), it should return a 'success' code (200) and the message is automatically removed from the queue
If the Lambda function is unable to process the message, it should return a 'failure' code (eg 400) and Amazon SQS will automatically attempt to re-process the message (unless it has exceeded the retry count)
If the Lambda function fails (eg due to a timeout), Amazon SQS will automatically attempt to re-process the message (unless it has exceeded the retry count)
If a message has exceeded its retry count, Amazon SQS will move the message to the Dead Letter Queue
To answer your questions:
If you wish to take responsibility for these activities yourself, you can use the ApproximateReceiveCount attribute on the message. In the request, it appears that you should add AttributeNames=['ApproximateReceiveCount'], but the documentation is a bit contradictory. You might need to use All instead.
Since you are sending a new message to the queue, Amazon SQS is not aware that it is the same message. The message is not 're-enqueued' since it is a new message.
When your Lambda function returns 'success' (200), the message is being deleted off the queue for you.
You might consider using the standard functionality for retries and Dead Letter Queues rather than implementing that logic yourself.
I'm sending messages that failed in my lambda to a dead letter queue using aws sdk. I want to wait for few hours before sending the message back to the main queue for reprocessing. I have a lambda attached to my dead letter queue. I can use delay for sending messages to the dead letter queue. But the maximum delay is 15 minutes. But I want to wait for more time. Has anyone done this before?
Amazon SQS is not intended to be used in this manner. Its primary purpose is to store messages and then provide them back when requested.
Some other options:
Store the message in a database and have the application search for relevant messages based on a timestamp field, or
Do some tricky stuff with delays on AWS Step Functions (which has a delay feature)
As shown in this answer you can extend the delay, by giving each message a timestamp and processing only those that are in queue for a while now.
message = SQS.poll_messages
if message.perform_message_at > Time.now
SQS.push_to_queue({perform_message_at : "Thursday November
2022"},delay:15 mins)
else
process_message(message)
end
I would like to publish a message on SQS and process that message after a few hours.
How can i schedule a message delivery or select messages from SQS based on some attribute?
I've implemented a SQS consumer but I'm receiving every message from SQS queue. Is possible to implement something like that on SQS? I was thinking about to receive every message and send to queue again if it's not time to process that message.
There is a feature called as Delay Queues in SQS, wherein if you set the delay on the queue then any message that is put on queue is available to consumers only after the delay duration has elapsed. However, the maximum delay that you can set there is 15 minutes and if you are looking for a delay of few hours this may not directly work for you.
The other option is to set a visibilty timeout for the messages higher than the delay time that you want. Then when you read the message you can get the message timestamp. If there is still some time left for your delay then you can sleep your consumer for the remaining time and after it has woken up you can process that message. However this is not a recommended way and would be highly inefficient because your threads are getting blocked. In fact what can as well be done is if there is still some time left for your delay then you just hold the message in a local List/Array and check for other messages and process this message after your delay. But all this would require entire logic to reside in your code and you don't get any ready-made feature from AWS
For example, a message is sent to a queue at 2016-05-19 23:29:05. And it being received by a consumer at 2016-05-19 23:30:05. After processed, the message is deleted at 2016-05-19 23:30:08. That means it takes 60 seconds from message sent to queue to consumer received it, and it takes 63 seconds from the message born to dead.
This metrics is helpful to monitor how fast consumer can consume messages.
To get lifetime of a message, I have to manually add timestamp attribute to each message when it is sent to queue. Consumer will retrieve the timestamp and calculate the time interval.
Does SQS provide similar API to get lifetime of a message (I checked SQS API and with no luck), or is there any other good idea?
There are attributes on an SQS message which will allow you to measure the time spent by a message in the queue.
Some of the attributes you can use are
SentTimestamp, CreatedTimestamp, LastModifiedTimestamp etc
I was looking at the javascript SDK, and it seems like some attributes are only returned when explicitly requested for
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SQS.html#receiveMessage-property
For the list of all attributes available within an SQS message please refer to http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html
While using AWS SQS, does "SentTimestamp" attribute of a message change after it is received from the queue, but not deleted and returned back to the queue after the visibility timeout expired?
No, and that behavior would be undesirable because SentTimestamp specifically describes when the message was first sent to the queue.
This documentation that describes the visibility timeout offers some insight:
Immediately after the component receives the message, the message is
still in the queue. However, you don't want other components in the
system receiving and processing the message again. Therefore, Amazon
SQS blocks them with a visibility timeout, which is a period of time
during which Amazon SQS prevents other consuming components from
receiving and processing that message.
The important takeaway here is that the message never really leaves the queue, it is simply hidden from other clients that are receiving messages. So message contents like the MessageID and SentTimeout won't change. On the other hand, things related to receiving the message like RecieptHandle and Receive Count do change each time the message is received.
You can self-verify this from the AWS web console by:
Creating a message in a queue.
Viewing the message.
Waiting for the visibility timeout to expire. Once it has, open the SQS console again in a new tab.
Viewing the message again, in the new tab. Compare the contents of both received messages.