In AWS SQS console,
I have created Standard SQS Queue and configured as following:
Message retention period: 4 hours
Default visibility timeout: 1 hour
Receive message wait time: 0 seconds
Delivery Delay: 0 seconds
Poll settings as following:
Polling Duration: 60 seconds
Maximum message count: 500
But, What if the count of messages sent to the queue is 1500?
There's a lambda that's processing the messages every half an hour and deleting the (read) SQS messages.
Will other 1000 messages get lost or will they get into SQS whenever another messages in SQS are getting deleted?
From docs:
A single Amazon SQS message queue can contain an unlimited number of messages.
So they will not get deleted from your SQS. Instead they will be send to your lambda as a second batch.
Related
I have a FIFO Queue in the AWS SQS, which is trigger's Lambda function.
I want to process each messages in Lambda function without parallel execution (one message at a time)
For example: If I have a message A, B, C in the queue. My lambda should complete A, then start B etc.,
My current config of the FIFO queue is
Message retention period: 4 Days
Default visibility timeout: 1 Hour 30 Minutes
Delivery delay: 0 sec
Receive message wait time: 0 Second
set up another lamba(add/setup trigger to invoke this lamba for every one minute using Amazon event brige rule).
In the lamba funtion using receivemessage api call( https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html) you can fetch desired number of messages from sqs queue (max 10 messages) and delete after processing each message.
I have lambda with a SQS trigger with
batch size 10
batch window 60 secs
Lambda timeout 5 mins
Visibility Timeout 6 mins
zero lambda retries
After triggering 30 messages in the queue, all the message are becoming inflight and the lambda process 15 to 16 messages and once the visibility time is left over the messages are going to DLQ.
Tried with 100 and 1 batch size, it stops unexpectedly after processing some messages and drop rest to DLQ.
Not able to understand why it stop processing after certain time.
Any help will be highly appreciated
I have a lambda that is triggered by an SQS but I want to disable the event trigger in the lower environments.
If there are other parts that publishes to that SQS, what would happens to the the message? There is no DLQ on the SQS. Will the message disappear after the MessageRetentionPeriod is up?
Per the SQS FAQ:
Q: How long can I keep my messages in Amazon SQS message queues?
A: You can configure the Amazon SQS message retention period to a value from 1 minute to 14 days. The default is 4 days. Once the message retention quota is reached, your messages are
automatically deleted.
The word 'quota' is a bad choice, imo, but it means the message retention period.
Note: if a message happens to have been moved from a regular SQS queue to its associated Dead Letter Queue then the retention period is considered to start when the message first arrived on the underlying SQS queue, not when it was transferred to the DLQ.
I created below resources using CloudFormation
SQS (DelayDelivery : 0)
Lambda
Roles
LambdaFunctionEventSourceMapping (SQS Triggers a Lambda function)
Logs
Resources created successfully.
When I send a message to SQS with Delay deliver for 30 seconds, SQS triggers Lambda instantaneously. Instead, it should have trigger after 30 seconds.
FYI : I am sending message using AWS console.
As per below link, it should have override the SQS delay to individual message delay.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-timers.html
Is there any other way to achieve this.
This is possibly a bug with the AWS console, because I encounter similar issue (lambda invoked instantaneously) when I send the message to SQS via AWS console. However, I can achieve the desired behaviour if I send the message to SQS via CLI:
aws sqs send-message --queue-url https://sqs.ap-southeast-1.amazonaws.com/{account}/{sqs} --message-body "msg 1 delay 60 secs" --delay-seconds 60
aws sqs send-message --queue-url https://sqs.ap-southeast-1.amazonaws.com/{account}/{sqs} --message-body "msg 2 delay 30 secs" --delay-seconds 30
Then this is what I see in the lambda Cloudwatch log:
2019-02-19T02:26:36.189Z 510df77b-d1e5-5297-b0a0-a39eba727c61 msg 2 delay 30 secs
2019-02-19T02:26:36.189Z 510df77b-d1e5-5297-b0a0-a39eba727c61 sent 2019-02-19T02:26:06.124Z
2019-02-19T02:26:36.189Z 510df77b-d1e5-5297-b0a0-a39eba727c61 1st receive 2019-02-19T02:26:36.124Z
2019-02-19T02:26:57.729Z fedcb590-2e14-596a-bc4b-e17545a46d91 msg 1 delay 60 secs
2019-02-19T02:26:57.729Z fedcb590-2e14-596a-bc4b-e17545a46d91 sent 2019-02-19T02:25:57.667Z
2019-02-19T02:26:57.729Z fedcb590-2e14-596a-bc4b-e17545a46d91 1st receive 2019-02-19T02:26:57.667Z
As you can see from the log, the messages delivery are properly delayed, and lambdas are only invoked once the message is received.
How does the Amazon SQS's DLQ work when it comes to old messages?
Do messages older than 14days() get moved to DLQ instead of being deleted ?
I dont see any documentation relating to how older messages are handled.
From the documentation it looks like just the errored messages are moved to DLQ, is my assumption right ?
Your understanding is correct; Messages that are older than the retention period you have set (max of 14 days), will be deleted, not moved to the DLQ.
SQS automatically deletes messages that have been in a queue for more
than maximum message retention period. The default message retention
period is 4 days. However, you can set the message retention period to
a value from 60 seconds to 1209600 seconds (14 days) with
SetQueueAttributes.
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/MessageLifecycle.html
All the messages that are not processed/consumed will gets pushed to DLQ.
Amazon SQS supports dead-letter queues (DLQ), which other queues (source queues) can target for messages that can't be processed (consumed) successfully.
Reference - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html