So I have 10 message groups in FIFO queue having 2 messages per group and I have also reserved lambda concurrency set to 5. (Lambda completes execution in 1 min and SQS visibility timeout set to 2 mins)
when all 20 messages are pushed to queue, SQS inflight messages gets set to 10 and then after the execution time, 5 messages gets processed successfully and other 5 moves to DLQ.
And then the next executions inflight messages gets set to 5 (as reserved lambda concurrency set to 5.) and processes as expected (This should be the expected behaviour right?)
Any particular reason why this is happening?
Related
I have a FIFO Queue in the AWS SQS, which is trigger's Lambda function.
I want to process each messages in Lambda function without parallel execution (one message at a time)
For example: If I have a message A, B, C in the queue. My lambda should complete A, then start B etc.,
My current config of the FIFO queue is
Message retention period: 4 Days
Default visibility timeout: 1 Hour 30 Minutes
Delivery delay: 0 sec
Receive message wait time: 0 Second
set up another lamba(add/setup trigger to invoke this lamba for every one minute using Amazon event brige rule).
In the lamba funtion using receivemessage api call( https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html) you can fetch desired number of messages from sqs queue (max 10 messages) and delete after processing each message.
I have lambda with a SQS trigger with
batch size 10
batch window 60 secs
Lambda timeout 5 mins
Visibility Timeout 6 mins
zero lambda retries
After triggering 30 messages in the queue, all the message are becoming inflight and the lambda process 15 to 16 messages and once the visibility time is left over the messages are going to DLQ.
Tried with 100 and 1 batch size, it stops unexpectedly after processing some messages and drop rest to DLQ.
Not able to understand why it stop processing after certain time.
Any help will be highly appreciated
An AWS SQS FIFO queue has a batch setting of 1 and a delay of 1 second. Every item received is associated with a MessageGroup.
All at once the queue receives 30 messages across 10 different message groups with each message group accounting for 3 messages...
Will the delay of one second apply at the queue level i.e. the 30 messages will take an elapsed time of 30 seconds to deliver?
Or will the queue spin up 10 consumers, one for each message group, emptying the queue in 3 seconds?
Will the delay of one second apply at the queue level i.e. the 30 messages will take an elapsed time of 30 seconds to deliver?
For FIFO, the delay is applied at queue level:
FIFO queues don't support per-message delays, only per-queue delays. If your application sets the same value of the DelaySeconds parameter on each message, you must modify your application to remove the per-message delay and set DelaySeconds on the entire queue instead.
Number of comsumer lambdas working in parallel does not need to be 10 as written bellow:
In SQS FIFO queues, using more than one MessageGroupId enables Lambda to scale up and process more items in the queue using a greater concurrency limit. Total concurrency is equal to or less than the number of unique MessageGroupIds in the SQS FIFO queue.
Thus you can have still fewer consumer lambdas than groups. But in ideal situation you would have 10 lambdas working in parallel. How it works with lambda is explained in the following AWS blog post:
New for AWS Lambda – SQS FIFO as an event source
I am working with AWS SQS and Lambda. I wanted to know that if the batchsize = 5 and sqs messages left = 3.
Will the Lambda be triggered by a batch of 3 messages or will sqs wait for the message count to become 5?
From docs:
Batch size – The number of items to read from the queue in each batch, up to 10. The event might contain fewer items if the batch that Lambda read from the queue had fewer items.
Thus, based on this, you should get 3 messages. Lambda should not be waiting for 5.
My Lambda configuration is as below
Lambda Concurrency is set to 50
And SQS trigger batch size is set to 1
Issue:
When my queue is flooded with 200+ messages, some of the sqs triggers are missed and the message from the queue goes to inflight state without even triggering the lambda. This is adding a latency in processing by the timeout value set for lambda as I need to wait for the message to come out of flight for it to be reprocessed.
Any inputs will be highly appreciated.
SQS is integrated with Lambda through event source mappings.
Thanks to the mappings, the Lambda service is long polling the SQS queue, and invoking your function on your behalf. What's more it automatically removes the messages from the queue if your Lambda successfully processes them.
Since you want to process 200+ messages, and you set concurrency to 50 with batch size of 1, it means that you can process only 50 messages in parallel. The rest will be throttled. When this happens:
If your function is throttled, returns an error, or doesn't respond, the message becomes visible again. All messages in a failed batch return to the queue, so your function code must be able to process the same message multiple times without side effects.
To rectify the issue, the following two immediate actions can be considered:
increase concurrency of your function to 200 or more.
increase batch size to 10. With the batch size and concurrency of 50, you can process 500 (10 x 50) messages concurrently.
Also since you are heavily throttled, setting up a dead-letter queue can be useful. The DLQ helps captures problematic or missed messages from the queue, so that you can process them later or inspect:
If a message fails to be processed multiple times, Amazon SQS can send it to a dead-letter queue. When your function returns an error, Lambda leaves it in the queue. After the visibility timeout occurs, Lambda receives the message again. To send messages to a second queue after a number of receives, configure a dead-letter queue on your source queue.