I deployed a lambda, SQS standard queue and Dead letter queue on AWS. And I configured maxReceiveCount in the queue to retry before putting events to DLQ. Lambda pulls events from SQS queue in batch and process each event sequently. My question is about how retry works in case of error. There are two retries, one is on lambda maximumRetryAttempts, the other is on SQS and DLQ. Should I disable the lambda one?
In function, when it processes one event it calls deleteMessage on sqs to delete it. If there is any event that throws exception, the function throws it to lambda to make the retry happen so that it won't retry the success events.
But lambda itself has a maximumRetryAttempts and should I set it to 0? otherwise, will it retry before return to SQS? If I don't disable it, will the retry to process the whole batch of events including the success one?
Not sure which maximumRetryAttempts on lambda you are referring to. But When you use SQS with Lambda through event source mapping, as its done by default, there is no any retry parameter on lambda.
The only retry that applies is set at SQS, not lambda.
The retry option for lambda I can think of, and maybe you are thinning off as well, is for asynchronous invocation. This does not apply for SQS, as your lambda is invoked synchronously with SQS:
Lambda polls the queue and invokes your Lambda function synchronously with an event that contains queue messages.
Lambda Function can be invoked in three different ways:
Lambda reads from and invokes function. Ex: From SQS, Kinesis, etc.
Function invoked synchronously. Ex: From ApiGateway, ELB, etc.
Function invoked asynchronously. Ex: From S3 Events, SNS, Cloudwatch events etc.
Below Retry attempts is applicable for Asynchronous invocations(option 3 above)
For SQS Failures, we have two options:
DLQ on SQS itself.
Destination on Lambda. This could be SNS, another lambda, event bridge and another SQS queue. With this option, we can send both failures and success events.
Note: We don't need to call deleteMessage within lambda, lambda poller will delete message from SQS, when lambda returns success.
Related
I have a lambda function with SQS as its trigger. when lambda executes, either it throws an error or not. it will put the job back in the queue and creates a loop and you know about the AWS bill for sure :)
should I return something in lambda function to let SQS know that I got the message(done the job)? how should I ack the message? as far as I know we don't have ack and nack in SQS.
Is there any option in the SQS configuration to only retry N time if any job fails?
For standard uses cases you do not have to actively manage success-failure communication between lambda and SQS. If the lambda returns without error within the timeout period, SQS will know the message was successfully processed. If the function returns an error, then SQS will retry a configurable number of times and finally direct still-failing messages to a Dead Letter Queue (if configured).
Docs: Amazon SQS supports dead-letter queues, which other queues (source queues) can target for messages that can't be processed (consumed) successfully.
Important: Add your DLQ to the SQS queue, not the Lambda. Lambda DLQs are a way to handle errors for async (event-driven) invocation.
When a file is added to my S3 bucket an S3PUT Event is triggered which puts a message into SQS. I've configured a Lambda to be triggered as soon as a message is available.
In the lambda function, I'm sending an API request to run a task on an ECS Fargate container with environment variables containing the message received from SQS. In the container I'm using the message to download the file from S3, do processing and on successful processing I wish to delete the message from SQS.
However the message gets deleted from SQS automatically after my lambda executes.
Is there any way that I can configure the lambda not to automatically delete the SQS message (other than raising an exception and failing the lambda purposely), so that I can programmatically delete the message from my container?
Update:
Consider this scenario which I wish to achieve.
Message enters SQS queue
Lambda takes the message & runs ECS API and finishes without deleting the msg from queue.
Msg is in-flight.
ECS container runs the task and deletes msg from queue on successful processing.
If container fails, after the visibility timeout the message will re-enter the queue and the lambda will be triggered again and the cycle will repeat from step 1.
If container fails more than a certain number of times, only then will message go from in-flight to DLQ.
This all currently works only if I purposely raise an exception on the lambda and I'm looking for a similar solution without doing this.
The behaviour is intended and as long as SQS is configured as a Lambda trigger, once the function returns (i.e. completes execution) the message is automatically deleted.
The way I see it, to achieve the behaviour you're describing you have 4 options:
Remove SQS as Lambda trigger and instead execute the Lambda Function on a schedule and poll the queue yourself. The lambda will read messages that are available but unless you delete them explicitly they will become available again once their visibility timeout is expired. You can achieve this with a CloudWatch schedule.
Remove SQS as Lambda trigger and instead execute the Lambda Function explicitly. Similar to the above but instead of executing on a schedule all the time, the Lambda function could be triggered by the producer of the message itself.
Keep the SQS Lambda trigger and store the message in an alternative SQS Queue (as suggested by #jarmod in a comment above).
Configure the producer of the message to publish a message to an SNS Topic and subscribe 2 SQS Queue to this topic. One of the two queues will trigger a Lambda Function, the other one will be used by your ECS tasks.
Update
Based on the new info provided, you have another option:
Leave the event flow as it is and let the message in the SQS be deleted by Lambda. Then, in your ECS Task, handle the failure state and put a new message in the SQS with the same payload/body. This will allow you to retry indefinitely.
There's no reason why the SQS message has to be the exact same, what you're interested is the body/payload.
You might want to consider adding a mechanism to set a limit to these retries and post a message to a DLQ.
One solution I can think of is: remove lambda triggered by the sqs queue, create an alarm that on sqs queue. When the alarm triggers, scale out the ecs task. When there's no item in the queue, scale down the ecs task. Let the ecs task just poll the queue and handle all the messages.
I have structure similar to this
SQS -> Lambda -> DLQ
When Lambda is invoked asynchronously like below then failure messages are getting added successfully to DLQ.
$ aws lambda invoke --function-name my-function --invocation-type Event --payload '{ "key": "value" }' response.json
But when lambda gets triggered on adding new messages to SQS then on failure, messages doesn't store in DLQ.
I found that events triggered when new message published to SQS are synchronous in nature.
Lambda polls the queue and invokes your function synchronously with an
event that contains queue messages.
Reference - https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html.
So I want either-
SQS event trigger invoke lambda asynchronously
or
Messages gets stored in DLQ on synchronous invocation of Lambda as well.
DLQ is used not to store error messages but to store failed events, that can be handled again later.
CloudWatch is used to store and show logs (including errors) for Lambda and any other AWS service.
The idea behind SQS triggering Lambda is that in case of Lambda failure, the event will be handled again by Lambda later.
It's the same idea as DLQ but implemented differently.
I'm trying to send s3event message to rabbitmq by invoking AWS lambda function. I have configured SQS as my dead letter queue(DLQ).
I know the message is sent to DLQ when there failure in invocation of lambda or situations like timeouts or resource constraints.
My question is ,I want to send event message to DLQ from inside lambda function on certain condition like if rabbitmq is down or some other condition of my interest.
Is there any possiblity for the same? Should I throw exception or there is some other better approach to send event message to DLQ.
I'm using java for development and connecting to rabbitmq from my lambda function.
The DLQ is simply an SQS Queue, so you could send a message to it like you would any other queue. You would want it to be formatted the same way that Lambda natively puts message in the DLQ so that whatever processing you have on the DLQ can perform the same way for all messages. You would want to ensure that you treat the lambda as successfully executed in this instance though so that the normal DLQ process doesn't pick up the same message twice.
In the DLQ setting of a Lambda you specify a SNS topic or a SQS Queue. In your setup you have configured the DLQ to be a SQS queue. This is a regular SQS Queue. Using the SQS Java SDK you can post a message to that SQS Queue.
here are few references:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-send-message.html
To get the Queue URL you can use these:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_GetQueueUrl.html
Or through Java:
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQSClient.html#getQueueUrl-java.lang.String-
I am using AWS Lambda function to process the messages in Queue it's working fine. But i need to execute this Lambda function when messages available or added in SQS queue.
Is it possible to trigger the Lambda function based on SQS queue.Please suggest one method to achieve this goal.
Invoking Lambda functions from SQS queues is not directly supported. You can see the list of available triggers here: http://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html
Possible solutions:
Replace SQS queue with Kinesis or DynamoDB. Both can trigger Lambda functions on updates.
Inject SNS before SQS. SNS can add items to SQS and trigger Lambda function.
If you don't need near real-time processing, these two options are also valid:
Create CloudWatch Event Rule that will trigger the Lambda function every N minutes (e.g. every minute).
Create CloudWatch alarm watching ApproximateNumberOfMessagesVisible parameter for your SQS queue. This alarm should publish to an SNS topic, which in turn will trigger Lambda function.
Lambda now supports SQS as a native event source
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html