The architecture for a service includes the following:
[S3 event notification -> EventBridge rule triggered -> same message sent to both Lambdas]
There is a DLQ configured for the EventBridge and a DLQ for each Lambda. The former DLQ is for issues with delivery of the message to the Lambdas and the latter DLQs are for issues with the Lambdas itself.
Let's say Lambda A processes the message successfully but Lambda B does not and after exhausting the retry policy, Lambda B sends the message from the EventBridge to its own DLQ since this error is due to a code issue with the Lambda B function.
After updating the code for Lambda B, what is the recommended approach to send the message in the DLQ back to the EventBridge for Lambda B to process it again without sending this message to Lambda A.
Note: The EventBridge sends the same S3 notification message to both Lambdas.
Related
I have an SQS queue (Q) that receives messages via the onFailure "destinations" setting from a Lambda function (F). The Lambda function is triggered by and EventBridge event bus "rule".
My question is: Can I configure the redrive policy of queue Q so that I can redrive messages directly to function F?
Currently, I have set the redrive allow policy to allowAll, but the "Start DLQ Redrive" button is disabled in the console. Looking at the configuration settings for the drive allow policy, I get the feeling that only other queues can be a target for a redrive.
What confuses me about this is that my goal here was to use the onFailure function of the "destinations" feature. Destinations can only be used when a function is called asynchronously and queues trigger lambdas synchronously. So if I were to put a queue in front of my lambda function F that could be a target for a redrive, then I would not be able to use the onFailure destination.
It's not possible to send an event payload from Queue Q to Lambda F with redrive. Redrive works by sending messages from the DLQ back to the source queue, not directly to a Lambda target. Consider, too that the SQS message structure differs from that of EventBridge events, which would confuse your Lambda.
Check out Event replay as an alternative. Or add a Lambda to periodically read from the DLQ and resubmit the events.
I deployed a lambda, SQS standard queue and Dead letter queue on AWS. And I configured maxReceiveCount in the queue to retry before putting events to DLQ. Lambda pulls events from SQS queue in batch and process each event sequently. My question is about how retry works in case of error. There are two retries, one is on lambda maximumRetryAttempts, the other is on SQS and DLQ. Should I disable the lambda one?
In function, when it processes one event it calls deleteMessage on sqs to delete it. If there is any event that throws exception, the function throws it to lambda to make the retry happen so that it won't retry the success events.
But lambda itself has a maximumRetryAttempts and should I set it to 0? otherwise, will it retry before return to SQS? If I don't disable it, will the retry to process the whole batch of events including the success one?
Not sure which maximumRetryAttempts on lambda you are referring to. But When you use SQS with Lambda through event source mapping, as its done by default, there is no any retry parameter on lambda.
The only retry that applies is set at SQS, not lambda.
The retry option for lambda I can think of, and maybe you are thinning off as well, is for asynchronous invocation. This does not apply for SQS, as your lambda is invoked synchronously with SQS:
Lambda polls the queue and invokes your Lambda function synchronously with an event that contains queue messages.
Lambda Function can be invoked in three different ways:
Lambda reads from and invokes function. Ex: From SQS, Kinesis, etc.
Function invoked synchronously. Ex: From ApiGateway, ELB, etc.
Function invoked asynchronously. Ex: From S3 Events, SNS, Cloudwatch events etc.
Below Retry attempts is applicable for Asynchronous invocations(option 3 above)
For SQS Failures, we have two options:
DLQ on SQS itself.
Destination on Lambda. This could be SNS, another lambda, event bridge and another SQS queue. With this option, we can send both failures and success events.
Note: We don't need to call deleteMessage within lambda, lambda poller will delete message from SQS, when lambda returns success.
What is the difference between setting a Dead Letter Queue on an SNS Topic or on a Lambda function?
I was wondering, because if you set the DLQ on the SNS subscription, then that subscription message will fail over to DLQ when the Lambda (the subscriber) fails, correct? So in that scenario would setting the DLQ in these two places have the same effect?
I have set a DLQ on a SNS Topic Subscription, and it didn't "automagically" appear as the DLQ on the Lambda screen settings, so I assume there may be some difference?
SNS dead letter queue ref: https://docs.aws.amazon.com/sns/latest/dg/sns-dead-letter-queues.html
In general, message delivery fails when Amazon SNS can't access a subscribed endpoint due to a client-side or server-side error.
Lambda dead letter queue ref: https://aws.amazon.com/about-aws/whats-new/2016/12/aws-lambda-supports-dead-letter-queues/
AWS Lambda will write the event object invoking the Lambda function to this [DLQ] endpoint after the standard retry policy (2 additional retries on failure) is exhausted.
Lambda:
SNS subscription:
The SNS DLQ is specific for SNS, the benefit here is that it also accounts for client errors such as the Lambda service being down. This will ensure if the Lambda service is down the messages can be replayed to it later, whereas if the DLQ is attached to the Lambda this would only account for a replay if the service is running.
However as I mentioned the SNS DLQ is only for notifications that are from SNS, whereas Lambda can support the DLQ from any incoming events. This means if you have multiple SNS topics, or an SNS topic and some SQS queues you only need to apply it to the Lambda itself.
Both services use SQS for their DLQs so the ingestion/retrieval from both would be identical. If you have a DLQ on both services then yes you might end up with 2 copies of the event/notification, however it is unlikely that you will get both as in theory would the Lambda endpoint acknowledges it SNS would treat as sent, this would then be Lambdas responsibility to add to the DLQ if it fails.
SNS DLQ and Lambda DLQ protect your workload from different failure modes. The former is for failed message delivery from a topic, whereas the latter is for failed function execution. You should use both simultaneously. Here more info: https://aws.amazon.com/blogs/compute/designing-durable-serverless-apps-with-dlqs-for-amazon-sns-amazon-sqs-aws-lambda/
I have structure similar to this
SQS -> Lambda -> DLQ
When Lambda is invoked asynchronously like below then failure messages are getting added successfully to DLQ.
$ aws lambda invoke --function-name my-function --invocation-type Event --payload '{ "key": "value" }' response.json
But when lambda gets triggered on adding new messages to SQS then on failure, messages doesn't store in DLQ.
I found that events triggered when new message published to SQS are synchronous in nature.
Lambda polls the queue and invokes your function synchronously with an
event that contains queue messages.
Reference - https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html.
So I want either-
SQS event trigger invoke lambda asynchronously
or
Messages gets stored in DLQ on synchronous invocation of Lambda as well.
DLQ is used not to store error messages but to store failed events, that can be handled again later.
CloudWatch is used to store and show logs (including errors) for Lambda and any other AWS service.
The idea behind SQS triggering Lambda is that in case of Lambda failure, the event will be handled again by Lambda later.
It's the same idea as DLQ but implemented differently.
I'm trying to send s3event message to rabbitmq by invoking AWS lambda function. I have configured SQS as my dead letter queue(DLQ).
I know the message is sent to DLQ when there failure in invocation of lambda or situations like timeouts or resource constraints.
My question is ,I want to send event message to DLQ from inside lambda function on certain condition like if rabbitmq is down or some other condition of my interest.
Is there any possiblity for the same? Should I throw exception or there is some other better approach to send event message to DLQ.
I'm using java for development and connecting to rabbitmq from my lambda function.
The DLQ is simply an SQS Queue, so you could send a message to it like you would any other queue. You would want it to be formatted the same way that Lambda natively puts message in the DLQ so that whatever processing you have on the DLQ can perform the same way for all messages. You would want to ensure that you treat the lambda as successfully executed in this instance though so that the normal DLQ process doesn't pick up the same message twice.
In the DLQ setting of a Lambda you specify a SNS topic or a SQS Queue. In your setup you have configured the DLQ to be a SQS queue. This is a regular SQS Queue. Using the SQS Java SDK you can post a message to that SQS Queue.
here are few references:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-send-message.html
To get the Queue URL you can use these:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_GetQueueUrl.html
Or through Java:
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQSClient.html#getQueueUrl-java.lang.String-