How to implement a Dead Letter Queue using AWS - amazon-web-services

Question

I'm trying to implement a dead letter queue for failed lambda events. I created an SQS queue and two lambdas. One lambda only throws an error and the other listens to the SQS queue and is triggered when the queue receives a message and logs the event to Cloudwatch.
In the AWS management console, I am able to see that the .function lambda (the one that throws an error) has a dead letter queue configured.
I am able to see that the .dispatcher function (the one that listens to SQS) is configured correctly and if I send a message to the SQS queue I can also see Cloudwatch logs showing it's being triggered as expected.
Although everything seems to be configured correctly, I am unable to get the .function to send an event to SQS when it's failed in order to trigger the .dispatcher lambda.
I've watched multiple videos, read documentation and I have not been able to figure out why this is not working. I've also tried the onError configuration but, no luck there either.
handler.js
module.exports.function = async (event) => {
throw new Error('Throwing error 🚨')
}
module.exports.dispatcher = async (event) => {
try {
console.log(`Dispached Event:${JSON.stringify(event)}`)
} catch (error) {
console.log(error)
return error
}
}
serverless.yml
service: dlq-example
frameworkVersion: '3'
provider:
name: aws
runtime: nodejs14.x
iam:
role:
statements:
# Allow functions to list all buckets
- Effect: Allow
Action: '*'
Resource: '*'
plugins:
- serverless-plugin-lambda-dead-letter
functions:
hello:
handler: handler.function
description: Throws error.
deadLetter:
targetArn:
GetResourceArn: DeadLetterQueue
dispatcher:
handler: handler.dispatcher
description: Dispatcher function. Configured to be triggered by events sent to SQS and handle them.
events:
- sqs:
arn:
Fn::GetAtt:
- DeadLetterQueue
- Arn
resources:
Resources:
DeadLetterQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: dlq
Here are images of what I see in the AWS management console:
Screenshot of handler.function lambda (the one that has the DLQ configured)
Screenshot of the handler.dispatcher function (The one that listens to SQS and handles each message)
Screenshot of the SQS queue and the lambda trigger configuration.

Answer 1

It seems to me, there might be disconnect in understanding on how the DLQ behavior works for Lambda.
First and foremost, DLQ feature in AWS Lambda is only available for triggers which are invoked asynchronously like S3, SNS, IOT etc
For the event source AWS SQS, Lambda polls the queue and invokes Lambda function synchronously with an event that contains queue messages.
Hence setting a DLQ on lambda for event source of SQS would not work, it would never be triggered by lambda.
From the context of your problem, I think, what you are trying to achieve are:
1. Have a mechanism to capture SQS messages which fails to process in lambda function. (Preferably to a designated DLQ)
2. Have a mechanism to re-process these failed message using a different lambda function.
Assuming my understanding of the premise of this problem is correct, you may follow below steps to achieve it.
SQS queue provide a feature where if the consumers of SQS fails to process the message for a certain number of times, These failed message can be moved to DLQ. (These DLQ can be configured at primary queue level).
You can follow your existing pattern and create a trigger on this DLQ to process the failed messages from the dispatcher lambda. You may use this helpful guide to set these up.
In this approach, you are configuring DLQ at Queue level instead on lambda.
I hope this would help you in your design.

Answer 2

For anyone who stumbles across this issue in the future. My configuration was correct, I was missing a SQS queue policy. Once I configured a policy everything worked as expected.