Here is sam yaml snippet:
SnsTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: !Sub ${Env}-sns-topic
TopicName: !Sub ${Env}-sns-topic
Queue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub ${Env}-queue
VisibilityTimeout: 300
MessageRetentionPeriod: 1209600
ReceiveMessageWaitTimeSeconds: 20
RedrivePolicy:
deadLetterTargetArn: !GetAtt Dlq.Arn
maxReceiveCount: 5
Dlq:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub ${Env}-dlq
VisibilityTimeout: 300
MessageRetentionPeriod: 1209600
ReceiveMessageWaitTimeSeconds: 0
TestSubscription:
Type: AWS::SNS::Subscription
DependsOn:
- SnsTopic
- Queue
- Dlq
Properties:
Protocol: sqs
TopicArn: !Ref SnsTopic
Endpoint: !GetAtt
- Queue
- Arn
RawMessageDelivery: true
I am trying to create an SNS that is subscribed to an SQS with a Dead Letter Queue (DLQ). After deployment, I can see that the SNS and SQS have been created successfully, but the SQS subscription to the SNS does not appear to be working. When I check the subscriptions for the SQS, I can see that the SNS that was created in the same stack is listed, but I have to manually add the subscription for it to work. I'm wondering what could be causing this issue or if there's something missing in my example.
You have to setup AWS::SQS::QueuePolicy to allow SNS to send messages to the queue. Check AWS docs for examples of how to do it.
Related
I have a service that uses both Lambda function and SQS. One Lambda function sends messages via SQS to trigger another function. Because the receiver function is going to call an API which blocks frequent visits (status code = 403), I need to reserve the concurrent executions to avoid that. However, that also results in the message not being consumed and gone missed in the queue. So is there another way to make sure no messages go missed while there's maximum concurrency?
Function properties:
VideoRouter:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub router-${Environment}
Handler: lambda_function.lambda_handler
Runtime: python3.7
CodeUri: router/
Description: something
MemorySize: 256
Timeout: 900
ReservedConcurrentExecutions : 5
Role:
Fn::GetAtt:
- LambdaRole
- Arn
Environment:
Variables:
LOGGING_LEVEL: INFO
APP_NAME: APPS
APP_ENV: !Ref Environment
BUCKET: !Sub my-bucket-${Environment}
PITCH_AI_DB: !Sub db-${Environment}
PROPLAY_CREDENTIALS: /credentials/
Tags:
env:
Ref: Environment
service: my-service
function_name: !Sub my-router-${Environment}
Events:
MySQSEvent:
Type: SQS
Properties:
Queue: !GetAtt QueueA.Arn
BatchSize: 1
Queue Properties:
RouterQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub router-queue-${Environment}
VisibilityTimeout: 900
RedrivePolicy:
deadLetterTargetArn:
Fn::GetAtt: RouterDeadLetterQueue.Arn
maxReceiveCount: 10
RouterDeadLetterQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub router-dead-letter-queue-${Environment}
MessageRetentionPeriod: 1209600
I have A service where we have sns topics and in B service sqs queue event.
from B service cloud formation I need to write the cloud formation YAML file to subscription between SNS event topic and SNS event queue.
sns topic name : sns-event-topic
subscribed to queue name: abcd-events
Resources:
AbcdEventQueue:
Type: "AWS::SQS::Queue"
Properties:
QueueName: "abcd-events"
AbcdEventQueuePolicy:
Type: "AWS::SQS::QueuePolicy"
Properties:
Queues:
- Ref: "AbcdEventQueue"
PolicyDocument:
Statement:
- Effect: "Allow"
Principal:
AWS: '*'
Action:
- sqs:SendMessage
- sqs:ReceiveMessage
- sqs:DeleteMessage
- sqs:GetQueueUrl
- sqs:GetQueueAttributes
- sqs:ListQueueTags
- sqs:ChangeMessageVisibility
Resource:
- !GetAtt AbcdEventQueue.Arn
Assuming you have the SNS topic already you would create a AWS::SNS::Subscription resource.
It would look like the below structure
Subscription:
Type: 'AWS::SNS::Subscription'
Properties:
TopicArn: !Ref TopicArn #You will need to provide the SNS Topic Arn here
Endpoint: !GetAtt
- AbcdEventQueue
- Arn
Protocol: sqs
RawMessageDelivery: 'true'
If the SNS topic does not share the same stack you will need to pass this into your template, this can be done either as a parameter or by using the Export feature to define a global value that you can use by referencing it with the Fn::ImportValue intrinsic function.
in lambda
Subscription:
Type: AWS::Lambda::EventSourceMapping
Properties:
EventSourceArn: !ImportValue sns-topic-arn
FunctionName: !GetAtt Function.Arn
Enabled: true
BatchSize: 1
I am trying to create an AWS SQS Dead Letter Queue, using the serverless framework
The idea is to have a SQS to trigger a Lambda function,
and have another SQS as a DeadLetterQueue, ie. to pick up the message in case the Lambda fails or timesout
I did the following to create a test project -
mkdir dlq
cd dlq/
serverless create --template aws-nodejs
Following is my serverless.yaml -
service: dlq
provider:
name: aws
runtime: nodejs12.x
region: ap-southeast-1
role: arn:aws:iam::xxxx:role/dlqLambdaRole
plugins:
- serverless-plugin-lambda-dead-letter
functions:
dlq:
handler: handler.hello
events:
- sqs:
arn:
Fn::GetAtt:
- MainQueue
- Arn
deadLetter:
targetArn:
GetResourceArn: DeadLetterQueue
resources:
Resources:
MainQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: main
DeadLetterQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: dlq
I also tried the following -
service: dlq
provider:
name: aws
runtime: nodejs12.x
region: ap-southeast-1
role: arn:aws:iam::xxxx:role/dlqLambdaRole
plugins:
- serverless-plugin-lambda-dead-letter
functions:
dlq:
handler: handler.hello
events:
- sqs:
arn:
Fn::GetAtt:
- MainQueue
- Arn
deadLetter:
sqs: dlq
resources:
Resources:
MainQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: main
But in both these cases, the framework is just creating a normal SQS
I am following this document -
https://www.serverless.com/plugins/serverless-plugin-lambda-dead-letter
Better late than never. Hope this helps you or someone searching for this problem.
When you configure SQS to trigger Lambda, the DLQ is supposed to be configured on the SQS(Since it would not be an Asynchronous invocation).
Notice the 'Note' section in the link
Source
Hence your serverless.yaml needs to declare the ReddrivePolicy in the main queue to refer to the DLQ. (Below)
service: dlq
provider:
name: aws
runtime: nodejs12.x
region: ap-southeast-1
role: arn:aws:iam::xxxx:role/dlqLambdaRole
functions:
dlq:
handler: handler.hello
events:
- sqs:
arn:
Fn::GetAtt:
- MainQueue
- Arn
deadLetter:
targetArn:
GetResourceArn: DeadLetterQueue
resources:
Resources:
MainQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: main
RedrivePolicy:
deadLetterTargetArn:
Fn::GetAtt:
- "DeadLetterQueue"
- "Arn"
maxReceiveCount: 5
DeadLetterQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: dlq
maxReceiveCount is set to 5 as per AWS Documentation.
To give you some background, Dead Letter Queue is just that, a normal SQS queue. it's the configuration at AWS Lambda that informs it to push message to this Queue whenever there is any error while processing the message.
You can verify this from the management console by referring to the "Dead-letter queue service" under "Asynchronous invocation"
I have a cloudformation stack containing a FIFO queue, and its associated dead letter queue. Previously this was not a FIFO queue, and it deployed fine, the dead letter queue dependency was made first, then the "source queue". After switching it to FIFO, it no longer works. I get this error:
"Template error: SQSQueue https://sqs.us-east-1.amazonaws.com/1234/dev-assignments-dlq doesn't exist",
So it seems the dead letter queue is no longer being created first.
AWSTemplateFormatVersion: "2010-09-09"
Resources:
SourceQueue:
Type: AWS::SQS::Queue
Properties:
FifoQueue: true
QueueName: 'dev-push-notifications.fifo'
RedrivePolicy:
deadLetterTargetArn:
Fn::GetAtt:
- 'DeadLetterQueue'
- 'Arn'
maxReceiveCount: 5
VisibilityTimeout: 30
DeadLetterQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: 'dev-push-notifications-dlq'
Turns out that the dead letter queue must be of the same type as its source.
Changing the cloudformation stack to this worked:
AWSTemplateFormatVersion: "2010-09-09"
Resources:
SourceQueue:
Type: AWS::SQS::Queue
Properties:
FifoQueue: true
QueueName: 'dev-push-notifications.fifo'
RedrivePolicy:
deadLetterTargetArn:
Fn::GetAtt:
- 'DeadLetterQueue'
- 'Arn'
maxReceiveCount: 5
VisibilityTimeout: 30
DeadLetterQueue:
Type: AWS::SQS::Queue
Properties:
FifoQueue: true
QueueName: 'dev-push-notifications-dlq.fifo'
That's weird, as Cloudformation should detect the dependency because of the GetAtt. You can try declaring it explicitly using the DependsOn attribute:
AWSTemplateFormatVersion: "2010-09-09"
Resources:
SourceQueue:
Type: AWS::SQS::Queue
DependsOn: DeadLetterQueue
Properties:
# ...
I'm writing AWS Lambda to read message from SQS and then save some messages to S3 after filtering. Since 80% messages will be filtered out, I decided write S3 once for 100000 SQS messages.
Is it possible to trigger the Lambda only when the messages in SQS reach 10000?
It's possible with help of AWS CloudWatch.
You could configure an AWS CloudWatch Alarm which triggers an "AlarmAction" as soon as your SQS queue got 100000 visible messages. In case of an "Alarm" you are notifying a SNS Topic which then triggers your AWS Lambda.
If you are using AWS CloudFormation it might look as the following:
AWSTemplateFormatVersion: 2010-09-09
Resources:
Queue:
Type: AWS::SQS::Queue
QueueVisibleMessagesTopic:
Type: AWS::SNS::Topic
LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Lambda:
Type: AWS::Lambda::Function
Properties:
Handler: handler.index
Role: !GetAtt LambdaExecutionRole.Arn
Runtime: nodejs8.10
MemorySize: 128
Timeout: 10
LambdaSubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint: !GetAtt Lambda.Arn
Protocol: lambda
TopicArn: !Ref QueueVisibleMessagesTopic
LambdaSubscriptionPermissions:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt Lambda.Arn
Action: lambda:InvokeFunction
Principal: sns.amazonaws.com
SourceArn: !Ref QueueVisibleMessagesTopic
QueueVisibleMessagesAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
Namespace: AWS/SQS
MetricName: ApproximateNumberOfMessagesVisible
Dimensions:
- Name: QueueName
Value: !GetAtt Queue.QueueName
Statistic: Sum
Period: 300
EvaluationPeriods: 1
ComparisonOperator: GreaterThanOrEqualToThreshold
Threshold: 100000
AlarmActions:
- !Ref QueueVisibleMessagesTopic