I've defined an SNS topic, an SQS queue, and an SNS subscription resource in a Cloudformation stack. All three are in the same stack, same region, and same AWS account.
Resources:
SqsQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: 'some-queue'
SnsTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: 'some-topic'
SnsSubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint: !GetAtt [SqsQueue, Arn]
Protocol: sqs
TopicArn: !Ref SnsTopic
When I run the stack, all three resources are created successfully, but when I publish a message from SNS, it's never received by the SQS queue.
I've been following this link (https://aws.amazon.com/premiumsupport/knowledge-center/sqs-sns-subscribe-cloudformation/) and to my knowledge I've done everything I've needed to. What else am I missing?
Thanks!
Additional info
If I delete the subscription that Cloudformation created via the console and then create a new one via the console, messages are published fine. So it must be something incorrect about the subscription.
I used the AWS CLI to compare the properties of the subscription created by the Cloudformation template to the one created by the console. They are the exact same.
You need to add a policy to allow the SNS topic to publish to your queue. Something like this:
SnsToQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref SqsQueue
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: allow-sns-messages
Effect: Allow
Principal: '*'
Resource: !GetAtt SqsQueue.Arn
Action: SQS:SendMessage,
Condition:
ArnEquals:
aws:SourceArn: !Ref SnsTopic
Related
I've got a serverless file which creates an eventbridge rule on the default event bus:
StepFunctionErrorEvent:
Type: AWS::Events::Rule
Properties:
Name: ${self:custom.resourcePrefix}-step-function-error-event-rule
Description: Event bus rule coordinating what targets receive Step Function error events
EventPattern:
source:
- "aws.states"
"detail-type":
- "Step Functions Execution Status Change"
detail:
state:
- "FAILED"
- "TIMED_OUT"
- "ABORTED"
Targets:
- Arn: ${cf:${self:custom.resourcePrefix}-service-internal-slack-integration.PostSlackMessageLambdaArn}
Id: "ErrorSlackMessage"
DeadLetterConfig:
Arn: !GetAtt DefaultErrorTargetDLQ.Arn
DefaultErrorTargetDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: ${self:custom.resourcePrefix}-DefaultErrorTargetDL
And in a seperate serverless file which also gets deployed I'm adding the following Lambda permission to pl-us-east-2-pilot-post-slack-message:
resources:
Resources:
TriggerPostSlackMessageLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt PostSlackMessageLambdaFunction.Arn
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn: !Sub arn:aws:events:${AWS::Region}:${AWS::AccountId}:rule/pl-us-east-2-pilot-step-function-error-event-rule
However despite pl-us-east-2-pilot-post-slack-message lambda having the above listed as a permission under 'Resource based policy' (in the Lambda console) the EventBridge rule does not trigger when there is a Lambda failure. It does trigger if I create a new rule using the AWS Console, but for whatever reason it's not able to successfully trigger using serverless/CloudFormation.
Every post I seem to read about this topic makes mention of the same thing - that is to have the permission set on your Lambda, but I've done that and it's still not working. Does anyone have any idea what could be the reason why it's not triggering?
hard one to spot, but since i was using step functions
detail:
state:
- "FAILED"
- "TIMED_OUT"
- "ABORTED"
should be
detail:
status:
- "FAILED"
- "TIMED_OUT"
- "ABORTED"
In Azure, if I were to configure a function app to post to a storage account queue, I would used managed identity for the authentication. This would remove any need to store credentials for the queue.
Does AWS have something equivalent to managed identity that could be used for a Lambda posting to SQS?
With Serverless Framework
provider:
iamRoleStatements:
- Effect: Allow
Action:
- sqs:SendMessage
- sqs:DeleteMessage
Resource:
- !GetAtt MyQueue.Arn
resources:
Resources:
MyQueue:
Type: "AWS::SQS::Queue"
Properties:
QueueName: ${self:custom.sqs.name}
I am trying to create an AWS SQS queue using the serverless framwework,
But I am getting the following error on deploying the severless.yaml
The provided execution role does not have permissions to call SendMessage on SQS
The issue is the IAM role is created by serverless framework and I have no control of what permissions the framework adds to the role,
Ideally, if the function trigger is an SQS, or needs a DLQ configured,
I was hoping the framework would add Send and Receive message permissions to the role, but I guess it did not
Serveless.yaml -
service: dlq
provider:
name: aws
runtime: nodejs12.x
profile: csStage
region: ap-southeast-1
plugins:
- serverless-plugin-lambda-dead-letter
functions:
dlqFunction:
handler: handler.hello
deadLetter:
sqs: dl-queue
You have complete control over the permissions added to that role. You can add an iamRoleStatements section to your serverless.yml file under provider that describes the permissions you wish to apply to the role applied to functions. It would look something like:
provider:
iamRoleStatements:
- Effect: Allow
Action:
- "sqs:SendMessage"
Resource:
- arn:aws:sqs:region:accountid:queueid
You can find out more in the official documentation here: https://www.serverless.com/framework/docs/providers/aws/guide/iam/#iam/
You can use iamRoleStatements to give that permission to your Lambda function. The following template worked for me:
service: dlq
provider:
name: aws
runtime: nodejs12.x
profile: csStage
region: ap-southeast-1
iamRoleStatements:
- Effect: Allow
Action:
- sqs:SendMessage
Resource: !GetAtt DLQ.Arn
plugins:
- serverless-plugin-lambda-dead-letter
functions:
dlqFunction:
handler: handler.hello
deadLetter:
targetArn:
GetResourceArn: DLQ
resources:
Resources:
DLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: dl-queue
I have to make an HTTP request that takes a long time to receive a response. I don't want AWS Lambda to make this request as I will be charged for the time it is waiting for the response. Is there any way to use AWS Lambda to handle the response without being charged while waiting?
Based on my comment, I would recommend hitting the long poll endpoint often as opposed to staying connected for long periods of time. You can use a CloudWatch Rule to trigger the lambda function every 5 minutes (or whatever interval you choose). You can give the Lambda a short timeout, say 5-10 seconds, which should prevent it from running for too long. I'm assuming the long poll endpoint will guarantee delivery at least once.
Here is some CloudFormation YAML to get you started on the setup. Far from complete, but should get you on the right track.
Description: Automatically hit long poll endpoint
Resources:
#################################################
# IAM Role for Lambda
#################################################
ROLELAMBDADEFAULT:
Type: AWS::IAM::Role
Properties:
RoleName: your-lambda-default
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- edgelambda.amazonaws.com
- lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
- arn:aws:iam::aws:policy/service-role/AWSLambdaRole
Policies: []
#################################################
# Lambda function
#################################################
LFUNC:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: bucket-with-code
S3Key: code.zip
Description: Some function name
FunctionName: my-function-name
Handler: index.handler
MemorySize: 256
Role: !GetAtt ROLELAMBDADEFAULT.Arn
#choose your runtime here
Runtime: nodejs8.10
Timeout: 6
#################################################
# Rule to trigger the lambda
#################################################
RULE1:
Type: AWS::Events::Rule
Properties:
Name: custom-trigger
Description: Trigger my lambda
ScheduleExpression: rate(5 minutes)
State: ENABLED
Targets:
- Arn: !GetAtt LFUNC.Arn
Id: uniqueid1
After creating a Lambda function in Cloudformation, I would like to be able to setup the Cloudwatch Logs expiration in the same Cloudformation script.
eg:
MyLambdaRole:
Type: AWS::Iam::Role
...
Properties:
...
Policies:
-
PolicyName: "myPolicy"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- "logs:CreateLogGroup"
- "logs:CreateLogStream"
- "logs:PutLogEvents"
Resource: "arn:aws:logs:*:*:*"
MyLambda:
Type: AWS::Lambda::Function
Properties:
...
Role: !GetAtt [ MyLambdaRole, Arn ]
However, CloudFormation does not allow to modify/update Logs that are reserved for AWS: "Log groups starting with AWS/ are reserved for AWS."
Is there a workaround for this? Since there is no way to setup the log name in the Lambda resource creation, maybe there is some way to specify it in the Role definition I can't find.
Try this and use RetentionInDays attribute to change the logs expire after time
LogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Join ['/', ['/aws/lambda', !Ref MyLambda]]
RetentionInDays: 7 # days
Note: the issue of the LogGroup failing to create will appear if the log group name already exists( will exist if MyLambda already exists). The workaround would be to delete and create stack.
No, there is not. As you wrote, it's a log group owned by AWS and you can't give yourself more permissions in a role than AWS would allow. Therefore, you can't allow yourself to modify their log group.
Use the AWS Serverless application Model, takes care of the deployment, roles and logs outbox and you always can add your custom cloudformation code https://github.com/awslabs/serverless-application-model
they already have a lot of examples ready to go.