I've recently started learning and implementing services using AWS services. So, I guess I'm missing some small steps which I can't figure it.
I'm trying to implement the following diagram using the Cloudformation template. Everything is working fine unless. The Lambda and SQS queue are subscribed to the SNS topic successfully. Whenever a file is stored at the bucket, or even when I publish a message to the SNS topic manually, the lambda function is triggered successfully, but the message is not published to the SQS queue. I've also added the AWS::SQS::QueuePolicy to allow SNS to send messages to SQS, but it still does not work.
template.yml:
...
Resources:
S3ObjectPutTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: !Sub ${AppName}-vrp-creation-${Environment}-topic
BucketToSNSPermission:
Type: AWS::SNS::TopicPolicy
...
Bucket:
Type: AWS::S3::Bucket
...
Lambda:
Type: AWS::Serverless::Function
...
Queue:
Type: AWS::SQS::Queue
Properties:
DelaySeconds: 0
MaximumMessageSize: 262144
MessageRetentionPeriod: 864000
QueueName: !Sub ${AppName}-${Environment}-queue
ReceiveMessageWaitTimeSeconds: 0
VisibilityTimeout: 90
TopicToQueuePermission:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref Queue
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sqs:SendMessage
Resource: !GetAtt Queue.Arn
Condition:
ArnEquals:
aws:SourceArn: !Ref S3ObjectPutTopic
TopicToQueueSubscription:
Type: AWS::SNS::Subscription
Properties:
Protocol: sqs
TopicArn: !Ref S3ObjectPutTopic
Endpoint: !GetAtt Queue.Arn
RawMessageDelivery: true
The full Cloudformation template.yaml file: template.yaml
You have mentioned Service: s3.amazonaws.com instead of Service: sns.amazonaws.com in your SQS policy. Update the template and try.
TopicToQueuePermission:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref Queue
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sqs:SendMessage
Resource: !GetAtt Queue.Arn
Condition:
ArnEquals:
aws:SourceArn: !Ref S3ObjectPutTopic
Related
I am trying to create 2 SQS using a single cloudformation template. I want to apply same Queue policy to both the queues. Currently I have the below template to create 1 queue and its corresponding dlq. Do I have to write all the resources twice or can I reuse the same queue policy for Queue2?
Resources:
Queue1:
Type: AWS::SQS::Queue
Properties:
RedrivePolicy:
deadLetterTargetArn: !GetAtt DeadLetterQueue.Arn
maxReceiveCount: 10
QueueName: queue1
MessageRetentionPeriod: 604800
VisibilityTimeout: 150
ReceiveMessageWaitTimeSeconds: 0
DeadLetterQueue1:
Type: AWS::SQS::Queue
Properties:
QueueName: queue1-dlq
MessageRetentionPeriod: 1209600
QueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: allow-service-to-read-and-write-messages
Effect: Allow
Principal:
AWS:
- !Sub arn:aws:iam::${AWS::AccountId}:role/User
Resource: !GetAtt Queue1.Arn
Action:
- SQS:ChangeMessageVisibility
- SQS:DeleteMessage
- SQS:GetQueueUrl
- SQS:ReceiveMessage
- SQS:SendMessage
Queues:
- !Ref Queue1
DeadLetterQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: allow-service-to-read-messages
Effect: Allow
Principal:
AWS:
- !Sub arn:aws:iam::${AWS::AccountId}:role/User
Resource: !GetAtt DeadLetterQueue1.Arn
Action:
- SQS:ChangeMessageVisibility
- SQS:DeleteMessage
- SQS:GetQueueUrl
- SQS:ReceiveMessage
- Sid: allow-queue-to-send-messages
Effect: Allow
Principal: '*'
Resource: !GetAtt DeadLetterQueue1.Arn
Action:
- SQS:SendMessage
Condition:
ArnEquals:
"aws:SourceArn": !Ref Queue1
Queues:
- !Ref DeadLetterQueue1
Outputs:
SourceQueueURL:
Description: URL of source queue
Value:
Ref: Queue1
SourceQueueARN:
Value: !GetAtt Queue1.Arn
Description: Arn of created SQS
DeadLetterQueueURL:
Description: URL of dead-letter queue
Value:
Ref: DeadLetterQueue1
DeadLetterQueueARN:
Value: !GetAtt DeadLetterQueue1.Arn
Description: Arn of created SQS
Tried with just 1 queue and its working fine
You have to create the policy separately for the second queue as it will have different !GetAtt DeadLetterQueue1.Arn. If you do not want to do this, you can extract the policy to a different template and use that template as a nested stack in your template with queue.
I have event rule configured in multiple accounts in same region which will get the events from cloudtrail and trigger the events. Since there are so many accounts I cannot configure SNS topic in each and every account and confirm the subscription manually.
How do I have a centralized SNS topic in one account so that the event rule will be sent from all the accounts to that one topic which can trigger the alerts? I haven't worked on this topic earlier and hoping to get some light.
Note: I want to achieve this in CF template.
Here is the cf template that I use.
AWSTemplateFormatVersion: 2010-09-09
Description: ''
Parameters:
DestinationEmailAddress:
Description: ''
Type: String
AllowedPattern: ^[A-Za-z._%+-]+#[A-Za-z.-]+\.[A-za-z]{2,}$
SNSTopicName:
Description: The name of the SNS Topic.
Type: String
AllowedPattern: ^[\w+=,.#/-]+$
Resources:
SnsTopic:
Type: 'AWS::SNS::Topic'
Properties:
Subscription:
- Endpoint: !Ref DestinationEmailAddress
Protocol: email
TopicName: !Ref SNSTopicName
TopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: DefaultStatement
Effect: Allow
Principal:
Service:
- events.amazonaws.com
- cloudwatch.amazonaws.com
- config.amazonaws.com
- cloudformation.amazonaws.com
Action:
- "SNS:GetTopicAttributes"
- "SNS:SetTopicAttributes"
- "SNS:AddPermission"
- "SNS:RemovePermission"
- "SNS:DeleteTopic"
- "SNS:Subscribe"
- "SNS:ListSubscriptionsByTopic"
- "SNS:Publish"
- "SNS:Receive"
Resource: !Ref SnsTopic
Condition:
StringEquals:
"AWS:SourceOwner": 'AWS::'
- Sid: AllowEventRuletoPosttoSNSTopic
Effect: Allow
Principal:
Service: "events.amazonaws.com"
Action: "sns:Publish"
Resource: !Ref SnsTopic
Topics:
- !Ref SnsTopic
EventRule:
Type: 'AWS::Events::Rule'
Properties:
Name: ''
Description: ''
State: ENABLED
Targets:
- Arn: !Ref SnsTopic
Id: SNSTopic
EventPattern:
source:
- aws.kms
detail-type:
- AWS API Call via CloudTrail
detail:
eventSource:
- kms.amazonaws.com
eventName:
- DisableKey
Currently my serverless.yml file looks like this:
service: bbb
provider:
name: aws
runtime: go1.x
stage: dev
package:
exclude:
- ./**
include:
- ./bin/**
functions:
ccc:
handler: bin/executable
name: my1minutelambda
role:
'Fn::GetAtt':
- mylambdaexecutionrole
- Arn
resources:
Resources:
mylambdaexecutionrole:
Type: AWS::IAM::Role
Properties:
RoleName: my-basiclambdaexec-role
Description: This is my basiclambdaexecution role
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: Allow
Principal:
Service:
- "lambda.amazonaws.com"
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
myminschedulerevent:
Type: AWS::Events::Rule
Properties:
Description: This is my 1 minute rate scheduler.
Name: my-1-min-trigger-event-scheduler
ScheduleExpression: rate(1 hour)
Targets:
-
Arn: "arn:aws:lambda:us-east-1:111111111111:function:my1minutelambda" #update your a/c Id
Id: "TargetFunctionV1"
command used to deploy: sls deploy
After deployment finished, I can see on aws management console that all my resources got created.
BUT I am not able to see cloudwatch trigger extablishment for my lambda function.
See below screenshot:
CloudWatch Event Rule created successfully. (Target section pointing to my lambda function)
Trigger link not established for my lambda:
Please let me know what i am missing here. Thank you.
Update#1:
After adding following lines (as suggested by Marcin), I am able to see "CloudWatch event".
EventsPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: my1minutelambda
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceAccount: !Ref 'AWS::AccountId'
SourceArn: !GetAtt myminschedulerevent.Arn
But, I can't see CloudWatch logs!! So, I can't findout if my lambda function is executing. Please see image below:
I tried to replicate the issue using serverless framework.
To do so I added the following AWS::Lambda::Permission to the end of your template:
EventsPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: dsfgsdfg # <-- REPLACE this with your function name my1minutelambda
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn: !GetAtt myminschedulrevent.Arn
After adding the permissions, the console showed the trigger as expected:
If all you are trying to do is get a Lambda function to execute on a schedule, the Serverless Framework already includes an event type expressly for that purpose:
functions:
crawl:
handler: crawl
events:
- schedule: rate(2 hours)
- schedule: cron(0 12 * * ? *)
It will set everything up for you with no need to add additional CloudFormation. You can find the documentation here: https://www.serverless.com/framework/docs/providers/aws/events/schedule/#schedule/
ScheduledRule:
Type: AWS::Events::Rule
Properties:
Name: "SphEvent"
Description: "ScheduledRule"
ScheduleExpression: "rate(1 hour)"
State: "ENABLED"
Targets:
- Arn: !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:lambda-name"
Id: "TargetFunctionV1"
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref "LambdaFunction"
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn:
Fn::GetAtt:
- "ScheduledRule"
- "Arn"
I'm writing AWS Lambda to read message from SQS and then save some messages to S3 after filtering. Since 80% messages will be filtered out, I decided write S3 once for 100000 SQS messages.
Is it possible to trigger the Lambda only when the messages in SQS reach 10000?
It's possible with help of AWS CloudWatch.
You could configure an AWS CloudWatch Alarm which triggers an "AlarmAction" as soon as your SQS queue got 100000 visible messages. In case of an "Alarm" you are notifying a SNS Topic which then triggers your AWS Lambda.
If you are using AWS CloudFormation it might look as the following:
AWSTemplateFormatVersion: 2010-09-09
Resources:
Queue:
Type: AWS::SQS::Queue
QueueVisibleMessagesTopic:
Type: AWS::SNS::Topic
LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Lambda:
Type: AWS::Lambda::Function
Properties:
Handler: handler.index
Role: !GetAtt LambdaExecutionRole.Arn
Runtime: nodejs8.10
MemorySize: 128
Timeout: 10
LambdaSubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint: !GetAtt Lambda.Arn
Protocol: lambda
TopicArn: !Ref QueueVisibleMessagesTopic
LambdaSubscriptionPermissions:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt Lambda.Arn
Action: lambda:InvokeFunction
Principal: sns.amazonaws.com
SourceArn: !Ref QueueVisibleMessagesTopic
QueueVisibleMessagesAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
Namespace: AWS/SQS
MetricName: ApproximateNumberOfMessagesVisible
Dimensions:
- Name: QueueName
Value: !GetAtt Queue.QueueName
Statistic: Sum
Period: 300
EvaluationPeriods: 1
ComparisonOperator: GreaterThanOrEqualToThreshold
Threshold: 100000
AlarmActions:
- !Ref QueueVisibleMessagesTopic
I'm trying to implement this simple architecture with Serverless Framework :
File Upload to S3 Bucket --> SNS Topic --> 2 Lambda Functions
Actually i'm just testing with just 1 lambda function and here is my code:
service: MyImageLibrary
provider:
name: aws
runtime: python3.6
stage: dev
region: us-west-2
package:
individually: true
functions:
handler: index.lambda_handler
resources:
Resources:
ImageUploadedTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: SNSTopicImageUploadedTopic
ImageUploadedTopicSubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint:
Fn::GetAtt: [ TestLambdaFunction , "Arn" ]
Protocol: lambda
TopicArn:
arn:aws:sns:us-west-2:xxxxxxxxxxx:SNSTopicImageUploadedTopic
ImageUploadedTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: AllowBucketToPushNotificationEffect
Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sns:Publish
Resource: "*"
Topics:
- arn:aws:sns:us-west-2:xxxxxxxxxxx:SNSTopicImageUploadedTopic
MyImagesBucket:
Type: AWS::S3::Bucket
DependsOn: ImageUploadedTopicPolicy
Properties:
BucketName: ${self:custom.bucketName}
NotificationConfiguration:
TopicConfigurations:
- Event: s3:ObjectCreated:*
Topic:
arn:aws:sns:us-west-2:xxxxxxxxxxxx:SNSTopicImageUploadedTopic
CorsConfiguration:
CorsRules:
- AllowedMethods:
- GET
- PUT
- POST
- HEAD
AllowedOrigins:
- "*"
AllowedHeaders:
- "*"
In the console i can see that the Lambda function is subscribed to the the Topic :
But in the Lambda Function the SNS topic is not set in the Trigger section :
Am i missing something ?
Ok, i GOT IT !
I had to add another permission to allow the Lambda function to be triggered by the SNS Topic :
TestLambdaFunctionPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
Principal: sns.amazonaws.com
SourceArn:
Ref: ImageUploadedTopic
FunctionName:
Fn::GetAtt: [ TestLambdaFunction, "Arn" ]
And now in the console i can the SNS as trigger for my Lambda function :)