I'm trying to implement this simple architecture with Serverless Framework :
File Upload to S3 Bucket --> SNS Topic --> 2 Lambda Functions
Actually i'm just testing with just 1 lambda function and here is my code:
service: MyImageLibrary
provider:
name: aws
runtime: python3.6
stage: dev
region: us-west-2
package:
individually: true
functions:
handler: index.lambda_handler
resources:
Resources:
ImageUploadedTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: SNSTopicImageUploadedTopic
ImageUploadedTopicSubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint:
Fn::GetAtt: [ TestLambdaFunction , "Arn" ]
Protocol: lambda
TopicArn:
arn:aws:sns:us-west-2:xxxxxxxxxxx:SNSTopicImageUploadedTopic
ImageUploadedTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: AllowBucketToPushNotificationEffect
Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sns:Publish
Resource: "*"
Topics:
- arn:aws:sns:us-west-2:xxxxxxxxxxx:SNSTopicImageUploadedTopic
MyImagesBucket:
Type: AWS::S3::Bucket
DependsOn: ImageUploadedTopicPolicy
Properties:
BucketName: ${self:custom.bucketName}
NotificationConfiguration:
TopicConfigurations:
- Event: s3:ObjectCreated:*
Topic:
arn:aws:sns:us-west-2:xxxxxxxxxxxx:SNSTopicImageUploadedTopic
CorsConfiguration:
CorsRules:
- AllowedMethods:
- GET
- PUT
- POST
- HEAD
AllowedOrigins:
- "*"
AllowedHeaders:
- "*"
In the console i can see that the Lambda function is subscribed to the the Topic :
But in the Lambda Function the SNS topic is not set in the Trigger section :
Am i missing something ?
Ok, i GOT IT !
I had to add another permission to allow the Lambda function to be triggered by the SNS Topic :
TestLambdaFunctionPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
Principal: sns.amazonaws.com
SourceArn:
Ref: ImageUploadedTopic
FunctionName:
Fn::GetAtt: [ TestLambdaFunction, "Arn" ]
And now in the console i can the SNS as trigger for my Lambda function :)
Related
I have event rule configured in multiple accounts in same region which will get the events from cloudtrail and trigger the events. Since there are so many accounts I cannot configure SNS topic in each and every account and confirm the subscription manually.
How do I have a centralized SNS topic in one account so that the event rule will be sent from all the accounts to that one topic which can trigger the alerts? I haven't worked on this topic earlier and hoping to get some light.
Note: I want to achieve this in CF template.
Here is the cf template that I use.
AWSTemplateFormatVersion: 2010-09-09
Description: ''
Parameters:
DestinationEmailAddress:
Description: ''
Type: String
AllowedPattern: ^[A-Za-z._%+-]+#[A-Za-z.-]+\.[A-za-z]{2,}$
SNSTopicName:
Description: The name of the SNS Topic.
Type: String
AllowedPattern: ^[\w+=,.#/-]+$
Resources:
SnsTopic:
Type: 'AWS::SNS::Topic'
Properties:
Subscription:
- Endpoint: !Ref DestinationEmailAddress
Protocol: email
TopicName: !Ref SNSTopicName
TopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: DefaultStatement
Effect: Allow
Principal:
Service:
- events.amazonaws.com
- cloudwatch.amazonaws.com
- config.amazonaws.com
- cloudformation.amazonaws.com
Action:
- "SNS:GetTopicAttributes"
- "SNS:SetTopicAttributes"
- "SNS:AddPermission"
- "SNS:RemovePermission"
- "SNS:DeleteTopic"
- "SNS:Subscribe"
- "SNS:ListSubscriptionsByTopic"
- "SNS:Publish"
- "SNS:Receive"
Resource: !Ref SnsTopic
Condition:
StringEquals:
"AWS:SourceOwner": 'AWS::'
- Sid: AllowEventRuletoPosttoSNSTopic
Effect: Allow
Principal:
Service: "events.amazonaws.com"
Action: "sns:Publish"
Resource: !Ref SnsTopic
Topics:
- !Ref SnsTopic
EventRule:
Type: 'AWS::Events::Rule'
Properties:
Name: ''
Description: ''
State: ENABLED
Targets:
- Arn: !Ref SnsTopic
Id: SNSTopic
EventPattern:
source:
- aws.kms
detail-type:
- AWS API Call via CloudTrail
detail:
eventSource:
- kms.amazonaws.com
eventName:
- DisableKey
I've recently started learning and implementing services using AWS services. So, I guess I'm missing some small steps which I can't figure it.
I'm trying to implement the following diagram using the Cloudformation template. Everything is working fine unless. The Lambda and SQS queue are subscribed to the SNS topic successfully. Whenever a file is stored at the bucket, or even when I publish a message to the SNS topic manually, the lambda function is triggered successfully, but the message is not published to the SQS queue. I've also added the AWS::SQS::QueuePolicy to allow SNS to send messages to SQS, but it still does not work.
template.yml:
...
Resources:
S3ObjectPutTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: !Sub ${AppName}-vrp-creation-${Environment}-topic
BucketToSNSPermission:
Type: AWS::SNS::TopicPolicy
...
Bucket:
Type: AWS::S3::Bucket
...
Lambda:
Type: AWS::Serverless::Function
...
Queue:
Type: AWS::SQS::Queue
Properties:
DelaySeconds: 0
MaximumMessageSize: 262144
MessageRetentionPeriod: 864000
QueueName: !Sub ${AppName}-${Environment}-queue
ReceiveMessageWaitTimeSeconds: 0
VisibilityTimeout: 90
TopicToQueuePermission:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref Queue
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sqs:SendMessage
Resource: !GetAtt Queue.Arn
Condition:
ArnEquals:
aws:SourceArn: !Ref S3ObjectPutTopic
TopicToQueueSubscription:
Type: AWS::SNS::Subscription
Properties:
Protocol: sqs
TopicArn: !Ref S3ObjectPutTopic
Endpoint: !GetAtt Queue.Arn
RawMessageDelivery: true
The full Cloudformation template.yaml file: template.yaml
You have mentioned Service: s3.amazonaws.com instead of Service: sns.amazonaws.com in your SQS policy. Update the template and try.
TopicToQueuePermission:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref Queue
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sqs:SendMessage
Resource: !GetAtt Queue.Arn
Condition:
ArnEquals:
aws:SourceArn: !Ref S3ObjectPutTopic
Currently my serverless.yml file looks like this:
service: bbb
provider:
name: aws
runtime: go1.x
stage: dev
package:
exclude:
- ./**
include:
- ./bin/**
functions:
ccc:
handler: bin/executable
name: my1minutelambda
role:
'Fn::GetAtt':
- mylambdaexecutionrole
- Arn
resources:
Resources:
mylambdaexecutionrole:
Type: AWS::IAM::Role
Properties:
RoleName: my-basiclambdaexec-role
Description: This is my basiclambdaexecution role
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: Allow
Principal:
Service:
- "lambda.amazonaws.com"
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
myminschedulerevent:
Type: AWS::Events::Rule
Properties:
Description: This is my 1 minute rate scheduler.
Name: my-1-min-trigger-event-scheduler
ScheduleExpression: rate(1 hour)
Targets:
-
Arn: "arn:aws:lambda:us-east-1:111111111111:function:my1minutelambda" #update your a/c Id
Id: "TargetFunctionV1"
command used to deploy: sls deploy
After deployment finished, I can see on aws management console that all my resources got created.
BUT I am not able to see cloudwatch trigger extablishment for my lambda function.
See below screenshot:
CloudWatch Event Rule created successfully. (Target section pointing to my lambda function)
Trigger link not established for my lambda:
Please let me know what i am missing here. Thank you.
Update#1:
After adding following lines (as suggested by Marcin), I am able to see "CloudWatch event".
EventsPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: my1minutelambda
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceAccount: !Ref 'AWS::AccountId'
SourceArn: !GetAtt myminschedulerevent.Arn
But, I can't see CloudWatch logs!! So, I can't findout if my lambda function is executing. Please see image below:
I tried to replicate the issue using serverless framework.
To do so I added the following AWS::Lambda::Permission to the end of your template:
EventsPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: dsfgsdfg # <-- REPLACE this with your function name my1minutelambda
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn: !GetAtt myminschedulrevent.Arn
After adding the permissions, the console showed the trigger as expected:
If all you are trying to do is get a Lambda function to execute on a schedule, the Serverless Framework already includes an event type expressly for that purpose:
functions:
crawl:
handler: crawl
events:
- schedule: rate(2 hours)
- schedule: cron(0 12 * * ? *)
It will set everything up for you with no need to add additional CloudFormation. You can find the documentation here: https://www.serverless.com/framework/docs/providers/aws/events/schedule/#schedule/
ScheduledRule:
Type: AWS::Events::Rule
Properties:
Name: "SphEvent"
Description: "ScheduledRule"
ScheduleExpression: "rate(1 hour)"
State: "ENABLED"
Targets:
- Arn: !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:lambda-name"
Id: "TargetFunctionV1"
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref "LambdaFunction"
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn:
Fn::GetAtt:
- "ScheduledRule"
- "Arn"
Considering this lambda function on a serverless.yml file:
functions:
s3toEc2Lambda:
handler: s3toec2lambda.S3toEc2Lambda
name: "${self:service}-s3toEc2Lambda"
role: S3toEc2LambdaRole
And considering this SNS created on resources section: Does someone knows how to inform the Sns ARN Endpoint from the lambda function s3toEc2Lambda ?
resources:
Resources:
WordpressFrontEndSnsS3toEc2:
Type: AWS::SNS::Topic
Properties:
TopicName: "wordpress-front-end-s3-ec2"
WordpressFrontEndSnsS3toEc2Lambda:
Type: AWS::SNS::Subscription
Properties:
Endpoint: { "Fn::GetAtt": ["s3toEc2Lambda", "Arn" ] } <------ HERE <------
#Endpoint: ${self:functions.s3toEc2Lambda} <------ OR HERE <------
#Endpoint: { "Fn::GetAtt": ["${self:functions.s3toEc2Lambda}", "Arn" ] } <------ OR HERE <------
Protocol: lambda
TopicArn: !Ref 'WordpressFrontEndSnsS3toEc2'
For me always appear a error message like this: "Template error: instance of Fn::GetAtt references undefined resource s3toEc2Lambda"
Thank You !
CloudFormation resources created by serverless have known format. For lambda function this is:
{normalizedFunctionName}LambdaFunction
Thus you should be able to reference your function using the following:
"Fn::GetAtt": [ S3toEc2LambdaLambdaFunction, Arn ]
More example about this are here
We can create Function Roles, Functions Policy and Lambda functions SAM template.yml file by this
Type: AWS::IAM::Role
Properties:
RoleName: UatAdminUserStatsLambda
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: 'Allow'
Principal:
Service:
- 'lambda.amazonaws.com'
Action:
- 'sts:AssumeRole'
FunctionPolicy:
Type: AWS::IAM::Policy
DependsOn: FunctionRole
Properties:
PolicyName: UserStatsPolicy
Roles:
- !Ref FunctionRole
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: 'Allow'
Action:
- 'iam:GetUser'
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:GetLogEvents'
- 'logs:PutLogEvents'
- 's3:GetObject'
- 's3:PutObject'
Resource: '*'
adminUsersList:
Type: AWS::Serverless::Function
Properties:
CodeUri: src/admin/
Role: !GetAtt FunctionRole.Arn
Handler: adminUsersList.adminUsersList
Layers:
- !Ref NodeDependenciesLayer
Events:
adminUsersListEvent:
Type: Api
Properties:
Path: /api/admins
Method: GET
Here is example for lambda invocation by event:
EventRule:
Type: AWS::Events::Rule
Properties:
Description: "EventRule"
EventPattern:
source:
- "aws.ec2"
detail-type:
- "EC2 Instance State-change Notification"
detail:
state:
- "stopping"
State: "ENABLED"
Targets:
-
Arn:
Fn::GetAtt:
- "LambdaFunction"
- "Arn"
Id: "TargetFunctionV1"
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Ref: "LambdaFunction"
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn:
Fn::GetAtt:
- "EventRule"
- "Arn"
How to modify this tp be able to invoke state machine not lambda? I have replaced "LambdaFunction with "MyStateMachine"
but how modify PermissionForEventsToInvokeLambda? There is no AWS::StepFunctions::Permission in cloudformation :-(
Here's an snippet from the template we used to have to trigger state machine processing from an S3 event:
InboundBucketPutObjectRule:
Type: 'AWS::Events::Rule'
Properties:
Name: !Sub 'inbound-bucket-put-object-rule'
EventPattern:
source:
- aws.s3
detail-type:
- AWS API Call via CloudTrail
detail:
eventSource:
- s3.amazonaws.com
eventName:
- PutObject
- CopyObject
- CompleteMultipartUpload
requestParameters:
bucketName:
- !Ref InboundBucket
Targets:
- Id: ProcessNewObject
Arn: !Ref StateMachine
RoleArn: !GetAtt
- StateMachineStartExecutionRole
- Arn
StateMachineStartExecutionPolicy:
Type: 'AWS::IAM::ManagedPolicy'
Properties:
ManagedPolicyName: !Sub 'state-machine-start-execution'
Roles:
- !Ref StateMachineStartExecutionRole
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'states:StartExecution'
Resource:
- !Ref StateMachine
StateMachineStartExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Sub 'state-machine-start-execution'
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: 'sts:AssumeRole'
StateMachine:
Type: 'AWS::StepFunctions::StateMachine'
Properties:
StateMachineName: !Sub 'state-machine'
Taken from real template so I had to anonymize this which might have introduced errors.
Template structure:
InboundBucketPutObjectRule is the CloudWatch event which gets triggered on file uploads.
StateMachineStartExecutionPolicy + StateMachineStartExecutionRole essentially allow CloudWatch event to start execution of a state machine.
StateMachine - the state machine which should be started (definition omitted).
I wrote above that we used to have this - but not anymore. Please see the following answer for our current solution:
https://stackoverflow.com/a/57563395/303810