I've followed the AWS SAM documentation for hands-on learning experience and have reached this particular section: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-s3.html
Above page in the documentation explains how to map the S3 events to any resource and I've done something similar on my local machine. My local template below.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
CreateThumbnail:
Type: AWS::Serverless::Function
Properties:
CodeUri: hello_world/
Handler: app.lambda_handler
Runtime: python3.6
Timeout: 60
Policies: AWSLambdaExecute
Events:
CreateThumbnailEvent:
Type: S3
Properties:
Bucket: !Ref testBucket
Events: s3:ObjectCreated:*
testBucket:
Type: AWS::S3::Bucket
Now, when I build and deploy above SAM template, while a new S3 bucket is getting created, the 'object creation' event is not mapped to the Lambda function created. PSB screenshot.
I haven't found any articles or blogs on this error (probably, I'm doing a silly mistake here)
Please help.
Thanks.
I realized that for some reason, the AWS Lambda console is not showing the trigger event but the mapping is successful. I've also validated by uploading sample image to the bucket which triggered the lambda.
The event mapping can be seen in the properties section of the S3 bucket created.
S3 test bucket created -> Properties -> Events
You can add a AWS::Lambda::Permission to your Resources as described in this AWS SAM Github issue
LambdaInvokePermission:
Type: 'AWS::Lambda::Permission'
Properties:
FunctionName: !GetAtt MyFunction.Arn
Action: 'lambda:InvokeFunction'
Principal: 's3.amazonaws.com'
SourceAccount: !Sub ${AWS::AccountId}
SourceArn: !GetAtt MyBucket.Arn
First create bucket where you plan to save packaged code
aws s3 mb s3://youtbucketname
After sam build
go for
sam package --template-file template.yaml --s3-bucket <yourbucketname>
Related
I'm trying to provide my Lambda function with the S3FullAccessPolicy policy. Note the target bucket is not configured within the template.yaml - it already exists. Considering the syntax examples from this documentation I have three options:
1.AWS managed policy named:
Policies:
- S3FullAccessPolicy
2.AWS SAM policy template (SQSPollerPolicy) defined:
Policies:
- S3FullAccessPolicy:
BucketName: abc-bucket-name
3.Or an inline policy document:
Policies:
- Statement:
...
In trying #1 I get an error that says it seems to suggest I need to provide an arn. If this is the case where would I provide it? The error:
1 validation error detected: Value 'S3FullAccessPolicy' at 'policyArn' failed to satisfy constraint:
Member must have length greater than or equal to 20
For #2 I provide the bucket name but it says that the policy is 'invalid'. I've tried adding quotes and replacing the name with an arn - but no luck.
And #3 - I can find the code for the policy here but that's in yaml so I wonder if that's even what I'm supposed to be using.
What am I missing here? I'm open to using any one of these options but right now I'm 0/3.
The full Lambda function:
testFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: lambda/testFunction/
Handler: app.lambda_handler
Runtime: python3.8
Timeout: 900
Policies:
- S3FullAccessPolicy
I used below template without any issues.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: ./hello_world/
Handler: app.lambda_handler
Runtime: python3.8
Tracing: Active
Policies:
- S3FullAccessPolicy:
BucketName: existingbucketname # bucket name without arn
Ran it using below command and it deployed successfully.
sam deploy --stack-name sample-stack --s3-bucket bucket-to-deploy --capabilities CAPABILITY_IAM
I'm new to SAM templates. I have the following snippet of my SAM Template where I used to pass the name of bucket name as a parameter from outside of this SAM YAML file :-
SAM Template:-
MyLambda:
Type: AWS::Serverless::Function
Properties:
CodeUri: ./functions/test/dist/
Handler: index.lambdaHandler
Runtime: nodejs12.x
Events:
S3PutObjectEvent:
Type: S3
Properties:
Bucket: !Ref S3BucketName
Events: s3:ObjectCreated:*
Parameter.YAML:-
DeploymentEnvironment:
default:
S3BucketName: my-awesome-s3-bucket
Now, I do not create any S3 Bucket using SAM Template or Infrastructure as a code (IaC). Bucket creation is done by Lambda code itself hence there is no S3 Object Type declaration in my SAM Template.
When I execute this command, sam validate to validate the SAM Template then I get this error:-
/template.yaml' was invalid SAM Template.
Error: [InvalidResourceException('MyLambda', 'Event with id [S3PutObjectEvent] is invalid. S3 events must reference an S3 bucket in the same template.')] ('MyLambda', 'Event with id [S3PutObjectEvent] is invalid. S3 events must reference an S3 bucket in the same template.')
I really need your help in achieving this as I tried hard in getting it solved. I read various forums, not sure if we can pass the bucket name from outside of the SAM template or not.
Is there any way workaround? This is really critical issue for me. Appreciate your help on this. thanks
Bucket creation is done by Lambda code itself
I'd recommend against this pattern, as your Lambda even source won't get created if the Bucket doesn't already exist.
Try creating the bucket in your SAM template, and pass the bucket name to your function as an environment variable.
Optionally you can set different environment names on your bucket name (addressing comment) using Parameters.
Parameters:
Env:
Type: String
AllowedValues:
- dev
- qa
- prod
Default: dev
Resources:
MyBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub 'My-unique-bucket-name-${Env}'
MyLambda:
Type: AWS::Serverless::Function
Properties:
CodeUri: ./functions/test/dist/
Handler: index.lambdaHandler
Runtime: nodejs12.x
Environment:
Variables:
BUCKET_NAME: !Ref MyBucket # passed to Lambda as environment variable
Events:
S3PutObjectEvent:
Type: S3
Properties:
Bucket: !Ref MyBucket
Events: s3:ObjectCreated:*
And get the bucket name in your function
const bucket = process.env.BUCKET_NAME
I want to create a deployment script for some lambda functions using AWS SAM. Two of those functions will be deployed into one account(account A) but will be triggered by an s3 bucket object creation event in a second account(account B). From what I know the only way to do this is by using adding a resource based policy to my lambda. But I don't know how to do that in AWS SAM. My current yaml file looks like this.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
deploy-test-s3-triggered-lambda
Parameters:
AppBucketName:
Type: String
Description: "REQUIRED: Unique S3 bucket name to use for the app."
Resources:
S3TriggeredLambda:
Type: AWS::Serverless::Function
Properties:
Role: arn:aws:iam::************:role/lambda-s3-role
Handler: src/handlers/s3-triggered-lambda.invokeAPI
CodeUri: src/handlers/s3-triggered-lambda.js.zip
Runtime: nodejs10.x
MemorySize: 128
Timeout: 60
Policies:
S3ReadPolicy:
BucketName: !Ref AppBucketName
Events:
S3NewObjectEvent:
Type: S3
Properties:
Bucket: !Ref AppBucket
Events: s3:ObjectCreated:*
AppBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref AppBucketName
What do I need to add to this yaml file in order to tie a resource based policy that allows for cross account access to my lambda function?
This can be done achieved with the help of AWS::Lambda::Permission using aws_cdk.aws_lambda.CfnPermission.
For example, to allow your lambda to be called from a role in another account, add the following to your CDK:
from aws_cdk import aws_lambda
aws_lambda.CfnPermission(
scope,
"CrossAccountInvocationPermission",
action="lambda:InvokeFunction",
function_name="FunctionName",
principal="arn:aws:iam::111111111111:role/rolename",
)
If your bucket and your Lambda function exist in separate accounts I don't know if it's possible to modify both of them from SAM / a single CloudFormation template.
Don't think cross account s3 event is possible with SAM, may need to go back to CFN.
Can a S3 bucket and triggered Lambda be created in separate CloudFormation templates. I want to keep long running resources stack separate from the likes of Lambda which get updated quite frequently
When tried to create Lambda separately it says that bucket defined in lambda event should be defined in same template and cannot be referenced.
S3 events must reference an S3 bucket in the same template.
GetFileMetadata:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub '${targetenv}-lambdaname'
CodeUri: target-file-0.0.1-SNAPSHOT.jar
Handler: LambdaFunctionHandler::handleRequest
Runtime: java8
Timeout: 30
MemorySize: 512
Environment:
Variables:
STAGE: !Sub '${targetenv}'
Events:
S3Event:
Type: S3
Properties:
Bucket:
Ref: MyS3Bucket
Events:
- 's3:ObjectCreated:*'
MyS3Bucket:
Type: 'AWS::S3::Bucket'
DependsOn: BucketPermission
Properties:
BucketName: !Sub 'bucketname-${targetenv}'
On November 21 2021, AWS announced S3 Event Notifications with Amazon EventBridge. Consequently, you can deploy one stack with an S3 bucket with EventBridge integration enabled and then a second stack with a Lambda function that is triggered by EventBridge events for the specific bucket.
Persistence Stack:
AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: 'Stack with S3 bucket with EventBridge event notification enabled'
Parameters:
BucketName:
Type: String
Description: 'Name of the bucket to be created'
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref BucketName
NotificationConfiguration:
EventBridgeConfiguration:
EventBridgeEnabled: true
# Alternatively shorthand config
# EventBridgeConfiguration: {}
Application Stack:
AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Stack with Lambda for procesing S3 events via EventBridge
Parameters:
BucketName:
Type: String
Description: Name of the bucket to listen events from
Resources:
S3EventProcessor:
Type: AWS::Serverless::Function
Properties:
FunctionName: S3EventListener
Architectures:
- arm64
Runtime: nodejs14.x
Handler: index.handler
InlineCode: |
exports.handler = (event, context) => {
console.log('event:', JSON.stringify(event));
}
Events:
S3EventBridgeRule:
Type: EventBridgeRule
Properties:
Pattern:
source:
- aws.s3
detail:
bucket:
name:
- !Ref BucketName
By configuring the Pattern, you can filter the events stream for more specific events such as Object Create or Object Deleted, file names, file extension, etc. Please find more info in the EventBridge userguide
This could not be done when this answer was originally written, but there has been progress in this area. Since then, S3 has added support for SNS and SQS Event as AWS::S3::Bucket NotificationConfiguration which could be declared in one stack and then imported to the other stack. More recently, AWS has also added EventBridge as yet another option, please see my other answer.
This is not possible in SAM version 2016-10-31. Copied from the S3 event source type in the SAM documentation:
NOTE: To specify an S3 bucket as an event source for a Lambda function, both resources have to be declared in the same template. AWS SAM does not support specifying an existing bucket as an event source.
The template is creating a bucket (MyS3Bucket).
Then, the serverless function is referencing it:
Bucket:
Ref: MyS3Bucket
If you want to refer to that bucket from another template, you can export the bucket name from the first stack:
Outputs:
S3Bucket:
Description: Bucket that was created
Value: !Ref MyS3Bucket
Export:
Name: Stack1-Bucket
Then, import it into the second stack:
Bucket:
Fn::ImportValue:
Stack1-Bucket
See: Exporting Stack Output Values - AWS CloudFormation
I followed this tutorial to setup an AWS Lambda function that is invoked upon an upload to S3 and populates DynamoDB.
I'm trying to achieve the same with AWS SAM for which I need to define a template.yaml file with the configuration information. I keep getting this error when deploying with Cloudformation -
Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Circular dependency between resources: [LambdaPerm]
I could not find a lot of information on this so I'm struggling to debug. What's causing this error and how can I resolve this? Here's my template configuration -
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
Gradebook:
Type: AWS::Serverless::Function
Properties:
FunctionName: LambdaGradebookServerless
Handler: serverless.LambdaGradebook
Runtime: java8
CodeUri: ./target/serverless-0.0.1-SNAPSHOT.jar
Role: arn:aws:iam::xxxxxxxxxxxx:role/lambda-s3-execution-role
LambdaPerm:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName:
Ref: Gradebook
Principal: s3.amazonaws.com
SourceAccount:
Ref: AWS::xxxxxxxxxxxx
SourceArn:
Fn::Join:
- ':'
- - arn
- aws
- s3
- ''
- ''
- Ref: gradebookBucket
gradebookBucket:
Type: AWS::S3::Bucket
Properties:
Bucket: gradebook-lambda
NotificationConfiguration:
LambdaConfigurations:
- Event: s3:ObjectCreated:*
Function:
Ref: Gradebook
To avoid this circular dependency, create the S3 bucket and the Lambda function independently, then update the stack with an S3 notification configuration.
If anyone comes searching for this, I wrote a blog to address this scenario.
https://aws.amazon.com/blogs/mt/resolving-circular-dependency-in-provisioning-of-amazon-s3-buckets-with-aws-lambda-event-notifications/
The implementation approach is similar to what jarmod describes in the accepted answer. The event notification is setup later using a CloudFormation custom resource.
I got the circular dependency error and it turns out it was a missing parameter that I was referencing from the resource on the CloudFormation template!.