I followed this tutorial to setup an AWS Lambda function that is invoked upon an upload to S3 and populates DynamoDB.
I'm trying to achieve the same with AWS SAM for which I need to define a template.yaml file with the configuration information. I keep getting this error when deploying with Cloudformation -
Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Circular dependency between resources: [LambdaPerm]
I could not find a lot of information on this so I'm struggling to debug. What's causing this error and how can I resolve this? Here's my template configuration -
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
Gradebook:
Type: AWS::Serverless::Function
Properties:
FunctionName: LambdaGradebookServerless
Handler: serverless.LambdaGradebook
Runtime: java8
CodeUri: ./target/serverless-0.0.1-SNAPSHOT.jar
Role: arn:aws:iam::xxxxxxxxxxxx:role/lambda-s3-execution-role
LambdaPerm:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName:
Ref: Gradebook
Principal: s3.amazonaws.com
SourceAccount:
Ref: AWS::xxxxxxxxxxxx
SourceArn:
Fn::Join:
- ':'
- - arn
- aws
- s3
- ''
- ''
- Ref: gradebookBucket
gradebookBucket:
Type: AWS::S3::Bucket
Properties:
Bucket: gradebook-lambda
NotificationConfiguration:
LambdaConfigurations:
- Event: s3:ObjectCreated:*
Function:
Ref: Gradebook
To avoid this circular dependency, create the S3 bucket and the Lambda function independently, then update the stack with an S3 notification configuration.
If anyone comes searching for this, I wrote a blog to address this scenario.
https://aws.amazon.com/blogs/mt/resolving-circular-dependency-in-provisioning-of-amazon-s3-buckets-with-aws-lambda-event-notifications/
The implementation approach is similar to what jarmod describes in the accepted answer. The event notification is setup later using a CloudFormation custom resource.
I got the circular dependency error and it turns out it was a missing parameter that I was referencing from the resource on the CloudFormation template!.
Related
I have the following CloudFormation template:
AWSTemplateFormatVersion: '2010-09-09'
Resources:
HostingBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: "my-hosting-bucket"
LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
MyFunction:
Type: AWS::Lambda::Function
Properties:
Handler: index.handler
Runtime: nodejs12.x
Role: !GetAtt 'LambdaExecutionRole.Arn'
Timeout: 300
Code:
S3Bucket: !Ref 'HostingBucket'
S3Key: !Sub 'lambda/backend.zip'
But while deploying this I get the following error:
Resource handler returned message: "Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist. (Service: Lambda, Status Code: 400, Request ID: SOME-REQUEST-ID)"
Now I understand why the error occurs: my lambda is trying to get the deployed code, but it's not there yet because the S3 is created in the same stack. Probably I will need the following steps:
Create S3 bucket (via CloudFormation)
Deploy lambda code to created S3 bucket
Create lambda (via CloudFormation)
How can I deploy this CloudFormation template before deploying my Lambda handler to S3? I'd like to use the following flow:
Create S3 bucket and lambda (via CloudFormation)
Deploy lambda code to created S3 bucket
Have lambda pick up the newest code (which happens automatically iirc)
You would typically deploy your artifact into S3 during build time in a pipeline.
After deploying a Lambda through SAM, I was getting a 403 inside the Lambda when attempting to download from S3. I checked the Lambda's Role in IAM Management Console, and I saw that the Role only had AWSLambdaBasicExecutionRole. However, it should also have the policies from the SAM template for S3 read/write.
Here is a snippet from my SAM template (with some things renamed):
MyFunction:
Type: AWS::Serverless::Function
Properties:
PackageType: Image
Metadata:
DockerTag: my-tag
DockerContext: ./src/stuff
Dockerfile: Dockerfile
Policies:
- Version: '2012-10-17'
- S3ReadPolicy:
BucketName: !Ref MyBucket
- S3WritePolicy:
BucketName: !Ref MyBucket
Shouldn't the S3ReadPolicy and S3WritePolicy be a part of the Lambda's Role?
Is there something I'm missing?
I know I could manually add the policies needed, but obviously I want as much as possible to be happening automatically via SAM.
Other details: I'm not sure if it matters, but for the sake of additional context, the Lambda is part of a Step Function state machine. I'm using the boto3 library for making the request to download from S3. I get a {'Code': '403', 'Message': 'Forbidden'} error from boto3.
(answering my own question). The issue was the Policies section needed to be under Properties.
MyFunction:
Type: AWS::Serverless::Function
Properties:
PackageType: Image
Policies:
- Version: '2012-10-17'
- S3ReadPolicy:
BucketName: !Ref MyBucket
- S3WritePolicy:
BucketName: !Ref MyBucket
Metadata:
DockerTag: my-tag
DockerContext: ./src/stuff
Dockerfile: Dockerfile
I'm trying to use Cloudformation to package and deploy a simple "hello world" serverless app that uses a single Lambda Layer. The issue I'm having is that the LayerVersion section in my CF template file doesn't seem to like the fact that I'm using a !Ref to specify the S3Bucket and S3Key values. I don't want to hard-code these; nothing I've found in the documentation suggests that what I'm trying to do won't work, but it doesn't work :(
Here's the output of the deploy command that's failing:
aws cloudformation deploy --template-file out.yml --stack-name cftest-lambda --parameter-overrides S3BucketNameParameter=cftest-0eddf3f0b289f2c2 S3LambdaLayerNameParameter=cftest-lambda-layer-1602434332.zip --capabilities CAPABILITY_NAMED_IAM
Waiting for changeset to be created..
Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [libs] is invalid. property Content not defined for resource of type AWS::Serverless::LayerVersion
Here is the full CF template file:
cat template.yml
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: An AWS Lambda application
Parameters:
S3BucketNameParameter:
Type: String
Description: Bucket name for deployment artifacts
S3LambdaLayerNameParameter:
Type: String
Description: Object name for lambda layer deployment artifact
Resources:
helloworldfunction:
Type: AWS::Serverless::Function
Properties:
Handler: lambda_function.lambda_handler
Runtime: python3.8
CodeUri: hello-world-with-layer/.
Description: Hello world function to test cf using layers
Timeout: 10
# Function's execution role
Policies:
- AWSLambdaBasicExecutionRole
- AWSLambdaReadOnlyAccess
- AWSXrayWriteOnlyAccess
Tracing: Active
Layers:
- !Ref libs
libs:
Type: AWS::Serverless::LayerVersion
Properties:
Content:
S3Bucket: !Ref S3BucketNameParameter
S3Key: !Ref S3LambdaLayerNameParameter
CompatibleRuntimes:
- python3.8
LayerName: hello-world-lib
Description: Dependencies for the hello-world-with-layer app.
Any suggestions on how to approach this correctly?
The correct properties for LayerContent are:
Bucket: String
Key: String
Version: String
However, you are using (different names):
S3Bucket: String
S3Key: String
I've followed the AWS SAM documentation for hands-on learning experience and have reached this particular section: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-s3.html
Above page in the documentation explains how to map the S3 events to any resource and I've done something similar on my local machine. My local template below.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
CreateThumbnail:
Type: AWS::Serverless::Function
Properties:
CodeUri: hello_world/
Handler: app.lambda_handler
Runtime: python3.6
Timeout: 60
Policies: AWSLambdaExecute
Events:
CreateThumbnailEvent:
Type: S3
Properties:
Bucket: !Ref testBucket
Events: s3:ObjectCreated:*
testBucket:
Type: AWS::S3::Bucket
Now, when I build and deploy above SAM template, while a new S3 bucket is getting created, the 'object creation' event is not mapped to the Lambda function created. PSB screenshot.
I haven't found any articles or blogs on this error (probably, I'm doing a silly mistake here)
Please help.
Thanks.
I realized that for some reason, the AWS Lambda console is not showing the trigger event but the mapping is successful. I've also validated by uploading sample image to the bucket which triggered the lambda.
The event mapping can be seen in the properties section of the S3 bucket created.
S3 test bucket created -> Properties -> Events
You can add a AWS::Lambda::Permission to your Resources as described in this AWS SAM Github issue
LambdaInvokePermission:
Type: 'AWS::Lambda::Permission'
Properties:
FunctionName: !GetAtt MyFunction.Arn
Action: 'lambda:InvokeFunction'
Principal: 's3.amazonaws.com'
SourceAccount: !Sub ${AWS::AccountId}
SourceArn: !GetAtt MyBucket.Arn
First create bucket where you plan to save packaged code
aws s3 mb s3://youtbucketname
After sam build
go for
sam package --template-file template.yaml --s3-bucket <yourbucketname>
I wrote resources in my serverless.yml like below:
resources:
Resources:
RestApi :
Type : AWS::ApiGateway::RestApi
Properties :
Body : ${file(./swagger.yaml)}
LoginApiToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName: login
Action: lambda:InvokeFunction
Principal: apigateway.amazonaws.com
When I sls deploy, below error occured:
An error occurred: LoginApiToInvokeLambda - Function not found: arn:aws:lambda:ap-northeast-1:xxxxxxxxxxxx:function:api-dev-login (Service: AWSLambda; Status Code: 404; Error Code: ResourceNotFoundException
In the initial deployment, I thought that permissions were set before creating lambda functions. Therefore, I commented out LoginApiToInvokeLambda in my serverless.yml. I sls deploy again, it succeeded. But ApiGateway does not have permission to invoke lambda. After that I restored the commented out part, and sls deploy. Finally I was able to give ApiGateway the permission of Lambda invoke.
Is there a way to accomplish this at the same time?
You can use DependsOn functionality of CloudFormation in the resources section.
resources:
Resources:
# ...
LoginApiToInvokeLambda:
Type: AWS::Lambda::Permission
DependsOn: LoginLambdaFunction
Properties:
FunctionName: login
Action: lambda:InvokeFunction
Principal: apigateway.amazonaws.com
I've assumed your lambda function key is login which gets translated to LoginLambdaFunction. If not, check the serverless documentation on how the resources get named.
In short serverless translates your configuration to a CloudFormation template, and the resources section allows you to customise what gets generated, which is why you can use DependsOn to solve your issue.