I am trying to get my lambda to run when an image is added to a "folder" in an s3 bucket. Here is the template
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: 1. Creates the S# bucket that sotres the images from the camera.\n
2. Resizes the images when a new image shows up from a camera.\n
3. Adds a record of the image in the DB.
Globals:
Function:
Timeout: 10
Parameters:
DeploymentStage:
Type: String
Default: production
Resources:
CameraImagesBucket:
Type: 'AWS::S3::Bucket'
Properties:
BucketName: !Sub
- com.wastack.camera.images.${stage}
- { stage: !Ref DeploymentStage }
CreateThumbnailFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: image_resize/
Handler: app.lambda_handler
Runtime: python3.8
Description: Creates a thumbnail of images in the camare_images bucket
Policies:
- S3ReadPolicy:
BucketName: !Sub
- com.wastack.camera.images.${stage}
- { stage: !Ref DeploymentStage }
- S3WritePolicy:
BucketName: !Sub
- com.wastack.camera.images.${stage}
- { stage: !Ref DeploymentStage }
Events:
CameraImageEvent:
Type: S3
Properties:
Bucket:
Ref: CameraImagesBucket
Events:
- 's3:ObjectCreated:*'
Filter:
S3Key:
Rules:
- Name: prefix
Value: camera_images
When I look at the lambda created on the AWS console, I do not see the trigger even in the lambda visualiser. The lambda doesn't event have the s3 read and write policies attached to it.
The s3 bucket and the lambda are created, but the policies and triggers that are supposed to connect them are not created.
I did not get any error when I run sam deploy
Question: why did it not attach the s3 trigger event or the s3 access policies to the lambda function?
Policies for s3 So the template is straight forward. If you place the full template in does it work. If that is also failing, check the permissions on what you're running SAM as. Also there's an open ticket on github, This appears to be your issue. See comments.
I'm stuck in a weird issue. I have created an AWS S3 bucket using following cloud formation template:-
AWSTemplateFormatVersion: '2010-09-09'
Metadata:
License: Unlicensed
Description: >
This template creates a global unique S3 bucket in a specific region which is unique.
The bucket name is formed by the environment, account id and region
Parameters:
# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html
Environment:
Description: This paramenter will accept the environment details from the user
Type: String
Default: sbx
AllowedValues:
- sbx
- dev
- qa
- e2e
- prod
ConstraintDescription: Invalid environment. Please select one of the given environments only
Resources:
# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket.html
MyS3Bucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
Properties:
BucketName: !Sub 'global-bucket-${Environment}-${AWS::Region}-${AWS::AccountId}' # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html
AccessControl: Private
LoggingConfiguration:
DestinationBucketName: !Ref 'LoggingBucket'
LogFilePrefix: 'access-logs'
Tags:
- Key: name
Value: globalbucket
- Key: department
Value: engineering
LoggingBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
Properties:
BucketName: !Sub 'global-loggings-${Environment}-${AWS::Region}-${AWS::AccountId}'
AccessControl: LogDeliveryWrite
Outputs:
GlobalS3Bucket:
Description: A private S3 bucket with deletion policy as retain and logging configuration
Value: !Ref MyS3Bucket
Export:
Name: global-bucket
If you note in the template above then I'm exporting this S3 bucket in the Outputs section by the name called global-bucket.
Now, my intention is to refer to this existing bucket going forward in my AWS account whenever any new resource like Lambda, etc wants an S3 bucket. Here is an example using AWS SAM (Serverless Application Model), I'm trying to create an AWS Lambda and trying to refer to this existing S3 bucket using property !ImportValue and the export name as global-bucket as shown below:-
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
hellolambda
Sample SAM Template for hellolambda
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs12.x
Events:
HelloLambdaEvent:
Type: S3
Properties:
Bucket: !Ref SrcBucket
Events: s3:ObjectCreated:*
SrcBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !ImportValue global-bucket
Now, the problem is when I execute the command like sam build and then sam deploy --guided and select the same region (where my previous CloudFormation stack output is present) then I get the following error:-
global-bucket-sbx-ap-southeast-1-088853283839 already exists in stack arn:aws:cloudformation:ap-southeast-1:088853283839:stack/my-s3-global-bucket/aabd20e0-f57d-11ea-80bf-06f1487f6a64
The screenshot below:-
The problem is AWS CloudFormation is trying to create the S3 bucket rather than referring to the existing one.
But, if I try to update this SAM template like and then execute sam deploy, I get the following error:-
Waiting for changeset to be created..
Error: Failed to create changeset for the stack: my-lambda-stack, ex: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [HelloWorldFunction] is invalid. Event with id [HelloLambdaEvent] is invalid. S3 events must reference an S3 bucket in the same template.
I'm blocked by both ends. I would really appreciate it if someone can assist to guide me writing the SAM template correctly in my Lambda so that I can refer the existing bucket properly instead of creating the new one.
Thank you
Any items listed under the Resources section refer to the resources the stack is responsible for maintaining.
When you list SrcBucket you are asking for CloudFormation to create a new S3 bucket with the name being the value of !ImportValue global-bucket which is the name of an S3 bucket you have already created.
Assuming that this is the bucket name you can simply reference it in your template as shown below.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
hellolambda
Sample SAM Template for hellolambda
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs12.x
Events:
HelloLambdaEvent:
Type: S3
Properties:
Bucket: !ImportValue global-bucket
Events: s3:ObjectCreated:*
I want to set a stage name for the API Gateway in a SAM template.yaml.
But whatever I try I'm not succeeding. Without trying to name my stage, everything works as expected but with the default stage names Prod and Stage.
My sam-cli version is 0.47.0
I did find three comparable questions here on Stackoverflow but none of the answers work for me.
How can I change the name of the API stage in a SAM template?
How can I use api gateway stages via cloudformation or sam?
Using SAM file to remove default “Stages” in AWS ApiGateway?
I always get an error something like this:
Unresolved resource dependencies [ServerlessRestApi] in the Outputs block of the template
So how do I get a stage name I choose myself. I don't care much if Prod and Stage coexist with my chosen name.
Just to be complete, my template.yaml file is below:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sam-app
Sample SAM Template for sam-app
Globals:
Function:
Timeout: 3
Api:
Cors:
AllowMethods: "'OPTIONS,PUT'"
AllowHeaders: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
AllowOrigin: "'*'"
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs12.x
Events:
HelloWorld:
Type: Api
Properties:
Path: /hello-world
Method: put
Outputs:
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/dev/hello-world/"
HelloWorldFunction:
Description: "Hello World Lambda Function ARN"
Value: !GetAtt HelloWorldFunction.Arn
HelloWorldFunctionIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt HelloWorldFunctionRole.Arn
I probably don't understand the intended workflow behind this. Why have 2 stage names when the lambda function the API Gateway is pointing to, is the same?
I will have 'dev' and 'prod' environments but they will use different stack names so I can never mix up the different environments.
I always use deploy-dev.sh and deploy-pod.sh scripts that check if I'm on the development or master (production) branch before actually deploying something. So those scripts would point to a different template.yaml files because they are called from different git branches. I'm using this way for deployment already for a long time and it works well for me.
On a side note: Why the existing stage names start with a capital? It looks so ugly and unusual.
So I found my own answer which is sort of a combination of two answers to the questions I found on StackOverflow that are mentioned in my question.
I still don't understand why this is so needlessly complicated.
I added a parameter to the top level of the template.yaml file. The use of a parameter is not strictly needed. I added this so I can have a single template file that is called from both my deploy-dev.sh and deploy-prod.sh scripts.
Below is the parameter declaration:
Parameters:
Stage:
Type: String
Default: dev
Then, under the Resources group, I added a new ApiDeployment resource. The name you use is totally up to you as long as you use the exact same name elsewhere as a !Ref. The only reason to add this resource is that you are not allowed to simply use the StageName in the properties of the Api section of the function event. You are also not allowed to put StageName in the Globals Api section.
ApiDeployment:
Type: AWS::Serverless::Api
Properties:
StageName: !Ref Stage <- this is referencing the parameter but it could be a fixed value
Then, under the Events section of the Lambda function I added a property RestApiId that is referencing the ApiDeployment resource. The last line in the block below.
HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs12.x
Events:
HelloWorld:
Type: Api
Properties:
Path: /hello-world
Method: put
RestApiId: !Ref ApiDeployment
As I mentioned in my question, I got errors complaining about the output section of the yaml file. It turns out that the output section is optional anyway. So when I commented it out, everything worked.
But I used the output section in my deploy script to show me the URL of the API Gateway so with some trying I got that working too.
The error was caused in the 4th line. It originally had ${ServerlessRestApi}. Just replace it with the new resource name I added to the yaml file: ${ApiDeployment} and everything is fine.
Outputs:
ApiDeployment:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ApiDeployment}.execute-api.${AWS::Region}.amazonaws.com/${Stage}/hello-world/"
HelloWorldFunction:
Description: "Hello World Lambda Function ARN"
Value: !GetAtt HelloWorldFunction.Arn
HelloWorldFunctionIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt HelloWorldFunctionRole.Arn
Because I use a parameter in the yaml file, you need to call sam deploy with the parameter name and value. The exact syntax for this is, like a lot of AWS's documentation, very well hidden. So below is the way you start your deployment:
sam deploy --parameter-overrides "ParameterKey=Stage,ParameterValue=dev"
You probably still have the Stage stage in the API Gateway console under Stages but you can delete that without any repercussions.
For completeness here is my full template.yaml file which is, by the way, the file you get when you do sam init
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sam-app: Sample SAM Template for sam-app
Parameters:
Stage:
Type: String
Default: dev
Globals:
Function:
Timeout: 3
Api:
Cors:
AllowMethods: "'OPTIONS,PUT'"
AllowHeaders: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
AllowOrigin: "'*'"
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs12.x
Events:
HelloWorld:
Type: Api
Properties:
Path: /hello-world
Method: put
RestApiId: !Ref ApiDeployment
ApiDeployment:
Type: AWS::Serverless::Api
Properties:
StageName: !Ref Stage
Outputs:
ApiDeployment:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ApiDeployment}.execute-api.${AWS::Region}.amazonaws.com/${Stage}/hello-world/"
HelloWorldFunction:
Description: "Hello World Lambda Function ARN"
Value: !GetAtt HelloWorldFunction.Arn
HelloWorldFunctionIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt HelloWorldFunctionRole.Arn
Here is an update for anyone landing on this question.
I'm not using SAM anymore. I switched to AWS CDK.
The CDK lets you define your entire AWS infrastructure in Code!
You can use Javascript, Typescript, Python, C# and Java although most examples seem to be in Typescript.
It's not a very easy switch but it is worthwhile. SAM seems to be a dead-end street.
SAM only covers a small subset of all AWS has to offer but the CDK covers everything.
It's quite new and it's a moving target, plus the devs don't give a rat's ass about breaking updates as they are still moving things around between modules.
But after a few days, you'll start to get a grasp and it has infinite flexibility because you are using a normal programming language to set up things like API Gateways, Lambdas, Custom Domains, IAM rules, etc.
It's also very compact (compared to SAM templates).
I used it to have different stacks for staging and production based upon the got branch I'm in. So when I deploy while my repo is on the dev branch I'll have a different environment (including different domain names and so) then when I would deploy from the master or prod branch. The names of the different services also differ, depending on the git branch.
To deploy, you just run "cdk deploy"
To get started look at this excellent workshop: https://cdkworkshop.com/
Below is an example of this branch switching. I only show parts of the method for the dev branch. For the prod branch, I just have a copy of the same method in the same file but the method has a different name and the variables for the service names also differ. Just look at the piece of (incomplete) code and you should get an idea of how it works.
import * as branchName from 'current-git-branch'
const branch = branchName()
/*-------- This is the development stack --------*/
export class StripePaymentsDev extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props)
if (branch !== 'dev') {
throw new Error('Not on dev branch')
}
// Env vars from Parameter Store
const STRIPE_PUBLIC = StringParameter.valueForStringParameter(this, `/${branch}/STRIPE_PUBLIC`)
const STRIPE_SECRET = StringParameter.valueForStringParameter(this, `/${branch}/STRIPE_SECRET`)
const STRIPE_API_VERSION = StringParameter.valueForStringParameter(this, `/${branch}/STRIPE_API_VERSION_PAYMENTS`)
// Names for the dev environment
const domainMapName = 'PaymentsDev'
const eventBusName = 'WebhooksBusDev'
const ruleName = 'WebhooksRuleDev'
const eventBus = new EventBus(stackScope, eventBusName, { eventBusName })
const cert = Certificate.fromCertificateArn(stackScope, certName, certArn)
const stackScope = this
// IAM rules
const lambdaPolicy = new iam.PolicyStatement({
actions: ['events:*'],
resources: ['*']
})
const sqsPolicy = new iam.PolicyStatement({
actions: ['sqs:*'],
resources: ['*']
})
const webhooks = new lambda.Function(stackScope, lambdaWebhooksName, {
runtime: lambda.Runtime.NODEJS_12_X,
code: lambda.Code.fromAsset('webhook-handler'),
handler: 'webhooks.handler',
timeout: Duration.seconds(600),
description: 'Processes Stripe Webhooks',
retryAttempts: 0,
environment: {
STRIPE_PUBLIC,
STRIPE_SECRET,
STRIPE_API_VERSION,
MONGO_URL,
MONGO_DB,
MONGO_PORT,
DEBUG
}
})
webhooks.addToRolePolicy(sqsPolicy)
const rule = new Rule(stackScope, ruleName, {
description: 'Triggers lambda to process stipe webhooks',
enabled: true,
eventBus: eventBus,
eventPattern: {
detailType: ['transaction'],
source: ['custom.payments']
},
ruleName: ruleName
})
rule.addTarget(new eventTargets.LambdaFunction(webhooks))
new HttpApi(stackScope, apiName, {
defaultIntegration: new LambdaProxyIntegration({ handler: payments }),
defaultDomainMapping: {
domainName: new DomainName(stackScope, domainMapName, {
domainName: PAYMENT_DOMAIN,
certificate: cert
})
}
})
}
}
This way creates only your specified stage instead of creating one more stage named Stage.
This setup did trick.
Globals:
Api:
OpenApiVersion: 3.0.1
I've also created the new AWS::Serverless::Api named RestApi to overwrite implicit ServerlessRestApi. Remember to set RestApi into RestApiId of every API event.
template.yaml
AWSTemplateFormatVersion: 2010-09-09
Description: >-
app-sam
Transform:
- AWS::Serverless-2016-10-31
# ====================================
# PARAMETERS SETUP
# ====================================
Parameters:
StageName:
Type: String
Default: dev
Description: (Required) Enter dev, prod. Default is dev.
AllowedValues:
- dev
- prod
ProjectName:
Type: String
Default: sam-api
Description: (Required) The name of the project
MinLength: 3
MaxLength: 50
AllowedPattern: ^[A-Za-z_-]+$
ConstraintDescription: "Required. Can be characters, hyphen, and underscore only. No numbers or special characters allowed."
ExistingTable:
Type: String
Default: example-table
Description: (Required) The name of existing DynamoDB
MinLength: 3
MaxLength: 50
AllowedPattern: ^[A-Za-z_-]+$
ConstraintDescription: "Required. Can be characters, hyphen, and underscore only. No numbers or special characters allowed."
# ====================================
# GLOBAL SETUP
# ====================================
Globals:
Api:
OpenApiVersion: 3.0.1
Function:
Runtime: nodejs14.x
Timeout: 180
MemorySize: 256
Environment:
Variables:
TABLE_NAME: !Ref ExistingTable
Resources:
# Reference this one to overwrite implicit stage
# https://github.com/aws/serverless-application-model/issues/191#issuecomment-580412747
RestApi:
Type: AWS::Serverless::Api
Properties:
Name: !Ref ProjectName
StageName: !Ref StageName
# This is a Lambda function config associated with the source code: get-all-items.js
getAllItemsFunction:
Type: AWS::Serverless::Function
Properties:
Handler: src/handlers/get-all-items.getAllItemsHandler
Description: A simple example includes a HTTP get method to get all items from a DynamoDB table.
Policies:
# Give Create/Read/Update/Delete Permissions to the ExistingTable
- DynamoDBCrudPolicy:
TableName: !Ref ExistingTable
Events:
Api:
Type: Api
Properties:
Path: /
Method: GET
RestApiId: !Ref RestApi
Outputs:
WebEndpoint:
Description: "API Gateway endpoint URL for Prod stage"
Value: !Sub "https://${RestApi}.execute-api.${AWS::Region}.amazonaws.com/${StageName}/"
Is there a way to do conditional template urls in cloudformation?
This fails because it is not evaluated until the aws cloudformation deploy step and it errors out saying that the templateURL must be an s3 link. When I hard code in one of the urls it will upload that relative file to s3 and in the packaged final template it will just have the s3 url in place.
This doesn't work (pre aws cloudformation package step)
Vpc:
Type: 'AWS::CloudFormation::Stack'
Properties:
Parameters:
AlertingModule: !GetAtt 'Alerting.Outputs.StackName'
NatGateways: 'true'
TemplateURL: !If [IsProduction, './default-vpc-module.yml', './production-vpc-module.yml']
When a url is hardcoded in it will package into this (post aws cloudformation pacakge step)
VpcModule:
Fn::GetAtt:
- Vpc
- Outputs.StackName
AlertingModule:
Fn::GetAtt:
- Alerting
- Outputs.StackName
Priority: '2'
HealthCheckPath: /health-check.php
TemplateURL: https://s3.amazonaws.com/my-cicd-bucket-/fe556ff9386a28c063c4a110b31b.template
Yes you absolutely can achieve what your after, the way i did it was reference the s3 url, by using an aws cloudformaiton package but also a s3 cp in the codebuild stage to enforce naming conventions on the url. As long as your template url paths are distinguished differently.
To provide a very flexible example, you could use !Sub with an !If replacement statement on the dynamic name component, which will also allow your to use !Ref inside the !If statement:
Parameters:
StageProd:
Description: Environment
Default: "production"
Type: String
.......
......
TemplateURL: !Sub
- https://${CfnBucketName}.s3-ap-southeast-2.amazonaws.com/${CfnKeyPrefix}/SomeFileName-${Stage}.yaml
- { Stage: !If [ IsProduction, !Ref StageProd, "default"]}
The above should meet almost any sort of combination of dynamic naming you want to achieve; however, you could also very much simplify the above with a simple !Sub replacing the stage name.
Here is what I ended up doing, but I don't like it.
Vpc:
Condition: IsProduction
Type: 'AWS::CloudFormation::Stack'
Properties:
Parameters:
AlertingModule: !GetAtt 'Alerting.Outputs.StackName'
NatGateways: 'true' # reduce costs
TemplateURL: ./production-vpc-module.yml
Vpc:
Condition: IsNotProduction
Type: 'AWS::CloudFormation::Stack'
Properties:
Parameters:
AlertingModule: !GetAtt 'Alerting.Outputs.StackName'
NatGateways: 'true' # reduce costs
TemplateURL: ./default-vpc-module.yml
I'm trying to write a transform macro within cloudformation which will format a given string into an S3 bucket name prefix. I wrote out the script as a lambda and created the macro, but I get an error saying that my transform macro name does not exist.
BucketNameFormattingMacro:
Type: AWS::CloudFormation::Macro
Properties:
Description: Changes strings to be formatted properly to be added to S3 bucket names.
FunctionName: !GetAtt BucketNameFormattingScript.Arn
TransformFunctionPermissions:
Type: AWS::Lambda::Permission
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !GetAtt BucketNameFormattingScript.Arn
Principal: 'cloudformation.amazonaws.com'
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName:
Fn::Transform:
- Name: BucketNameFormattingMacro
Parameters:
InputString: !Sub '{ENV}-bucket'
Running the script throws this:
"No transform named 699790013825::BucketNameFormattingMacro found.. Rollback requested by user."
There is a string of numbers before the name of the macro that I didn't put there that I suspect is part of the issue. Why are those numbers there and how do I properly reference the transform to use it within my bucket name?
Edit: Unfortunately, you can't access a macro within a cloudformation script if it's being created within the same script.