API Gateway with SAM isn't updated correctly - amazon-web-services

We use Cloud Formation for define a bunch of Lambda functions:
AWSTemplateFormatVersion: '2010-09-09'
Transform:
- 'AWS::Serverless-2016-10-31'
Resources:
MyLambda:
Type: 'AWS::Serverless::Function'
Properties:
Handler: com.handler::MyLambda
Runtime: java8
CodeUri: .
Description: Some desc
MemorySize: 512
Timeout: 15
Role: !Ref LambdaRole
FunctionName: MyLambda
Events:
MyLambdaEvt:
Type: Api
Properties:
RestApiId: !Ref MyApiDef
Path: /lambda/my
Method: get
MyApiDef:
Type: AWS::Serverless::Api
Properties:
DefinitionUri: s3://a-bucket/api-gateway.yml
StageName: prod
Outputs:
ApiUrl:
Description: URL of your API endpoint
Value: !Join
- ''
- - https://
- !Ref MyApiDef
- '.execute-api.'
- !Ref 'AWS::Region'
- '.amazonaws.com/prod'
A CodePipeline generate a changeset and execute it.
In this way all the Lambda function are correctly updated but the API Gateway endpoint are not update correctly and we need to import and deploy the YML in s3://a-bucket/api-gateway.yml manually.

Why the API doesn't update (an educated guess)
In order for a change to be added to a a change set, CloudFormation has to detect a change. If the only thing that changes (for MyApiDef) between deployments is the contents of the .yaml file out on S3, CloudFormation isn't going to detect a change that it needs to add to the change set.
If this API definition lived in the CF template, rather than a file on S3, CF would (obviously) detect every change and update the API for you.
Since the definition lives in S3, and the file name hasn't changed, no change is detected, so nothing gets updated.
Possible work arounds
You have to convince CloudFormation that something has changed with your API definition. These two things worked for me:
Updating the MyApiDef key itself each run works. (MyApiDefv2,
MyApiDefv3, etc)
Updating the DefinitionUri works. (i.e. version the
filename in S3).
Neither of these is great, but appending a version to the filename in S3 seems more reasonable than the other option.
There are probably other ways to convince CloudFormation a change has taken place. Notably, I could not get Variables to work for this purpose.

Related

Getting unexpected results in API Gateway from cloudformation

I am somewhat new to writing yaml files and deploying them. I have one built from things that I have worked on in the past, but it is producing results that I don't understand and I believe that this is causing a CORS error when I used the API Gateway (It works using insomnia/postman).
Here is the YAML file.
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Storygraf backend API
Globals:
Function:
Timeout: 3
Resources:
ExpressApi:
Type: AWS::Serverless::Api
Properties:
StageName: prod
StageName: dev
ExpressLambdaFunction:
Type: AWS::Serverless::Function
Properties:
Environment:
Variables:
CodeUri: ./
Handler: lambda.handler
MemorySize: 512
Runtime: nodejs14.x
Timeout: 30
Events:
ProxyApiRoot:
Type: Api
Properties:
RestApiId: !Ref ExpressApi
Path: /
Method: ANY
cors: true
ProxyApiGreedy:
Type: Api
Properties:
RestApiId: !Ref ExpressApi
Path: /{proxy+}
Method: ANY
cors: true
I don't really understand the ProxyAPIRoot and ProxyAPIGreedy settings, but I have run the deployment with and without that section and gotten the same results.
I am also attempting to create two stages on the API gateway - one for production and one for development but it occurs to me that I don't really understand the stages very well and the result that I am getting has a "dev" stage (which is correct) and a "Stage" stage, which is not correct.
Here are the resources and stages for the resulting gateway.
Why is there are "Stage" stage and not a "prod" stage?
What is the proxy? Is it causing my CORS problem?
You have to create separate SAM YAML files for different environments. You cannot specify two stages in the same YAML file. If you want to deploy the complete stack in one YAML file, I suggest you switch to pure Cloudformation template, then you would have a lot more flexibility on defining resources.
You can checkout this link and create multiple stages.
Regarding the /{proxy+} resource, it will capture all the requests, like /products, /users/1. Whereas, the / resource will only respond to requests to the endpoint root url, i.e / only.

how can I use lambda versions and aliases to version my API with CloudFormation?

We have an API which needs to be versioned. We are using API Gateway and Lambda on AWS. All code is deployed using CICD, and we use SAM templates on top of CloudFormation.
NB we are using API Gateway stages to handle different environments (e.g. stage "dev1" points to lambda "dev1-MyService".
We have a resource 'v1' defined as follows:
V1Resource:
Type: AWS::ApiGateway::Resource
Properties:
RestApiId: !Ref Api
ParentId: !GetAtt Api.RootResourceId
PathPart: 'v1'
# proxy traffic to a lambda:
ProxyMethod:
Type: AWS::ApiGateway::Method
Properties:
HttpMethod: ANY
...
Integration:
Type: AWS_PROXY
Uri: arn:aws:apigateway:...:function:MyFunction:v1/invocations'
And a lambda defined like this:
ApiFunction:
Type: AWS::Serverless::Function
Properties:
AutoPublishAlias: "v1"
FunctionName: MyFunction
...
What we want to do is keep the v1 lambda in the system, but when we have breaking changes to the API, we can define a /v2/ resource, pointing to the v2 alias of the lambda
My plan was to define a second resource (V2Resource) and second proxy method, referencing the v2 alias of the lambda, which I could create by bumping AutoPublishAlias to v2 in the same change.
Then the v2 alias remains pointing to the old version (forever in theory) and each time we deploy from then on, the latest version of the API is aliased to v2 (until we need to do another breaking change and introduce v3).
Unfortunately, when I do that, it deletes the old alias v1.
So how am I supposed to use Lambda aliases to do API versioning with API Gateway? With CloudFormation and SAM templates?
You could use AWS::Lambda::Version resource, from the documentation:
creates a version from the current code and configuration of a function. Use versions to create a snapshot of your function code and configuration that doesn't change.
LambdaVersion:
Type: AWS::Lambda::Version
Properties:
FunctionName: !Ref ApiFunction
Description: v1
Where the Description property gets fixed because it does not accept updates. Additionally, you can create some aliases for these version like dev, stage, prod. For example, for dev version this would be
DevAlias:
Type: AWS::Lambda::Alias
Properties:
Description: Dev Alias for Lambda function
FunctionName: !Ref ApiFunction
FunctionVersion: !GetAtt LambdaVersion.Version
Name: DEV

Dynamically change event properties on aws cloudformation templates

We are building a serverless app using aws and we want to enable lambda warmers only on the production environment.
Our cloudformation parameters:
Parameters:
Environment:
Description: Environment name
Type: String
EnableWarmer:
Description: Flag to enable/disable warmup Events
Default: DISABLED
Type: String
Our lambdas yaml file looks like this:
MyLambda:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub '${Environment}-my-lambda'
CodeUri: src
Handler: lambda.handler
Runtime: nodejs12.x
MemorySize: 128
Timeout: 100
Description: Creates a new something
Layers:
- !Ref InternalDependencyLayer
- !Ref ExternalDependencyLayer
Role: !Ref LambdaRoleArn
Events:
Api:
Type: Api
Properties:
Path: /url
Method: POST
RestApiId: !Ref ApiGateway
WarmingSchedule:
Type: Schedule
Properties:
Enabled: !Ref EnableWarmer
Schedule: rate(5 minutes)
Input: '{ "warmer":true, "concurrency": 2 }'
And then we deploy the dev environment with these params:
- Key: Environment
Value: dev
- Key: EnableWarmer
Value: DISABLED
Similarly for the production environment we deploy with these params:
- Key: Environment
Value: production
- Key: EnableWarmer
Value: ENABLED
According to aws documentation parameters can't be of type boolean which is the required type of the enabled attribute of the schedule event.
Fortunately amazon states:
Enabled Indicates whether the rule is enabled.
To disable the rule, set this property to False.
Type: Boolean
Required: No
AWS CloudFormation compatibility: This property is similar to the
State property of an AWS::Events::Rule resource. If this property is
set to True then AWS SAM passes ENABLED, otherwise it passes DISABLED
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-schedule.html
However when we deploy to the dev environment the warmers are enabled.
The problem with the Enabled property here seems to be that it does not support using CloudFormation's If function and a Condition to do something like that:
WarmingSchedule:
Enabled: !If [WarmerCondition, true, false]
(At least I couldn't figure it out, but maybe I just did something wrong) This is similar to what #raul-barreto suggests with !Equals. However, in my tests it always sets the rule's status to ENABLED.
It seems like there are two options left what you can do:
a) (Not recommended) Copy the CloudFormation code of your Lambda function and have a second one available in your CloudFormation resources. Now set a condition to both of them so you only create one of them, depending on your environment. Note: I don't say this is a good approach but it'd be possible. Problems are you have to maintain the definition of both Lambda functions and if you repeat this approach, you'll faster reach CloudFormations template size limits.
b) If you want to make sure you have a minimum number of instances available to your function (at least that's what I interpret from your question), consider using AWS Lambda's Provisioned Concurrency feature instead. You can define it like this in your AWS Lambda function using SAM:
MyLambda:
Type: AWS::Serverless::Function
Properties:
AutoPublishAlias: 'LATEST' #Setting AutoPublishAlias is important, otherwise you can not use provisioned concurrency
ProvisionedConcurrencyConfig:
ProvisionedConcurrentExecutions: 2
This snippet will provision concurrency for the function's alias 'LATEST' which points to the latest version of your function. Read more about AutoPublishAlias in SAM (Note: the link's content is written in the context of traffic shifting but it explains what the property does). It makes sure that you always have 2 instances of your Lambda functions available.
The accepted solution works, but as stated in comments, it is a bit difficult to control the enabling and disabling of the event. The following seems to work in a more robust way.
Events:
Scheduler:
Type: Schedule
Properties:
Schedule: !If [EnableWarmer, "rate(5 minutes)",
"cron(00 00 01 01 ? 1970)"]
According to the documentation, Enabled must a Boolean variable.
You can still have a String parameter and convert it to Boolean inside the CloudFormation.
WarmingSchedule:
Type: Schedule
Properties:
Enabled: !Equals [!Ref EnableWarmer, ENABLED]
Schedule: rate(5 minutes)
Input: '{ "warmer":true, "concurrency": 2 }'
This way you still can send ENABLED or DISABLED as a parameter but the input of WarmingSchedule.Enabled will be a Boolean.

How do you get the API endpoint's URL when it's created within the Events property of Cloudformation's Lambda definition

I'm creating a Lambda function via CloudFormation (AWS' SAM: Serverless Application Model) and have defined an API endpoint via the Lambda-function's Events property.
...
MyFunction:
Type: AWS::Serverless::Function
Properties:
Description: Do amazing things
CodeUri: my_function/
Events:
Trigger:
Type: Api
Properties:
Path: /p1/{v1}
Method: post
Handler: app.run
...
I'd now like to use the URL of the endpoint that's been created in another part of the CloudFormation YAML file. I've tried to use the SAM documentation for Lambda but the only return values have to do with the Function's ARN and resource name.
Specifically, thought I think unrelated to the exact question, I want to use the API endpoint as a subscription for an SNS Topic.
¿How can I get the URL of the API Endpoint?
you can directly reference the RestApi resource like this.
Resources:
apiGateway:
Type: AWS::ApiGateway::RestApi
Properties:
Name: !Sub ${AWS::StackName}-my-api
Description: my-api-edge
Outputs:
apiGatewayInvokeURL:
Value: !Sub "https://${apiGateway}.execute-api.${AWS::Region}.amazonaws.com/${apiGatewayStageName}"
lambdaArn:
Value: !GetAtt "lambdaFunction.Arn"
I think I found the answer across a couple of places.
This Stack Overflow post shows that there is an implicit reference created that you can use as follows
!Ref ServerlessRestApi
This was supported in practice, by a SAM Respository App
And then I re-read, more closely, the SAM API documentation which shows the RestApiId Property. It says
...Typically, this is set to reference an AWS::Serverless::Api resource defined in this template. If not defined, a default AWS::Serverless::Api resource is created..."
So it looks like you can reference it as !Ref ServerlessRestApi without any modification to the YAML in the original question, or you could add the following Property, RestApiId: MyAPI, and reference it as !Ref MyAPI.
However, to get the actual URL, it looks like you have to use a Fn::Sub to glue together a couple of parts. Pahud Hsieh does it in his SAM Repository app above
Outputs:
APIUrlPrefix:
Value:
Fn::Sub:
- https://${ServerlessRestApi}.execute-api.${Region}.amazonaws.com/Prod/incomingwebhooks/
- Region:
Ref: AWS::Region
ServerlessRestApi:
Ref: ServerlessRestApi
...

Difference between SAM template and Cloudformation template

I'm finding it hard to understand the difference between SAM template and Cloudformation template. I know that SAM template can be used to define Serverless Applications like Lambda, but how does that make it different from Cloudformation template? Is the syntax different? I can still specify the Lambda definitions in cloudformation template. So, my question is why should I care about SAM? Won't knowing about just cloud formation template be sufficient?
From CloudFormation's perspective, SAM is a transform. Meaning: SAM templates are syntactically equivalent, but they allow you to define your serverless app with more brevity. The SAM template eventually gets expanded into full CFN behind the scenes. If you already know CFN, but want to write less YAML code, SAM may be beneficial to you. The idea is to reduce your effort.
SAM templates are a superset of Cloudformation. Any Cloudformation template can be run through SAM unchanged, and it will work. SAM supports all the types available in Cloudformation templates, so you can think of SAM as "CloudFormation++".
However, SAM also gives you additional "transforms" that allow you to define certain concepts succinctly, and SAM will figure out what you mean and fill in the the missing pieces to create a full, expanded, legal Cloudformation template.
Example: For SAM (and Serverless Framework) users, who deal mostly in Lambda functions, one of the more most useful transforms is the Events property on the Lambda function -- SAM will add all the objects needs to access that function through an API path in API Gateway.
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: HelloWorldFunction
Handler: app.lambdaHandler
Runtime: nodejs12.x
Events: # <--- "Events" property is not a real Cloudformation Lambda property
HelloWorld:
Type: Api
Properties:
Path: /hello
Method: get
The SAM template snippet shown above gets transformed/expanded into several API Gateway objects (a RestApi, a deployment, and a stage). The AWS::Serverless::Function type used in this snippet is not a real Cloudformation type -- you won't find it in the docs. SAM expands it into a Cloudformation template containing a AWS::Lambda::Function object and several different AWS::ApiGateway::* objects that Cloudformation understands.
To give you an idea of how much manual coding this saves you, here's what the expanded version of the above SAM template looks like as a full Cloudformation template:
Resources:
HelloWorldFunctionHelloWorldPermissionProd:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
Principal: apigateway.amazonaws.com
FunctionName:
Ref: HelloWorldFunction
SourceArn:
Fn::Sub:
- arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/GET/hello
- __Stage__: "*"
__ApiId__:
Ref: ServerlessRestApi
HelloWorldFunctionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Tags:
- Value: SAM
Key: lambda:createdBy
ServerlessRestApiProdStage:
Type: AWS::ApiGateway::Stage
Properties:
DeploymentId:
Ref: ServerlessRestApiDeployment_NNN
RestApiId:
Ref: ServerlessRestApi
StageName: Prod
ServerlessRestApiDeployment_NNN:
Type: AWS::ApiGateway::Deployment
Properties:
RestApiId:
Ref: ServerlessRestApi
Description: 'RestApi deployment id: ???'
StageName: Stage
ServerlessRestApi:
Type: AWS::ApiGateway::RestApi
Properties:
Body:
info:
version: '1.0'
title:
Ref: AWS::StackName
paths:
"/hello":
get:
x-amazon-apigateway-integration:
httpMethod: POST
type: aws_proxy
uri:
Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloWorldFunction.Arn}/invocations
responses: {}
swagger: '2.0'
HelloWorldFunction:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: aws-sam-cli-managed-default-samclisourcebucket-???
S3Key: temp/???
Tags:
- Value: SAM
Key: lambda:createdBy
Handler: app.lambdaHandler
Role:
Fn::GetAtt:
- HelloWorldFunctionRole
- Arn
Timeout: 3
Runtime: nodejs12.x
Previously, if you were authoring pure Cloudformation, you would have had to code all this by hand, over and over, for each API Gateway endpoint that you wanted to create. Now, with a SAM template, you define the API as an "Event" property of the Lambda function, and SAM (or Serverless Framework) takes care of the drudgery.
In the old days, when we had to do all this by hand, it totally sucked. But now, everything is glorious again.
Like #Luis Colon said, SAM is a transform. What that means, is that at the top of a SAM Template there is a Transform statement that lets CloudFormation know to run an intrinsic function, Transform, on this SAM template to turn it into a CloudFormation template. So, all SAM Templates will eventually be converted into CF templates, but for the end-user in most cases it is easier to just use the SAM template. For instance, for a simple application with Lambdas triggered by a new API you're creating, the SAM template will let you accomplish this in fewer lines than CloudFormation.
To extend this, the Serverless Framework behaves similarly. Serverless is designed to work across platforms (AWS, Azure, etc.). It's syntax looks very similar to SAM, and it too converts the template into the target platform's (ie. AWS) fuller version of the template (ie. CloudFormation template).
You can imagine SAM as an extended form of CloudFormation. SAM makes Serverless/Lambda deployments easier.
Even CloudFormation can deploy lambda scripts using inline scripts but it has a limitation of 4096 characters and you cannot pack custom dependencies, python libraries.
So to make Lambda/Serverless deployments easy SAM is used. SAM is a CLI tool. You cannot find SAM in AWS Console.
In case of python deployment, sam will read the requirements.txt file build a package, and will deploy the package when you wish to sam deploy
So at the end of the day you can write as much lengthy Lambda Code, use as many libraries you want and even import your custom libraries i.e. complete flexibility.