I want to use a macro in my Cloudformation template, however I would like to use the macro only in production environment.
I see I can use a conditional statement, however the Transform requires a key name.
This is my template:
AWSTemplateFormatVersion: "2010-09-09"
Transform: [
"AWS::Serverless-2016-10-31",
!If [IsProduction, AddCloudWatchAlarms, !Ref "AWS::NoValue" ]
]
Parameters:
Environment:
Type: String
AllowedValues:
- prod
- stag
- dev
ConstraintDescription: invalid environment, only [prod, stag, dev] are allowed
Conditions:
IsProduction: !Equals [ !Ref Environment, prod ]
As you can see I am passing AWS::NoValue.
This is working perfectly when the I am deploying prod environment, however when I try to deploy in dev I get this error.
Error: Failed to create changeset for the stack: server-hosting-rust-dev, An error occurred (ValidationError) when calling the CreateChangeSet operation: Transforms defined as maps require Name key.
How can I achieve this?
I could modify the code of the macro to skip its processing if the environment is dev, but it is third party open source, it would take quite a while for my pull request to get approved or even create a fork (Which I will eventually do), so I am asking if there is a faster method to achieve this.
Thanks
Related
I am using serverless framework to deploy api gateway. I don't want to apply VPC to every stage in serverless. Is there a way for me to add below configuration based on stage name?
provider:
name: aws
runtime: nodejs12.x
...
endpointType: PRIVATE
vpcEndpointIds:
- 'Fn::ImportValue': 'api-gateway-endpoint'
resourcePolicy:
- Effect: Deny
...
There are a few ways to do conditional deployments in serverless.yml, some are more brittle than others and there are pros/cons to each but here is a list of methods I have collected:
Variables to control conditionals
NOTE: We use a custom regex variable syntax to separate Serverless variables from cloudformation variable syntax. All of the following snippets use this modified syntax:
provider:
name: aws
# Changes serverless variable to ${{}} double curly braces
variableSyntax: "\\${{([ ~:a-zA-Z0-9._\\'\",\\-\\/\\(\\)]+?)}}"
Serverless Custom Variables
custom:
scheduleEnabled:
dev: true
stage: true
prod: true
# Then In lambda function declaration
events:
- schedule:
name: MyScheduleName
description: SomeDescription
rate: cron(0/5 * * * ? *)
# Use custom variable and the serverless `stage` supplied
# via your deployment command to choose whether this feature is enabled
enabled: ${self:custom.scheduleEnabled.${self:custom.stage}}
input: {"_keepwarm": true}
Using Conditional Operations
Setting up CloudFormation Conditionals
resources:
- Conditions:
# True if they are equal ==
MyCondition: [!Equals ["${{env:SOMETHING}}","SOME_STRING"]]
# True if they are not equal !=
MyConditionTwo: !Not [!Equals ["${{env:SOMETHING_ELSE}}","SOME_OTHER_STRING"]]
# Using a custom serverless variable
IsProd: [!Equals ["${{self:provider.stage}}","prod"]]
Using Cloudformation Conditionals
If your conditional has two options, you can write it as:
# If true choose X, if False choose Y
Source:
Type: !If
- MyCondition # Conditional Name
- GITHUB # If condition is true
- GITHUB_ENTERPRISE # if condition is false
If your conditional is something you want to toggle ON or OFF, you can write it as:
# If True do nothing, If False choose X
MyCodebuildSetup:
Type: "AWS::CodeBuild::Project"
VpcConfig: !If
- MyCondition # Condition Name
- !Ref AWS::NoValue # If True, AWS will not attach a VPC Config
- VpcId: <My_VPC_ID> # If False, Use this VPC Config
Subnets:
- <my_VPC_Subnet>
SecurityGroupIds:
- <my_VPC_security_group
Weird Conditional Hacks (document usage well if you decide to implement them)
Conditionally deploying a large series of resources
Using Serverless to selectively deploy entire files of resources using strings set via environment variables.
resources:
- ${{file(some_dir/conditional_file_${{env:MY_CONDITION}}.yml)}}
This is also a way to toggle deployment of many resources via conditionals.
conditional_file_A.yaml can contain all the resources you toggle deployment of, while conditional_file_B.yaml can contain an empty Resources list (If you don't want a "file not found" warning from serverless:
Resources:
Using Serverless Variables in buildspec.yml files or .json files that will be sent to cloudformation (parameter store files, etc)
If you want to conditionally modify a .json or .yml file that will be sent deployed as a value for some CloudFormation key (such as deploying a .json file for parameter store, or submitting buildspec.yml files for CodePipeline), you can't actually use ${} serverless syntax in these external files. This is because serverless gets confused with it's own method of calling Key/Values from external files.
In order to use Serverless Variables in these files, you can make them .txt extensions as long as the actual file is formatted as proper .json or .yml
Source:
Type: CODEPIPELINE
# This file can't have any serverless variables in it
BuildSpec: ${{file(my_buildspec_file.yml)}}
Source:
Type: CODEPIPELINE
# This file CAN have serverless variables because it is
# interpreted by serverless as a .txt, the variables
# are resolved and then it is sent to cloudformation as a string anyway
BuildSpec: ${{file(my_buildspec_file_yaml_formatted.txt)}}
You may check the following answers: Conditional serverless.yml based on stage?
Therefore, you get something like:
resources:
Conditions:
IsProd:
Fn::Equals:
- ${opt:stage}
- prod
Resources:
SomeIAMRole:
Type: AWS::IAM::Role
Condition: IsProd
Properties:
etc
etc
Alternatively, there is a Serverless Plugin Ifelse
Add the plugin:
plugins:
- serverless-plugin-ifelse
and then add your conditions:
custom:
.....
....
serverlessIfElse:
- If: '"${opt:stage}" == "production"'
Set:
functions.helloWorld.cors: true
see more info Serveless plugin if-else
We are building a serverless app using aws and we want to enable lambda warmers only on the production environment.
Our cloudformation parameters:
Parameters:
Environment:
Description: Environment name
Type: String
EnableWarmer:
Description: Flag to enable/disable warmup Events
Default: DISABLED
Type: String
Our lambdas yaml file looks like this:
MyLambda:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub '${Environment}-my-lambda'
CodeUri: src
Handler: lambda.handler
Runtime: nodejs12.x
MemorySize: 128
Timeout: 100
Description: Creates a new something
Layers:
- !Ref InternalDependencyLayer
- !Ref ExternalDependencyLayer
Role: !Ref LambdaRoleArn
Events:
Api:
Type: Api
Properties:
Path: /url
Method: POST
RestApiId: !Ref ApiGateway
WarmingSchedule:
Type: Schedule
Properties:
Enabled: !Ref EnableWarmer
Schedule: rate(5 minutes)
Input: '{ "warmer":true, "concurrency": 2 }'
And then we deploy the dev environment with these params:
- Key: Environment
Value: dev
- Key: EnableWarmer
Value: DISABLED
Similarly for the production environment we deploy with these params:
- Key: Environment
Value: production
- Key: EnableWarmer
Value: ENABLED
According to aws documentation parameters can't be of type boolean which is the required type of the enabled attribute of the schedule event.
Fortunately amazon states:
Enabled Indicates whether the rule is enabled.
To disable the rule, set this property to False.
Type: Boolean
Required: No
AWS CloudFormation compatibility: This property is similar to the
State property of an AWS::Events::Rule resource. If this property is
set to True then AWS SAM passes ENABLED, otherwise it passes DISABLED
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-schedule.html
However when we deploy to the dev environment the warmers are enabled.
The problem with the Enabled property here seems to be that it does not support using CloudFormation's If function and a Condition to do something like that:
WarmingSchedule:
Enabled: !If [WarmerCondition, true, false]
(At least I couldn't figure it out, but maybe I just did something wrong) This is similar to what #raul-barreto suggests with !Equals. However, in my tests it always sets the rule's status to ENABLED.
It seems like there are two options left what you can do:
a) (Not recommended) Copy the CloudFormation code of your Lambda function and have a second one available in your CloudFormation resources. Now set a condition to both of them so you only create one of them, depending on your environment. Note: I don't say this is a good approach but it'd be possible. Problems are you have to maintain the definition of both Lambda functions and if you repeat this approach, you'll faster reach CloudFormations template size limits.
b) If you want to make sure you have a minimum number of instances available to your function (at least that's what I interpret from your question), consider using AWS Lambda's Provisioned Concurrency feature instead. You can define it like this in your AWS Lambda function using SAM:
MyLambda:
Type: AWS::Serverless::Function
Properties:
AutoPublishAlias: 'LATEST' #Setting AutoPublishAlias is important, otherwise you can not use provisioned concurrency
ProvisionedConcurrencyConfig:
ProvisionedConcurrentExecutions: 2
This snippet will provision concurrency for the function's alias 'LATEST' which points to the latest version of your function. Read more about AutoPublishAlias in SAM (Note: the link's content is written in the context of traffic shifting but it explains what the property does). It makes sure that you always have 2 instances of your Lambda functions available.
The accepted solution works, but as stated in comments, it is a bit difficult to control the enabling and disabling of the event. The following seems to work in a more robust way.
Events:
Scheduler:
Type: Schedule
Properties:
Schedule: !If [EnableWarmer, "rate(5 minutes)",
"cron(00 00 01 01 ? 1970)"]
According to the documentation, Enabled must a Boolean variable.
You can still have a String parameter and convert it to Boolean inside the CloudFormation.
WarmingSchedule:
Type: Schedule
Properties:
Enabled: !Equals [!Ref EnableWarmer, ENABLED]
Schedule: rate(5 minutes)
Input: '{ "warmer":true, "concurrency": 2 }'
This way you still can send ENABLED or DISABLED as a parameter but the input of WarmingSchedule.Enabled will be a Boolean.
I can't seem to get Ref or Fn:GetAtt to return a valid value for use with setting up a resource.
serverless.yml
...etc...
functions:
bearerTokenAuthentication:
handler: app.bearerTokenAuthentication
name: ${self:service}-auth-bearer
resources:
- ${file(./serverless_resources.yml)}
serverless_resources.yml
Resources:
ApiGateway:
Type: AWS::ApiGateway::RestApi
Properties:
Name: restapi-${self:provider.stage}
Description: Endpoints
ApiKeySourceType: HEADER # (to read the API key from the X-API-Key header of a request)
ApiGatewayBearerAuthorizer:
Type: AWS::ApiGateway::Authorizer
Properties:
Type: token
IdentitySource: method.request.header.Authorization
Name: BearerAuthorization
AuthorizerResultTtlInSeconds: 300
AuthorizerUri: !Join #arn:aws:apigateway:${self:provider.region}:lambda:path/${self:functions.bearerTokenAuthentication.name}
- ''
- - 'arn:aws:apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/'
- !GetAtt
- bearerTokenAuthentication # also tried !Ref bearerTokenAuthentication and '${self:functions.bearerTokenAuthentication.name}'
- Arn
- /invocations
RestApiId: !Ref ApiGateway
No matter what I do, GetAtt cannot find the ARN for the Lambda function declared in bearerTokenAuthentication. I just keep getting this error:
Error: The CloudFormation template is invalid: Template error: instance of Fn::GetAtt references undefined resource bearerTokenAuthentication
... or if trying Ref ...
Error: The CloudFormation template is invalid: Template format error: Unresolved resource dependencies [bearerTokenAuthentication] in the Resources block of the template
Is it possible to reference Lambda ARNs from the resource section? It seems by the error messages it is looking for "resource" names. I always thought the lambda function declaration was also considered a resource (besides the obvious Resources: block of course), perhaps I am misunderstanding something.
I figured it out. I had a NodeJS project and was using the "serverless" command line (sls) to deploy using serverless.yml. It turns out it creates a .serverless sub-directroy with some files in it. One of them is a compiled template for AWS Cloud Formation called cloudformation-template-update-stack.json. It appears that the utility likes to mangle the names by making the first character uppercase and adding "LambdaFunction" to all the function names (for whatever reason). In this case, bearerTokenAuthentication was renamed to BearerTokenAuthenticationLambdaFunction (the actual resource name). After looking into the compiled template it all became clear. The utility also seems to figure out the dependencies as well, which was good to know. This was the final result:
AuthorizerUri: !Join
- ''
- - 'arn:aws:apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/'
- !GetAtt [ BearerTokenAuthenticationLambdaFunction, Arn ]
- '/invocations'
Other "Gotchas":
DO NOT define the AWS::ApiGateway::RestApi resource (like I did in my question) if you are also using event mappings with the functions, otherwise you will get 2 APIs created. event entries automatically cause an API to be created called "ApiGatewayRestApi" - which is the resource name generated by the sls utility. The last line of the last file was changed to this:
RestApiId: !Ref ApiGatewayRestApi
And my ApiGateway: section was removed.
Credit goes to this post which helped make it more clear to me what was really going on: https://forum.serverless.com/t/fixed-how-do-i-get-reference-api-gateway-restapi-id-in-serverless-yml/3397/5
Previous Answer:
I found another way as well. This is what I resorted to doing until I found the proper (shorter) way. I was able to pull the lambda name and manually stitch together the required URI:
AuthorizerUri: !Join
- ''
- - 'arn:aws:apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/arn:aws:lambda:'
- !Ref 'AWS::Region'
- ':'
- !Ref 'AWS::AccountId'
- ':function:'
- '${self:functions.bearerTokenAuthentication.name}'
- '/invocations'
I hope that helps save someone some time trying to understand the complicated .yml file. I also cannot understand why it is so hard to make it simple to understand. All someone had to do is say (for me) was "sls takes a 'serverless.yml' file, and optional include files (such as declarations specific to the cloud system itself, like AWS Cloud Formation), and generates a template JSON file that is used by the target cloud services system to deploy your solution. Also, the names you give may get mangled, so check the template." I'm also surprised that no one has created an editor to make all this easier by now - perhaps something I'll look into myself one day. ;)
You can always go to the deployed lambda and look for the aws:cloudformation:logical-id tag. That way you get the logical ID you should be using in your serverless.yaml. (Don't like this behind-the-scenes names tricks either...)
I have one CFT that creates an EBS volume. Then I have a second CFT that attaches the volume to the instance. I use a Custom Resource which runs a lambda to find the EbsVolumeId and the InstanceId
The 2 CFT's are part of a jenkins pipeline and its possible that the playbook that uses the CFT to create the EBS volume can be skipped and thus there is no EbsVolumeId to reference.
So in the CFT that attaches the EBS volume, I tried adding a Condition
Resources:
MountPoint:
Type: AWS::EC2::VolumeAttachment
Condition: AttachEBS
Properties:
InstanceId: !GetAtt SCSHelper.InstanceId
VolumeId: !GetAtt SCSHelper.EbsVolumeId
The custom resource looks like
SCSHelper:
Type: Custom::SCSHelper
So I have to define the condition before the resources:
Conditions:
AttachEBS: !Not [!Equals http:// !GetAtt SCSHelper.EbsVolumeId , None ]
The problem is the Conditions is failing with:
An error occurred (ValidationError) when calling the CreateStack operation: Template format error: Unresolved dependencies https://forums.aws.amazon.com/. Cannot reference resources in the Conditions block of the template An error occurred (ValidationError) when calling the CreateStack operation: Template format error: Unresolved dependencies https://forums.aws.amazon.com/.
So it appears that the SCSHelper.EbsVolumeId attribute is not available for the condition to use.
Is there a way to make this work or is there a better way to conditionally run the CFT that attaches the EBS volume?
Thanks in advance...
The important section of the Conditions documentation is this
Parameters
Define the input values that you want to evaluate in your conditions. Conditions will result in true or false based on values from these input parameters.
Conditions in CloudFormation can be based on Parameters alone. They can't take in-flight resource values into account, as these values aren't known at compile time.
I'm creating a Nodejs microservice for AWS Lambda. I scaffolded by project using AWS Codestar, and that set me up with a CI/CD pipeline that automatically deploys the lambda function. Nice.
The issue is that every time it deploys the lambda function it must delete and recreate the function, thus deleting any versions or aliases I made.
This means I really can't roll back to other releases. I basically have use git to actually revert the project, push to git, wait for the super-slow AWS Code Pipeline to flow through successfully, and then have it remake the function. To me that sounds like a pretty bad DR strategy, and I would think the right way to rollback should be simple and fast.
Unfortunately, it looks like the CloudFormation section of AWS doesn't offer any help here. When you drill into your stack on the first CloudFormation page it only shows you information about the latest formation that occurred. Dear engineers of AWS CloudFormation: if there was a page for each stack that showed a history of CloudFormation for this stack and an option to rollback to it, that would be really awesome. For now, though, there's not. There's just information about the latest formation that's been clouded. One initially promising option was "Rollback Triggers", but this is actually just something totally different that lets you send a SNS notification if your build doesn't pass.
When I try to change the CodePipeline stage for deploy from CREATE_CHANGE_SET to CREATE_UPDATE I then get this error when it tries to execute:
Action execution failed UpdateStack cannot be used with templates
containing Transforms. (Service: AmazonCloudFormation; Status Code:
400; Error Code: ValidationError; Request ID:
bea5f687-470b-11e8-a616-c791ebf3e8e1)
My template.yml looks like this by the way:
AWSTemplateFormatVersion: 2010-09-09
Transform:
- AWS::Serverless-2016-10-31
- AWS::CodeStar
Parameters:
ProjectId:
Type: String
Description: AWS CodeStar projectID used to associate new resources to team members
Resources:
HelloWorld:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
Runtime: nodejs8.10
Environment:
Variables:
NODE_ENV: staging
Role:
Fn::ImportValue:
!Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]
Events:
GetEvent:
Type: Api
Properties:
Path: /
Method: get
PostEvent:
Type: Api
Properties:
Path: /
Method: post
The only options in the CodePipeline "Deploy" action are these:
It would be really great if someone could help me to see how in AWS you can make Lambda functions with CodePipeline in a way that they are easy and fast to rollback. Thanks!