Problem:
I have a cloudformation template that is supposed to retrieve code found in CodeCommit and push it to a Lambda. The code in CodeCommit also contains a SAM template with a few parameters. The SAM template has the following setup
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: SAM Template for Deploy Python code to Lambda
Parameters:
ArtifactsBucket:
Description: The artifact bucket to get the lambda code
Type: String
Name:
Description: Name of the lambda function
Type: String
SqsARN:
Description: AWS SQS Arn to act as a trigger for the lambda function
Type: String
...
and the CodePipeline Cloudformation template has the following to override the 3 parameters present in the SAM template.
...
- Name: Deploy
Actions:
- Name: Deploy
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: 1
Configuration:
ActionMode: CREATE_UPDATE
Capabilities: 'CAPABILITY_IAM,CAPABILITY_AUTO_EXPAND,CAPABILITY_NAMED_IAM'
ChangeSetName: !Join
- '-'
- - lambda
- !Ref CodeCommitRepoName
ParameterOverrides: !Sub |
{
"ArtifactsBucket": "${ArtifactsBucket}",
"Name": "${CodeCommitRepoName}",
"SqsARN": {"Fn::ImportValue": "My-queue:us-east-1:Queue:Arn"}
}
...
The ArtifactBucket and the Name parameters are easily changed by the !Sub function but I am not able to have a valid value for SqsARN which is an imported value.
QuestionIs there anyway to include ImportValue in conjungtion with a Sub function within ParametersOverride?
Attempts I also tried to switch from
{"Fn::ImportValue": "My-queue:us-east-1:Queue:Arn"}
to
!ImportValue": "My-queue:us-east-1:Queue:Arn"
which also did not work. Remove the !Sub function and using a !Ref function yields the same output/problem as with ImportValue.
Key thing to remember is, using JSON within YAML
From the documentation
You can't use the short form of !ImportValue when it contains a !Sub.
It is valid for AWS CloudFormation, but not valid
for YAML:
Assuming Environment as Parameter, Here is a working example of Sub >> ImportValue >> Sub
Value: !Sub
- '{
"sqsUrl": "${sqsUrl}",
}'
- {
sqsUrl: { 'Fn::ImportValue': { 'Fn::Sub': 'QueUrl-${Environment}' } }
}
Applying it to above example might look something like below(using ssm for testing)
Parameters:
Environment:
Type: String
Default: DV
ArtifactsBucket:
Type: String
Default: TestBucket
CodeCommitRepoName:
Type: String
Default: Test
Resources:
SmsLambdaParameter:
Type: 'AWS::SSM::Parameter'
Properties:
Name: !Sub
- '/${EnvFullUpper}/My-Param/Test'
- { EnvFullUpper: !Ref Environment }
Type: 'String'
Value: !Sub
- '{
"ArtifactsBucket": "${ArtifactsBucket}",
"Name": "${CodeCommitRepoName}",
"SqsARN": "${SqsArn}"
}'
- { SqsARN: { 'Fn::ImportValue': { 'Fn::Sub': 'QueueArn-${Environment}' } } }
You can use array form of Fn::Sub:
!Sub
- String
- Var1Name: Var1Value
Var2Name: Var2Value
which would result in:
ParameterOverrides: !Sub
- |
{
"ArtifactsBucket": "${ArtifactsBucket}",
"Name": "${CodeCommitRepoName}",
"SqsARN": "${SqsARNImport}"
}
- SqsARNImport: !ImportValue <name-of-your-queue-export>
Related
Parameters:
ClusterName:
Type: String
ClusterVersion:
Type: Number
AllowedValues: [1.21, 1.20, 1.19, 1.18]
RoleArnValue:
Type: String
ListOfSubnetIDs:
Description: Array of Subnet IDs
Type: List<AWS::EC2::Subnet::Id>
ListOfSecurityGroupIDs:
Description: Array of security group ids
Type: List<AWS::EC2::SecurityGroup::Id>
Resources:
EKSCluster:
Type: AWS::EKS::Cluster
Properties:
Name: !Sub ${ClusterName}
Version: !Sub ${ClusterVersion}
RoleArn: !Sub ${RoleArnValue}
ResourcesVpcConfig:
SecurityGroupIds:
- !Sub ${ListOfSecurityGroupIDs}
SubnetIds:
- !Sub ${ListOfSubnetIDs}
Above is the .yaml clouldformation template I have created so i can spin up eks cluster. Then i am using aws cli to spin up the cluster using the following command.
aws cloudformation deploy --template-file eks.yaml --stack-name cluster-test --parameter-overrides ClusterName=Dev ClusterVersion=1.21 ListOfSubnetIDs=subnet-11111d11b11b011f4,subnet-99999d237f87f11d7,subnet-222222c110c7e4be7,subnet-88888884de8d25176 ListOfSecurityGroupIDs=sg-01111111a21221 RoleArnValue=arn:aws:iam::123456546456:role/cluster-ServiceRole-WMIC72AOWSP0 --capabilities CAPABILITY_NAMED_IAM
I get the following error
An error occurred (ValidationError) when calling the CreateChangeSet operation: Template error: variable ListOfSecurityGroupIDs in Fn::Sub expression does not resolve to a string
I am not sure why. Am i using !sub in correctly? Would really appreciate input on this.
Since you want to reference the parameters you provided the template as they are, you should use the Ref function.
Here's an example of a valid template:
Parameters:
ClusterName:
Type: String
RoleArnValue:
Type: String
ListOfSubnetIDs:
Description: Array of Subnet IDs
Type: List<AWS::EC2::Subnet::Id>
ListOfSecurityGroupIDs:
Description: Array of security group ids
Type: List<AWS::EC2::SecurityGroup::Id>
Resources:
EKSCluster:
Type: AWS::EKS::Cluster
Properties:
Name: !Ref ClusterName
RoleArn: !Ref RoleArnValue
ResourcesVpcConfig:
SecurityGroupIds: !Ref ListOfSecurityGroupIDs
SubnetIds: !Ref ListOfSubnetIDs
and here's how I deployed it:
aws cloudformation deploy --template-file eks.yml --stack-name cluster-test --parameter-overrides ClusterName=Dev ListOfSubnetIDs=subnet-be0a99c4,subnet-c71046ae ListOfSecurityGroupIDs=sg-009690ac6b3bff6df,sg-009a3f1cb63943941 -RoleArnValue=...
Sub should be used when you want to perform string manipulation. Checkout the examples from the documentation.
I am trying to deploy lambda having a zip(contains jar file). Now if the static value of the artifact in CodeUri is provided, it works fine but the problem is that the artifact is not static in nature i.e the version of the jar file (along with its name ex: abc-<1.x.x>-prod.jar) will change whenever their is new build.
So, I want to pass the artifact name in CodeUri as dynamic value rather than static value.
I had tried splitting Bucket, Key & pass the value as parameter but it fails saying NoSuchKey while deployment.
Edit: Adding Sample Template
Transform: AWS::Serverless-2016-10-31
Description: engine-service
Parameters:
Environment:
Type: String
Default: ""
SecurityGroupIds:
Type: String
Default: ""
SubnetIds1:
Type: String
Default: ""
SubnetIds2:
Type: String
Default: ""
DBSubnetGroupName:
Type: String
Default: ""
RDSSecret:
Type: String
Default: ""
RDSInstance:
Type: String
Default: ""
API:
Type: String
Default: ""
Globals:
Function:
Timeout: 120
Resources:
TranslationEngineLambda:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub "translation-engine-service-${Environment}"
CodeUri: target/abc-**1.0.0-SNAPSHOT**-prod.jar
Handler: com.abc.Main
Runtime: java11
MemorySize: 1024
Environment:
Variables:
BUCKET_NAME: "abc-dummy"
DB_SECRET: "abc-dummy"
FUNCTION_NAME: TranslateFunction
SPRING_PROFILES_ACTIVE: db
TEXT_EXTRACT_LAMBDA: !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:text-extract-service-${Environment}
TRANSLATE_OPTION: AWS
VpcConfig:
SecurityGroupIds:
- !Ref SecurityGroupIds
SubnetIds:
- !Ref SubnetIds1
- !Ref SubnetIds2
Policies:
- AWSLambda_FullAccess
- AmazonEC2FullAccess
- SecretsManagerReadWrite
- AmazonS3ReadOnlyAccess
- AmazonRDSFullAccess
TranslationEngineLambdaInvoke:
Type: "AWS::Lambda::Permission"
Properties:
Action: "lambda:InvokeFunction"
FunctionName: !GetAtt "TranslationEngineLambda.Arn"
Principal: "apigateway.amazonaws.com"
SourceArn: !Join ['', ['arn:aws:execute-api:MyRegion:MyAccountNumber:', Fn::ImportValue: !Ref API, '/*/POST/language-translator/v1/translate']]
Outputs:
TranslationEngineLambda:
Description: "Hello World Lambda Function ARN"
Value: !GetAtt TranslationEngineLambda.Arn
TranslationEngineLambdaIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt TranslationEngineLambdaRole.Arn`
Your question encompasses a few things.
First, if you're using CodeUri as you're doing in the template with a relative URL, AWS SAM will use that path to search from the directory in which the template resides to find the required files. If you're using Bucket/Key, AWS SAM will look in S3 to the specified Bucket and search for the Key. This is of course an entirely different way of working and assumes that you've already uploaded the artefact to that location yourself. You've presumably not done this, which results in the NoSuchKey error.
One of the more useful things about AWS SAM is that you can not only use it to deploy your code, but you can also use it to build your artefacts themselves. In that case, you have to point your CodeUri to the root of the folder in which your Lambda function code resides. AWS SAM will then - in the build step - create the necessary artefact (be it a jar of a zip). During the deployment, it will upload those artefacts to S3, update the CodeUris to reflect this and deploy the CloudFormation stack.
I don't think you can use CloudFormation parameters (with !Sub, !Join or similar) when using a relative CodeUri URL since the parameters are only interpreted in the cloud, and not during the AWS SAM build or package steps. So if you do not want to rely on AWS SAM to build your artefacts, you're probably better of also uploading them yourself.
Parameters:
Bucket:
Default: bucket_1
Type: String
AllowedValues:
- bucket_1
- bucket_2
Resources:
StepFunction:
Type: 'AWS::StepFunctions::StateMachine'
Properties:
DefinitionString: |-
{
...
}
RoleArn: ...
StateMachineName: ...
EventRule:
Type: 'AWS::Events::Rule'
Properties:
ScheduleExpression: cron(...)
Targets:
- RoleArn: ...
Arn: !Ref StepFunction
Id: "ScheduleStepFunction"
InputTransformer:
InputPathsMap:
Bucket: !Ref Bucket
InputTemplate: "{\"Bucket\": <Bucket>}"
I am trying to pass input parameter named Bucket, defined as a parameter in the CloudFormation Template itself, to a StepFunction Target of the Event Rule.
I get this error: InputPath for target ScheduleStepFunction is invalid.
Note: The optional parameter 'InputPath' has not been provided also.
Targets:
- RoleArn: ...
Arn: !Ref StepFunction
Id: "ScheduleStepFunction"
InputTransformer:
InputPathsMap:
Bucket: !Ref Bucket
InputTemplate: "{\"Bucket\": <Bucket>}"
It seems like your Targets have some issue because there is no Input is mention in your target as suggested by the Amazon eventRule documentation. Go to the provided link to use Input in target as suggested by AWS UserGuide.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-events-rule-target.html
I have a CloudFormation template that looks something like the following:
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
env:
Type: String
Default: NONE
Resources:
GraphQLAPI:
Type: AWS::AppSync::GraphQLApi
Properties:
Name: !Sub 'my-api-${env}'
AuthenticationType: AMAZON_COGNITO_USER_POOLS
UserPoolConfig:
UserPoolId: <something>
AwsRegion: !Ref AWS::Region
DefaultAction: ALLOW
Suppose that I already have a SSM parameter named /dev/cognitoUserPoolId. When I create this template, passing env=dev, I want to use the value of that parameter as the UserPoolId. I want to avoid manually passing a new CFN parameter for every SSM parameter, as there may be quite a few in practice.
You should be able to use dynamic references to the SSM parameters in your template.
Something like:
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
env:
Type: String
Default: NONE
Resources:
GraphQLAPI:
Type: AWS::AppSync::GraphQLApi
Properties:
Name: !Sub 'my-api-${env}'
AuthenticationType: AMAZON_COGNITO_USER_POOLS
UserPoolConfig:
UserPoolId: !Sub '{{resolve:ssm:/${env}/cognitoUserPoolId:1}}'
AwsRegion: !Ref AWS::Region
DefaultAction: ALLOW
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html
I create CloudFormation template for my AWS Lambda function and I need to specify different values of environment variables for different lambda aliases.
My template looks like:
AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: Lambda function configuration
Resources:
EndpointLambda:
Type: "AWS::Lambda::Function"
Properties:
FunctionName: "endpoint-lambda"
Handler: "com.test.aws.RequestHandler::handleRequest"
Runtime: java8
Code:
S3Bucket: "lambda-functions"
S3Key: "test-endpoint-lambda-0.0.1.jar"
Description: Test Lambda function
MemorySize: 256
Timeout: 60
Environment:
Variables:
ES_HOST: test-es-host-url
ES_ON: true
ES_PORT: 443
ES_PROTOCOL: https
REDIS_URL: test-redis-host-url
QaLambdaAlias:
Type: "AWS::Lambda::Alias"
Properties:
FunctionName: !Ref EndpointLambda
FunctionVersion: 1
Name: "QA"
Description: "QA alias"
ProdLambdaAlias:
Type: "AWS::Lambda::Alias"
Properties:
FunctionName: !Ref EndpointLambda
FunctionVersion: 1
Name: "Prod"
Description: "Production alias"
As you see, I have two aliases - QA and Prod and bunch of environment variables. I specified variables with common values in lambda function declaration. But I need to declare for QA alias env. variable's values related to QA, and for Prod alias - values for Prod environment. Any ideas how can I do that?
You can use CloudFormation Parameters to do this. As a quick example:
Parameters:
LambdaRuntime:
Type: String
Default: 'java8'
Description: What Lambda runtime do we use?
Resources:
QaLambdaAlias:
Type: "AWS::Lambda::Alias"
Properties:
FunctionName:
Ref: EndpointLambda
FunctionVersion: 1
Name: "QA"
Description: "QA alias"
Runtime:
Ref: LambdaRuntime
Then, if you want to use a different parameter, when you deploy via CLI, you can override with parameter-overrides like this:
aws cloudformation deploy --stack-name MyStack --template-file \
CloudFormation/MyStack.yaml --capabilities CAPABILITY_IAM \
--parameter-overrides LambdaRuntime=nodejs8.10