I have a step function inside cloudformation. The cloudformation stack also create Lambdas which i will use as resource in step function. I have something like
TestLambda:
Type: "AWS::Lambda::Function"
Properties:
Handler: "test_lambda.lambda_handler"
Role: "arn:aws:iam::1234342334:role/Lambda"
Code:
ZipFile: !Sub |
from __future__ import print_function
import boto3
def lambda_handler(event, context):
print(event)
Runtime: "python2.7"
....
TestStateMachine:
Type: AWS::StepFunctions::StateMachine
Properties:
StateMachineName: "Test"
DefinitionString: |-
{
"StartAt": "State1",
"States": {
"State1" : {
"Type" : "Task",
"Resource" : "${!GetAtt TestLambda.Arn}",
"Next": "State2?"
},
...
...
all inside one cloudformation template.
"SCHEMA_VALIDATION_FAILED: Value is not a valid resource ARN"
I also tried !GetAtt TestLambda.Arn, it didn't work. I want the lambda and stepfunction created inside the single cloudformation template. Please let me know if there is better, cleaner way of doing it.
Thank you
You should use Fn::Sub function for that:
TestStateMachine:
Type: AWS::StepFunctions::StateMachine
Properties:
StateMachineName: "Test"
DefinitionString:
Fn::Sub:
|-
{
"StartAt": "State1",
"States": {
"State1" : {
"Type" : "Task",
"Resource" : "${TestLambda.Arn}",
"Next": "State2?"
},
Related
I've been all over the web searching for an answer to this.
Essentially, we're spinning up an API using Swagger, which is awesome and works great, but one thing doesn't work... When we make a call to an Endpoint, we get a 500 error (it's not a 500 error that we're providing either it's one from AWS). The error states "Execution failed due to configuration error: Invalid permissions on Lambda function" (https://youtu.be/H4LM_jw5zzs <- This is a video, from another user, of the error I'm getting).
I've gone down many ratholes, and have found an answer... It involves using the AWS CLI and looks a bit like this:
aws lambda add-permission \
--function-name FUNCTION_NAME \
--statement-id STATEMENT_ID \
--action lambda:InvokeFunction \
--principal apigateway.amazonaws.com \
--source-arn "arn:aws:execute-api:us-east-1:ACCOUNT_ID:API_ID/*/METHOD/ENDPOINT"
This is great and all, but we are using CloudFormation to spin up everything and we want this to be automated. Is there an easier way to go about this? Is there something in CloudFormation that will give us the resource policy that we need?
I'm hitting a bit of a wall with this, but I've been working on it for a few hours today and it's a bit of a blocker for our API release, so any help would be much appreciated. :)
There is a CloudFormation solution to this problem. See the following CloudFormation snippet:
"Permission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": { "Fn::GetAtt": [ "Lambda", "Arn" ] },
"Action": "lambda:InvokeFunction",
"Principal": "apigateway.amazonaws.com",
"SourceArn": { "Fn::Join": [ "", [
"arn:aws:execute-api:",
{ "Ref": "AWS::Region" }, ":",
{ "Ref": "AWS::AccountId" }, ":",
{ "Ref": "API" },
"/*/*/*"
] ] }
}
}
This grants API Gateway permissions to launch your Lambda function. Variables in this snippet you need to change are Lambda (line 4) and API (line 11).
For the invoke permissions:
"APIInvokePermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {
"Ref": "YOUR_LAMBDA_FUNCTION_RESOURCE_NAME"
},
"Action": "lambda:InvokeFunction",
"Principal": "apigateway.amazonaws.com",
"SourceArn": {
"Fn::Sub": "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${YOUR_REST_API_RESOURCE_NAME}/*/*/*"
}
}
},
Thanks https://twitter.com/edjgeek for helping me get this straight in my head.
This GIST shows how to use AWS::Serverless:Function with Events to automatically generate the needed AWS::Lambda::Permission to allow APIGateway (for a given route) to invoke your Lambda:
https://gist.github.com/rainabba/68df1567cbd0c4930d428c8953dc2316
Both of the following approaches assume an api such as (many fields omitted for readability):
MyApi:
Type: 'AWS::Serverless::Api'
Properties:
DefinitionBody:
Fn::Transform:
Name: AWS::Include
Parameters:
Location: openapi.yaml
Using "SAM Events"
The most relevant bit (I've omitted many required fields):
MyLambdaFunction:
Type: 'AWS::Serverless::Function'
Properties:
Events:
MyRouteEventToProxy:
Type: Api
Properties:
Method: POST
Path: '/some-route/{pathParm}'
RestApiId: !Ref MyApi # ResourceName of AWS::Serverless::Api
Auth:
Authorizer: NONE
Using "openapi binding"
If you'd rather declare the binding in the openapi.yaml, then see the following project (no Lambda/Events required). This approach requires an explict role to allow invoke.
template.yaml relevant bits:
MyLambdaFunction:
Type: 'AWS::Serverless::Function'
Properties:
# Events: # No need for Events when binding from openapi.yaml
MyHttpApiRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service: "apigateway.amazonaws.com"
Action:
- "sts:AssumeRole"
openapi.yaml relevant bits:
paths:
post:
x-amazon-apigateway-integration:
httpMethod: POST
uri:
Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyLambdaFunction.Arn}:live/invocations"
contentHandling: "CONVERT_TO_TEXT"
type: aws_proxy
credentials:
Fn::GetAtt: [MyHttpApiRole, Arn]
https://github.com/aws-samples/sessions-with-aws-sam/tree/master/http-api-direct-integration
I'm trying to create some resources using Cloudformation with serverless framework, In which I need to substitute resource name from another resource. Tried to use !Sub but still I couldn't get Arn of another resource created.
Tried all the approaches in this stackoverflow question How to use Sub and GetAtt functions at the same time in CloudFormation template? to no avail.
I appreciate any help.
Resources:
BasicParameter:
Type: AWS::SSM::Parameter
Properties:
Name: /data/config-name
Type: String
Value:
Fn::Base64:
!Sub |
{
"filter_configs": [{
"stream_name": !GetAtt tpRecordDeliveryStream.Arn,
"events": [
{
"name": "event_name1",
"stream": "streamname1"
},
{
"name": "event_name2"
}
]
}]
}
Description: Configuration for stream filters
Tags:
project: projectname
team: data
owner: owner_name
This was resolved by using serverless-pseudo-parameters serverless plugin. Serverless framework also uses ${} placeholder and it conflicts with Cloudformation placeholders. serverless-pseudo-parameters solves that by allowing us to replace those place holders with #{} which are replaced during sls deploy with cloud formation templates
Resources:
streamConfig:
Type: AWS::SSM::Parameter
Properties:
Name: config_name
Type: String
Value:
Fn::Base64: |
{
"filter_configs": [{
"firehose_stream_arn": "#{tpRecordDeliveryStream.Arn}",
"events": [
{
"name": "config0",
"filter1": "value1"
},
{
"name": "config1"
}
]
}]
}
Description: Configuration for stream filters
Since you have !Sub |, instead of
"stream_name": !GetAtt tpRecordDeliveryStream.Arn,
the following should be enough
"stream_name": "${tpRecordDeliveryStream.Arn}"
The alternative using !Sub in array notation:
Value:
Fn::Base64:
!Sub
- |
{
"filter_configs": [{
"stream_name": "${tpRecordDeliveryStreamArn}",
"events": [
{
"name": "event_name1",
"stream": "streamname1"
},
{
"name": "event_name2"
}
]
}]
}
- tpRecordDeliveryStreamArn: !GetAtt tpRecordDeliveryStream.Arn
I'm trying to deploy stepfunctions with CloudFormation, and I'd like to reference the actual stepfunction definition from an external file in S3.
Here's how the template looks like:
StepFunction1:
Type: "AWS::StepFunctions::StateMachine"
Properties:
StateMachineName: !Ref StepFunction1SampleName
RoleArn: !GetAtt StepFunctionExecutionRole.Arn
DefinitionString:
Fn::Transform:
Name: AWS::Include
Parameters:
Location:
Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunction1/definition.json'
However, this doesn't seem to be supported, as we are getting error
Property validation failure: [Value of property {/DefinitionString} does not match type {String}]
I am doing something similar for APIs, referencing the actual API definition from an external swagger file, and that seems to be working fine.
Example:
SearchAPI:
Type: "AWS::Serverless::Api"
Properties:
Name: myAPI
StageName: latest
DefinitionBody:
Fn::Transform:
Name: AWS::Include
Parameters:
Location:
Fn::Sub: 's3://${ArtifactsBucketName}/ApiGateway/myAPI/swagger.yaml'
How can I make this work?
The trick is to escape the StepFunction DefinitionString property, and include the actual property, DefinitionString, in the external CloudFormation referenced file. Escaping only the stepfunction definition string would fail, CloudFormation complaining that the referenced Transform/Include template, is not a valid yaml/json.
Here's how it looks like:
Template:
StepFunction1:
Type: "AWS::StepFunctions::StateMachine"
Properties:
StateMachineName: !Ref StepFunction1SampleName
RoleArn: !GetAtt StepFunctionExecutionRole.Arn
Fn::Transform:
Name: AWS::Include
Parameters:
Location:
Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunction1/definition.json'
External stepfunction definition file:
{
"DefinitionString" : {"Fn::Sub" : "{\r\n \"Comment\": \"A Retry example of the Amazon States Language using an AWS Lambda Function\",\r\n \"StartAt\": \"HelloWorld\",\r\n \"States\": {\r\n \"HelloWorld\": {\r\n \"Type\": \"Task\",\r\n \"Resource\": \"arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}\", \r\n \"End\": true\r\n }\r\n }\r\n}"}
}
Now, although this solves the problem, it's a bit more difficult to maintain the StepFunction definition, in this form, in source control.
So I've thought about using a CloudFormation custom resource backed by a lambda function. The lambda function would deal with the actual StepFunction DefinitionString escaping part.
Here's how it looks like:
Template:
StepFunctionParser:
Type: Custom::AMIInfo
Properties:
ServiceToken: myLambdaArn
DefinitionString:
Fn::Transform:
Name: AWS::Include
Parameters:
Location:
Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunctionX/definition.json'
StepFunctionX:
Type: "AWS::StepFunctions::StateMachine"
Properties:
StateMachineName: StepFunction1SampleNameX
RoleArn: !GetAtt StepFunctionExecutionRole.Arn
DefinitionString: !GetAtt StepFunctionParser.DefinitionString
External StepFunction definition file:
{
"Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
"StartAt": "HelloWorld",
"States": {
"HelloWorld": {
"Type": "Task",
"Resource": {"Fn::Sub" : "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}" },
"End": true
}
}
}
Here's the documentation for creating AWS Lambda-backed Custom Resources.
There's still a problem with this.
Transform/Include converts external template boolean properties into string properties.
Therefore, DefinitionString
"DefinitionString": {
"States": {
"HelloWorld": {
"Type": "Task",
"Resource": "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}",
**"End": true**
}
},
"Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
"StartAt": "HelloWorld"
}
becomes
"DefinitionString": {
"States": {
"HelloWorld": {
"Type": "Task",
"Resource": _realLambdaFunctionArn_,
**"End": "true"**
}
},
"Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
"StartAt": "HelloWorld"
}
CloudFormation then complains about the StepFunction definition not being valid:
Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: Expected value of type Boolean at /States/HelloWorld/End'
Is this a CloudFormation Transform/Include issue? Can someone from AWS give a statement on this?
In the mean time, one solution could be to use yaml instead of json, to store the stepfunction definition externally. No string escaping:
DefinitionString:
Fn::Sub: |
{
"Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
"StartAt": "HelloWorld",
"States": {
"HelloWorld": {
"Type": "Task",
"Resource": "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}",
"End": true
}
}
}
Here you can find another approach to this problem: https://github.com/aws-samples/lambda-refarch-imagerecognition
And here is a brief summary:
As part of your CloudFormation template file (let's say it's named template.yml) you define the state machine as follows:
LambdaA:
...
LambdaB:
...
LambdaC:
...
MyStateMachine:
Type: AWS::StepFunctions::StateMachine
Properties:
StateMachineName: !Sub 'MyMachine-${SomeParamIfNeeded}'
DefinitionString:
!Sub
- |-
##{{STATEMACHINE_DEF}}
- {
lambdaArnA: !GetAtt [ LambdaA, Arn ],
lambdaArnB: !GetAtt [ LambdaB, Arn ],
lambdaArnC: !GetAtt [ LambdaC, Arn ]
}
RoleArn: !GetAtt [ StatesExecutionRole, Arn ]
Then you will need replacement script, here you can find one: inject_state_machine_cfn.py
Then you can create your state machine definition as a separate file, say state_machine.json with a content:
{
"Comment": "This is an example of Parallel State Machine type",
"StartAt": "Parallel",
"States": {
"Parallel": {
"Type": "Parallel",
"Next": "Final State",
"Branches": [
{
"StartAt": "Run Lambda A",
"States": {
"Run Lambda A": {
"Type": "Task",
"Resource": "${lambdaArnA}",
"ResultPath": "$.results",
"End": true
}
}
},
{
"StartAt": "Run Lambda B",
"States": {
"Run Login Functionality Test": {
"Type": "Task",
"Resource": "${lambdaArnB}",
"ResultPath": "$.results",
"End": true
}
}
}
]
},
"Final State": {
"Type": "Task",
"Resource": "${lambdaArnC}",
"End": true
}
}
}
Once you have all these elements you can invoke the transformation function:
python inject_state_machine_cfn.py -s state_machine.json -c template.yml -o template.complete.yml
I'm using a nested Cloudformation template structured like this:
masterTemplate -> childA -> childB
Each of the JSON template files is stored in S3 with a bucket name of "${masterStackName}-env-upload". This works fine from the parent template, as I can simply do:
"TemplateURL": {
"Fn::Join": [
"",
[
"https://s3.amazonaws.com/",
{
"Ref": "AWS::StackName"
},
"-env-upload/device-specific-template-HALO-BUILD-VERSION.json"
]
]
},
However, when childA attempts to do the same thing to launch the childB template, "AWS::StackName" becomes the generated name for childA - meaning that it is trying to access a non-existent bucket.
My question is: how can I pass down the name of the master/parent stack to the child stacks? I attempted to do this as a Parameter, but "Ref" is not allowed for parameter values (so I couldn't do "Ref" : "AWS::StackName" for the value).
Any help is appreciated!
It is in fact possible to pass the parent stack name to the child stacks using parameters:
Here's an exemple:
parent.yml
---
AWSTemplateFormatVersion: '2010-09-09'
Resources:
ChildA:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: ...test/test-child-a.yml
Parameters:
ParentStackName: !Ref AWS::StackName
test-child-a.yml
---
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
ParentStackName:
Type: String
Resources:
TestBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref ParentStackName
One option is to decouple the stack name from the S3 bucket name, and specify the S3 bucket as a parameter in the masterTemplate stack, and then reference it in the Outputs section. The
In the master:
"Outputs": {
"EnvUploadBucketName" : {
"Value" : { "Ref" : "paramEnvUploadBucketName" }
}
}
In the child:
"TemplateURL": {
"Fn::Join": [
"",
[
"https://s3.amazonaws.com/",
{ "Fn::GetAtt" : [ "masterTemplate", "Outputs.EnvUploadBucketName" ] }
"/device-specific-template-HALO-BUILD-VERSION.json"
]
]
}
In this case, EnvUploadBucketName would be the name of the upload bucket, passed as an output from the masterTemplate stack.
When I try to create custom authorizerer for my AWS API Gateway using CloudFormation, it freezes trying all the time to execute CreateAuthorizer call, but fails. Here is the minimum CloudFormation template with which I can reproduce that behavior:
AWSTemplateFormatVersion: "2010-09-09"
Resources:
ApiGatewayV1:
Type: "AWS::ApiGateway::RestApi"
Properties:
Name: "test"
ApiAuthorizerV1:
Type: "AWS::ApiGateway::Authorizer"
Properties:
RestApiId: !Ref "ApiGatewayV1"
Name: "test"
Type: "TOKEN"
AuthorizerUri: !Sub "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda::${AWS::AccountId}:function:${!stageVariables.AuthorizerLambdaName}/invocations"
IdentitySource: "method.request.header.Authorization"
AuthorizerResultTtlInSeconds: 3600
CloudTrail log:
"errorCode": "InternalFailure",
"errorMessage": "An unknown error occurred",
"requestParameters": {
"restApiId": "lweme6j3wk",
"createAuthorizerInput": {
"providerARNs": [],
"identitySource": "method.request.header.Authorization",
"authorizerResultTtlInSeconds": 3600,
"type": "TOKEN",
"name": "test",
"authorizerUri": "arn:aws:apigateway:eu-central-1:lambda:path/2015-03-31/functions/arn:aws:lambda::<ACCOUNT_ID>:function:${stageVariables.AuthorizerLambdaName}/invocations"
},
"template": false
},
"responseElements": null,
"requestID": "470e2efa-d3c1-11e7-b0cc-b7fd2383ef6b",
"eventID": "2ceccaa5-9b97-4b1e-93e5-3c4e6bca419d",
Ok, that was supper bizzare. When I explicitely specified region in the target lambda ARN it worked!
Replaced:
arn:aws:lambda::<ACCOUNT_ID>
With:
arn:aws:lambda:<REGION>:<ACCOUNT_ID>
(and yes, it works with ${AWS::Region}:${AWS::AccountId} placeholder, I used fixed values to check.