Deploy StepFunctions with CloudFormation, from external definition file - amazon-web-services

Question

I'm trying to deploy stepfunctions with CloudFormation, and I'd like to reference the actual stepfunction definition from an external file in S3.
Here's how the template looks like:
StepFunction1:
Type: "AWS::StepFunctions::StateMachine"
Properties:
StateMachineName: !Ref StepFunction1SampleName
RoleArn: !GetAtt StepFunctionExecutionRole.Arn
DefinitionString:
Fn::Transform:
Name: AWS::Include
Parameters:
Location:
Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunction1/definition.json'
However, this doesn't seem to be supported, as we are getting error
Property validation failure: [Value of property {/DefinitionString} does not match type {String}]
I am doing something similar for APIs, referencing the actual API definition from an external swagger file, and that seems to be working fine.
Example:
SearchAPI:
Type: "AWS::Serverless::Api"
Properties:
Name: myAPI
StageName: latest
DefinitionBody:
Fn::Transform:
Name: AWS::Include
Parameters:
Location:
Fn::Sub: 's3://${ArtifactsBucketName}/ApiGateway/myAPI/swagger.yaml'
How can I make this work?

Answer 1

The trick is to escape the StepFunction DefinitionString property, and include the actual property, DefinitionString, in the external CloudFormation referenced file. Escaping only the stepfunction definition string would fail, CloudFormation complaining that the referenced Transform/Include template, is not a valid yaml/json.
Here's how it looks like:
Template:
StepFunction1:
Type: "AWS::StepFunctions::StateMachine"
Properties:
StateMachineName: !Ref StepFunction1SampleName
RoleArn: !GetAtt StepFunctionExecutionRole.Arn
Fn::Transform:
Name: AWS::Include
Parameters:
Location:
Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunction1/definition.json'
External stepfunction definition file:
{
"DefinitionString" : {"Fn::Sub" : "{\r\n \"Comment\": \"A Retry example of the Amazon States Language using an AWS Lambda Function\",\r\n \"StartAt\": \"HelloWorld\",\r\n \"States\": {\r\n \"HelloWorld\": {\r\n \"Type\": \"Task\",\r\n \"Resource\": \"arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}\", \r\n \"End\": true\r\n }\r\n }\r\n}"}
}
Now, although this solves the problem, it's a bit more difficult to maintain the StepFunction definition, in this form, in source control.
So I've thought about using a CloudFormation custom resource backed by a lambda function. The lambda function would deal with the actual StepFunction DefinitionString escaping part.
Here's how it looks like:
Template:
StepFunctionParser:
Type: Custom::AMIInfo
Properties:
ServiceToken: myLambdaArn
DefinitionString:
Fn::Transform:
Name: AWS::Include
Parameters:
Location:
Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunctionX/definition.json'
StepFunctionX:
Type: "AWS::StepFunctions::StateMachine"
Properties:
StateMachineName: StepFunction1SampleNameX
RoleArn: !GetAtt StepFunctionExecutionRole.Arn
DefinitionString: !GetAtt StepFunctionParser.DefinitionString
External StepFunction definition file:
{
"Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
"StartAt": "HelloWorld",
"States": {
"HelloWorld": {
"Type": "Task",
"Resource": {"Fn::Sub" : "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}" },
"End": true
}
}
}
Here's the documentation for creating AWS Lambda-backed Custom Resources.
There's still a problem with this.
Transform/Include converts external template boolean properties into string properties.
Therefore, DefinitionString
"DefinitionString": {
"States": {
"HelloWorld": {
"Type": "Task",
"Resource": "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}",
**"End": true**
}
},
"Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
"StartAt": "HelloWorld"
}
becomes
"DefinitionString": {
"States": {
"HelloWorld": {
"Type": "Task",
"Resource": _realLambdaFunctionArn_,
**"End": "true"**
}
},
"Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
"StartAt": "HelloWorld"
}
CloudFormation then complains about the StepFunction definition not being valid:
Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: Expected value of type Boolean at /States/HelloWorld/End'
Is this a CloudFormation Transform/Include issue? Can someone from AWS give a statement on this?

Answer 2

In the mean time, one solution could be to use yaml instead of json, to store the stepfunction definition externally. No string escaping:
DefinitionString:
Fn::Sub: |
{
"Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
"StartAt": "HelloWorld",
"States": {
"HelloWorld": {
"Type": "Task",
"Resource": "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}",
"End": true
}
}
}

Answer 3

Here you can find another approach to this problem: https://github.com/aws-samples/lambda-refarch-imagerecognition
And here is a brief summary:
As part of your CloudFormation template file (let's say it's named template.yml) you define the state machine as follows:
LambdaA:
...
LambdaB:
...
LambdaC:
...
MyStateMachine:
Type: AWS::StepFunctions::StateMachine
Properties:
StateMachineName: !Sub 'MyMachine-${SomeParamIfNeeded}'
DefinitionString:
!Sub
- |-
##{{STATEMACHINE_DEF}}
- {
lambdaArnA: !GetAtt [ LambdaA, Arn ],
lambdaArnB: !GetAtt [ LambdaB, Arn ],
lambdaArnC: !GetAtt [ LambdaC, Arn ]
}
RoleArn: !GetAtt [ StatesExecutionRole, Arn ]
Then you will need replacement script, here you can find one: inject_state_machine_cfn.py
Then you can create your state machine definition as a separate file, say state_machine.json with a content:
{
"Comment": "This is an example of Parallel State Machine type",
"StartAt": "Parallel",
"States": {
"Parallel": {
"Type": "Parallel",
"Next": "Final State",
"Branches": [
{
"StartAt": "Run Lambda A",
"States": {
"Run Lambda A": {
"Type": "Task",
"Resource": "${lambdaArnA}",
"ResultPath": "$.results",
"End": true
}
}
},
{
"StartAt": "Run Lambda B",
"States": {
"Run Login Functionality Test": {
"Type": "Task",
"Resource": "${lambdaArnB}",
"ResultPath": "$.results",
"End": true
}
}
}
]
},
"Final State": {
"Type": "Task",
"Resource": "${lambdaArnC}",
"End": true
}
}
}
Once you have all these elements you can invoke the transformation function:
python inject_state_machine_cfn.py -s state_machine.json -c template.yml -o template.complete.yml