Related
Step Function is defined like that:
{
"StartAt": "Decision_Maker",
"States": {
"Decision_Maker":{
"Type": "Choice",
"Choices": [
{
"Variable": "$.body.MyData",
"StringEquals": "null", //that doesn't work :(
"Next": "Run_Task1"
}],
"Default": "Run_Task2"
},
"Run_Task1": {
"Type": "Task",
"Resource": "url_1",
"Next": "Run_Task2"
},
"Run_Task2": {
"Type": "Task",
"Resource": "url_2",
"End": true
}
}
}
Basically it's a choice between 2 tasks.
Input data is like this:
{
"body": {
"prop1": "value1",
"myData": {
"otherProp": "value"
}
}
}
The problem is that sometimes there's no myData in JSON. So input may come like this:
{
"body": {
"prop1": "value1",
"myData": null
}
}
How do I check whether or not myData is null?
As of August 2020, Amazon States Language now has an isNull and isPresent Choice Rule. Using these you can natively check for null or the existence of a key in the state input inside a Choice state.
Example:
{ "Variable": "$.possiblyNullValue", "IsNull": true }
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-choice-state.html#amazon-states-language-choice-state-rules
The order matters. Set the "IsPresent": false first, then the "IsNull": true, finally the scalar comparison last.
"Check MyValue": {
"Comment": "Check MyValue",
"Type": "Choice",
"Default": "ContinueWithMyValue",
"Choices": [
{
"Or": [
{
"Variable": "$.MyValue",
"IsPresent": false
},
{
"Variable": "$.MyValue",
"IsNull": true
},
{
"Variable": "$.MyValue",
"BooleanEquals": false
}
],
"Next": "HaltProcessing"
},
{
"Variable": "$.MyValue",
"BooleanEquals": true,
"Next": "ContinueWithMyValue"
}
]
},
As per my experience, Choice Type can't deal nulls. Best way could be pre-processing your input using a lambda in the very first state and return the event formatting it as "null". Below code snippet might help.
def lambda_handler(event, context):
if event['body']['MyData']:
return event
else:
event['body']['MyData']="null"
return event
Note: This also handles empty string.
I am writing an AWS step function, and for one of the steps, I wish to call a lambda that accepts an array as one of the inputs. However, if I try to pass in a JsonPath into the array, I get
The value for the field 'arrayField.$' must be a STRING that contains a JSONPath but was an ARRAY
My step function definition:
{
"StartAt": "First",
"States": {
"First": {
"Type": "Pass",
"Parameters": {
"type": "person"
},
"ResultPath": "$.output",
"Next": "Second"
},
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
"Parameters": {
"regularParameter": "some string",
"arrayParameter.$": ["$.output.type"]
},
"Next": "Succeed"
},
"Succeed": {
"Type": "Succeed"
}
}
}
How can I use jsonPath inside the array?
Since a new release you could use the intrinsic function States.Array:
"arrayParameter.$": "States.Array($.output.type)"
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html
As #Seth Miller mentioned JsonPath resolution within arrays doesn't work unfortunately. If the amount of values to replace in the array is small and known there's a simple workaround (in my case I needed an array of size 1).
The steps are:
Initialise the array with the number of values you need;
Replace each value using "ResultPath": "$.path.to.array[n]";
Use "$.path.to.array" in your task.
Simple, working example:
{
"StartAt": "First",
"States": {
"First": {
"Type": "Pass",
"Parameters": {
"type": "person"
},
"ResultPath": "$.output",
"Next": "Initialise Array"
},
"Initialise Array": {
"Comment": "Add an entry for each value you intend to have in the final array, the values here don't matter.",
"Type": "Pass",
"Parameters": [
0
],
"ResultPath": "$.arrayParameter",
"Next": "Fill Array"
},
"Fill Array": {
"Comment": "Replace the first entry of array with parameter",
"Type": "Pass",
"InputPath": "$.output.type",
"ResultPath": "$.arrayParameter[0]",
"End": true
}
}
}
And to use the resulting array in your task example:
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
"Parameters": {
"regularParameter": "some string",
"arrayParameter.$": "$.arrayParameter"
},
"Next": "Succeed"
},
Another way to approach this is by using Parallel state that outputs an array of objects and then use jsonPath to convert it to a simple array:
{
"StartAt": "Parallel",
"States": {
"Parallel": {
"Type": "Parallel",
"Next": "Use Array",
"ResultPath": "$.items",
"Branches": [
{
"StartAt": "CreateArray",
"States": {
"CreateArray": {
"Type": "Pass",
"Parameters": {
"value": "your value"
},
"End": true
}
}
}
]
},
"Use Array": {
"Type": "Pass",
"Parameters": {
"items.$": "$.items[*].value"
},
"End": true
}
}
}
In this example, Parallel state outputs the following json:
{
"items": [
{
"value": "your value"
}
]
}
And "Use Array" state produces:
{
"items": [
"your value"
]
}
JSONPath inside parameters field need to be a string. So if you want to pass to lambda function a parameter called arrayParameter, you´ll need to make a jsonPath query that extract that array.
For example, if inside the key output is a key called outputArray with the array as its value.
Input JSON:
{
"pre": "sdigf",
"output": {
"result": 1,
"outputArray": ["test1","test2","test.."]
}
}
The parameter sintax:
"arrayParameter.$": "$.output.outputArray"
Reasonable advice
I ran into a use case for JsonPath resolution within arrays today and found (like you have) that the functionality does not exist today. I ended up deciding that doing the data massaging in code was simpler and cleaner. For example, you could create a small Lambda that takes in the object emitted by First and massages it to a format acceptable to Second and adds it to the output (WaterKnight mentions this solution in a comment to another question).
This assumes that you are, for some reason, unable to change the format of the input to that Lambda in Second (which would be the absolute shortest path here).
Unreasonable advice
That said, if you want a way to do this completely within Step Functions that is fairly gross, you can use the result of a Map state that executes Pass states. The output of the Map state is an array that aggregate the output of each constituent Pass state. These Pass states simply emit the value(s) you want in the final array using the Parameters attribute. An example Step Function definition follows. I did warn that it is gross and that I went a different way to solve the problem.
{
"StartAt": "First",
"Comment": "Please don't actually do this",
"States": {
"First": {
"Type": "Pass",
"Parameters": {
"type": "person"
},
"ResultPath": "$.output",
"Next": "Add Array"
},
"Add Array": {
"Comment": "A Map state needs some array to loop over in order to work. We will give it a dummy array. Add an entry for each value you intend to have in the final array. The values here don't matter.",
"Type": "Pass",
"Result": [
0
],
"ResultPath": "$.dummy",
"Next": "Mapper"
},
"Mapper": {
"Comment": "Add a Pass state with the appropriate Parameters for each field you want to map into the output array",
"Type": "Map",
"InputPath": "$",
"ItemsPath": "$.dummy",
"Parameters": {
"output.$": "$.output"
},
"Iterator": {
"StartAt": "Massage",
"States": {
"Massage": {
"Type": "Pass",
"Parameters": {
"type.$": "$.output.type"
},
"OutputPath": "$.type",
"End": true
}
}
},
"ResultPath": "$.output.typeArray",
"Next": "Second"
},
"Second": {
"Comment": "The Lambda in your example is replaced with Pass so that I could test this",
"Type": "Pass",
"Parameters": {
"regularParameter": "some string",
"arrayParameter.$": "$.output.typeArray"
},
"Next": "Succeed"
},
"Succeed": {
"Type": "Succeed"
}
}
}
As many answers correctly pointed out, it's not possible to do it exactly the way you need. But I would suggest another solution: an array of dictionaries. It's not exactly what you need, but is native and not hacky.
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
"Parameters": {
"regularParameter": "some string",
"arrayParameter": [{"type.$": "$.output.type"}]
},
"Next": "Succeed"
},
The result would be
{
"regularParameter": "some string",
"arrayParameter": [{"type": "SingleItemWrappedToAnArray"}]
}
It can not be described with Parallel State in AWS Step Function.
B and C should be in parallel.
C sends messages to both D and E.
D and E should be in parallel.
{
"StartAt": "A",
"States": {
"A": {
"Type": "Pass",
"Next": "Parallel State 1"
},
"Parallel State 1": {
"Type": "Parallel",
"Branches": [{
"StartAt": "B",
"States": {
"B": {
"Type": "Pass",
"End": true
}
}
},
{
"StartAt": "C",
"States": {
"C": {
"Type": "Pass",
"End": true
}
}
}
],
"Next": "Parallel State 2"
},
"Parallel State 2": {
"Type": "Parallel",
"Branches": [{
"StartAt": "D",
"States": {
"D": {
"Type": "Pass",
"End": true
}
}
},
{
"StartAt": "E",
"States": {
"E": {
"Type": "Pass",
"End": true
}
}
}
],
"Next": "F"
},
"F": {
"Type": "Pass",
"End": true
}
}
}
Answer is No , inside step function no state can set multiple states (invokes both successors)to its Next task. As per AWS step function cannot start State Machine as StartAt by providing multiple State names.
You can tweak your logic and use The Parallel state and achive same ,If you share your usecase may be help to solve problems.
How to specify multiple result path values in AWS Step Functions
A Parallel state provides each branch with a copy of its own input
data (subject to modification by the InputPath field). It generates
output that is an array with one element for each branch, containing
the output from that branch.
https://aws.amazon.com/blogs/aws/new-step-functions-support-for-dynamic-parallelism/
Example of state function
{
"Comment": "An example of the Amazon States Language using a choice state.",
"StartAt": "FirstState",
"States": {
"FirstState": {
"Type": "Task",
"Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
"Next": "ChoiceState"
},
"ChoiceState": {
"Type" : "Choice",
"Choices": [
{
"Variable": "$.foo",
"NumericEquals": 1,
"Next": "FirstMatchState"
},
{
"Variable": "$.foo",
"NumericEquals": 2,
"Next": "SecondMatchState"
}
],
"Default": "DefaultState"
},
"FirstMatchState": {
"Type" : "Task",
"Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:OnFirstMatch",
"Next": "NextState"
},
"SecondMatchState": {
"Type" : "Task",
"Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:OnSecondMatch",
"Next": "NextState"
},
"DefaultState": {
"Type": "Fail",
"Error": "DefaultStateError",
"Cause": "No Matches!"
},
"NextState": {
"Type": "Task",
"Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
"End": true
}
}
}
https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-example
https://sachabarbs.wordpress.com/2018/10/30/aws-step-functions/
As I answered in How to simplify complex parallel branch interdependencies for Step Functions, what you asked is better to be modeled as DAG but not state machine.
Depends on your use case, you might be able to workaround it (just as #horatiu-jeflea 's answer), but it's a workaround (not the straightforward way) anyway.
I am trying to call a couple of steps in my step function in a loop but I am unable to get my head around how I need to do this. Here's what I have till now: I need to add another lambda function(GetReviews) which will then call CreateReview, SendNotification in a loop. How would I go about doing this?
I am referring to the "Iterating a Loop Using Lambda" document, which shows it is possible.
Step function Defination:
{
"Comment": "Scheduling Engine",
"StartAt": "CreateReview",
"States": {
"CreateReview": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:529627678433:function:CreateReview",
"Next": "CreateNotification",
"InputPath": "$",
"ResultPath": "$.CreateReviewResult",
"OutputPath": "$"
},
"CreateNotification": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:529627678433:function:CreateNotification",
"InputPath": "$",
"ResultPath": "$.CreateNotificationResult",
"OutputPath": "$",
"End": true
}
}
}
I am contributing to this answer because I am using a little different approach to be able to loop inside of a step-function without having to rely on a lambda to increment. If someone in the future needs a generic solution, this can be a good reference.
Here is the example with code:
{
"Comment": "A description of my state machine",
"StartAt": "InitVariables",
"States": {
"InitVariables": {
"Type": "Pass",
"Parameters": {
"index": 0,
"incrementor": 1,
"ArrayLength.$": "States.ArrayLength($.inputArray)"
},
"ResultPath": "$.iterator",
"Next": "LoopChoice"
},
"LoopChoice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.iterator.ArrayLength",
"NumericGreaterThanPath": "$.iterator.index",
"Next": "IncrementVariable"
}
],
"Default": "End"
},
"IncrementVariable": {
"Type": "Pass",
"Parameters": {
"index.$": "States.MathAdd($.iterator.index, $.iterator.incrementor)",
"incrementor": 1,
"ArrayLength.$": "$.iterator.ArrayLength"
},
"ResultPath": "$.iterator",
"Next": "LoopChoice"
},
"End": {
"Type": "Pass",
"End": true
}
} }
This is the base for the loop, I use the States.MathAdd($.iterator.index, $.iterator.incrementor) intrinsic function to add two values, in this case, increment the index with a increment amount defined in the initVariables state. And also get the length of the array that I want to loop. You get the array length by also using a intrinsic function, States.ArrayLength("$.path.to.array"). The array is passed in the input.
To get the value of the array we can use the intrinsic function, States.ArrayGetItem($.inputArray, $.iterator.index).
All the custom logic should be put between the loopChoice state and the IncrementVariable State.
Hope this helps someone in the future.
Sorry for the late reply. You've probably solved it in between, but here you are
So, when looping in Step Functions, I quite simply add a Choice State (see Choice State Rules).
One of your State would need to output wether or not you have finished looping, or the number of items iterated on and the total number of items.
In the first case, it would be something like
{
"Comment": "Scheduling Engine",
"StartAt": "CreateReview",
"States": {
"GetReviews": {
whatever
"Next": "LoopChoiceState"
},
"LoopChoiceState": {
"Type" : "Choice",
"Choices": [
{
"Variable": "$.loopCompleted",
"BooleanEquals": false,
"Next": "GetReviews"
}
],
"Default": "YourEndState"
},
"CreateReview": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-westz2:529627678433:function:CreateReview",
"Next": "CreateNotification",
"InputPath": "$",
"ResultPath": "$.CreateReviewResult",
"OutputPath": "$"
},
"CreateNotification": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:529627678433:function:CreateNotification",
"InputPath": "$",
"ResultPath": "$.CreateNotificationResult",
"OutputPath": "$",
"End": true
}
}
}
Second case:
{
"Comment": "Scheduling Engine",
"StartAt": "CreateReview",
"States": {
"GetReviews": {
whatever
"Next": "LoopChoiceState"
},
"LoopChoiceState": {
"Type" : "Choice",
"Choices": [
{
"Variable": "$.iteratedItemsCount",
"NumericEquals": "$.totalItemsCount",
"Next": "CreateNotification"
}
],
"Default": "CreateReview"
},
"CreateReview": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:529627678433:function:CreateReview",
"Next": "CreateNotification",
"InputPath": "$",
"ResultPath": "$.CreateReviewResult",
"OutputPath": "$"
},
"CreateNotification": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:529627678433:function:CreateNotification",
"InputPath": "$",
"ResultPath": "$.CreateNotificationResult",
"OutputPath": "$",
"End": true
}
}
}
You could also use indexes (current index and last index) instead of the the number of items iterated over; it would help you keep track of where you in create reviews.
Three options are presented below. Here is a visual summary:
#1 Map: Repeat a set of steps for each element of an array (without a loop, optionally concurrently)
Map State is an alternative to looping when you want to run a set of steps for each element of an input array. Each element runs in parallel by default. Set MaxConcurrency: 1 to mimic the serial execution of #NunoGamaFreire's loop-based solution.
{
"StartAt": "MockArray",
"States": {
"MockArray": {
"Type": "Pass",
"Result": [ { "name": "Zaphod" }, { "name": "Arthur" }, { "name": "Trillian" } ],
"ResultPath": "$.Items",
"Next": "MapState"
},
"MapState": {
"Type": "Map",
"ResultPath": "$.MapResult",
"End": true,
"Iterator": {
"StartAt": "MockWork",
"States": {
"MockWork": {
"Type": "Pass",
"Parameters": {
"output.$": "States.Format('Hello, {}!', $.name)"
},
"OutputPath": "$.output",
"End": true
}
}
},
"ItemsPath": "$.Items"
}
}
}
One output is produced for each element in the MockArray, processed concurrently:
"MapResult": [ "Hello, Zaphod!", "Hello, Arthur!", "Hello, Trillian!" ]
#2 Repeat a set of steps X times (loop without lambda, serially)
This option involves proper looping! Repeat a set of tasks serially X number of times until a Choice State determines that an incrementing counter variable has reached X. Use the new States.MathAdd intrinsic function to increment without a Lambda. This option is suited for custom retry logic or other cases when you may want to break the loop early.
{
"StartAt": "InitializeCounter",
"States": {
"InitializeCounter": {
"Type": "Pass",
"Comment": "Initialize the counter at 0. Move all inputs to Payload.Input",
"Parameters": {
"Counter": 0
},
"Next": "IncrementCounter"
},
"IncrementCounter": {
"Type": "Pass",
"Comment": "Increment the Counter by 1",
"Parameters": {
"Counter.$": "States.MathAdd($.Counter, 1)"
},
"Next": "MockWork"
},
"MockWork": {
"Type": "Pass",
"Comment": "Simulate some work. Optionally break early from the loop with ExitNow: true",
"Result": false,
"ResultPath": "$.ExitNow",
"Next": "Loop?"
},
"Loop?": {
"Type": "Choice",
"Choices": [{ "Or": [
{ "Variable": "$.Counter", "NumericGreaterThanEqualsPath": "$$.Execution.Input.workCount" },
{ "Variable": "$.ExitNow", "BooleanEquals": true } ],
"Next": "Success"
}
],
"Default": "IncrementCounter"
},
"Success": {
"Type": "Succeed"
}
},
"TimeoutSeconds": 3
}
Counter iterates by one for each loop. The loop breaks if a task returns ExitNow: true.
{ "Counter": 4, "ExitNow": false }
#3 Repeat a set of steps X times (without a loop, *concurrently*)
This option is a hybrid of the first two. Like #2, we start with a desired number of iterations from the $.workCount input . Like #1, we map over an array concurrently. This time, though, the state machine creates the array with another intrinsic function, States.ArrayRange(1, $.workCount, 1).
{
"StartAt": "Iterations",
"States": {
"Iterations": {
"Type": "Pass",
"Parameters": {
"Iterations.$": "States.ArrayRange(1, $.workCount, 1)"
},
"Next": "MapState"
},
"MapState": {
"Type": "Map",
"ResultPath": "$.MapResult",
"End": true,
"Iterator": {
"StartAt": "MockWork",
"States": {
"MockWork": {
"Type": "Pass",
"Parameters": {
"output.$": "States.Format('Hello from iteration #{}', States.JsonToString($))"
},
"OutputPath": "$.output",
"End": true
}
}
},
"ItemsPath": "$.Iterations"
}
}
}
Tasks run concurrently, once for each item in Iterations.
"Iterations": [ 1, 2, 3, 4 ],
"MapResult": [ "Hello from iteration #1", "Hello from iteration #2", "Hello from iteration #3", "Hello from iteration #4" ]
I need to develop a 'State Machine' using 'AWS Step Functions' that accomplishes the following:
Call a Lambda function that will connect to DynamoDb & retrieve a list of rows. (I know how to do this.)
For each row from previous step, I need to call another Lambda function until all rows are read.
How do I do step #2 above in AWS Step Functions? In other words, how do I iterate over the results from the previous step.
It's not pretty, but this can be accomplished out of the box / without an iterator lambda by using JSONPath's slice operator and a Sentinel value.
Here's an example state machine:
{
"Comment": "Example of how to iterate over an arrray of items in Step Functions",
"StartAt": "PrepareSentinel",
"States": {
"PrepareSentinel": {
"Comment": "First, prepare a temporary array-of-arrays, where the last value has a special SENTINEL value.",
"Type": "Pass",
"Result": [
[
],
[
"SENTINEL"
]
],
"ResultPath": "$.workList",
"Next": "GetRealWork"
},
"GetRealWork": {
"Comment": "Next, we'll populate the first array in the temporary array-of-arrays with our actual work. Change this from a Pass state to a Task/Activity that returns your real work.",
"Type": "Pass",
"Result": [
"this",
"stage",
"should",
"return",
"your",
"actual",
"work",
"array"
],
"ResultPath": "$.workList[0]",
"Next": "FlattenArrayOfArrays"
},
"FlattenArrayOfArrays": {
"Comment": "Now, flatten the temporary array-of-arrays into our real work list. The SENTINEL value will be at the end.",
"Type": "Pass",
"InputPath": "$.workList[*][*]",
"ResultPath": "$.workList",
"Next": "GetNextWorkItem"
},
"GetNextWorkItem": {
"Comment": "Extract the first work item from the workList into currentWorkItem.",
"Type": "Pass",
"InputPath": "$.workList[0]",
"ResultPath": "$.currentWorkItem",
"Next": "HasSentinelBeenReached"
},
"HasSentinelBeenReached": {
"Comment": "Check if the currentWorkItem is the SENTINEL. If so, we're done. Otherwise, do something.",
"Type": "Choice",
"Choices": [
{
"Variable": "$.currentWorkItem",
"StringEquals": "SENTINEL",
"Next": "Done"
}
],
"Default": "DoWork"
},
"DoWork": {
"Comment": "Do real work using the currentWorkItem. Change this to be an activity/task.",
"Type": "Pass",
"Next": "RemoveFirstWorkItem"
},
"RemoveFirstWorkItem": {
"Comment": "Use the slice operator to remove the first item from the list.",
"Type": "Pass",
"InputPath": "$.workList[1:]",
"ResultPath": "$.workList",
"Next": "GetNextWorkItem"
},
"Done": {
"Type": "Succeed"
}
}
}