AWS Step Function: check for null - amazon-web-services

Step Function is defined like that:
{
"StartAt": "Decision_Maker",
"States": {
"Decision_Maker":{
"Type": "Choice",
"Choices": [
{
"Variable": "$.body.MyData",
"StringEquals": "null", //that doesn't work :(
"Next": "Run_Task1"
}],
"Default": "Run_Task2"
},
"Run_Task1": {
"Type": "Task",
"Resource": "url_1",
"Next": "Run_Task2"
},
"Run_Task2": {
"Type": "Task",
"Resource": "url_2",
"End": true
}
}
}
Basically it's a choice between 2 tasks.
Input data is like this:
{
"body": {
"prop1": "value1",
"myData": {
"otherProp": "value"
}
}
}
The problem is that sometimes there's no myData in JSON. So input may come like this:
{
"body": {
"prop1": "value1",
"myData": null
}
}
How do I check whether or not myData is null?

As of August 2020, Amazon States Language now has an isNull and isPresent Choice Rule. Using these you can natively check for null or the existence of a key in the state input inside a Choice state.
Example:
{ "Variable": "$.possiblyNullValue", "IsNull": true }
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-choice-state.html#amazon-states-language-choice-state-rules

The order matters. Set the "IsPresent": false first, then the "IsNull": true, finally the scalar comparison last.
"Check MyValue": {
"Comment": "Check MyValue",
"Type": "Choice",
"Default": "ContinueWithMyValue",
"Choices": [
{
"Or": [
{
"Variable": "$.MyValue",
"IsPresent": false
},
{
"Variable": "$.MyValue",
"IsNull": true
},
{
"Variable": "$.MyValue",
"BooleanEquals": false
}
],
"Next": "HaltProcessing"
},
{
"Variable": "$.MyValue",
"BooleanEquals": true,
"Next": "ContinueWithMyValue"
}
]
},

As per my experience, Choice Type can't deal nulls. Best way could be pre-processing your input using a lambda in the very first state and return the event formatting it as "null". Below code snippet might help.
def lambda_handler(event, context):
if event['body']['MyData']:
return event
else:
event['body']['MyData']="null"
return event
Note: This also handles empty string.

Related

Can I access the TaskToken from a Map state with ItemSelector where the iteration step uses lambda:invoke.waitForTaskToken?

I am using AWS step function to iterate over a list in an input document where for each iteration, I need to invoke an external service. So I want to iterate over each item and run a step using lambda:invoke.waitForTaskToken and pass the TaskToken into the execution of each iteration.
The problem I'm running into is how to use both an ItemSelector at the Map state level but also inject the TaskToken during the internal step. I need to use an ItemSelector because I want each item to also contain information from the input to Map state. The AWS Docs state:
The ItemSelector field replaces the Parameters field within the Map state. If you use the Parameters field in your Map state definitions to create custom input, we highly recommend that you replace them with ItemSelector.
But they also say:
During an execution, the context object is populated with relevant data for the Parameters field from where it is accessed. The value for a Task field is null if the Parameters field is outside of a task state.
These two statements seem to imply that what I'm trying to do is impossible.
So, what I want is something like:
{
"StartAt": "ExampleMapState",
"States": {
"ExampleMapState": {
"Type": "Map",
"ItemsPath": "$.items",
"ItemSelector": {
"dynamic.$": "$.dynamic",
"ContextIndex.$": "$$.Map.Item.Index",
"ContextValue.$": "$$.Map.Item.Value"
},
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "TestPass",
"States": {
"TestPass": {
"Type": "Task",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-west-2:123456789012:function:echo-lambda",
"Payload": {
"item.$": "$",
"token.$": "$$.Task.Token"
}
},
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"End": true
}
}
},
"End": true
}
}
}
But this doesn't work because the ItemSelector overrides the Payload of the internal TestPass state. Is there a way to get this to work?
ETA: I figured I would try putting $$.Task.Token in ItemSelector just in case it would magically work but it ended up throwing an error because $$.Task does not exist in the context object at that level.
Example with this (invalid) configuration:
{
"StartAt": "ExampleMapState",
"States": {
"ExampleMapState": {
"Type": "Map",
"ItemsPath": "$.items",
"ItemSelector": {
"dynamic.$": "$.dynamic",
"ContextIndex.$": "$$.Map.Item.Index",
"ContextValue.$": "$$.Map.Item.Value",
"token.$": "$$.Task.Token"
},
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "TestPass",
"States": {
"TestPass": {
"Type": "Task",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-west-2:123456789012:function:echo-lambda"
},
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"End": true
}
}
},
"End": true
}
}
}
Based on my research I don't think what I'm trying to do is possible. What I ended up implementing is a workaround. I modified the function providing input to this step function to put the dynamic info that I needed into every item in the list I am iterating over. So my step function definition now looks something like this
{
"StartAt": "ExampleMapState",
"States": {
"ExampleMapState": {
"Type": "Map",
"ItemsPath": "$.items",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "TestPass",
"States": {
"TestPass": {
"Type": "Task",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-west-2:123456789012:function:echo-lambda",
"Payload": {
"item.$": "$",
"token.$": "$$.Task.Token"
}
},
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"End": true
}
}
},
"End": true
}
}
}
And an example input to this step function looks like:
{
"dynamic": "info",
"items": [
{
"dynamic": "info",
"resize": "true",
"format": "jpg"
},
{
"dynamic": "info",
"resize": "false",
"format": "png"
},
{
"dynamic": "info",
"resize": "true",
"format": "jpg"
}
]
}
It's not great because I have to repeat info into every item ahead of time but it works.

Access Map State ( item ) in Step functions

I am trying to access item properties which I am iterating over using Map state. I keep getting this error:
Value ($['Map'].snapshot_id.S) for parameter snapshotId is invalid. Expected: 'snap-...'. (Service: Ec2, Status Code: 400, Request ID: 6fc02935-c161-49df-8606-bc6f3e2934a6)
I have gone through the docs, which seems to suggest access using $.Map.snapshot_id.S but doesn't seem to work. Meanwhile, an input item to Map is:
{
"snapshot_id": {
"S": "snap-01dd5ee46df84119e"
},
"file_type": {
"S": "bash"
},
"id": {
"S": "64e6893261d94669b7a8ca425233d68b"
},
"script_s3_link": {
"S": "df -h"
}
}
Map state definition:
"Map": {
"Type": "Map",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "Parallel State",
"States": {
"Parallel State": {
"Comment": "A Parallel state can be used to create parallel branches of execution in your state machine.",
"Type": "Parallel",
"Branches": [
{
"StartAt": "CreateVolume",
"States": {
"CreateVolume": {
"Type": "Task",
"Parameters": {
"AvailabilityZone": "us-east-2b",
"SnapshotId": "$$.Map.snapshot_id.S"
},
"Resource": "arn:aws:states:::aws-sdk:ec2:createVolume",
"Next": "AttachVolume"
},
"AttachVolume": {
"Type": "Task",
"Parameters": {
"Device": "MyData",
"InstanceId": "MyData",
"VolumeId": "MyData"
},
"Resource": "arn:aws:states:::aws-sdk:ec2:attachVolume",
"End": true
}
}
}
],
"End": true
}
}
},
"Next": "Final Result",
"ItemsPath": "$.Items",
"MaxConcurrency": 40
},
TL;DR "SnapshotId.$": "$.snapshot_id"
By default, each inline map iteration's input is one item from the ItemsPath array, accessible simply as $.
Your state machine definition implies that $.Items is an array of objects with a snapshot_id key (and other keys). If so, each iteration's snapshot id is at $.snapshot_id.
Finally, adding .$ to the parameter name (SnapshotId.$) tells Step Functions the value is not a literal, but rather a path value to be substituted.

How to Pass a parameter based on value of one of the input parameter in AWS Step Function?

For AWS Step Function, I am trying to pass in new parameter which will be derived from another input parameter. Here is an example:
Input param to Step Function will be:
{
"abc": <abc_value>
OR
"def": <def_value>
}
This should be input params to further steps, if abc is present and value is A:
{
"abc": "A",
"def": "X"
}
This should be input params to further steps, if abc is present and value is B:
{
"abc": "B",
"def": "Y"
}
I tried to search the solution for it and seems like it can be done with combination of Pass and Choice states like following and couldn't found any other solution:
{
"StartAt": "InputRouterStep",
"States": {
"InputRouterStep": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.abc",
"IsPresent": true,
"Next": "DeriveDef"
}
],
"Default": "MainSteps"
},
"DeriveDef": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.abc",
"StringEquals": "A",
"Next": "PassDefForA"
},
{
"Variable": "$.abc",
"StringEquals": "B",
"Next": "PassDefForB"
}
],
"Default": "PassDefForA"
},
"PassDefForA": {
"Type": "Pass",
"Parameters": {
"abc.$": "$.abc"
"def": "X"
},
"Next": "MainSteps"
},
"PassDefForB": {
"Type": "Pass",
"Parameters": {
"abc.$": "$.abc"
"def": "Y"
},
"Next": "MainSteps"
},
...
Can someone please help me out here to understand if there is any more efficient way to do this? Probably Lambda can be one, but it is very short computation, may be above solution is better?
If I understand correctly, your solution is pretty "efficient" in the sense you won't need to invoke an additional lambda and incur the extra state transitions of doing so.
From a maintanence standpoint, the following (imho) might be easier to update down the road for future values of $.abc, needing to only the valid values in the ConditionalDefaultForDEF and the corresponding values in the InputRouteStep.
{
"StartAt": "ConditionalDefaultForDEF",
"States": {
"ConditionalDefaultForDEF": {
"Type": "Pass",
"Result": [
{
"key": "A",
"value": "X"
},
{
"key": "B",
"value": "Y"
}
],
"ResultPath": "$.abc_mapping",
"Next": "InputRouterStep"
},
"InputRouterStep": {
"Type": "Choice",
"Choices": [
{
"And": [
{
"Variable": "$.abc",
"IsPresent": true
},
{
"And": [
{
"Or": [
{
"Variable": "$.abc",
"StringEquals": "A"
},
{
"Variable": "$.abc",
"StringEquals": "B"
}
]
}
]
}
],
"Next": "PassDef"
}
],
"Default": "MainSteps"
},
"PassDef": {
"Type": "Pass",
"InputPath": "$.abc_mapping[?(#.key == $['abc'])].value",
"Parameters": {
"def.$": "$[0]"
},
"Next": "MainSteps"
},
"...": "..."
}
}

How to pass different output from Choice state in AWS Step Function?

Let say part of my Step Function looks like next:
"ChoiceStateX": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.value",
"NumericEquals": 0,
"Next": "ValueIsZero"
}
],
"Default": "DefaultState"
},
"ValueIsZero": {
"Type" : "Task",
"Resource": "arn:aws:lambda:******:function:Zero",
"Next": "NextState"
},
"DefaultState": {
"Type" : "Task",
"Resource": "arn:aws:lambda:******:function:NotZero",
"Next": "NextState"
}
Let assume that input to this state is:
{
"value": 0,
"output1": object1,
"output2": object2,
}
My issue is that I have to pass output1 to ValueIsZero state and output2 to DefaultState. I know that it is possible to change InputPath in ValueIsZero and DefaultState states. But this way isn't acceptable for me because I am calling these states from some other states also.
I tried to modify ChoiceStateX state like next:
"ChoiceStateX": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.value",
"NumericEquals": 0,
"OutputPath": "$.output1",
"Next": "ValueIsZero"
}
],
"Default": "DefaultState"
}
I got next error in this case: Field OutputPath is not supported.
How is it possible to implement this functionality?
PS: In the current moment I am using 'proxy' states between ChoiceStateX and ValueIsZero/DefaultState where modifying the output.
I have checked:
Input and Output Processing
Choice
but haven't found a solution yet.
It looks like it isn't possible to specify different OutputPath for one state.
The solution with proxy states doesn't look graceful.
I have solved this issue in another way in the state before ChoiceStateX. I am setting instances of different types in output property and only route it on ChoiceStateX state.
My input of ChoiceStateX state looks like:
{
"value": value,
"output": value==0 ? object1 : object2
}
End final version of ChoiceStateX state:
"ChoiceStateX": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.value",
"NumericEquals": 0,
"Next": "ValueIsZero"
}
],
"OutputPath": "$.output",
"Default": "DefaultState"
}
It is still isn't perfect, because I implement the same logic in two places.

Loop inside a Step Function

I am trying to call a couple of steps in my step function in a loop but I am unable to get my head around how I need to do this. Here's what I have till now: I need to add another lambda function(GetReviews) which will then call CreateReview, SendNotification in a loop. How would I go about doing this?
I am referring to the "Iterating a Loop Using Lambda" document, which shows it is possible.
Step function Defination:
{
"Comment": "Scheduling Engine",
"StartAt": "CreateReview",
"States": {
"CreateReview": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:529627678433:function:CreateReview",
"Next": "CreateNotification",
"InputPath": "$",
"ResultPath": "$.CreateReviewResult",
"OutputPath": "$"
},
"CreateNotification": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:529627678433:function:CreateNotification",
"InputPath": "$",
"ResultPath": "$.CreateNotificationResult",
"OutputPath": "$",
"End": true
}
}
}
I am contributing to this answer because I am using a little different approach to be able to loop inside of a step-function without having to rely on a lambda to increment. If someone in the future needs a generic solution, this can be a good reference.
Here is the example with code:
{
"Comment": "A description of my state machine",
"StartAt": "InitVariables",
"States": {
"InitVariables": {
"Type": "Pass",
"Parameters": {
"index": 0,
"incrementor": 1,
"ArrayLength.$": "States.ArrayLength($.inputArray)"
},
"ResultPath": "$.iterator",
"Next": "LoopChoice"
},
"LoopChoice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.iterator.ArrayLength",
"NumericGreaterThanPath": "$.iterator.index",
"Next": "IncrementVariable"
}
],
"Default": "End"
},
"IncrementVariable": {
"Type": "Pass",
"Parameters": {
"index.$": "States.MathAdd($.iterator.index, $.iterator.incrementor)",
"incrementor": 1,
"ArrayLength.$": "$.iterator.ArrayLength"
},
"ResultPath": "$.iterator",
"Next": "LoopChoice"
},
"End": {
"Type": "Pass",
"End": true
}
} }
This is the base for the loop, I use the States.MathAdd($.iterator.index, $.iterator.incrementor) intrinsic function to add two values, in this case, increment the index with a increment amount defined in the initVariables state. And also get the length of the array that I want to loop. You get the array length by also using a intrinsic function, States.ArrayLength("$.path.to.array"). The array is passed in the input.
To get the value of the array we can use the intrinsic function, States.ArrayGetItem($.inputArray, $.iterator.index).
All the custom logic should be put between the loopChoice state and the IncrementVariable State.
Hope this helps someone in the future.
Sorry for the late reply. You've probably solved it in between, but here you are
So, when looping in Step Functions, I quite simply add a Choice State (see Choice State Rules).
One of your State would need to output wether or not you have finished looping, or the number of items iterated on and the total number of items.
In the first case, it would be something like
{
"Comment": "Scheduling Engine",
"StartAt": "CreateReview",
"States": {
"GetReviews": {
whatever
"Next": "LoopChoiceState"
},
"LoopChoiceState": {
"Type" : "Choice",
"Choices": [
{
"Variable": "$.loopCompleted",
"BooleanEquals": false,
"Next": "GetReviews"
}
],
"Default": "YourEndState"
},
"CreateReview": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-westz2:529627678433:function:CreateReview",
"Next": "CreateNotification",
"InputPath": "$",
"ResultPath": "$.CreateReviewResult",
"OutputPath": "$"
},
"CreateNotification": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:529627678433:function:CreateNotification",
"InputPath": "$",
"ResultPath": "$.CreateNotificationResult",
"OutputPath": "$",
"End": true
}
}
}
Second case:
{
"Comment": "Scheduling Engine",
"StartAt": "CreateReview",
"States": {
"GetReviews": {
whatever
"Next": "LoopChoiceState"
},
"LoopChoiceState": {
"Type" : "Choice",
"Choices": [
{
"Variable": "$.iteratedItemsCount",
"NumericEquals": "$.totalItemsCount",
"Next": "CreateNotification"
}
],
"Default": "CreateReview"
},
"CreateReview": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:529627678433:function:CreateReview",
"Next": "CreateNotification",
"InputPath": "$",
"ResultPath": "$.CreateReviewResult",
"OutputPath": "$"
},
"CreateNotification": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:529627678433:function:CreateNotification",
"InputPath": "$",
"ResultPath": "$.CreateNotificationResult",
"OutputPath": "$",
"End": true
}
}
}
You could also use indexes (current index and last index) instead of the the number of items iterated over; it would help you keep track of where you in create reviews.
Three options are presented below. Here is a visual summary:
#1 Map: Repeat a set of steps for each element of an array (without a loop, optionally concurrently)
Map State is an alternative to looping when you want to run a set of steps for each element of an input array. Each element runs in parallel by default. Set MaxConcurrency: 1 to mimic the serial execution of #NunoGamaFreire's loop-based solution.
{
"StartAt": "MockArray",
"States": {
"MockArray": {
"Type": "Pass",
"Result": [ { "name": "Zaphod" }, { "name": "Arthur" }, { "name": "Trillian" } ],
"ResultPath": "$.Items",
"Next": "MapState"
},
"MapState": {
"Type": "Map",
"ResultPath": "$.MapResult",
"End": true,
"Iterator": {
"StartAt": "MockWork",
"States": {
"MockWork": {
"Type": "Pass",
"Parameters": {
"output.$": "States.Format('Hello, {}!', $.name)"
},
"OutputPath": "$.output",
"End": true
}
}
},
"ItemsPath": "$.Items"
}
}
}
One output is produced for each element in the MockArray, processed concurrently:
"MapResult": [ "Hello, Zaphod!", "Hello, Arthur!", "Hello, Trillian!" ]
#2 Repeat a set of steps X times (loop without lambda, serially)
This option involves proper looping! Repeat a set of tasks serially X number of times until a Choice State determines that an incrementing counter variable has reached X. Use the new States.MathAdd intrinsic function to increment without a Lambda. This option is suited for custom retry logic or other cases when you may want to break the loop early.
{
"StartAt": "InitializeCounter",
"States": {
"InitializeCounter": {
"Type": "Pass",
"Comment": "Initialize the counter at 0. Move all inputs to Payload.Input",
"Parameters": {
"Counter": 0
},
"Next": "IncrementCounter"
},
"IncrementCounter": {
"Type": "Pass",
"Comment": "Increment the Counter by 1",
"Parameters": {
"Counter.$": "States.MathAdd($.Counter, 1)"
},
"Next": "MockWork"
},
"MockWork": {
"Type": "Pass",
"Comment": "Simulate some work. Optionally break early from the loop with ExitNow: true",
"Result": false,
"ResultPath": "$.ExitNow",
"Next": "Loop?"
},
"Loop?": {
"Type": "Choice",
"Choices": [{ "Or": [
{ "Variable": "$.Counter", "NumericGreaterThanEqualsPath": "$$.Execution.Input.workCount" },
{ "Variable": "$.ExitNow", "BooleanEquals": true } ],
"Next": "Success"
}
],
"Default": "IncrementCounter"
},
"Success": {
"Type": "Succeed"
}
},
"TimeoutSeconds": 3
}
Counter iterates by one for each loop. The loop breaks if a task returns ExitNow: true.
{ "Counter": 4, "ExitNow": false }
#3 Repeat a set of steps X times (without a loop, *concurrently*)
This option is a hybrid of the first two. Like #2, we start with a desired number of iterations from the $.workCount input . Like #1, we map over an array concurrently. This time, though, the state machine creates the array with another intrinsic function, States.ArrayRange(1, $.workCount, 1).
{
"StartAt": "Iterations",
"States": {
"Iterations": {
"Type": "Pass",
"Parameters": {
"Iterations.$": "States.ArrayRange(1, $.workCount, 1)"
},
"Next": "MapState"
},
"MapState": {
"Type": "Map",
"ResultPath": "$.MapResult",
"End": true,
"Iterator": {
"StartAt": "MockWork",
"States": {
"MockWork": {
"Type": "Pass",
"Parameters": {
"output.$": "States.Format('Hello from iteration #{}', States.JsonToString($))"
},
"OutputPath": "$.output",
"End": true
}
}
},
"ItemsPath": "$.Iterations"
}
}
}
Tasks run concurrently, once for each item in Iterations.
"Iterations": [ 1, 2, 3, 4 ],
"MapResult": [ "Hello from iteration #1", "Hello from iteration #2", "Hello from iteration #3", "Hello from iteration #4" ]