Passing Input Parameters from one Step Function to another - amazon-web-services

I do have an Step Function A - which executes a lambda and pull some results.
Same Function has a Map Stage which is iterating over results and should call another Step Function from Map State.
While calling another Step Function B from the map state i am not able to pass the parameter or that one record as Input to Step Function B.
Please suggest how can i use Input for second step function.
Below is the example i am using , orderServiceResponse has a List of orders which I need to iterate and pass that one order to next step function.
"Validate-All" : {
"Type" : "Map",
"InputPath" : "$.orderServiceResponse",
"ItemsPath" : "$.orders",
"MaxConcurrency" : 5,
"ResultPath" : "$.orders",
"Iterator" : {
"StartAt" : "Validate" ,
"States" :{
"Validate" : {
"Type" : "Task"
"Resource" : "arn:aws:states:::states:startExecution.sync:2",
"Parameters" {
"Input" : "$.orders",
"StateMachineArn" : "{arn of Step Function B }
},
"End" : true
}
}

TL;DR Use Parameters with Map Context to add the full input object to each Map element iteration.
You have an array of data you want to process elementwise in a Map State. By default, Map only passes
the array element's data to the map iterator. But we can add additional context to each iteration.
Here is an example - the important bits are commented:
{
"StartAt": "MapState",
"States": {
"MapState": {
"Type": "Map",
"ResultPath": "$.MapResult",
"Next": "Success",
// the map's elements of each get the following:
"Parameters": {
"Index.$": "$$.Map.Item.Index", // the array element's data (we only get this by default)
"Order.$": "$$.Map.Item.Value", // the array element's index 0,1,2...
"FullInput.$": "$" // a copy of the the full input object <-- this is what you were looking for
},
"Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "Fail" }],
// substitute your iterator:
"Iterator": {
"StartAt": "MockTask",
"States": {
"MockTask": {
"End": true,
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxxxx",
"Parameters": {
"expression": "`Order ${$.Order.OrderID} was ordered by ${$.FullInput.CustomerName}`",
"expressionAttributeValues": {
"$.Order.OrderID.$": "$.Order.OrderID",
"$.FullInput.CustomerName.$": "$.FullInput.CustomerName"
}
}
}
}
},
"ItemsPath": "$.Orders"
},
"Success": { "Type": "Succeed" },
"Fail": { "Type": "Fail" }
}
}
Execution Input, 3 Orders:
{
"CustomerID": 1,
"CustomerName": "Morgan Freeman",
"OtherInfo": { "Foo": "Bar" },
"Orders": [{ "OrderID": "A", "Status": "Fulfilled" }, { "OrderID": "B", "Status": "Pending" }, { "OrderID": "C", "Status": "Cancelled" }]
}
Map Iteration 0 Input:
{
"Order": { "OrderID": "A", "Status": "Fulfilled" },
"Index": 0,
"FullInput": { "CustomerID": 1, "CustomerName": "Morgan Freeman", "OtherInfo": { "Foo": "Bar" }, "Orders": [{...
Execution Output MapResult key
{
"MapResult": [
"Order A was ordered by Morgan Freeman",
"Order B was ordered by Morgan Freeman",
"Order C was ordered by Morgan Freeman"
]
...
}

Related

Rendering Values in a Step Function: extract YYYY-MM from 2022-10-20

I am trying to get a date prefix from a date in an AWS Step Function. When I try to call a variable from a previous step, that variable doesn't render. Here is the step function code:
{
"Comment": "Set and use a new variable in Step Functions",
"StartAt": "Set Date Prefix",
"States": {
"Set Date Prefix": {
"Type": "Pass",
"Result": {
"date_prefix": "${$$.Execution.Input.date.substr(0,7)}"
},
"ResultPath": "$.date_prefix",
"Next": "Use Date Prefix"
},
"Use Date Prefix": {
"Type": "Pass",
"Result": {
"date_prefix_used": "$.date_prefix"
},
"End": true
}
}
}
When I pass the following input:
{
"date": "2022-10-20"
}
I get the following as output:
{
"date_prefix_used": "$.date_prefix"
}
when I should have gotten:
{
"date_prefix": "2022-10",
"date_prefix_used": "2022-10"
}
What am I doing wrong?
You can intinsically extract the YYYY-MM from a date string input like 2022-10-20 with two Pass states. A Pass state can apply substitutions and State Machine intrinsic functions, but in the Parameters field, not the Result field. Remember the .$ suffix on keys whose values have substitutions. Otherwise, the values will be treated as literals.
{
"Comment": "Set and use a new variable in Step Functions",
"StartAt": "SplitDate",
"States": {
"SplitDate": {
"Type": "Pass",
"Parameters": {
"date_components.$": "States.StringSplit($.date, '-')"
},
"Next": "GetPrefix"
},
"GetPrefix": {
"Type": "Pass",
"InputPath": "$.date_components",
"Parameters": {
"date_prefix.$": "States.Format('{}-{}',States.ArrayGetItem($, 0), States.ArrayGetItem($, 1))"
},
"End": true
}
}
}
The Split Date task splits the date with States.StringSplit.
{
"date_components": [ "2022", "10", "20" ]
}
The GetPrefix task reassembles the the year and month with States.Format and States.ArrayGetItem:
{
"date_prefix": "2022-10"
}

Access Map State ( item ) in Step functions

I am trying to access item properties which I am iterating over using Map state. I keep getting this error:
Value ($['Map'].snapshot_id.S) for parameter snapshotId is invalid. Expected: 'snap-...'. (Service: Ec2, Status Code: 400, Request ID: 6fc02935-c161-49df-8606-bc6f3e2934a6)
I have gone through the docs, which seems to suggest access using $.Map.snapshot_id.S but doesn't seem to work. Meanwhile, an input item to Map is:
{
"snapshot_id": {
"S": "snap-01dd5ee46df84119e"
},
"file_type": {
"S": "bash"
},
"id": {
"S": "64e6893261d94669b7a8ca425233d68b"
},
"script_s3_link": {
"S": "df -h"
}
}
Map state definition:
"Map": {
"Type": "Map",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "Parallel State",
"States": {
"Parallel State": {
"Comment": "A Parallel state can be used to create parallel branches of execution in your state machine.",
"Type": "Parallel",
"Branches": [
{
"StartAt": "CreateVolume",
"States": {
"CreateVolume": {
"Type": "Task",
"Parameters": {
"AvailabilityZone": "us-east-2b",
"SnapshotId": "$$.Map.snapshot_id.S"
},
"Resource": "arn:aws:states:::aws-sdk:ec2:createVolume",
"Next": "AttachVolume"
},
"AttachVolume": {
"Type": "Task",
"Parameters": {
"Device": "MyData",
"InstanceId": "MyData",
"VolumeId": "MyData"
},
"Resource": "arn:aws:states:::aws-sdk:ec2:attachVolume",
"End": true
}
}
}
],
"End": true
}
}
},
"Next": "Final Result",
"ItemsPath": "$.Items",
"MaxConcurrency": 40
},
TL;DR "SnapshotId.$": "$.snapshot_id"
By default, each inline map iteration's input is one item from the ItemsPath array, accessible simply as $.
Your state machine definition implies that $.Items is an array of objects with a snapshot_id key (and other keys). If so, each iteration's snapshot id is at $.snapshot_id.
Finally, adding .$ to the parameter name (SnapshotId.$) tells Step Functions the value is not a literal, but rather a path value to be substituted.

Access an Array Item by index in AWS Dynamodb Query Results "Items" in Step Function

I have this dynamodb:Query in my step function:
{
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:dynamodb:query",
"Next": "If nothing returned by query Or Study not yet Zipped",
"Parameters": {
"TableName": "TEST-StudyProcessingTable",
"ScanIndexForward": false,
"Limit": 1,
"KeyConditionExpression": "OrderID = :OrderID",
"FilterExpression": "StudyID = :StudyID",
"ExpressionAttributeValues": {
":OrderID": {
"S.$": "$.body.order_id"
},
":StudyID": {
"S.$": "$.body.study_id"
}
}
},
"ResultPath": "$.processed_files"
}
The results comes in as an array called Items which is nested under my ResultPath
processed_files.Items:
{
"body": {
"order_id": "1001",
"study_id": "1"
},
"processed_files": {
"Count": 1,
"Items": [
{
"Status": {
"S": "unzipped"
},
"StudyID": {
"S": "1"
},
"ZipFileS3Key": {
"S": "path/to/the/file"
},
"UploadSet": {
"S": "4"
},
"OrderID": {
"S": "1001"
},
"UploadSet#StudyID": {
"S": "4#1"
}
}
],
"LastEvaluatedKey": {
"OrderID": {
"S": "1001"
},
"UploadSet#StudyID": {
"S": "4#1"
}
},
"ScannedCount": 1
}
}
My question is how do i access the items inside this array from a choice state in a step function?
I need to query then decide something based on the results by checking the item in a condition in a choice state.
The problem is that since this is an array I can't access it using regular JsonPath (like with Items.item), and in my next step the choice condition does NOT accept an index like processed_files.Items['0'].Status
Ok so the answer was so simple all you need to do is use a number instead of string for the array index like this.
processed_files.Items[0].Status
I was originally mislead by an error I received which said that it expected a ' or '[' after the first '['. I mistakenly thought this meant it only accepts strings.
I was wrong, it works like any other array.
I hope this helps somebody one day.

How to use jsonPath inside array in AWS Step Functions

I am writing an AWS step function, and for one of the steps, I wish to call a lambda that accepts an array as one of the inputs. However, if I try to pass in a JsonPath into the array, I get
The value for the field 'arrayField.$' must be a STRING that contains a JSONPath but was an ARRAY
My step function definition:
{
"StartAt": "First",
"States": {
"First": {
"Type": "Pass",
"Parameters": {
"type": "person"
},
"ResultPath": "$.output",
"Next": "Second"
},
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
"Parameters": {
"regularParameter": "some string",
"arrayParameter.$": ["$.output.type"]
},
"Next": "Succeed"
},
"Succeed": {
"Type": "Succeed"
}
}
}
How can I use jsonPath inside the array?
Since a new release you could use the intrinsic function States.Array:
"arrayParameter.$": "States.Array($.output.type)"
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html
As #Seth Miller mentioned JsonPath resolution within arrays doesn't work unfortunately. If the amount of values to replace in the array is small and known there's a simple workaround (in my case I needed an array of size 1).
The steps are:
Initialise the array with the number of values you need;
Replace each value using "ResultPath": "$.path.to.array[n]";
Use "$.path.to.array" in your task.
Simple, working example:
{
"StartAt": "First",
"States": {
"First": {
"Type": "Pass",
"Parameters": {
"type": "person"
},
"ResultPath": "$.output",
"Next": "Initialise Array"
},
"Initialise Array": {
"Comment": "Add an entry for each value you intend to have in the final array, the values here don't matter.",
"Type": "Pass",
"Parameters": [
0
],
"ResultPath": "$.arrayParameter",
"Next": "Fill Array"
},
"Fill Array": {
"Comment": "Replace the first entry of array with parameter",
"Type": "Pass",
"InputPath": "$.output.type",
"ResultPath": "$.arrayParameter[0]",
"End": true
}
}
}
And to use the resulting array in your task example:
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
"Parameters": {
"regularParameter": "some string",
"arrayParameter.$": "$.arrayParameter"
},
"Next": "Succeed"
},
Another way to approach this is by using Parallel state that outputs an array of objects and then use jsonPath to convert it to a simple array:
{
"StartAt": "Parallel",
"States": {
"Parallel": {
"Type": "Parallel",
"Next": "Use Array",
"ResultPath": "$.items",
"Branches": [
{
"StartAt": "CreateArray",
"States": {
"CreateArray": {
"Type": "Pass",
"Parameters": {
"value": "your value"
},
"End": true
}
}
}
]
},
"Use Array": {
"Type": "Pass",
"Parameters": {
"items.$": "$.items[*].value"
},
"End": true
}
}
}
In this example, Parallel state outputs the following json:
{
"items": [
{
"value": "your value"
}
]
}
And "Use Array" state produces:
{
"items": [
"your value"
]
}
JSONPath inside parameters field need to be a string. So if you want to pass to lambda function a parameter called arrayParameter, you´ll need to make a jsonPath query that extract that array.
For example, if inside the key output is a key called outputArray with the array as its value.
Input JSON:
{
"pre": "sdigf",
"output": {
"result": 1,
"outputArray": ["test1","test2","test.."]
}
}
The parameter sintax:
"arrayParameter.$": "$.output.outputArray"
Reasonable advice
I ran into a use case for JsonPath resolution within arrays today and found (like you have) that the functionality does not exist today. I ended up deciding that doing the data massaging in code was simpler and cleaner. For example, you could create a small Lambda that takes in the object emitted by First and massages it to a format acceptable to Second and adds it to the output (WaterKnight mentions this solution in a comment to another question).
This assumes that you are, for some reason, unable to change the format of the input to that Lambda in Second (which would be the absolute shortest path here).
Unreasonable advice
That said, if you want a way to do this completely within Step Functions that is fairly gross, you can use the result of a Map state that executes Pass states. The output of the Map state is an array that aggregate the output of each constituent Pass state. These Pass states simply emit the value(s) you want in the final array using the Parameters attribute. An example Step Function definition follows. I did warn that it is gross and that I went a different way to solve the problem.
{
"StartAt": "First",
"Comment": "Please don't actually do this",
"States": {
"First": {
"Type": "Pass",
"Parameters": {
"type": "person"
},
"ResultPath": "$.output",
"Next": "Add Array"
},
"Add Array": {
"Comment": "A Map state needs some array to loop over in order to work. We will give it a dummy array. Add an entry for each value you intend to have in the final array. The values here don't matter.",
"Type": "Pass",
"Result": [
0
],
"ResultPath": "$.dummy",
"Next": "Mapper"
},
"Mapper": {
"Comment": "Add a Pass state with the appropriate Parameters for each field you want to map into the output array",
"Type": "Map",
"InputPath": "$",
"ItemsPath": "$.dummy",
"Parameters": {
"output.$": "$.output"
},
"Iterator": {
"StartAt": "Massage",
"States": {
"Massage": {
"Type": "Pass",
"Parameters": {
"type.$": "$.output.type"
},
"OutputPath": "$.type",
"End": true
}
}
},
"ResultPath": "$.output.typeArray",
"Next": "Second"
},
"Second": {
"Comment": "The Lambda in your example is replaced with Pass so that I could test this",
"Type": "Pass",
"Parameters": {
"regularParameter": "some string",
"arrayParameter.$": "$.output.typeArray"
},
"Next": "Succeed"
},
"Succeed": {
"Type": "Succeed"
}
}
}
As many answers correctly pointed out, it's not possible to do it exactly the way you need. But I would suggest another solution: an array of dictionaries. It's not exactly what you need, but is native and not hacky.
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
"Parameters": {
"regularParameter": "some string",
"arrayParameter": [{"type.$": "$.output.type"}]
},
"Next": "Succeed"
},
The result would be
{
"regularParameter": "some string",
"arrayParameter": [{"type": "SingleItemWrappedToAnArray"}]
}

Merging JSON outputs of parallel states in Step Function

I have a parallel block in a Step Function. The input to the parallel block looks like
{ "a": null, "b": null }
The 2 parallel states are populateA and populateB. The output from populateA is
{ "a": "A", "b": null }
And the output from populateB is
{ "a": null, "b": "B" }
The output from the parallel block is an array
[ { "a": "A", "b": null }, { "a": null, "b": "B" } ]
How do I merge the two into
{ "a": "A", "b": "B" }
If PopulateA always only populates the value of a and PopulateB only the value of b, you can use a Pass state with Parameters (instead of the Result field) after your Parallel State to combine the inputs. Parameters allow you to transform your input by specifying key-value pairs using JSONPath.
Here is an example State Machine:
{
"StartAt": "Parallel",
"States": {
"Parallel": {
"Type": "Parallel",
"ResultPath": "$.CombinedOutput",
"Next": "MergeOutputs",
"Branches": [{
"StartAt": "populateA",
"States": {
"populateA": {
"Type": "Pass",
"Result": {
"a": "A",
"b": null
},
"End": true
}
}
},
{
"StartAt": "populateB",
"States": {
"populateB": {
"Type": "Pass",
"Result": {
"a": null,
"b": "B"
},
"End": true
}
}
}
]
},
"MergeOutputs": {
"Type": "Pass",
"Parameters": {
"a.$": "$.CombinedOutput[0].a",
"b.$": "$.CombinedOutput[1].b"
},
"Next": "EndState"
},
"EndState": {
"Type": "Pass",
"End": true
}
}
}
I'm afraid that it is not possible (currently) to do that without additional state. You can add additional Task after Parallel state which will merge array elements.
The code of the additional task would look like this (Python):
def merge_dicts(a, b):
result = a.copy()
for key, value in b.items():
if value is not None:
result[key] = value
return result
def handler(event, context):
result = {}
for item in event:
result = merge_dicts(result, item)
return result
handler takes output from the Parallel states and merges it into single dictionary. The state should be directly after Parallel.
Keep in mind that this code handles only flat-dict (not nested ones) so it will work in simple cases but not in complex ones (I've added it here only to show the general idea).