Rendering Values in a Step Function: extract YYYY-MM from 2022-10-20 - amazon-web-services

I am trying to get a date prefix from a date in an AWS Step Function. When I try to call a variable from a previous step, that variable doesn't render. Here is the step function code:
{
"Comment": "Set and use a new variable in Step Functions",
"StartAt": "Set Date Prefix",
"States": {
"Set Date Prefix": {
"Type": "Pass",
"Result": {
"date_prefix": "${$$.Execution.Input.date.substr(0,7)}"
},
"ResultPath": "$.date_prefix",
"Next": "Use Date Prefix"
},
"Use Date Prefix": {
"Type": "Pass",
"Result": {
"date_prefix_used": "$.date_prefix"
},
"End": true
}
}
}
When I pass the following input:
{
"date": "2022-10-20"
}
I get the following as output:
{
"date_prefix_used": "$.date_prefix"
}
when I should have gotten:
{
"date_prefix": "2022-10",
"date_prefix_used": "2022-10"
}
What am I doing wrong?

You can intinsically extract the YYYY-MM from a date string input like 2022-10-20 with two Pass states. A Pass state can apply substitutions and State Machine intrinsic functions, but in the Parameters field, not the Result field. Remember the .$ suffix on keys whose values have substitutions. Otherwise, the values will be treated as literals.
{
"Comment": "Set and use a new variable in Step Functions",
"StartAt": "SplitDate",
"States": {
"SplitDate": {
"Type": "Pass",
"Parameters": {
"date_components.$": "States.StringSplit($.date, '-')"
},
"Next": "GetPrefix"
},
"GetPrefix": {
"Type": "Pass",
"InputPath": "$.date_components",
"Parameters": {
"date_prefix.$": "States.Format('{}-{}',States.ArrayGetItem($, 0), States.ArrayGetItem($, 1))"
},
"End": true
}
}
}
The Split Date task splits the date with States.StringSplit.
{
"date_components": [ "2022", "10", "20" ]
}
The GetPrefix task reassembles the the year and month with States.Format and States.ArrayGetItem:
{
"date_prefix": "2022-10"
}

Related

Can I access the TaskToken from a Map state with ItemSelector where the iteration step uses lambda:invoke.waitForTaskToken?

I am using AWS step function to iterate over a list in an input document where for each iteration, I need to invoke an external service. So I want to iterate over each item and run a step using lambda:invoke.waitForTaskToken and pass the TaskToken into the execution of each iteration.
The problem I'm running into is how to use both an ItemSelector at the Map state level but also inject the TaskToken during the internal step. I need to use an ItemSelector because I want each item to also contain information from the input to Map state. The AWS Docs state:
The ItemSelector field replaces the Parameters field within the Map state. If you use the Parameters field in your Map state definitions to create custom input, we highly recommend that you replace them with ItemSelector.
But they also say:
During an execution, the context object is populated with relevant data for the Parameters field from where it is accessed. The value for a Task field is null if the Parameters field is outside of a task state.
These two statements seem to imply that what I'm trying to do is impossible.
So, what I want is something like:
{
"StartAt": "ExampleMapState",
"States": {
"ExampleMapState": {
"Type": "Map",
"ItemsPath": "$.items",
"ItemSelector": {
"dynamic.$": "$.dynamic",
"ContextIndex.$": "$$.Map.Item.Index",
"ContextValue.$": "$$.Map.Item.Value"
},
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "TestPass",
"States": {
"TestPass": {
"Type": "Task",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-west-2:123456789012:function:echo-lambda",
"Payload": {
"item.$": "$",
"token.$": "$$.Task.Token"
}
},
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"End": true
}
}
},
"End": true
}
}
}
But this doesn't work because the ItemSelector overrides the Payload of the internal TestPass state. Is there a way to get this to work?
ETA: I figured I would try putting $$.Task.Token in ItemSelector just in case it would magically work but it ended up throwing an error because $$.Task does not exist in the context object at that level.
Example with this (invalid) configuration:
{
"StartAt": "ExampleMapState",
"States": {
"ExampleMapState": {
"Type": "Map",
"ItemsPath": "$.items",
"ItemSelector": {
"dynamic.$": "$.dynamic",
"ContextIndex.$": "$$.Map.Item.Index",
"ContextValue.$": "$$.Map.Item.Value",
"token.$": "$$.Task.Token"
},
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "TestPass",
"States": {
"TestPass": {
"Type": "Task",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-west-2:123456789012:function:echo-lambda"
},
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"End": true
}
}
},
"End": true
}
}
}
Based on my research I don't think what I'm trying to do is possible. What I ended up implementing is a workaround. I modified the function providing input to this step function to put the dynamic info that I needed into every item in the list I am iterating over. So my step function definition now looks something like this
{
"StartAt": "ExampleMapState",
"States": {
"ExampleMapState": {
"Type": "Map",
"ItemsPath": "$.items",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "TestPass",
"States": {
"TestPass": {
"Type": "Task",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-west-2:123456789012:function:echo-lambda",
"Payload": {
"item.$": "$",
"token.$": "$$.Task.Token"
}
},
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"End": true
}
}
},
"End": true
}
}
}
And an example input to this step function looks like:
{
"dynamic": "info",
"items": [
{
"dynamic": "info",
"resize": "true",
"format": "jpg"
},
{
"dynamic": "info",
"resize": "false",
"format": "png"
},
{
"dynamic": "info",
"resize": "true",
"format": "jpg"
}
]
}
It's not great because I have to repeat info into every item ahead of time but it works.

Access Map State ( item ) in Step functions

I am trying to access item properties which I am iterating over using Map state. I keep getting this error:
Value ($['Map'].snapshot_id.S) for parameter snapshotId is invalid. Expected: 'snap-...'. (Service: Ec2, Status Code: 400, Request ID: 6fc02935-c161-49df-8606-bc6f3e2934a6)
I have gone through the docs, which seems to suggest access using $.Map.snapshot_id.S but doesn't seem to work. Meanwhile, an input item to Map is:
{
"snapshot_id": {
"S": "snap-01dd5ee46df84119e"
},
"file_type": {
"S": "bash"
},
"id": {
"S": "64e6893261d94669b7a8ca425233d68b"
},
"script_s3_link": {
"S": "df -h"
}
}
Map state definition:
"Map": {
"Type": "Map",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "Parallel State",
"States": {
"Parallel State": {
"Comment": "A Parallel state can be used to create parallel branches of execution in your state machine.",
"Type": "Parallel",
"Branches": [
{
"StartAt": "CreateVolume",
"States": {
"CreateVolume": {
"Type": "Task",
"Parameters": {
"AvailabilityZone": "us-east-2b",
"SnapshotId": "$$.Map.snapshot_id.S"
},
"Resource": "arn:aws:states:::aws-sdk:ec2:createVolume",
"Next": "AttachVolume"
},
"AttachVolume": {
"Type": "Task",
"Parameters": {
"Device": "MyData",
"InstanceId": "MyData",
"VolumeId": "MyData"
},
"Resource": "arn:aws:states:::aws-sdk:ec2:attachVolume",
"End": true
}
}
}
],
"End": true
}
}
},
"Next": "Final Result",
"ItemsPath": "$.Items",
"MaxConcurrency": 40
},
TL;DR "SnapshotId.$": "$.snapshot_id"
By default, each inline map iteration's input is one item from the ItemsPath array, accessible simply as $.
Your state machine definition implies that $.Items is an array of objects with a snapshot_id key (and other keys). If so, each iteration's snapshot id is at $.snapshot_id.
Finally, adding .$ to the parameter name (SnapshotId.$) tells Step Functions the value is not a literal, but rather a path value to be substituted.

Passing Input Parameters from one Step Function to another

I do have an Step Function A - which executes a lambda and pull some results.
Same Function has a Map Stage which is iterating over results and should call another Step Function from Map State.
While calling another Step Function B from the map state i am not able to pass the parameter or that one record as Input to Step Function B.
Please suggest how can i use Input for second step function.
Below is the example i am using , orderServiceResponse has a List of orders which I need to iterate and pass that one order to next step function.
"Validate-All" : {
"Type" : "Map",
"InputPath" : "$.orderServiceResponse",
"ItemsPath" : "$.orders",
"MaxConcurrency" : 5,
"ResultPath" : "$.orders",
"Iterator" : {
"StartAt" : "Validate" ,
"States" :{
"Validate" : {
"Type" : "Task"
"Resource" : "arn:aws:states:::states:startExecution.sync:2",
"Parameters" {
"Input" : "$.orders",
"StateMachineArn" : "{arn of Step Function B }
},
"End" : true
}
}
TL;DR Use Parameters with Map Context to add the full input object to each Map element iteration.
You have an array of data you want to process elementwise in a Map State. By default, Map only passes
the array element's data to the map iterator. But we can add additional context to each iteration.
Here is an example - the important bits are commented:
{
"StartAt": "MapState",
"States": {
"MapState": {
"Type": "Map",
"ResultPath": "$.MapResult",
"Next": "Success",
// the map's elements of each get the following:
"Parameters": {
"Index.$": "$$.Map.Item.Index", // the array element's data (we only get this by default)
"Order.$": "$$.Map.Item.Value", // the array element's index 0,1,2...
"FullInput.$": "$" // a copy of the the full input object <-- this is what you were looking for
},
"Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "Fail" }],
// substitute your iterator:
"Iterator": {
"StartAt": "MockTask",
"States": {
"MockTask": {
"End": true,
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxxxx",
"Parameters": {
"expression": "`Order ${$.Order.OrderID} was ordered by ${$.FullInput.CustomerName}`",
"expressionAttributeValues": {
"$.Order.OrderID.$": "$.Order.OrderID",
"$.FullInput.CustomerName.$": "$.FullInput.CustomerName"
}
}
}
}
},
"ItemsPath": "$.Orders"
},
"Success": { "Type": "Succeed" },
"Fail": { "Type": "Fail" }
}
}
Execution Input, 3 Orders:
{
"CustomerID": 1,
"CustomerName": "Morgan Freeman",
"OtherInfo": { "Foo": "Bar" },
"Orders": [{ "OrderID": "A", "Status": "Fulfilled" }, { "OrderID": "B", "Status": "Pending" }, { "OrderID": "C", "Status": "Cancelled" }]
}
Map Iteration 0 Input:
{
"Order": { "OrderID": "A", "Status": "Fulfilled" },
"Index": 0,
"FullInput": { "CustomerID": 1, "CustomerName": "Morgan Freeman", "OtherInfo": { "Foo": "Bar" }, "Orders": [{...
Execution Output MapResult key
{
"MapResult": [
"Order A was ordered by Morgan Freeman",
"Order B was ordered by Morgan Freeman",
"Order C was ordered by Morgan Freeman"
]
...
}

aws step function parallel with input parameters

I am trying to use AWS step functions to create parallel branches of execution.
One of the parallel branches starts another step function invocation, how can we pass input from this parallel branch to next step function execution
{
"Comment": "Parallel Example.",
"StartAt": "FunWithMath",
"States": {
"FunWithMath": {
"Type": "Parallel",
"End": true,
"Branches": [
{
"StartAt": "Add", /// This receives some json object here input {}
"States": {
"Add": {
"Type": "Task", ***//How to pass the received input to the following arn as input?***
"Resource": ""arn:aws:states:::states:startExecution",
Parameters: {
"StateMachineArn": "anotherstepfunctionarnpath"
}
"End": true
}
}
},
{
"StartAt": "Subtract",
"States": {
"Subtract": {
"Type": "Task",
"Resource": "some lambda arn here,
"End": true
}
}
}
]
}
}
}
anotherstepfunctionarnpath :
{
"Comment": "Second state machine",
"StartAt": "stage1",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters":{
"Arguments":{
"Variable1" :"???" / how to access the value of the input passed to here
}
}
}
You can use Input to pass output from one SFN to other one:
First SFN(It will call second SFN)
{
"Comment": "My first SFN",
"StartAt": "First SFN",
"States": {
"First SFN": {
"Type": "Task",
"ResultPath": "$.to_pass",
"Resource": "arn:aws:lambda:us-east-1:807278658150:function:test-lambda",
"Next": "Trigger Next SFN"
},
"Trigger Next SFN": {
"Type": "Task",
"Resource": "arn:aws:states:::states:startExecution",
"Parameters": {
"Input": {
"Comment.$": "$"
},
"StateMachineArn": "arn:aws:states:us-east-1:807278658150:stateMachine:MyStateMachine2"
},
"End": true
}
}
}
Second SFN (MyStateMachine2)
{
"Comment": "A Hello World example of the Amazon States Language using Pass states",
"StartAt": "Hello",
"States": {
"Hello": {
"Type": "Pass",
"Result": "Hello",
"Next": "World"
},
"World": {
"Type": "Pass",
"Result": "World",
"End": true
}
}
}
First SFN's Execution
Second SFN's Execution
Explanation
The Lambda test-lambda is returning:
{
"user": "stackoverflow",
"id": "100"
}
Which is stored in "ResultPath": "$.to_pass" here in to_pass variable. I am passing the same output to next state machine MyStateMachine2 which is done by
"Input": {
"Comment.$": "$"
}
In the next State Machine's execution you see that same data is received as input which was created by first Lambda.
You can read more about it here.

How to use jsonPath inside array in AWS Step Functions

I am writing an AWS step function, and for one of the steps, I wish to call a lambda that accepts an array as one of the inputs. However, if I try to pass in a JsonPath into the array, I get
The value for the field 'arrayField.$' must be a STRING that contains a JSONPath but was an ARRAY
My step function definition:
{
"StartAt": "First",
"States": {
"First": {
"Type": "Pass",
"Parameters": {
"type": "person"
},
"ResultPath": "$.output",
"Next": "Second"
},
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
"Parameters": {
"regularParameter": "some string",
"arrayParameter.$": ["$.output.type"]
},
"Next": "Succeed"
},
"Succeed": {
"Type": "Succeed"
}
}
}
How can I use jsonPath inside the array?
Since a new release you could use the intrinsic function States.Array:
"arrayParameter.$": "States.Array($.output.type)"
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html
As #Seth Miller mentioned JsonPath resolution within arrays doesn't work unfortunately. If the amount of values to replace in the array is small and known there's a simple workaround (in my case I needed an array of size 1).
The steps are:
Initialise the array with the number of values you need;
Replace each value using "ResultPath": "$.path.to.array[n]";
Use "$.path.to.array" in your task.
Simple, working example:
{
"StartAt": "First",
"States": {
"First": {
"Type": "Pass",
"Parameters": {
"type": "person"
},
"ResultPath": "$.output",
"Next": "Initialise Array"
},
"Initialise Array": {
"Comment": "Add an entry for each value you intend to have in the final array, the values here don't matter.",
"Type": "Pass",
"Parameters": [
0
],
"ResultPath": "$.arrayParameter",
"Next": "Fill Array"
},
"Fill Array": {
"Comment": "Replace the first entry of array with parameter",
"Type": "Pass",
"InputPath": "$.output.type",
"ResultPath": "$.arrayParameter[0]",
"End": true
}
}
}
And to use the resulting array in your task example:
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
"Parameters": {
"regularParameter": "some string",
"arrayParameter.$": "$.arrayParameter"
},
"Next": "Succeed"
},
Another way to approach this is by using Parallel state that outputs an array of objects and then use jsonPath to convert it to a simple array:
{
"StartAt": "Parallel",
"States": {
"Parallel": {
"Type": "Parallel",
"Next": "Use Array",
"ResultPath": "$.items",
"Branches": [
{
"StartAt": "CreateArray",
"States": {
"CreateArray": {
"Type": "Pass",
"Parameters": {
"value": "your value"
},
"End": true
}
}
}
]
},
"Use Array": {
"Type": "Pass",
"Parameters": {
"items.$": "$.items[*].value"
},
"End": true
}
}
}
In this example, Parallel state outputs the following json:
{
"items": [
{
"value": "your value"
}
]
}
And "Use Array" state produces:
{
"items": [
"your value"
]
}
JSONPath inside parameters field need to be a string. So if you want to pass to lambda function a parameter called arrayParameter, you´ll need to make a jsonPath query that extract that array.
For example, if inside the key output is a key called outputArray with the array as its value.
Input JSON:
{
"pre": "sdigf",
"output": {
"result": 1,
"outputArray": ["test1","test2","test.."]
}
}
The parameter sintax:
"arrayParameter.$": "$.output.outputArray"
Reasonable advice
I ran into a use case for JsonPath resolution within arrays today and found (like you have) that the functionality does not exist today. I ended up deciding that doing the data massaging in code was simpler and cleaner. For example, you could create a small Lambda that takes in the object emitted by First and massages it to a format acceptable to Second and adds it to the output (WaterKnight mentions this solution in a comment to another question).
This assumes that you are, for some reason, unable to change the format of the input to that Lambda in Second (which would be the absolute shortest path here).
Unreasonable advice
That said, if you want a way to do this completely within Step Functions that is fairly gross, you can use the result of a Map state that executes Pass states. The output of the Map state is an array that aggregate the output of each constituent Pass state. These Pass states simply emit the value(s) you want in the final array using the Parameters attribute. An example Step Function definition follows. I did warn that it is gross and that I went a different way to solve the problem.
{
"StartAt": "First",
"Comment": "Please don't actually do this",
"States": {
"First": {
"Type": "Pass",
"Parameters": {
"type": "person"
},
"ResultPath": "$.output",
"Next": "Add Array"
},
"Add Array": {
"Comment": "A Map state needs some array to loop over in order to work. We will give it a dummy array. Add an entry for each value you intend to have in the final array. The values here don't matter.",
"Type": "Pass",
"Result": [
0
],
"ResultPath": "$.dummy",
"Next": "Mapper"
},
"Mapper": {
"Comment": "Add a Pass state with the appropriate Parameters for each field you want to map into the output array",
"Type": "Map",
"InputPath": "$",
"ItemsPath": "$.dummy",
"Parameters": {
"output.$": "$.output"
},
"Iterator": {
"StartAt": "Massage",
"States": {
"Massage": {
"Type": "Pass",
"Parameters": {
"type.$": "$.output.type"
},
"OutputPath": "$.type",
"End": true
}
}
},
"ResultPath": "$.output.typeArray",
"Next": "Second"
},
"Second": {
"Comment": "The Lambda in your example is replaced with Pass so that I could test this",
"Type": "Pass",
"Parameters": {
"regularParameter": "some string",
"arrayParameter.$": "$.output.typeArray"
},
"Next": "Succeed"
},
"Succeed": {
"Type": "Succeed"
}
}
}
As many answers correctly pointed out, it's not possible to do it exactly the way you need. But I would suggest another solution: an array of dictionaries. It's not exactly what you need, but is native and not hacky.
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
"Parameters": {
"regularParameter": "some string",
"arrayParameter": [{"type.$": "$.output.type"}]
},
"Next": "Succeed"
},
The result would be
{
"regularParameter": "some string",
"arrayParameter": [{"type": "SingleItemWrappedToAnArray"}]
}