The AWS States Language specification describes the role of the InputPath and Parameters fields but does not give an example of the filters being used together.
My understanding is that, if specified, the JSON path given by the InputPath field is applied to the raw input producing the effective input. Then, if specified, the value of the parameters field is applied, modifying the effective input.
Extending the example given in the spec, given the following Task state definition:
"X": {
"Type": "Task",
"Resource": "arn:aws:swf:us-east-1:123456789012:task:X",
"Next": "Y",
"InputPath": "$.sub",
"Parameters": {
"flagged": true,
"parts": {
"first.$": "$.vals[0]",
"last3.$": "$.vals[3:]"
}
}
}
then, given the following input:
{
"flagged": 7,
"sub" : {
"vals": [0, 10, 20, 30, 40, 50]
}
}
the effective input to the code identified in the Resource field would be:
{
"flagged": true,
"parts": {
"first": 0,
"last3": [30, 40, 50]
}
}
Is my interpretation correct?
It is totally correct. Parameters is a Payload Template created to reshape input data to meet the format expectations of tasks, while ResultSelector is doing the same thing but for output data.
The value of "Parameters" MUST be a Payload Template which is a JSON object, whose input is the result of applying the InputPath to the raw input. If the "Parameters" field is provided, its payload, after the extraction and embedding, becomes the effective input.
Also sometimes specs could be a bit hard to read, a visual graph might be helpful then
Related
Basically, i have the following input:
{
"name": "abc",
"choice": "choice1"
}
My dynamoDB table has the following structure:
Partition key - "name"
Complex json with choices:
{
"choices":
{
"choice1": ......,
"choice2": ......
}
}
I want to directly read from dynamodb, and get a subitem under the relevant choice:
{
"StartAt": "Read Next Message from DynamoDB",
"States": {
"Read Next Message from DynamoDB": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:getItem",
"Parameters": {
"TableName": "my_table",
"Key": {
"customerName": {"S.$": "$.name"}
}
},
"OutputPath": "$.Item.choices.M.choice1.M.myvalue.S",
"Next": "World"
},
"World": {
"Type": "Pass",
"End": true
}
}
}
basically i want to do something like "$.Item.choices.M.{$.choice}.M.myvalue.S", and take one of the output's keys from the input. is this possible?
I think what you're looking for is JsonPath interpolation, but that is not supported as per this thread on AWS forums.
As far as I know Step Functions allow only path reference through $, . and [] operators (Reference Path).
I don't know how much control you have on the DynamoDB table's data but I think your problem can be solved easily if your choice types are modeled in following way
{
"choices": [{
"choiceType": "choice1",
........
},
{
"choiceType": "choice2",
........
}]
}
Now you can use the map state to iterate over the choices array. Note that don't forget to pass the expected choiceType to each iteration.
First state of the map iterator can be a choice state which compares choiceType and moves to appropriate next state. So, basically your rest of the workflow is modeled as iterator of the map state in step 1.
Now, if you don't have the control over DynamoDB table, then you can process the query result in an AWS Lambda.
How can I pass my input to my output in a task in AWS Step Functions?
I'm aware of this question, and the docs:
If the value of ResultPath is null, that means that the state’s own raw output is discarded and its raw input becomes its result.
But what I need is:
Given my input
{
"input": "my_input"
}
And my lambda output
{
"output": "my_output"
}
I need to pass to the next state the following json:
{
"input": "my_input",
"output": "my_output"
}
Two suggestions comes to mind, either Use ResultPath to Replace the Input with the Result, which allows you to
If you don't specify a ResultPath, the default behavior is as if you had specified "ResultPath": "$". Because this tells the state to replace the entire input with the result, the state input is completely replaced by the result coming from the task result.
For this option to work, the Lambda function must return the desired response:
{
"input": "my_input",
"output": "my_output"
}
Alternatively Use ResultPath to Include the Result with the Input in the Step Functions developer guide. Next, if if you change the return value from you Lambda to include just "my_output" you can specify "ResultPath": "$.output" to achieve the desired result:
{
"input": "my_input",
"output": "my_output"
}
For anyone who comes to this question, it's not possible to do what I asked, but what you can do is, according to the docs:
{
"States": {
"my-state": {
"Type": "task",
"ResultPath": "$.response"
}
}
}
This will append whatever the lambda returns into the response node, resulting in:
{
"input": "my_input",
"response": {
"output": "my_output"
}
}
For anyone coming to this question:
It is possible to do this using ResultSelector and Intrinsic Functions and Context Object.
There is no need to change output.
With input given as:
{ "input_key": "input_value" }
and state output given as:
{ "output_key": "output_value" }
then in the state add the following fields:
the $.Payload field comes from lambda's output, if the state is not a lambda call, then check the output to merge the proper key
"ResultSelector": {"response.$": "States.JsonMerge($.Payload,
$$.Execution.Input, false)"}, "OutputPath": "$.response"
as a result the task's output will contain the input and output merged flat, without some weird nested dictionaries, exactly as requested
How can I passthrough the input to a Task state in an AWS Step Functions to the output?
After reading the Input and Output Processing page in the AWS docs, I have played with various combinations of InputPath, ResultPath and OutputPath.
State definition:
"First State": {
"Type": "Task",
"Resource": "[My Lambda ARN]",
"Next": "Second State",
"InputPath": "$.someKey",
"OutputPath": "$"
}
Input:
{
"someKey": "someValue"
}
Expected Result
I would like the output of the First State (and thus the input of Second State) to be
{
"someKey": "someValue"
}
Actual Result
[empty]
What if the input is more complicated, e.g.
{
"firstKey": "firstValue",
"secondKey": "secondValue"
}
I would like to forward all of it without worrying about (sub) paths.
In the Amazon States Language spec it is stated that:
If the value of ResultPath is null, that means that the state’s own raw output is discarded and its raw input becomes its result.
Consequently, I updated my state definition to
"First State": {
"Type": "Task",
"Resource": "[My Lambda ARN]",
"Next": "Second State",
"ResultPath": null
}
As a result, when passing the input example Task input payload will be copied to the output, even for rich objects like:
{
"firstKey": "firstValue",
"secondKey": "secondValue"
}
For those who find themselves here using CDK, the solution is to use the explicit aws_stepfunctions.JsonPath.DISCARD enum rather than None/null.
from aws_cdk import (
aws_stepfunctions,
aws_stepfunctions_tasks,
)
aws_stepfunctions_tasks.LambdaInvoke(
self,
"my_function",
lambda_function=lambda_function,
result_path=aws_stepfunctions.JsonPath.DISCARD,
)
https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-stepfunctions.JsonPath.html#static-discard
I was looking for a solution from passing input from one parallel state to another parallel state and the above option worked really good.
For example my step function is like this...tas1->parallel task2 -> parallel trask3 -> task4. So when it start with parallel task3, the input values are wiped out, so ptask3 is failing. With the above option, i was able to pass in same input from ptask2 to ptas3.
I have a AWS Step Function State formatted as follows:
"MyState": {
"Type": "Task",
"Resource": "<MyLambdaARN>",
"ResultPath": "$.value1"
"Next": "NextState"
}
I want to add a second value but can't find out how anywhere. None of the AWS examples display multiple ResultPath values being added to the output.
Would I just add a comma between them?
"MyState": {
"Type": "Task",
"Resource": "<MyLambdaARN>",
"ResultPath": "$.value1, $.value2"
"Next": "NextState"
}
Or is there a better way to format these?
Let's answer this straight up: you can't specify multiple ResultPath values, because it doesn't make sense. Amazon does do a pretty bad job of explaining how this works, so I understand why this is confusing.
You can, however, return multiple result values from a State in your State Machine.
General Details
The input to any State is a JSON object. The output of the State is a JSON object.
ResultPath directs the State Machine what to do with the output (result) of the State. Without specifying ResultPath, it defaults to $ which means all the input to the State is lost, replaced by the output of the State.
If you want to allow data from the input JSON to pass through your State, you specify a ResultPath to describe a property to add/overwrite on the input JSON to pass to the next State.
Your scenario
In your case, $.value1 means the output JSON of your State is the input JSON with a new/overwritten property value1 containing the output JSON of your lambda.
If you want multiple values in your output, your lambda should return a JSON object containing the multiple values, which will be the value of the value1 property.
If you don't care about allowing input values passing through your State, leave the ResultPath as the default $ by omitting it. The output JSON containing your multiple values will be the input to the next State.
Support scenario
Here's a simple State machine I use to play with the inputs and outputs:
{
"StartAt": "State1",
"States": {
"State1": {
"Type": "Pass",
"Result": { "Value1": "Yoyo", "Value2": 1 },
"ResultPath": "$.Result",
"Next": "State2"
},
"State2": {
"Type": "Pass",
"Result": { "Value2": 5 },
"ResultPath": "$.Result",
"Next": "State3"
},
"State3": {
"Type": "Pass",
"Result": "Done",
"End": true
}
}
}
Execute this with the following input:
{
"Input 1": 10000,
"Input 2": "YOLO",
"Input 3": true
}
Examine the inputs and outputs of each Stage. You should observe the following:
The input is passed all the way through, because the ResultPath always directs output to a Result property of the input.
The output of State1 is overwritten by the Output of State2. The net effect is Result.Value1 disappears and Result.Value2 is "updated".
Hopefully this clarifies how to use ResultPath effectively.
You cannot specify several values in ResultPath, because ResultPath defines the path of your result value in the json. The close analogy for ResultPath is a return value of a function, as your step can return only 1 value it should be put into 1 node in the resulting json.
If you have an input json
{
"myValue": "value1",
"myArray": [1,2,3]
}
And define your ResultPath as $.myResult the overall resulting json will be
{
"myValue": "value1",
"myArray": [1,2,3],
"myResult": "result"
}
Now you can truncate this json to pass only part of it to the next step in your function using OutputPath (e.g. OutputPath: "$.myResult")
InputPath and OutputPath can have several nodes in their definition, but ResultPath should always have only 1 node.
Seems like today it can be done by using ResultsSelector.
I think it would be like the following, but I haven't actually done this so I can't say for sure.
"ResultPath": {
"var1" : "$.value1",
"var2" : "$.value2"
},
=====
After looking further into this, I am convinced that there is no direct way to do what you want to do. Here is a way that could give you the results that you want.
1) You would omit the InputPath, OutputPath, and ResultPath from your step. This would mean that all of $. would be passed in as an input to your step function and that all of the output from the lambda function would be stored as $. In the lambda function you could set the results fields to be whatever you want them to be. The lambda function must return the modified input as it output.
Is there a better way to return the record(s) after DS.Store#pushPayload is called? This is what I'm doing...
var payload = { id: 1, title: "Example" }
store.pushPayload('post', payload);
return store.getById('post', payload.id);
But, with regular DS.Store#push you get the inserted record returned. The only difference between the two, from what I can tell, is that DS.Store#pushPayload serializes the payload data with the correct serializers.
DS.Store#pushPayload is able to take an array of items, not just one, and may contain side-loaded data. It processes a full payload and expects root keys in the payload:
{
"posts": [{
"id": 1,
"title": "title",
"comments": [1]
}],
"comments": [
//.. and so on ...
]
}
DS.Store#push expects a single record which has been normalized and contains no side loaded data (notice there is no root key):
{
"id": 1,
"title": "title",
"comments": [1]
}
For this reason, it makes sense for push to return the record, but for pushPayload to return nothing.
When you use pushPayload, a second lookup of store.find('post', 1) (or store.getById('post', 1)) is the way to go, I don't believe there is a better way.
As of this PR pushPayload can now return an array of all the records pushed into the store, once the 'ds-pushpayload-return' feature flag has been enabled.
At the moment, this feature isn't available in a standard or beta release-- you'll have to use
"ember-data": "emberjs/data#master",
(i.e. Canary) in your package.json in order to access it. I'm not sure when the feature will be generally available.