Here is my step function:
{
"Comment": "A description of my state machine",
"StartAt": "lambda1",
"States": {
"lambda1": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": "$"
},
"Next": "lambda2"
},
"lambda2": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": {
"item_id.$": "$.item_id"
}
},
"Next": "lambda3"
},
"lambda3": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": {
"item_id.$": "" <------- How to access the id here
"data.$": $.data
}
},
"End": true
}
}
}
I am passing an item_id to lambda2, and that lambda returns some output data.
In lambda3, I need the output data of lambda2 as well as the item_id that was passed to lambda2 from lambda1.
I cannot modify the application code for lambda2 to also return the item_id passed to it.
Is it possible for me to pass the item_id from lambda1 all the way to lambda3?
In lambda3, I need the output data of lambda2 as well as the item_id that was passed to lambda2 from lambda1.
Yes, this is possible using ResultPath.
I've detailed the various outcomes of using ResultPath in my answer here.
In your case, you need a ResultPath of $.DataFromLambda1 set for lambda2 which would set the state input of lambda3 to:
the state input of lambda1 (which is the item_id)
+ the state output of lambda2.
"lambda2": {
...
"OutputPath": "$.Payload",
"ResultPath": "$.DataFromLambda1",
"Parameters": {
"Payload.$": {
"item_id.$": "$.item_id"
}
},
"Next": "lambda3"
}
Then change lambda3 to:
"lambda3": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": {
"item_id.$": "$.DataFromLambda1.item_id"
"data.$": $.data
}
},
"End": true
}
Related
I'm sure someone will point me to an immediate solution, but I've been at this for hours, so I'm just going to ask.
I cannot get a State Machine to accept an initial input. The intent is to set up an EventBridge trigger pointed at the State Machine with a static JSON passed to the SM to initiate with the proper parameters. In development, I'm just using Step Functions option to pass a JSON as the initial input when you select "New Execution".
This is the input:
{"event":{
"country": "countryA",
"landing_bucket": "aws-glue-countryA-inputs",
"landing_key": "countryA-Bucket/prefix/filename.csv",
"forecast_bucket": "aws-forecast-countryA",
"forecast_key": "inputs/",
"date_start": "2018-01-01",
"validation": "False",
"validation_size": 90
}
}
When looking at what is passed at the ExecutionStarted log entry:
{
"input": {
"country": "countryA",
"landing_bucket": "aws-glue-countryA-inputs",
"landing_key": "countryA-Bucket/prefix/filename.csv",
"forecast_bucket": "aws-forecast-countryA",
"forecast_key": "inputs/",
"date_start": "2018-01-01",
"validation": "False",
"validation_size": 90
}
,
"inputDetails": {
"truncated": false
},
"roleArn": "arn:aws:iam::a-valid-service-role"
}
This is the State Machine:
"Comment": "A pipeline!",
"StartAt": "Invoke Preprocessor",
"States": {
"Invoke Preprocessor": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"InputPath": "$.input",
"Parameters": {
"FunctionName": "arn:aws:lambda:my-lambda-arn:$LATEST"
},
"Next": "EndSM"
},
"EndSM": {
"Type": "Pass",
"Result": "Ended",
"End": true
}
}
}
I've tried nearly anything I can think of from changing the InputPath to assigning the "input" dictionary directly to a variable:
"$.event":"$.input"
To drilling down to the individual variables and assigning those directly like:
"$.country:"$.country". I've also used the new Step Functions Data Flow Simulator and can't get anywhere. If anyone has any thoughts, I'd really appreciate it.
Thanks!
Edited for correct solution:
You need to set the Payload.$ parameter to $. That will pass in the entire input object to the lambda.
{
"Comment": "A pipeline!",
"StartAt": "Invoke Preprocessor",
"States": {
"Invoke Preprocessor": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:my-lambda-arn:$LATEST",
"Payload.$": "$"
},
"Next": "EndSM"
},
"EndSM": {
"Type": "Pass",
"Result": "Ended",
"End": true
}
}
}
Another thing you could do is specify the input in the parameters, this will allow you to specify only all/certain parts of the json to pass in.
{
"Comment": "A pipeline!",
"StartAt": "Invoke Preprocessor",
"States": {
"Invoke Preprocessor": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"InputPath": "$",
"Parameters": {
"FunctionName": "arn:aws:lambda:my-lambda-arn:$LATEST",
"input_event.$": "$.event"
},
"Next": "EndSM"
},
"EndSM": {
"Type": "Pass",
"Result": "Ended",
"End": true
}
}
}
From the code perspective you could just reference it like so (python):
input = event['input_event']
I am trying to use AWS step functions to create parallel branches of execution.
One of the parallel branches starts another step function invocation, how can we pass input from this parallel branch to next step function execution
{
"Comment": "Parallel Example.",
"StartAt": "FunWithMath",
"States": {
"FunWithMath": {
"Type": "Parallel",
"End": true,
"Branches": [
{
"StartAt": "Add", /// This receives some json object here input {}
"States": {
"Add": {
"Type": "Task", ***//How to pass the received input to the following arn as input?***
"Resource": ""arn:aws:states:::states:startExecution",
Parameters: {
"StateMachineArn": "anotherstepfunctionarnpath"
}
"End": true
}
}
},
{
"StartAt": "Subtract",
"States": {
"Subtract": {
"Type": "Task",
"Resource": "some lambda arn here,
"End": true
}
}
}
]
}
}
}
anotherstepfunctionarnpath :
{
"Comment": "Second state machine",
"StartAt": "stage1",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters":{
"Arguments":{
"Variable1" :"???" / how to access the value of the input passed to here
}
}
}
You can use Input to pass output from one SFN to other one:
First SFN(It will call second SFN)
{
"Comment": "My first SFN",
"StartAt": "First SFN",
"States": {
"First SFN": {
"Type": "Task",
"ResultPath": "$.to_pass",
"Resource": "arn:aws:lambda:us-east-1:807278658150:function:test-lambda",
"Next": "Trigger Next SFN"
},
"Trigger Next SFN": {
"Type": "Task",
"Resource": "arn:aws:states:::states:startExecution",
"Parameters": {
"Input": {
"Comment.$": "$"
},
"StateMachineArn": "arn:aws:states:us-east-1:807278658150:stateMachine:MyStateMachine2"
},
"End": true
}
}
}
Second SFN (MyStateMachine2)
{
"Comment": "A Hello World example of the Amazon States Language using Pass states",
"StartAt": "Hello",
"States": {
"Hello": {
"Type": "Pass",
"Result": "Hello",
"Next": "World"
},
"World": {
"Type": "Pass",
"Result": "World",
"End": true
}
}
}
First SFN's Execution
Second SFN's Execution
Explanation
The Lambda test-lambda is returning:
{
"user": "stackoverflow",
"id": "100"
}
Which is stored in "ResultPath": "$.to_pass" here in to_pass variable. I am passing the same output to next state machine MyStateMachine2 which is done by
"Input": {
"Comment.$": "$"
}
In the next State Machine's execution you see that same data is received as input which was created by first Lambda.
You can read more about it here.
I've been researching on how to pass data from a Lambda to another Lambda in a Step Function and this is what i got.
I have this dummy lambda that pass the data name:
exports.lambdaHandler = async (event, context, callback) => {
const name = 'test';
callback(null, { name });
}
to another lambda, where i try to get the data in this way, but is not working:
const name = event.name; //this returns undefined
Based on this tutorial, this should be enough but it doesn't work. Can you point me in what direction should i go? Do i have to use the InputPath, ResultPath properties of the states machines?
[Update]
This is the State machine definition:
{
"Comment": "commen test",
"StartAt": "FunctionOne",
"States": {
"FunctionOne": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": <arn FunctionOne>
},
"Next": "FunctionTwo"
},
"FunctionTwo": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": <arn FunctionTwo>
},
"End": true
}
}
}
Try this
{
"Comment": "commen test",
"StartAt": "FunctionOne",
"States": {
"FunctionOne": {
"Type": "Task",
"Resource": "<arn FunctionOne>",
"Next": "FunctionTwo"
},
"FunctionTwo": {
"Type": "Task",
"Resource": "<arn FunctionTwo>",
"End": true
}
}
}
I have the below state machine. The requirement is to have a lambda to query DB and get all the ids. Next I have a parallel state call that calls more than five lambdas at once. Instead of passing all the ids fetched to all the lambdas, I need to pass the respective ids to each lambda.
In the below state language, first call is DB_CALL, lets say it returns {id1, id2, id3, id4, id5, id6}, I want to pass only id1 to First_Lambda and id2 to Second_Lambda etc...
The entire id object should get passed to all lambdas. Please suggest a way to achieve this.
{
"Comment": "Concurrent Lambda calls",
"StartAt": "StarterLambda",
"States": {
"StarterLambda": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:DB_CALL",
"Next": "ParallelCall"
},
"State": {
"ParallelCall": {
"Type": "Parallel",
"End": true,
"Branches": [
{
"StartAt": "First",
"States": {
"First": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:First_Lambda",
"TimeoutSeconds": 120,
"End": true
}
}
},
{
"StartAt": "Second",
"States": {
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:Second_Lambda",
"Retry": [ {
"ErrorEquals": ["States.TaskFailed"],
"IntervalSeconds": 1,
"MaxAttempts": 2,
"BackoffRate": 2.0
} ],
"End": true
}
}
},
{
"StartAt": "Third",
"States": {
"Third": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:Third_Lambda",
"Catch": [ {
"ErrorEquals": ["States.TaskFailed"],
"Next": "CatchHandler"
} ],
"End": true
},
"CatchHandler": {
"Type": "Pass",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:CATCH_HANDLER",
"End": true
}
}
},
{
"StartAt": "Fourth",
"States": {
"Fourth": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:Fourth_Lambda",
"TimeoutSeconds": 120,
"End": true
}
}
},
{
"StartAt": "Fifth",
"States": {
"Fifth": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:Fifth_Lambda",
"TimeoutSeconds": 120,
"End": true
}
}
},
{
"StartAt": "Sixth",
"States": {
"Sixth": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:Sixth_Lambda",
"TimeoutSeconds": 120,
"End": true
}
}
}
}
]
}
}
}
}
You can use Step Function parameter option.
This would allow you to send specific value or json to next lambda.
"Parameters": {
"toprocess.$": "$.MetaData.CorrelationId"
},
So input to this lambda would be smaller dto than compared to you first lambda. So while returning value from this lambda avoid assigning it back to Step function result.
"OutputPath": "$",
"ResultPath": "$.PartialResutl",
What you are looking for is the Map State. With this state, you pass in the iterator, in your case the path to the ids. The map state will run once for each item in the list. Within the map state, you have a full state machine, so you can call a Lambda or any other state. It has controls to limit how many are running at once if that is needed.
Working on an AWS StepFunction that gets an array of dates from a Lambda call, then passes to a Task that should take that array as a parameter to pass into a lambda.
The Get Date Range task works fine and outputs the date array:
{
"rng": [
"2019-05-07",
"2019-05-09"
]
}
...and the array gets passed into the ProcessDateRange task, but I cannot assign the array the range Parameter.
It literally tries to pass this: "$.rng" instead of this:
[
"2019-05-07",
"2019-05-09"
]
Here's the StateMachine:
{
"StartAt": "Try",
"States": {
"Try": {
"Type": "Parallel",
"Branches": [{
"StartAt": "Get Date Range",
"States": {
"Get Date Range": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:get-date-range",
"Parameters": {
"name": "thename",
"date_query": "SELECT date from sch.tbl_dates;",
"database": "the_db"
}
,
"ResultPath": "$.rng",
"TimeoutSeconds": 900,
"Next": "ProcessDateRange"
},
"ProcessDateRange": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:process-date-range",
"Parameters": {
"range": "$.rng"
},
"ResultPath": "$",
"Next": "Exit"
},
"Exit": {
"Type": "Succeed"
}
}
}],
"Catch": [{
"ErrorEquals": ["States.ALL"],
"ResultPath": "$.Error",
"Next": "Failed"
}],
"Next": "Succeeded"
},
"Failed": {
"Type": "Fail",
"Cause": "There was an error. Please review the logs.",
"Error": "error"
},
"Succeeded": {
"Type": "Succeed"
}
}
}
This is because you are using the wrong syntax for Lambda tasks. To specify the input you need to set the InputPath key, for example:
"ProcessDateRange": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:process-date-range",
"InputPath": "$.rng",
"ResultPath": "$",
"Next": "Exit"
},
If you want a parameter to be interpreted as a JSON path instead of a literal string, add ".$" to the end of the parameter name. To modify your example:
"ProcessDateRange": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:process-date-range",
"Parameters": {
"range.$": "$.rng"
},
"ResultPath": "$",
"Next": "Exit"
},
Relevant docs here: https://docs.aws.amazon.com/step-functions/latest/dg/connectors-parameters.html#connectors-parameters-path