I am trying to pass output of 1st trigger as input to my lambda function. After code run i am not able to see my lambda function is called. Although I seen that Input of 2nd step function is taken as string as it is. It is not passing actual value.
please see below image. Input & Output for 2nd trigger
{
"Comment": "A description of my state machine",
"StartAt": "Pass",
"States": {
"Pass": {
"Type": "Pass",
"Result": {
"method": "GET",
"body": "SELECT * from TestTable"
},
"ResultPath": "$.latest",
"OutputPath": "$.latest",
"Next": "NextPass"
},
"NextPass": {
"Type": "Task",
"Resource": "arn:aws:lambda:<location>:<key>:function:<function-Name>",
"Parameters": {
"method": "$.latest.method",
"body": "$.latest.body"
},
"End": true
}
}
}
To access the data from the input, you need to add ".$" to the end of the Parameter names. This tells Step Functions that you're reading from a JSON path.
So your code should look like the following snippet below:
"Parameters": {
"method.$": "$.latest.method",
"body.$": "$.latest.body"
}
Hope that helps!
Related
I'm submitting an AWS Batch job from a Step Function. The batch job takes command line parameters. I can run it in the step function like this:
{
"Comment": "Submit aws batch job with parameters",
"StartAt": "SubmitJob",
"States": {
"SubmitJob": {
"Type": "Task",
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobName": "my-job",
"JobDefinition": "arn:aws:batch:us-west-1:123456789000:job-definition/my-job-definition:1",
"JobQueue": "arn:aws:batch:us-west-1:123456789000:job-queue/my-job-queue",
"ContainerOverrides": {
"Command": [
"Hello",
"World"
]
}
},
"End": true
}
}
}
However, I would like to use the input to this step as the command line parameters. So I try to use the $.parameter notation in the Command.
My input is
{
"param_1": "Hello",
"param_2": "World"
}
and my step function is
{
"Comment": "Submit aws batch job with parameters",
"StartAt": "SubmitJob",
"States": {
"SubmitJob": {
"Type": "Task",
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobName": "my-job",
"JobDefinition": "arn:aws:batch:us-west-1:123456789000:job-definition/my-job-definition:1",
"JobQueue": "arn:aws:batch:us-west-1:123456789000:job-queue/my-job-queue",
"ContainerOverrides": {
"Command": [
"$.param_1",
"$.param_2"
]
}
},
"End": true
}
}
}
However, this results in my batch job receiving the raw strings like "$.param_1" rather than them being substituted by the input "Hello". Why? What is required instead?
This is one Solution/Workaraound, I'm not sure if it is the only option.
In the AWS Batch Job Definition, in the Container properties, set Command to be
["Ref::param_1","Ref::param_2"]
These "Ref::" links will capture parameters that are provided when the Job is run. Additionally, you can specify parameters in the job definition Parameters section but this is only necessary if you want to provide defaults.
In the Step Functions State Machine definition, instead of using ContainerOverrides, use Parameters. Overall it will look like:
{
"Comment": "Submit aws batch job with parameters",
"StartAt": "SubmitJob",
"States": {
"SubmitJob": {
"Type": "Task",
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobName": "my-job",
"JobDefinition": "arn:aws:batch:us-west-1:123456789000:job-definition/my-job-definition:2",
"JobQueue": "arn:aws:batch:us-west-1:123456789000:job-queue/my-job-queue",
"Parameters": {
"param_1.$": "$.param_1",
"param_2.$": "$.param_2"
}
},
"End": true
}
}
}
Thus the input into the step function gets passed to the Batch job which captures them and uses as Command.
reference: https://docs.aws.amazon.com/batch/latest/userguide/example-job-definitions.html
I'm trying two consecutive API Gateway calls from a Step Function, using values from the first result on the second. I tried already on a POST call and it worked, but now I can't do it on a GET call. This is my step function:
{
"Comment": "A Hello World example of the Amazon States Language using Pass states",
"StartAt": "Api Call",
"States": {
"Api Call": {
"Type": "Task",
"Resource": "arn:aws:states:::apigateway:invoke",
"Parameters": {
"ApiEndpoint": "***************.us-east-1.amazonaws.com",
"Method": "GET",
"Path": "universities",
"RequestBody": {},
"AuthType": "NO_AUTH"
},
"ResultSelector": {
"logWord.$": "$.StatusText"
},
"Next": "Api Call 2"
},
"Api Call 2": {
"Type": "Task",
"Resource": "arn:aws:states:::apigateway:invoke",
"Parameters": {
"ApiEndpoint": "***************.us-east-1.amazonaws.com",
"Method": "GET",
"Path": "logging",
"Headers": {
"Content-Type": ["application/json"]
},
"QueryParameters": {
"logWord.$": "$.logWord"
},
"AuthType": "NO_AUTH"
},
"End": true
}
}
}
The error that I get is the following:
{
"error": "States.Runtime",
"cause": "An error occurred while executing the state 'Api Call 2' (entered at the event id #7). The Parameters '{\"ApiEndpoint\":\"***************.us-east-1.amazonaws.com\",\"Method\":\"GET\",\"Path\":\"logging\",\"Headers\":{\"Content-Type\":[\"application/json\"]},\"QueryParameters\":{\"logWord\":\"OK\"},\"AuthType\":\"NO_AUTH\"}' could not be used to start the Task: [The value of the field 'QueryParameters' has an invalid format]"
}
According to the documentation, query parameters should be between brackets, but if I'm using a context variable I can't put the brackets. The console doesn't allow me to even save (The value for the field 'logWord.$' must be a STRING that contains a JSONPath but was an ARRAY).
If anyone knows how to make GET call using context variables would be greatly appreciated.
Looks like QueryParameters wants a list of strings. Use the States.Array intrinsic function to convert your interpolated string to a [string] list.
"QueryParameters": {
"logWord.$": "States.Array($.logWord)" // ["OK"]
},
I have a step function with a map task, as known the map have to work on an array from the ItemsPath, how can i pass the whole input to the lambda and not only the array.
{"StartAt": "Find","States": {
"Find": {
"Type": "Map",
"MaxConcurrency": 0,
"InputPath": "$",
"ItemsPath": "$.Payload.contacts",
"Iterator": {
"StartAt": "func",
"States": {
"func": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:....",
"Parameters": {
"Input": {
"Payload":{
"contact.$": "$"
}
}
},
"End": true
}
}
},
"ResultPath": "$.Input",
"End": true
}}} ,
i want the whole input to be passed in the event parameter
If you use Iterator it will pass the values from ItemsPath as input to Lambda. You can use the Parameters block to transform the input to the lambda and add the whole input. I haven't tried it myself but I'm pretty sure that should do it.
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html
I'm sure someone will point me to an immediate solution, but I've been at this for hours, so I'm just going to ask.
I cannot get a State Machine to accept an initial input. The intent is to set up an EventBridge trigger pointed at the State Machine with a static JSON passed to the SM to initiate with the proper parameters. In development, I'm just using Step Functions option to pass a JSON as the initial input when you select "New Execution".
This is the input:
{"event":{
"country": "countryA",
"landing_bucket": "aws-glue-countryA-inputs",
"landing_key": "countryA-Bucket/prefix/filename.csv",
"forecast_bucket": "aws-forecast-countryA",
"forecast_key": "inputs/",
"date_start": "2018-01-01",
"validation": "False",
"validation_size": 90
}
}
When looking at what is passed at the ExecutionStarted log entry:
{
"input": {
"country": "countryA",
"landing_bucket": "aws-glue-countryA-inputs",
"landing_key": "countryA-Bucket/prefix/filename.csv",
"forecast_bucket": "aws-forecast-countryA",
"forecast_key": "inputs/",
"date_start": "2018-01-01",
"validation": "False",
"validation_size": 90
}
,
"inputDetails": {
"truncated": false
},
"roleArn": "arn:aws:iam::a-valid-service-role"
}
This is the State Machine:
"Comment": "A pipeline!",
"StartAt": "Invoke Preprocessor",
"States": {
"Invoke Preprocessor": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"InputPath": "$.input",
"Parameters": {
"FunctionName": "arn:aws:lambda:my-lambda-arn:$LATEST"
},
"Next": "EndSM"
},
"EndSM": {
"Type": "Pass",
"Result": "Ended",
"End": true
}
}
}
I've tried nearly anything I can think of from changing the InputPath to assigning the "input" dictionary directly to a variable:
"$.event":"$.input"
To drilling down to the individual variables and assigning those directly like:
"$.country:"$.country". I've also used the new Step Functions Data Flow Simulator and can't get anywhere. If anyone has any thoughts, I'd really appreciate it.
Thanks!
Edited for correct solution:
You need to set the Payload.$ parameter to $. That will pass in the entire input object to the lambda.
{
"Comment": "A pipeline!",
"StartAt": "Invoke Preprocessor",
"States": {
"Invoke Preprocessor": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:my-lambda-arn:$LATEST",
"Payload.$": "$"
},
"Next": "EndSM"
},
"EndSM": {
"Type": "Pass",
"Result": "Ended",
"End": true
}
}
}
Another thing you could do is specify the input in the parameters, this will allow you to specify only all/certain parts of the json to pass in.
{
"Comment": "A pipeline!",
"StartAt": "Invoke Preprocessor",
"States": {
"Invoke Preprocessor": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"InputPath": "$",
"Parameters": {
"FunctionName": "arn:aws:lambda:my-lambda-arn:$LATEST",
"input_event.$": "$.event"
},
"Next": "EndSM"
},
"EndSM": {
"Type": "Pass",
"Result": "Ended",
"End": true
}
}
}
From the code perspective you could just reference it like so (python):
input = event['input_event']
I am writing an AWS step function, and for one of the steps, I wish to call a lambda that accepts an array as one of the inputs. However, if I try to pass in a JsonPath into the array, I get
The value for the field 'arrayField.$' must be a STRING that contains a JSONPath but was an ARRAY
My step function definition:
{
"StartAt": "First",
"States": {
"First": {
"Type": "Pass",
"Parameters": {
"type": "person"
},
"ResultPath": "$.output",
"Next": "Second"
},
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
"Parameters": {
"regularParameter": "some string",
"arrayParameter.$": ["$.output.type"]
},
"Next": "Succeed"
},
"Succeed": {
"Type": "Succeed"
}
}
}
How can I use jsonPath inside the array?
Since a new release you could use the intrinsic function States.Array:
"arrayParameter.$": "States.Array($.output.type)"
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html
As #Seth Miller mentioned JsonPath resolution within arrays doesn't work unfortunately. If the amount of values to replace in the array is small and known there's a simple workaround (in my case I needed an array of size 1).
The steps are:
Initialise the array with the number of values you need;
Replace each value using "ResultPath": "$.path.to.array[n]";
Use "$.path.to.array" in your task.
Simple, working example:
{
"StartAt": "First",
"States": {
"First": {
"Type": "Pass",
"Parameters": {
"type": "person"
},
"ResultPath": "$.output",
"Next": "Initialise Array"
},
"Initialise Array": {
"Comment": "Add an entry for each value you intend to have in the final array, the values here don't matter.",
"Type": "Pass",
"Parameters": [
0
],
"ResultPath": "$.arrayParameter",
"Next": "Fill Array"
},
"Fill Array": {
"Comment": "Replace the first entry of array with parameter",
"Type": "Pass",
"InputPath": "$.output.type",
"ResultPath": "$.arrayParameter[0]",
"End": true
}
}
}
And to use the resulting array in your task example:
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
"Parameters": {
"regularParameter": "some string",
"arrayParameter.$": "$.arrayParameter"
},
"Next": "Succeed"
},
Another way to approach this is by using Parallel state that outputs an array of objects and then use jsonPath to convert it to a simple array:
{
"StartAt": "Parallel",
"States": {
"Parallel": {
"Type": "Parallel",
"Next": "Use Array",
"ResultPath": "$.items",
"Branches": [
{
"StartAt": "CreateArray",
"States": {
"CreateArray": {
"Type": "Pass",
"Parameters": {
"value": "your value"
},
"End": true
}
}
}
]
},
"Use Array": {
"Type": "Pass",
"Parameters": {
"items.$": "$.items[*].value"
},
"End": true
}
}
}
In this example, Parallel state outputs the following json:
{
"items": [
{
"value": "your value"
}
]
}
And "Use Array" state produces:
{
"items": [
"your value"
]
}
JSONPath inside parameters field need to be a string. So if you want to pass to lambda function a parameter called arrayParameter, you´ll need to make a jsonPath query that extract that array.
For example, if inside the key output is a key called outputArray with the array as its value.
Input JSON:
{
"pre": "sdigf",
"output": {
"result": 1,
"outputArray": ["test1","test2","test.."]
}
}
The parameter sintax:
"arrayParameter.$": "$.output.outputArray"
Reasonable advice
I ran into a use case for JsonPath resolution within arrays today and found (like you have) that the functionality does not exist today. I ended up deciding that doing the data massaging in code was simpler and cleaner. For example, you could create a small Lambda that takes in the object emitted by First and massages it to a format acceptable to Second and adds it to the output (WaterKnight mentions this solution in a comment to another question).
This assumes that you are, for some reason, unable to change the format of the input to that Lambda in Second (which would be the absolute shortest path here).
Unreasonable advice
That said, if you want a way to do this completely within Step Functions that is fairly gross, you can use the result of a Map state that executes Pass states. The output of the Map state is an array that aggregate the output of each constituent Pass state. These Pass states simply emit the value(s) you want in the final array using the Parameters attribute. An example Step Function definition follows. I did warn that it is gross and that I went a different way to solve the problem.
{
"StartAt": "First",
"Comment": "Please don't actually do this",
"States": {
"First": {
"Type": "Pass",
"Parameters": {
"type": "person"
},
"ResultPath": "$.output",
"Next": "Add Array"
},
"Add Array": {
"Comment": "A Map state needs some array to loop over in order to work. We will give it a dummy array. Add an entry for each value you intend to have in the final array. The values here don't matter.",
"Type": "Pass",
"Result": [
0
],
"ResultPath": "$.dummy",
"Next": "Mapper"
},
"Mapper": {
"Comment": "Add a Pass state with the appropriate Parameters for each field you want to map into the output array",
"Type": "Map",
"InputPath": "$",
"ItemsPath": "$.dummy",
"Parameters": {
"output.$": "$.output"
},
"Iterator": {
"StartAt": "Massage",
"States": {
"Massage": {
"Type": "Pass",
"Parameters": {
"type.$": "$.output.type"
},
"OutputPath": "$.type",
"End": true
}
}
},
"ResultPath": "$.output.typeArray",
"Next": "Second"
},
"Second": {
"Comment": "The Lambda in your example is replaced with Pass so that I could test this",
"Type": "Pass",
"Parameters": {
"regularParameter": "some string",
"arrayParameter.$": "$.output.typeArray"
},
"Next": "Succeed"
},
"Succeed": {
"Type": "Succeed"
}
}
}
As many answers correctly pointed out, it's not possible to do it exactly the way you need. But I would suggest another solution: an array of dictionaries. It's not exactly what you need, but is native and not hacky.
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
"Parameters": {
"regularParameter": "some string",
"arrayParameter": [{"type.$": "$.output.type"}]
},
"Next": "Succeed"
},
The result would be
{
"regularParameter": "some string",
"arrayParameter": [{"type": "SingleItemWrappedToAnArray"}]
}