{
"Comment": "A description of my state machine",
"StartAt": "Batch SubmitJob",
"States": {
"Batch SubmitJob": {
"Type": "Task",
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobDefinition": "arn:aws:batch:us-east-1:XXXXXXXXXXX:job-definition/clientcopyjobdef:1",
"JobQueue": "arn:aws:batch:us-east-1:XXXXXXXXXX:job-queue/copyclientjq",
"ContainerOverrides": {
"Command.$": [
"dotnet",
"CopyClientJob.dll",
"$.input"
]
},
"JobName.$": "$.input"
},
"End": true
}
}
}
I was trying to create this state machine, it was working fine if I am passing command to JobDefinition directly, but here I am trying to override the command and want to pass command parameter from state input, so trying to pass like above code. For "JobName.$": "$.input" it is working but for
"Command.$": [
"dotnet",
"CopyClientJob.dll",
"$.input"
]
It is not working, the command is passing as it is to aws batch without transforming the parameter, can someone help on this.
Thanks
Got the solution, Actually I need to combine the all 3 using States.Array like below
{
"Comment": "A description of my state machine",
"StartAt": "Batch SubmitJob",
"States": {
"Batch SubmitJob": {
"Type": "Task",
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobDefinition": "arn:aws:batch:us-east-1:XXXXXXXXX:job-definition/clientcopyjobdef:1",
"JobQueue": "arn:aws:batch:us-east-1:XXXXXXXX:job-queue/copyclientjq",
"ContainerOverrides": {
"Command.$": "States.Array('dotnet', 'CopyClientJob.dll', $.input)"
},
"JobName": "test"
},
"End": true
}
}
}
Related
I am creating a Step Functions workflow which has various steps. I am referring to this topic in their documentation InputPath, ResultPath and OutputPath Examples. I am trying to check the identity and address of a person in my workflow as they've shown in their document. I'm passing the input for the Verify identity step within the state machine definition inside Parameters. My workflow looks like this.
Note: But when I run this, am getting the error -> An error occurred while executing the state 'Verify identity' (entered at the event id #19). Invalid path '$.identity' : Property ['identity'] not found in path $
What am I doing wrong here? Can someone please explain?
Thanks..
{
"StartAt": "Step1",
"States": {
"Step1": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
...something...
},
"Next": "Step2"
},
"Step2": {
"Type": "Choice",
"Choices": [
Do something...
],
"Default": "Step3.1"
},
"Step3.1": {
"Type": "Task",
...something...
}
},
"Next": "Step3.3"
},
...something...,
"Step4": {
"Type": "Parallel",
"Branches": [
{
"StartAt": "Verify identity",
"States": {
"Verify identity": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"InputPath": "$.identity",
"Parameters": {
"Payload": {
"identity": {
"email": "jdoe#example.com",
"ssn": "123-45-6789"
},
"firstName": "Jane",
"lastName": "Doe"
},
"FunctionName": "{Lambda ARN}"
},
"End": true
}
}
},
{
"StartAt": "Verify address",
"States": {
"Verify address": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"Payload": {
"street": "123 Main St",
"city": "Columbus",
"state": "OH",
"zip": "43219"
},
"FunctionName": "{Lambda ARN}"
},
"End": true
}
}
}
],
"Next": "Step5"
},
"Step5": {
"Type": "Task",
"Parameters": {
something...
},
"End": true
}
}```
You don't have an explicit transition in your example to call Step4 but assuming the order you have defined (step1 -> step2 -> step3.1 -> step3.3 -> step4)
This means the output from step3.3 should be something like
{
"cat": "meow",
"dog": "woof",
"identity": { // this is whats missing
"email": "jdoe#example.com",
"ssn": "123-45-6789"
}
}
this is what will get passed to each branch of your parallel state (Step4)
However, since you have anInputPath defined for Step4."Verify identity", the effective input to the task becomes
{
"email": "jdoe#example.com",
"ssn": "123-45-6789"
}
The error youre seeing
An error occurred while executing the state 'Verify identity' (entered at the event id #19). Invalid path '$.identity' : Property ['identity'] not found in path $
means the "identity" key (aka $.identity) isn't getting added to the output of Step3.3 (aka $)
How can use input in my state machine in mid of a string?
For example, I want to make a command using input params like this
["python test.py", "--name=$.name", "--age=$.age"]
But as per AWS documentation, I can't pass like above. I can only pass like
{"Command.$": "$.age"}
Why this strange structure is required. Why do I need to use .$ in the key. Why can't I use $.age freely anywhere?
What I want to achieve is something like this.
{
"Comment": "A description of my state machine",
"StartAt": "Pipeline",
"States": {
"Run Pipeline": {
"Type": "Task",
"Resource": "arn:aws:states:::ecs:runTask.sync",
"Parameters": {
"LaunchType": "FARGATE",
"Cluster": "aaaaaaaaaaaaaa",
"TaskDefinition": "aaaaaaaaaaaaaa",
"Overrides": {
"ContainerOverrides": [{
"Name": "my_container",
"Command": [
"python test.py",
"--name=$.name",
"--age=$.age"
]
}]
}
},
"End": true
}
}
}
where my step function input is
{"name": "Rahul", "age": 25}
The .$ tells step-functions that you are passing a path and not a literal string value.
Input/Ouput Path Params Doc
If you want to construct a string from inputs, you should be able to use intrinsic string formatting:
States.Format('python test.py --name={} --age={}', $.name, $.age)
Intrinsic Function Doc
{
"Comment": "A description of my state machine",
"StartAt": "Run Pipeline",
"States": {
"Run Pipeline": {
"Type": "Task",
"Resource": "arn:aws:states:::ecs:runTask.sync",
"Parameters": {
"LaunchType": "FARGATE",
"Cluster": "aaaaaaaaaaaaaa",
"TaskDefinition": "aaaaaaaaaaaaaa",
"Overrides": {
"ContainerOverrides": [{
"Name": "my_container",
"Command.$": "States.Array(States.Format('python test.py --name={} --age={}', $.name, $.age))"
}]
}
},
"End": true
}
}
}
States.Array makes an array out of a list of inputs.
States.Format lets you construct a string using literal and interpolated (variable) inputs.
I'm sure someone will point me to an immediate solution, but I've been at this for hours, so I'm just going to ask.
I cannot get a State Machine to accept an initial input. The intent is to set up an EventBridge trigger pointed at the State Machine with a static JSON passed to the SM to initiate with the proper parameters. In development, I'm just using Step Functions option to pass a JSON as the initial input when you select "New Execution".
This is the input:
{"event":{
"country": "countryA",
"landing_bucket": "aws-glue-countryA-inputs",
"landing_key": "countryA-Bucket/prefix/filename.csv",
"forecast_bucket": "aws-forecast-countryA",
"forecast_key": "inputs/",
"date_start": "2018-01-01",
"validation": "False",
"validation_size": 90
}
}
When looking at what is passed at the ExecutionStarted log entry:
{
"input": {
"country": "countryA",
"landing_bucket": "aws-glue-countryA-inputs",
"landing_key": "countryA-Bucket/prefix/filename.csv",
"forecast_bucket": "aws-forecast-countryA",
"forecast_key": "inputs/",
"date_start": "2018-01-01",
"validation": "False",
"validation_size": 90
}
,
"inputDetails": {
"truncated": false
},
"roleArn": "arn:aws:iam::a-valid-service-role"
}
This is the State Machine:
"Comment": "A pipeline!",
"StartAt": "Invoke Preprocessor",
"States": {
"Invoke Preprocessor": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"InputPath": "$.input",
"Parameters": {
"FunctionName": "arn:aws:lambda:my-lambda-arn:$LATEST"
},
"Next": "EndSM"
},
"EndSM": {
"Type": "Pass",
"Result": "Ended",
"End": true
}
}
}
I've tried nearly anything I can think of from changing the InputPath to assigning the "input" dictionary directly to a variable:
"$.event":"$.input"
To drilling down to the individual variables and assigning those directly like:
"$.country:"$.country". I've also used the new Step Functions Data Flow Simulator and can't get anywhere. If anyone has any thoughts, I'd really appreciate it.
Thanks!
Edited for correct solution:
You need to set the Payload.$ parameter to $. That will pass in the entire input object to the lambda.
{
"Comment": "A pipeline!",
"StartAt": "Invoke Preprocessor",
"States": {
"Invoke Preprocessor": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:my-lambda-arn:$LATEST",
"Payload.$": "$"
},
"Next": "EndSM"
},
"EndSM": {
"Type": "Pass",
"Result": "Ended",
"End": true
}
}
}
Another thing you could do is specify the input in the parameters, this will allow you to specify only all/certain parts of the json to pass in.
{
"Comment": "A pipeline!",
"StartAt": "Invoke Preprocessor",
"States": {
"Invoke Preprocessor": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"InputPath": "$",
"Parameters": {
"FunctionName": "arn:aws:lambda:my-lambda-arn:$LATEST",
"input_event.$": "$.event"
},
"Next": "EndSM"
},
"EndSM": {
"Type": "Pass",
"Result": "Ended",
"End": true
}
}
}
From the code perspective you could just reference it like so (python):
input = event['input_event']
I'm new to AWS step functions.
Trying to create a basic ETL flow of glue jobs. Upon completion of state machine definition im able to see the graph being generated , but getting a generic error "There are Amazon States Language errors in your state machine definition. Fix the errors to continue",
error message
that is not allowing me to proceed.
Here is the code and graph :
{
"Comment": "DRC downstream glue jobs execution step function:slf_aws_can_dbisdel_everyone_drc_amp",
"StartAt": "startFlow",
"States": {
"Comment": "various state types of the Amazon States Language",
"startFlow": {
"Comment": "Pass states are useful when constructing and debugging state machines.",
"Type": "Pass",
"Next": "stg_ods"
},
"stg_ods": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "stage_job_name"
},
"Next": "ods_job"
},
"ods_job": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "main_job_name"
},
"Next": "Wait 3 sec"
},
"Wait 3 sec": {
"Comment": "A Wait state delays the state machine from continuing for a specified time.",
"Type": "Wait",
"Seconds": 3,
"Next": "parallel_stg_adr"
},
"parallel_stg_adr": {
"Comment": "A Parallel state can be used to create parallel branches of execution in your state machine.",
"Type": "Parallel",
"Branches": [
{
"StartAt": "stg_job1",
"States": {
"stg_job1": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "stg_job_name1"
},
"End": true
}
}
},
{
"StartAt": "stg_job2",
"States": {
"stg_job2": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "stg_job_name2"
},
"End": true
}
}
}
],
"Next": "parallel_adr_job"
},
"parallel_adr_job": {
"Comment": "A Parallel state can be used to create parallel branches of execution in your state machine.",
"Type": "Parallel",
"Branches": [
{
"StartAt": "job1",
"States": {
"job1": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "some_glue_job",
"Arguments": {
"--target_table": "some_string_table",
"--calendar_key": "some_string"
}
},
"End": true
}
}
},
{
"StartAt": "job2",
"States": {
"job2": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "some_glue_job",
"Arguments": {
"--target_table": "some_string_table",
"--calendar_key": "some_string"
}
},
"End": true
}
}
}
],
"Next": "end_job"
},
"end_job": {
"Type": "Pass",
"End": true
}
}
}
Step function graph
"Comment": "various state types of the Amazon States Language",
This one at Line 5 seems to be incorrect. "States" map cannot have a "Comment" key. Remove it and then try. Rest of the config looks correct.
Edit 1
If the type of Step Function is Express, ".sync" functions won't work. Try changing the ARN to
"Resource": "arn:aws:states:::glue:startJobRun"
and you should be able to save your Step Function. You will then have to figure out how to setup a different Glue task.
I am trying to use AWS step functions to create parallel branches of execution.
One of the parallel branches starts another step function invocation, how can we pass input from this parallel branch to next step function execution
{
"Comment": "Parallel Example.",
"StartAt": "FunWithMath",
"States": {
"FunWithMath": {
"Type": "Parallel",
"End": true,
"Branches": [
{
"StartAt": "Add", /// This receives some json object here input {}
"States": {
"Add": {
"Type": "Task", ***//How to pass the received input to the following arn as input?***
"Resource": ""arn:aws:states:::states:startExecution",
Parameters: {
"StateMachineArn": "anotherstepfunctionarnpath"
}
"End": true
}
}
},
{
"StartAt": "Subtract",
"States": {
"Subtract": {
"Type": "Task",
"Resource": "some lambda arn here,
"End": true
}
}
}
]
}
}
}
anotherstepfunctionarnpath :
{
"Comment": "Second state machine",
"StartAt": "stage1",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters":{
"Arguments":{
"Variable1" :"???" / how to access the value of the input passed to here
}
}
}
You can use Input to pass output from one SFN to other one:
First SFN(It will call second SFN)
{
"Comment": "My first SFN",
"StartAt": "First SFN",
"States": {
"First SFN": {
"Type": "Task",
"ResultPath": "$.to_pass",
"Resource": "arn:aws:lambda:us-east-1:807278658150:function:test-lambda",
"Next": "Trigger Next SFN"
},
"Trigger Next SFN": {
"Type": "Task",
"Resource": "arn:aws:states:::states:startExecution",
"Parameters": {
"Input": {
"Comment.$": "$"
},
"StateMachineArn": "arn:aws:states:us-east-1:807278658150:stateMachine:MyStateMachine2"
},
"End": true
}
}
}
Second SFN (MyStateMachine2)
{
"Comment": "A Hello World example of the Amazon States Language using Pass states",
"StartAt": "Hello",
"States": {
"Hello": {
"Type": "Pass",
"Result": "Hello",
"Next": "World"
},
"World": {
"Type": "Pass",
"Result": "World",
"End": true
}
}
}
First SFN's Execution
Second SFN's Execution
Explanation
The Lambda test-lambda is returning:
{
"user": "stackoverflow",
"id": "100"
}
Which is stored in "ResultPath": "$.to_pass" here in to_pass variable. I am passing the same output to next state machine MyStateMachine2 which is done by
"Input": {
"Comment.$": "$"
}
In the next State Machine's execution you see that same data is received as input which was created by first Lambda.
You can read more about it here.