AWS Step Function not falling into catch block - amazon-web-services

I'm running a state machine that has a map state and this map iterate over items of the type task which trigger other state machines. Now, when the child state machines fail, the parent state machine doesn't catch the error even though I've configured wildcard(States.ALL) for any type of error.
What am I, doing wrong? How to catch the error from the child state machine?
{
"StartAt": "Child",
"States": {
"Child": {
"Type": "Map",
"MaxConcurrency": 15,
"ItemsPath": "$.child_items",
"Iterator": {
"StartAt": "Execute child state machine",
"States": {
"Execute forecast state machine": {
"Type": "Task",
"Resource": "arn:aws:states:::states:startExecution.sync:2",
"Parameters": {
"StateMachineArn": "${ChildStateMachineArn}",
"Input": {
"epoch.$": "$.epoch",
"execution_date.$": "$.execution_date"
}
},
"End": true
}
}
},
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "Failure"
}
],
"Next": "Next Module",
"ResultPath": null
},
"Next Module": {
"Type": "Map",
"MaxConcurrency": 10,
"ItemsPath": "$.next_module",
"Iterator": {
.
.
.
},
"Next": "Success",
"ResultPath": null
}
.
.
.
.
.
.
}
}

Related

Break an map loop execution in AWS step functions

I'm trying to build a step function with a loop (Map) inside that can be stopped whenever a specefic Error is thrown, something like this
"Job": {
"Type": "Map",
"InputPath": "$.content",
"ItemsPath": "$.data",
"MaxConcurrency": 0,
"Iterator": {
"StartAt": "Validate",
"States": {
"Validate": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:ship-val",
"Catch": [
{
"ErrorEquals": [
"ErrorOne"
],
"Next": "BreakLoop"
},
{
"ErrorEquals": ["States.ALL"],
"Next": "FailUncaughtError"
}
],
},
"FailUncaughtError":{
"Type": "Fail",
"Error": "Uncaught error"
},
"BreakLoop":{
"Type": "Fail",
"Error": "the loop should be stopped"
}
}
},
"ResultPath": "$.content.data",
"End": true
}
I tried to make the Next element of the Catch to a state outside the Map but I couldn't because the Map accept only states within it. Moreover, AFAIK there is no mention for a feature like this in AWS docs
Instead of catching the error inside the Map state, don't catch it and let the Map state to fail.
And add a catch to Map state and if the error is equal to what you are looking for continue to the next step:
{
"StartAt": "Map",
"States": {
"Map": {
"Type": "Map",
"ItemsPath": "$.array",
"Iterator": {
"StartAt": "FaultyLambda",
"States": {
"FaultyLambda": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "your function arn",
"Payload": {
"a": 1
}
},
"End": true
}
}
},
"Catch": [
{
"ErrorEquals": ["ErrorOne"],
"Next": "BreakLoop"
}
],
"Next": "BreakLoop"
},
"BreakLoop": {
"Type": "Pass",
"End": true
}
}
}
Any other error will not be catched and failed your entire execution.

Workflow has no terminal state

I am creating a workflow with AWS Step Function where I am first checking if a record exists in database, then based on the records there are two branches and each of them end at either Succeed or Failed state, but I am still getting Workflow has no end state error.
Following is the JSON for workflow
{
"Comment": "A demo state machine",
"StartAt": "FindCategory",
"States": {
"FindCategory": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxx:function:xxxxxx",
"Next": "Exists?"
},
"Exists?": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.exists",
"BooleanEquals": true,
"Next": "Yes"
},
{
"Variable": "$.exists",
"BooleanEquals": false,
"Next": "No"
}
]
},
"Yes": {
"Type": "Pass",
"Next": "GetQuestions"
},
"GetQuestions": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxxxxxx",
"Next": "ReplyWithPolls"
},
"ReplyWithPolls": {
"Type": "Map",
"MaxConcurrency": 2,
"Iterator": {
"StartAt": "SendPoll",
"States": {
"SendPoll": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxxxx:xxxxxxx",
"Next": "SendPoll"
}
}
},
"Next": "Succeed"
},
"No": {
"Type": "Pass",
"Next": "FailState"
},
"Succeed": {
"Type": "Succeed"
},
"FailState": {
"Type": "Fail",
"Error": "404",
"Cause": "Category not found"
}
}
}
I believe the problem is that your SendPoll state results in an infinite loop. It references itself as next. Instead, the state in the iterator should be a terminal state.
Replace the "Next" field in "SendPoll" state with an "End" field.
"SendPoll": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxxxx:xxxxxxx",
"End": true
}

Can AWS Step Function describe this kind of dataflow?

It can not be described with Parallel State in AWS Step Function.
B and C should be in parallel.
C sends messages to both D and E.
D and E should be in parallel.
{
"StartAt": "A",
"States": {
"A": {
"Type": "Pass",
"Next": "Parallel State 1"
},
"Parallel State 1": {
"Type": "Parallel",
"Branches": [{
"StartAt": "B",
"States": {
"B": {
"Type": "Pass",
"End": true
}
}
},
{
"StartAt": "C",
"States": {
"C": {
"Type": "Pass",
"End": true
}
}
}
],
"Next": "Parallel State 2"
},
"Parallel State 2": {
"Type": "Parallel",
"Branches": [{
"StartAt": "D",
"States": {
"D": {
"Type": "Pass",
"End": true
}
}
},
{
"StartAt": "E",
"States": {
"E": {
"Type": "Pass",
"End": true
}
}
}
],
"Next": "F"
},
"F": {
"Type": "Pass",
"End": true
}
}
}
Answer is No , inside step function no state can set multiple states (invokes both successors)to its Next task. As per AWS step function cannot start State Machine as StartAt by providing multiple State names.
You can tweak your logic and use The Parallel state and achive same ,If you share your usecase may be help to solve problems.
How to specify multiple result path values in AWS Step Functions
A Parallel state provides each branch with a copy of its own input
data (subject to modification by the InputPath field). It generates
output that is an array with one element for each branch, containing
the output from that branch.
https://aws.amazon.com/blogs/aws/new-step-functions-support-for-dynamic-parallelism/
Example of state function
{
"Comment": "An example of the Amazon States Language using a choice state.",
"StartAt": "FirstState",
"States": {
"FirstState": {
"Type": "Task",
"Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
"Next": "ChoiceState"
},
"ChoiceState": {
"Type" : "Choice",
"Choices": [
{
"Variable": "$.foo",
"NumericEquals": 1,
"Next": "FirstMatchState"
},
{
"Variable": "$.foo",
"NumericEquals": 2,
"Next": "SecondMatchState"
}
],
"Default": "DefaultState"
},
"FirstMatchState": {
"Type" : "Task",
"Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:OnFirstMatch",
"Next": "NextState"
},
"SecondMatchState": {
"Type" : "Task",
"Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:OnSecondMatch",
"Next": "NextState"
},
"DefaultState": {
"Type": "Fail",
"Error": "DefaultStateError",
"Cause": "No Matches!"
},
"NextState": {
"Type": "Task",
"Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
"End": true
}
}
}
https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-example
https://sachabarbs.wordpress.com/2018/10/30/aws-step-functions/
As I answered in How to simplify complex parallel branch interdependencies for Step Functions, what you asked is better to be modeled as DAG but not state machine.
Depends on your use case, you might be able to workaround it (just as #horatiu-jeflea 's answer), but it's a workaround (not the straightforward way) anyway.

How to capture output of a parallel state machine in AWS Step function

I need to run an AWS Step function that runs a parallel state machine running, say two state machines. My requirement is to check the final execution status of the parallel machine and if there is any failure, invoke an SNS service to send out an email. Pretty standard stuff but for the life of me, i can't figure out how to capture the combined error of a parallel step machine. This sample parallel machine runs
A "passtask" that is just a simple lambda pass function,
and
Runs a failtask that has a sleep timer for 5 seconds and is suppposed to fail after 5 seconds.
If I execute this machine, this machine correctly shows passtask as succeeded, failtask as cancelled, Overall Parallel Task as succeeded (?????), Notify Failure task as cancelled and the overall execution of state machine as "failed" as well.
I'd like to see passtask as succeeded, fail task as failed, overall Parallel Task as Failed, Notify Failure task as succeeded.
{
"Comment": "Parallel Example",
"StartAt": "Parallel Task",
"TimeoutSeconds": 120,
"States": {
"Parallel Task": {
"Type": "Parallel",
"Branches": [
{
"StartAt": "passtask",
"States": {
"passtask": {
"Type": "Task",
"Resource":"arn:xxxxxxxxxxxxxxx:function:passfunction",
"End": true
}
}
},
{
"StartAt": "failtask",
"States": {
"failtask": {
"Type": "Task",
"Resource":"arn: xxxxxxxxxxxxxxx:function:failfunction",
"End": true
}
}
}
],
"ResultPath": "$.status",
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "Notify Failure"
}
],
"Next": "Notify Success"
},
"Notify Failure": {
"Type": "Pass",
"InputPath": "$.input.Cause",
"End": true
},
"Notify Success": {
"Type": "Pass",
"Result": "This is a fallback from a task success",
"End": true
}
}
}
From your requirment "My requirement is to check the final execution status of the parallel machine and if there is any failure, invoke an SNS service to send out an email.", I understand that the "failtask" is just for debugging purposes and in the future it won't neccesarily fail. So the problem is, the moment Step Functions detect a failure in a branch all other branches are terminated and their outputs discarded, only the failed branch's output is used. So if you want to preserve the output of each Branch and check if a failure has occured, you will need to handle the errors in each branch and not report the whole branch as failed. Additionally you will need to add an output field to each branch which says if there was a failure or not (Choice State will give an error if a field does not exist). And also remember that the output of a ParralelState is an array with the output of each Branch, for example this State Machine should let each branch finish execution and handle the errors correctly:
{
"Comment": "Parallel Example",
"StartAt": "Parallel Task",
"TimeoutSeconds": 120,
"States": {
"Parallel Task": {
"Type": "Parallel",
"Branches": [{
"StartAt": "passtask",
"States": {
"passtask": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:XXXXXXXXXXXXXXXXX",
"Next": "SuccessBranch1",
"Catch": [{
"ErrorEquals": ["States.ALL"],
"Next": "FailBranch1"
}]
},
"SuccessBranch1": {
"Type": "Pass",
"Result": {
"Error": false
},
"ResultPath": "$.Status",
"End": true
},
"FailBranch1": {
"Type": "Pass",
"Result": {
"Error": true
},
"ResultPath": "$.Status",
"End": true
}
}
},
{
"StartAt": "failtask",
"States": {
"failtask": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:XXXXXXXXXXXXXXXXX",
"Next": "SuccessBranch2",
"Catch": [{
"ErrorEquals": ["States.ALL"],
"Next": "FailBranch2"
}]
},
"SuccessBranch2": {
"Type": "Pass",
"Result": {
"Error": false
},
"ResultPath": "$.Status",
"End": true
},
"FailBranch2": {
"Type": "Pass",
"Result": {
"Error": true
},
"ResultPath": "$.Status",
"End": true
}
}
}
],
"ResultPath": "$.ParralelOutput",
"Catch": [{
"Comment": "This catch should never catch any errors, as the error handling is done in the individual Branches",
"ErrorEquals": ["States.ALL"],
"ResultPath": "$.ParralelOutput",
"Next": "ChoiceStateX"
}],
"Next": "ChoiceStateX"
},
"ChoiceStateX": {
"Type": "Choice",
"Choices": [{
"Or": [{
"Variable": "$.ParralelOutput[0].Status.Error",
"BooleanEquals": true
},
{
"Variable": "$.ParralelOutput[1].Status.Error",
"BooleanEquals": true
}
],
"Next": "Notify Failure"
}],
"Default": "Notify Success"
},
"Notify Failure": {
"Type": "Pass",
"End": true
},
"Notify Success": {
"Type": "Pass",
"Result": "This is a fallback from a task success",
"End": true
}
}
}
For a more general case (although more complex) of the above as asked by Nisman in the comments. Instead of hardcoding the Choice State to check for every branch we can add a pass state with some JSONPath tricks to check for conditions not currently possible with a choice state alone.
Inside this Pass State we use Parameters to restructure our data in such a way that when we apply a JSONPath filter expression to this data using the OutputPath we are left with an array of either 2 (if no branches failed) or 3 (if some branches failed) elements, where the first element always contains the original input data and the second/third contains at least 1 key with the same name to be used by the choice state. Here's the State Machine JSON:
{
"Comment": "Parallel Example",
"StartAt": "Parallel Task",
"States": {
"Parallel Task": {
"Type": "Parallel",
"Branches": [
{
"StartAt": "passtask",
"States": {
"passtask": {
"Type": "Task",
"Resource": "<TASK RESOURCE>",
"End": true,
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"ResultPath": "$.error-info",
"Next": "FailBranch1"
}
]
},
"FailBranch1": {
"Type": "Pass",
"Parameters": {
"BranchOutput.$": "$",
"BranchError": true
},
"End": true
}
}
},
{
"StartAt": "failtask",
"States": {
"failtask": {
"Type": "Task",
"Resource": "<TASK RESOURCE>",
"End": true,
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"ResultPath": "$.error-info",
"Next": "FailBranch2"
}
]
},
"FailBranch2": {
"Type": "Pass",
"Parameters": {
"BranchOutput.$": "$",
"BranchError": true
},
"End": true
}
}
}
],
"ResultPath": "$.ParralelOutput",
"Next": "Pre-Process"
},
"Pre-Process": {
"Type": "Pass",
"Parameters": {
"OrderedArray": [
{
"OriginalData": {
"Input.$": "$",
"ShouldFilterData": false
}
},
{
"ValuesToCheck": {
"ListBranchErrors.$": "$.ParralelOutput[?(#.BranchError==true)].BranchError",
"BranchFailures": true
}
},
{
"DefaultAlwaysFalse": {
"ShouldFilterData": false,
"BranchFailures": false
}
}
]
},
"OutputPath": "$..[?(#.ShouldFilterData == false || #.ListBranchErrors[0] == true)]",
"Next": "ChoiceStateX"
},
"ChoiceStateX": {
"Type": "Choice",
"OutputPath": "$.[0].Input",
"Choices": [
{
"Variable": "$[1].BranchFailures",
"BooleanEquals": true,
"Next": "NotifyFailure"
},
{
"Variable": "$[1].BranchFailures",
"BooleanEquals": false,
"Next": "NotifySuccess"
}
],
"Default": "NotifyFailure"
},
"NotifyFailure": {
"Type": "Pass",
"End": true
},
"NotifySuccess": {
"Type": "Pass",
"Result": "This is a fallback from a task success",
"End": true
}
}
}

Parallel States Merge the output in Step Function

Is it possible to have following kind of Step Function graph, i.e. from 2 parallel state output, one combined state:
If yes, what would json for this looks like? If not, why?
A parallel task always outputs an array (containing one entry per branch).
You can tell AWS step functions to append the output into new (or existing) property in the original input with "ResultPath": "$.ParallelOut" in your parallel state definition, but this is not what you seem to be trying to achieve.
To merge the output of parallel task, you can leverage the "Type": "Pass" state to define transformations to apply to the JSON document.
For example, in the state machine below, I'm transforming a JSON array...
[
{
"One": 1,
"Two": 2
},
{
"Foo": "Bar",
"Hello": "World"
}
]
...into a few properties
{
"Hello": "World",
"One": 1,
"Foo": "Bar",
"Two": 2
}
{
"Comment": "How to convert an array into properties",
"StartAt": "warm-up",
"States": {
"warm-up": {
"Type": "Parallel",
"Next": "array-to-properties",
"Branches": [
{
"StartAt": "numbers",
"States": {
"numbers": {
"Type": "Pass",
"Result": {
"One": 1,
"Two" : 2
},
"End": true
}
}
},
{
"StartAt": "words",
"States": {
"words": {
"Type": "Pass",
"Result": {
"Foo": "Bar",
"Hello": "World"
},
"End": true
}
}
}
]
},
"array-to-properties": {
"Type": "Pass",
"Parameters": {
"One.$": "$[0].One",
"Two.$": "$[0].Two",
"Foo.$": "$[1].Foo",
"Hello.$": "$[1].Hello"
},
"End": true
}
}
}
It is possible as opposed diagram below
The parallel state should look like this
"MyParallelState": {
"Type": "Parallel",
"InputPath": "$",
"OutputPath": "$",
"ResultPath": "$.ParallelResultPath",
"Next": "SetCartCompleteStatusState",
"Branches": [
{
"StartAt": "UpdateMonthlyUsageState",
"States": {
"UpdateMonthlyUsageState": {
"Type": "Task",
"InputPath": "$",
"OutputPath": "$",
"ResultPath": "$.UpdateMonthlyUsageResultPath",
"Resource": "LambdaARN",
"End": true
}
}
},
{
"StartAt": "QueueTaxInvoiceState",
"States": {
"QueueTaxInvoiceState": {
"Type": "Task",
"InputPath": "$",
"OutputPath": "$",
"ResultPath": "$.QueueTaxInvoiceResultPath",
"Resource": "LambdaARN",
"End": true
}
}
}
The output of MyParallelState will be populated as in array, from each state in the Parallel state. They are populated within ParallelResultPath object and will be passed into the Next state
{
"ParallelResultPath": [
{
"UpdateMonthlyUsageResultPath": Some Output
},
{
"QueueTaxInvoiceResultPath": Some Output
}
]
}
We can use ResultSelector and Result Path to combine the result into one object
We have a parallel state like:
{
"StartAt": "ParallelBranch",
"States": {
"ParallelBranch": {
"Type": "Parallel",
"ResultPath": "$",
"InputPath": "$",
"OutputPath": "$",
"ResultSelector": {
"UsersResult.$": "$[1].UsersUpload",
"CustomersResult.$": "$[0].customersDataUpload"
},
"Branches": [
{
"StartAt": "customersDataUpload",
"States": {
"customersDataUpload": {
"Type": "Pass",
"ResultPath": "$.customersDataUpload.Output",
"Result": {
"CompletionStatus": "success",
"CompletionDetails": null
},
"Next": "Wait2"
},
"Wait2": {
"Comment": "A Wait state delays the state machine from continuing for a specified time.",
"Type": "Wait",
"Seconds": 2,
"End": true
}
}
},
{
"StartAt": "UsersUpload",
"States": {
"UsersUpload": {
"Type": "Pass",
"Result": {
"CompletionStatus": "success",
"CompletionDetails": null
},
"ResultPath": "$.UsersUpload.Output",
"Next": "Wait1"
},
"Wait1": {
"Comment": "A Wait state delays the state machine from continuing for a specified time.",
"Type": "Wait",
"Seconds": 1,
"End": true
}
}
}
],
"End": true
}
},
"TimeoutSeconds": 129600,
"Version": "1.0"
}
enter image description here
And the output will be like:
{
"UsersResult": {
"Output": {
"CompletionStatus": "success",
"CompletionDetails": null
}
},
"CustomersResult": {
"Output": {
"CompletionStatus": "success",
"CompletionDetails": null
}
}
}
Your diagram is technically wrong because no state can set multiple states to its Next task. You cannot start State Machine as StartAt by providing multiple State names. Also, even if it was possible I don't see any point why would you want to run two parallel states as opposed to one parallel state with all the sub states that you would split into two.
This worked for me
"Transform And Freeze": {
"Type": "Parallel",
"InputPath": "$",
"Branches": [
{
"StartAt": "Transform Status",
"States": {
"Transform Status": {
"Type": "Map",
"ItemsPath": "$",
"MaxConcurrency": 25,
"Iterator": {
"StartAt": "Transform",
"States": {
"Transform": {
"Type": "Task",
"Resource": "${TransformFunction}",
"End": true
}
}
},
"End": true
}
}
},
{
"StartAt": "Freeze Status",
"States": {
"Freeze Status": {
"Type": "Map",
"MaxConcurrency": 25,
"Iterator": {
"StartAt": "Freeze",
"States": {
"Freeze Transactions": {
"Type": "Task",
"Resource": "${FreezeFunction}",
"End": true
}
}
},
"End": true
}
}
}
],
"ResultPath" : "$.parts",
"Next": "SetParallelOutput",
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"ResultPath": "$.exception",
"Next": "Error Handler"
}
]
},
"SetParallelOutput": {
"Type": "Pass",
"Parameters": {
"foo.$": "$.foo",
"bar.$": "$.bar",
"parts.$": "$.parts[0]"
},
"Next": "Target Type"
},