Trying to update logic from checking fails within parallel flows in a AWS Step Function. The parallel flows have following logic to handle error:
"Subflow failed": {
"Type": "Pass",
"End": true,
"Result": true,
"ResultPath": "$.fail"
}
Once the parallel flows have finished the step function proceeds to check if any of the flows failed using the following logic:
"Any parallel flow fail?": {
"Type": "Choice",
"Choices": [
{
"Or": [
{
"And": [
{
"Variable": "$[0].fail",
"IsPresent": true
},
{
"Variable": "$[0].fail",
"BooleanEquals": true
}
]
},
This approach works but leads to an issue when developing new sub flows without adding a new block to check the fails in the array for new flow. I would like to find a more scalable method for checking the array if it contains the fail variable.
I have tried to use the States.ArrayContains but the usage and syntax is unfamiliar to me so I wasn't able to get valid syntax when trying to plug in into the check. Couldn't find any concrete examples of using the method online so any help is much appreciated.
Related
The below step function is executed in aws and when there is a missing of a required parameter it cancel the flow and throws States.Runtime Error. This is in catch phase of the step function but it is not catching the error as stated.
Defined Step function is as below,
{
"StartAt": "Log Start Step Function",
"Comment": "Executed with inputs",
"States": {
"Log Start Step Function": {
"Type": "Task",
"Resource": "arn:aws:lambda:eu-west-1:0000000:function:update",
"Parameters": {
"body": {
"itemID.$": "$.itemID",
"functionName.$": "$.stepFunctionName ",
"executionARN.$": "$$.Execution.Id",
"complete": false,
"inprogress": true,
"error": false
}
},
"Catch": [
{
"ErrorEquals": [
"States.Runtime"
],
"ResultPath": "$.taskresult",
"Next": "Log Failed Module"
},
{
"ErrorEquals": [
"States.ALL"
],
"ResultPath": "$.taskresult",
"Next": "Log Failed Module"
}
],
"ResultPath": "$.taskresult",
"Next": "Evaluate Module PA1"
}
}
}
Below is the step function,
And the error thrown is as below,
Runtime error is not executing Log failed module.
{
"ErrorEquals": [
"States.Runtime"
],
"ResultPath": "$.taskresult",
"Next": "Log Failed Module"
},
Is this AWS error or something wrong with the configuration which is done here or is there any other way to validate parameters in AWS Step Functions
From https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html
A States.Runtime error is not retriable, and will always cause the execution to fail. A retry or catch on States.ALL will not catch States.Runtime errors.
Your state machine is expecting the following as input:
"Parameters": {
"body": {
"itemID.$": "$.itemID",
"functionName.$": "$.stepFunctionName ",
"executionARN.$": "$$.Execution.Id",
"complete": false,
"inprogress": true,
"error": false
}
},
You need to pass them when you start a new execution instead of:
{
"Comment": "Insert your JSON here"
}
Which you are currently passing because it comes by default as the input body of a new execution in the AWS Console.
Read more about InputPath and Parameters here: https://docs.aws.amazon.com/step-functions/latest/dg/input-output-inputpath-params.html
I have the same problem.
I am beginning to think that the runtime error happens when the input path is processed, and before the catcher can be initialized. This means that try / catch to test for parameters present in the input is not possible. I also tried ChoiceState, to no avail.
So I think there is no solution but to provide every parameter you refer to in the state machine definition. But the documentation is not clear on this.
This caught me out too. My scenario was setting the output based on the results of S3 ListObjectVersions, with the versions to be deleted in a later task. In this case, $.Versions didn't exist because there was nothing in the bucket so States.Runtime was thrown.
{
"bucket.$": "$.Name",
"objects.$": "$.Versions"
}
To work around this -
I don't transform the result of ListObjectVersions task with ResultSelector. Instead this state simply outputs the unedited result.
I added a Choice state underneath with a rule to check if $.Versions is present.
If it is present, move to a Pass state and transform the input in the exact same was a I was originally transforming the result of the ListObjectVersions task using the ResultSelector (because the Pass state doesn't transform on output, only input).
If it is not present, move to a Success state because there is nothing to delete.
Here is a screen grab of the relevant section, just in case it's helpful to visualise.
I am trying to use AWS Step Functions to trigger operations many S3 files via Lambda. To do this I am invoking a step function with an input that has a base S3 key of the file and part numbers each file (each parallel iteration would operate on a different S3 file). The input looks something like
{
"job-spec": {
"base_file_name": "some_s3_key-",
"part_array": [
"part-0000.tsv",
"part-0001.tsv",
"part-0002.tsv", ...
]
}
}
My Step function is very simple, takes that input and maps it out, however I can't seem to get both the file and the array as input to my lambda. Here is my step function definition
{
"Comment": "An example of the Amazon States Language using a map state to process elements of an array with a max concurrency of 2.",
"StartAt": "Map",
"States": {
"Map": {
"Type": "Map",
"ItemsPath": "$.job-spec",
"ResultPath": "$.part_array",
"MaxConcurrency": 2,
"Next": "Final State",
"Iterator": {
"StartAt": "My Stage",
"States": {
"My Stage": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-east-1:<>:function:some-lambda:$LATEST",
"Payload": {
"Input.$": "$.part_array"
}
},
"End": true
}
}
}
},
"Final State": {
"Type": "Pass",
"End": true
}
}
}
As written above it complains that that job-spec is not an array for the ItemsPath. If I change that to $.job-spec.array I get the array I'm looking for in my lambda but the base key is missing.
Essentially I want each python lambda to get the base file key and one entry from the array to stitch together the complete file name. I can't just put the complete file names in the array due to the limit limit of how much data I can pass around in Step Functions and that also seems like a waste of data
It looks like the Parameters value can be used for this but I can't quite get the syntax right
Was able to finally get the syntax right.
"ItemsPath": "$.job-spec.part_array",
"Parameters": {
"part_name.$": "$$.Map.Item.Value",
"base_file_name.$": "$.job-spec.base_file_name"
},
It seems that Parameters can be used to create custom inputs for each stage. The $$ is accessing the context of the stage and not the actual input. It appears that ItemsPath takes the array and puts it into a context which can be used later.
UPDATE Here is some AWS Documentation showing this being used from the comments below
Update: Creating a step function from the Map State step template and running that also throws an error. This is strong evidence that the MaxConcurrency attribute together with the Parameters value is not working.
I am not able to use the MaxConcurrency attribute successfully in the step function definition.
This can be demonstrated by using the example provided in the documentation for the Map Task (new as of 18 sept 2019):
{
"StartAt": "ExampleMapState",
"States": {
"ExampleMapState": {
"Type": "Map",
"MaxConcurrency": 2,
"Parameters": {
"ContextIndex.$": "$$.Map.Item.Index",
"ContextValue.$": "$$.Map.Item.Value"
},
"Iterator": {
"StartAt": "TestPass",
"States": {
"TestPass": {
"Type": "Pass",
"End": true
}
}
},
"End": true
}
}
}
By executing the step function with the following input:
[
{
"who": "bob"
},
{
"who": "meg"
},
{
"who": "joe"
}
]
We can observe in the Execution event history that we get:
ExecutionStarted
MapStateEntered
MapStateStarted
MapIterationStarted (index 0)
MapIterationStarted (index 1)
PassStateEntered (index 0)
PassStateExited (index 0)
MapIterationSucceeded (index 0)
ExecutionFailed
The step function fails.
The ExecutionFailed step has the following output (execution id omitted):
{
"error": "States.Runtime",
"cause": "Internal Error (omitted)"
}
Trying to catch the error with a Catch step has no effect.
What am I doing wrong here? Is this a bug?
Response to a private ticket submitted to AWS this morning;
Thank you for contacting AWS Premium Support. My name is Akanksha and
I will be assisting you with this case.
I understand that you have been working with the new Map state feature
of step functions and have noticed that when we use Parameters along
with MaxConcurrency set to lower value than the number of iterations
(with only first iteration successful) it fails with ‘States.Runtime’
and looks like a bug with the functionality.
Thank you for providing the details. It helped me during
troubleshooting. In order to confirm the behavior, I used the below
state machine example with Pass:
{
"StartAt": "Map State",
"TimeoutSeconds": 3600,
"States": {
"Map State": {
"Type": "Map”,
"Parameters": {
“ContextValue.$”: "$$.Map.Item.Value"
},
"MaxConcurrency": 1,
"Iterator": {
"StartAt": "Run Task",
"States": {
"Run Task": {
"Type": "Pass",
"End": true
}
}
},
"Next": "Final State"
},
"Final State": {
"Type": "Pass",
"End": true
}
} }
I tested with multiple input lists and MaxConcurrency values and below
are my observations:
Input size list: 4 MaxConcurrency:1/2/3 - Fails and MaxConcurrency:0/4/5 or above - Works
Input size list: 3 MaxConcurrency: 1/2 - Fails and MaxConcurrency:0/3/4 or above - Works
Similarly, I performed tests by removing the parameters from state machine as well and could see that it works as expected with different
MaxConcurrency values.
I also tested the same by changing the Task type of “Pass” with “Lambda” and observed the same behavior.
Hence, I can confirm that the state machine fails when we have
parameters in the code and specify MaxConcurrency value as anything
other than zero or the number greater than or equal to the list size.
After doing some research regarding this behavior to check if this is
intended, I could not find much information regarding the same as this
is a new feature. So, I will be reaching out to the internal team with
all the details and the example state machine that you have provided.
Thank you for bringing this to our notice. I will get back to you as
soon as I have an update from the internal team. Please be assured
that I will regularly follow up with the team and work with them to
investigate further.
Meanwhile, if you have any other queries or concerns, please do let me
know.
Have a great day ahead!
I will update here when I get more information.
I have code that creates a segment when a queue is being read. In the first function (within the same lambda) I have this:
import * as AWSXRay from 'aws-xray-sdk'; // (using TypeScrpt)
AWSXRay.enableManualMode();
var segment1 = new AWSXRay.Segment("A");
In the second function (within the same lambda), called from the first, I have something like this:
var segment2 = new AWSXRay.Segment("B", segment1.trace_id, segment1.id);
Instead of seeing
*->A->B
On the AWS graph (on the website), I see:
*->A
*->B
...where they are not even associated, even though they have the same tracing ID, and the parent IDs are properly set. I seem to be missing something but not sure what...?
I even tried to pull X-Amzn-Trace-Id from the API request to use that as the root tracking ID for everything but that didn't work either.
This is the JSON for the first segment (A):
{
"Duration": 0.808,
"Id": "1-5d781a08-d41b49e35c3c0f38cdbd4912",
"Segments": [
{
"Document": {
"id": "74c99567f73185ce",
"name": "router",
"start_time": 1568152071.979,
"end_time": 1568152072.787,
"parent_id": "ef34fc0bcf23bbbe",
"aws": {
"xray": {
"sdk": "X-Ray for Node.js",
"sdk_version": "2.3.6",
"package": "aws-xray-sdk"
}
},
"service": {
"version": "unknown",
"runtime": "node",
"runtime_version": "v10.16.3",
"name": "unknown"
},
"trace_id": "1-5d781a08-d41b49e35c3c0f38cdbd4912"
},
"Id": "74c99567f73185ce"
}
]
}
This is the JSON for the second segment (B):
{
"Duration": 0.801,
"Id": "1-5d781a08-d9626abbab1cfbbfe4ff0dff",
"Segments": [
{
"Document": {
"id": "e2b4faaa6538bbb2",
"name": "handleCreateLoad",
"start_time": 1568152071.98,
"end_time": 1568152072.781,
"parent_id": "74c99567f73185ce",
"aws": {
"xray": {
"sdk": "X-Ray for Node.js",
"sdk_version": "2.3.6",
"package": "aws-xray-sdk"
}
},
"service": {
"version": "unknown",
"runtime": "node",
"runtime_version": "v10.16.3",
"name": "unknown"
},
"trace_id": "1-5d781a08-d9626abbab1cfbbfe4ff0dff",
"subsegments": [
{
"id": "08ccf2f374364066",
"name": "...-CreateLoad",
"start_time": 1568152071.981,
"end_time": 1568152072.781
}
]
},
"Id": "e2b4faaa6538bbb2"
}
]
}
It's quite clear the the parent ID for 'B' (74c99567f73185ce) points to "A"'s ID, but the graph does not connect them.
Also, I think _x_amzn_trace_id should be set when the lambda executes, but it is not. That may be root of my issues.
Turns out process.env._x_amzn_trace_id, required by the AWS XRay SDK, does NOT exist until the handler is called. It may help others to know what I went through:
At first I tried to get the trace details for the current lambda on start up (before the handler is called) to connect my new segments, but it didn't work. I have many handlers in the same project, so getting the lambda segment on startup is what I was hoping to do.
I then proceeded to create a main lambda segment (thinking I had to create the first segment myself) but all it did was create an orphaned segment. To make matters worse, each segment creates a new trace ID if one is not provided, and since I could not get the trace ID from the global start-up scope, nothing was connecting. The proper trace ID is important to pass along from start to finish for each request to make sure the calls down-stream are tracked properly.
Dumping of the environment variables before the handler is called and after clearly showed the trace ID is not provided until just before the handler gets called. It's sad that most of the online examples don't even bother to warn about this. I then moved the called to AWSXRay.getSegment() at the start of the lambda handler, then passed the details onto the child segments.
DO NOT set context.callbackWaitsForEmptyEventLoop = false while also calling the callback(error, response) callback passed to the lambda handler. Doing so will terminate the lambda without waiting for segment update events to flush to the daemon, resulting in orphaned segments. :(
Note: This documentation is lacking: https://docs.aws.amazon.com/xray-sdk-for-nodejs/latest/reference/
It states "You can retrieve the current segment or subsegment at any time" when in fact there are some times when you cannot. It's too bad there are no proper examples using actual working NodeJS Lambda code, instead of isolated lines of code thrown everywhere.
I have a lambda function that executes successfully with an intent called GetEvent that returns a specific string. I've created one utterance for this intent for testing purposes (one that is simple and doesn't require any of the optional slots for invoking the skill), but when using the service simulator to test the lambda function with this utterance for GetEvent I'm met with a lambda response that says "The response is invalid". Here is what the interaction model looks like:
#Intent Schema
{
"intents": [
{
"intent": "GetVessel",
"slots": [
{
"name": "boat",
"type": "LIST_OF_VESSELS"
},
{
"name": "location",
"type": "LIST_OF_LOCATIONS"
},
{
"name": "date",
"type": "AMAZON.DATE"
},
{
"name": "event",
"type": "LIST_OF_EVENTS"
}
]
},
{
"intent": "GetLocation",
"slots": [
{
"name": "event",
"type": "LIST_OF_EVENTS"
},
{
"name": "date",
"type": "AMAZON.DATE"
},
{
"name": "boat",
"type": "LIST_OF_VESSELS"
},
{
"name": "location",
"type": "LIST_OF_LOCATIONS"
}
]
},
{
"intent": "GetEvent",
"slots": [
{
"name": "event",
"type": "LIST_OF_EVENTS"
},
{
"name": "location",
"type": "LIST_OF_LOCATIONS"
}
]
}
]
}
With the appropriate custom skill type syntax and,
#First test Utterances
GetVessel what are the properties of {boat}
GetLocation where did {event} occur
GetEvent get me my query
When giving Alexa the utterance get me my query the lambda response should output the string as it did in the execution. I'm not sure why this isn't the case; this is my first project with the Alexa Skills Kit, so I am pretty new. Is there something I'm not understanding with how the lambda function, the intent schema and the utterances are all pieced together?
UPDATE: Thanks to some help from AWSSupport, I've narrowed the issue down to the area in the json request where new session is flagged as true. For the utterance to work this must be set to false (this works when inputting the json request manually, and this is also the case during the lambda execution). Why is this the case? Does Alexa really care about whether or not it is a new session during invocation? I've cross-posted this to the Amazon Developer Forums as well a couple of days ago, but have yet to get a response from someone.
This may or may not have changed -- the last time I used the service simulator (about two weeks ago at the time of writing) it had a pretty severe bug which would lead to requests being mapped to your first / wrong intent, regardless of actual simulated speech input.
So even if you typed in something random like wafaaefgae it simply tries to map that to the first intent you have defined, providing no slots to said intent which may lead to unexpected results.
Your issue could very well be related to this, triggering the same unexpected / buggy behavior because you aren't using any slots in your sample utterance
Before spending more time debugging this, I'd recommend trying the Intent using an actual echo or alternatively https://echosim.io/ -- interaction via actual speech works as expected, unlike the 'simulator'