I have a parallel task in the step function that contains two branches.
The input was:
{
"database": "test",
"userName": "tester",
"userID": "test123",
"adGroup": "testADGroup",
"dbGroup": "ReadGroup"
}
Each branch return a json result like the following
Branch 1 (I used "OutputPath": "$"):
{
"requestType": "GrantAccess",
"DBUser": "exists",
"ADUser": "exists"
}
Branch 2 (I used "ResultPath": "$.approvalStatus"):
{
"database": "test",
"userName": "tester",
"userID": "test123",
"adGroup": "testADGroup",
"dbGroup": "ReadGroup"
"approvalStatus": "Approved"
}
When both the branches complete, the output of the parallel task return:
[
{
"requestType": "GrantAccess",
"DBUser": "exists",
"ADUser": "exists"
},
{
"database": "test",
"userName": "tester",
"userID": "test123",
"adGroup": "testADGroup",
"dbGroup": "ReadGroup"
"approvalStatus": "Approved"
}
]
The next task is a choices,
"Choices": [
{
"Variable": "$.input[1].approvalStatus",
"StringEquals": "Approved",
"Next": "ProcessRequest"
},
{
"Variable": "$.input[1].approvalStatus",
"StringEquals": "Declined",
"Next": "SendDeclineNotification"
}
]
and it is keep giving me the following error:
"cause": "An error occurred while executing the state 'CheckApprovalStatus' (entered at the event id #16). Invalid path '$.input[1].approvalStatus': The choice state's condition path references an invalid value."
So here are my questions,
1) How should I reference it in the choice task to get the approvalStatus value?
2) Is there are anyway I can make the parallel task return in json format instead of an array?
Thanks in advance
I think you should use something like "$[1].approvalStatus" if you don't want to change the ResultPath.
Related
I am trying to access item properties which I am iterating over using Map state. I keep getting this error:
Value ($['Map'].snapshot_id.S) for parameter snapshotId is invalid. Expected: 'snap-...'. (Service: Ec2, Status Code: 400, Request ID: 6fc02935-c161-49df-8606-bc6f3e2934a6)
I have gone through the docs, which seems to suggest access using $.Map.snapshot_id.S but doesn't seem to work. Meanwhile, an input item to Map is:
{
"snapshot_id": {
"S": "snap-01dd5ee46df84119e"
},
"file_type": {
"S": "bash"
},
"id": {
"S": "64e6893261d94669b7a8ca425233d68b"
},
"script_s3_link": {
"S": "df -h"
}
}
Map state definition:
"Map": {
"Type": "Map",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "Parallel State",
"States": {
"Parallel State": {
"Comment": "A Parallel state can be used to create parallel branches of execution in your state machine.",
"Type": "Parallel",
"Branches": [
{
"StartAt": "CreateVolume",
"States": {
"CreateVolume": {
"Type": "Task",
"Parameters": {
"AvailabilityZone": "us-east-2b",
"SnapshotId": "$$.Map.snapshot_id.S"
},
"Resource": "arn:aws:states:::aws-sdk:ec2:createVolume",
"Next": "AttachVolume"
},
"AttachVolume": {
"Type": "Task",
"Parameters": {
"Device": "MyData",
"InstanceId": "MyData",
"VolumeId": "MyData"
},
"Resource": "arn:aws:states:::aws-sdk:ec2:attachVolume",
"End": true
}
}
}
],
"End": true
}
}
},
"Next": "Final Result",
"ItemsPath": "$.Items",
"MaxConcurrency": 40
},
TL;DR "SnapshotId.$": "$.snapshot_id"
By default, each inline map iteration's input is one item from the ItemsPath array, accessible simply as $.
Your state machine definition implies that $.Items is an array of objects with a snapshot_id key (and other keys). If so, each iteration's snapshot id is at $.snapshot_id.
Finally, adding .$ to the parameter name (SnapshotId.$) tells Step Functions the value is not a literal, but rather a path value to be substituted.
I am creating a Step Functions workflow which has various steps. I am referring to this topic in their documentation InputPath, ResultPath and OutputPath Examples. I am trying to check the identity and address of a person in my workflow as they've shown in their document. I'm passing the input for the Verify identity step within the state machine definition inside Parameters. My workflow looks like this.
Note: But when I run this, am getting the error -> An error occurred while executing the state 'Verify identity' (entered at the event id #19). Invalid path '$.identity' : Property ['identity'] not found in path $
What am I doing wrong here? Can someone please explain?
Thanks..
{
"StartAt": "Step1",
"States": {
"Step1": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
...something...
},
"Next": "Step2"
},
"Step2": {
"Type": "Choice",
"Choices": [
Do something...
],
"Default": "Step3.1"
},
"Step3.1": {
"Type": "Task",
...something...
}
},
"Next": "Step3.3"
},
...something...,
"Step4": {
"Type": "Parallel",
"Branches": [
{
"StartAt": "Verify identity",
"States": {
"Verify identity": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"InputPath": "$.identity",
"Parameters": {
"Payload": {
"identity": {
"email": "jdoe#example.com",
"ssn": "123-45-6789"
},
"firstName": "Jane",
"lastName": "Doe"
},
"FunctionName": "{Lambda ARN}"
},
"End": true
}
}
},
{
"StartAt": "Verify address",
"States": {
"Verify address": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"Payload": {
"street": "123 Main St",
"city": "Columbus",
"state": "OH",
"zip": "43219"
},
"FunctionName": "{Lambda ARN}"
},
"End": true
}
}
}
],
"Next": "Step5"
},
"Step5": {
"Type": "Task",
"Parameters": {
something...
},
"End": true
}
}```
You don't have an explicit transition in your example to call Step4 but assuming the order you have defined (step1 -> step2 -> step3.1 -> step3.3 -> step4)
This means the output from step3.3 should be something like
{
"cat": "meow",
"dog": "woof",
"identity": { // this is whats missing
"email": "jdoe#example.com",
"ssn": "123-45-6789"
}
}
this is what will get passed to each branch of your parallel state (Step4)
However, since you have anInputPath defined for Step4."Verify identity", the effective input to the task becomes
{
"email": "jdoe#example.com",
"ssn": "123-45-6789"
}
The error youre seeing
An error occurred while executing the state 'Verify identity' (entered at the event id #19). Invalid path '$.identity' : Property ['identity'] not found in path $
means the "identity" key (aka $.identity) isn't getting added to the output of Step3.3 (aka $)
I have 3 stages in my AWS Step Function:
Stage 1 - Lambda
Stage 2 - AWS Batch
Stage 3 - AWS Batch (Mandatory Cleanup)
Everything works fine in that if Stage 1 fails then it moves to the Cleanup stage. However, since the cleanup stage always passes, the Step Function's final result is always a Pass, whereas if Stage 1 or 2 fails, I need the Cleanup to be performed, yet the Step Function final result should be a fail.
Options investigated:
One way to solve this is to maintain a flag in a cache whether there is an error, but was wondering if there is an inbuilt way for this.
Another option is to use the Result Path to check for an error but I am not sure how to access this result from an AWS Batch.
Appreciate any advice on this, thanks.
I have added the following Catch block in Stage 1 and 2:
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "Cleanup"
}
]
The Cleanup stage is as follows:
"Cleanup": {
"Type": "Task",
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobDefinition": "arn:aws:batch:<region>:<account>:job-definition/MyCleanupJob",
"JobName": "cleanup",
"JobQueue": "arn:aws:batch:<region>:<account>:job-queue/MyCleanupQueue",
"ContainerOverrides": {
"Command": [
"java",
"-jar",
"cleanup.jar" ############ need to specify if an error occured as a command line parameter ###########
],
}
},
"End": true
}
Used below mechanism, credit for #LRutten for directing down this path.
For all success stages, append the response to the ResultPath else the previous results will be overwritten.
Set the error to the response path on an exception
Use a choice to decide if the step function should fail based on the presence of the error element
Here is the end output:
"MyLambda": {
"Type": "Task",
"Resource": "arn:aws:lambda:<region>:<account>:function:MyLambda",
"ResultPath": "$.mylambda", #### All results from the lambda are added to "mylambda" in the JSON
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"ResultPath": "$.error", #### If an error occurs it is appended to the result path as an "error" element
"Next": "Cleanup"
}
],
"Next": "MyBatch"
},
"MyBatch": {
"Type": "Task",
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobDefinition": "arn:aws:batch:<region>:<account>:job-definition/MyBatchJob",
"JobName": "cleanup",
"JobQueue": "arn:aws:batch:<region>:<account>:job-queue/MyBatchQueue",
"ContainerOverrides": {
"Command": [
"java",
"-jar",
"mybatch.jar"
],
}
},
"ResultPath": "$.mybatch",
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"ResultPath": "$.error",
"Next": "Cleanup"
}
],
"Next": "Cleanup"
},
"Cleanup": {
"Type": "Task",
"ResultPath": "$.cleanup",
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobDefinition": "arn:aws:batch:<region>:<account>:job-definition/MyCleanupJob",
"JobName": "cleanup",
"JobQueue": "arn:aws:batch:<region>:<account>:job-queue/MyCleanupQueue",
"ContainerOverrides": {
"Command": [
"java",
"-jar",
"cleanup.jar"
],
}
},
"Next": "Should Fail"
},
"Should Fail" :{
"Type" : "Choice",
"Choices" : [
{
"Variable" : "$.error", #### If an error element is present it means it is a Failure
"IsPresent": true,
"Next" : "Fail"
}
],
"Default" : "Pass"
},
"Fail" : {
"Type" : "Fail",
"Cause": "Step function failed"
},
"Pass" : {
"Type" : "Pass",
"Result": "Step function passed",
"End" : true
}
}
i got the error "The provided key element does not match the schema", while getting data from AWS dynamoDB using stepfunction.
stepfunction Defination
{
"Comment": "This is your state machine",
"StartAt": "Choice",
"States": {
"Choice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.data.Type",
"StringEquals": "GET",
"Next": "DynamoDB GetItem"
},
{
"Variable": "$.data.Type",
"StringEquals": "PUT",
"Next": "DynamoDB PutItem"
}
]
},
"DynamoDB GetItem": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:getItem",
"Parameters": {
"TableName": "KeshavDev",
"Key": {
"Email": {
"S": "$.Email"
}
}
},
"End": true
},
"DynamoDB PutItem": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"TableName": "KeshavDev",
"Item": {
"City": {
"S.$": "$.City"
},
"Email": {
"S.$": "$.Email"
},
"Address": {
"S.$": "$.Address"
}
}
},
"InputPath": "$.data",
"End": true
}
}
}
Input
{
"data": {
"Type": "GET",
"Email": "demo#gmail.com"
}
}
Error
{ "resourceType": "dynamodb", "resource": "getItem", "error":
"DynamoDB.AmazonDynamoDBException", "cause": "The provided key
element does not match the schema (Service: AmazonDynamoDBv2; Status
Code: 400; Error Code: ValidationException; Request ID:
a78c3d7a-ca3f-4483-b986-1735201d4ef2; Proxy: null)" }
I see some potential issues with the getItem task when compared to AWS documentation.
I think the Key field needs to be S.$ similar to what you have in your putItem task.
There is no ResultPath attribute to tell the state machine where to put the results.
Your path may not be correct, try $.data.Email
"DynamoDB GetItem": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:getItem",
"Parameters": {
"TableName": "KeshavDev",
"Key": {
"Email": {
"S.$": "$.data.Email"
}
}
},
"ResultPath": "$.DynamoDB",
"End": true
},
To be honest, I'm not sure if one of all of these are contributing to the validation error those are some things to experiment with.
On another note, there are some open source validators for Amazon State Language but for this case, they were not very helpful and said that your code was valid.
its working, above JD D mentoned steps and also by adding both key in step function definition.
DynamoDb have two key.
primary partition key
primary sort key
How do we use Parameters, ResultPath and ResultSelector to combine the results of a Task with the original input in the same JSON level?
I checked the documentation on AWS, but it seems that ResultSelector always create a new dictionary which puts it in 1-level below on the result.
Example input
{
"status": "PENDING",
"uuid": "00000000-0000-0000-0000-000000000000",
"first_name": "John",
"last_name": "Doe",
"email": "john.doe#email.com",
"orders": [
{
"item_uuid": "11111111-1111-1111-1111-111111111111",
"quantities": 2,
"price": 2.38,
"created_at": 16049331038000
}
]
}
State Machine definition
"Review": {
"Type": "Task",
"Resource": "arn:aws:states:us-east-1:123456789012:activity:Review",
"ResultPath": null,
"Next": "Processing",
"Parameters": {
"task_name": "REVIEW_REQUIRED",
"uuid.$": "$.uuid"
}
},
Example output from Review Activity
{
"review_status": "APPROVED"
}
Question
How do I update the State Machine definition to combined the result of Review Activity and the original input to something as below?
{
"status": "PENDING",
"uuid": "00000000-0000-0000-0000-000000000000",
"first_name": "John",
"last_name": "Doe",
"email": "john.doe#email.com",
"orders": [
{
"item_uuid": "11111111-1111-1111-1111-111111111111",
"quantities": 2,
"price": 2.38,
"created_at": 16049331038000
}
],
"review_status": "APPROVED"
}
NOTE
I don't have access to the Activity code, just the definition file.
I recommend NOT doing the way suggested above as you will drop all data that you do not include. It's not a long term approach, you can more easily do it like this:
Step Input
{
"a": "a_value",
"b": "b_value",
"c": {
"c": "c_value"
}
}
In your state-machine.json
"Flatten And Keep All Other Keys": {
"Type": "Pass",
"InputPath": "$.c.c",
"ResultPath": "$.c",
"Next": "Some Other State"
}
Step Output
{
"a": "a_value",
"b": "b_value",
"c": "c_value"
}
While Step Function does not allow you to do so, you can create a Pass state that flattens the input as a workaround.
Example Input:
{
"name": "John Doe",
"lambdaResult": {
"age": "35",
"location": "Eastern Europe"
}
}
Amazon State Language:
"Flatten": {
"State": "Pass",
"Parameters": {
"name.$" : "$.name",
"age.$" : "$.lambdaResult.age",
"location.$": "$.lambdaResult.location"
},
"Next": "MyNextState"
}
Output:
{
"name": "John Doe",
"age": "35",
"location": "Eastern Europe"
}
It's tedious, but it gets the job done.
Thanks for your question.
It looks like you don't necessarily need to manipulate the output in any way, and are looking for a way to combine the state's output with its input before passing it on to the next state. The ResultPath field allows you to combine a task result with task input, or to select one of these. The path you provide to ResultPath controls what information passes to the output.