AWS Step Function - Dynamically select branches to run in parallel - amazon-web-services

I am working on a solution where a state functions is suppose to fetch the required functions to be completed(based on input) and then needs to trigger those functions in parallel, so that i have a consolidated output in the end. This can be illustrated as below -
State Machine Graph
After the requirements based on the input has been fetched, the next state can be any combination of the states below but they do need to run in parallel so that i know all required tasks have been completed. I do not want to keep a bunch of choices or deciders, since the umber of required steps can grow in the future. Is it possible to to decide on which branches are required to be run in parallel and run them.

we will need a choice within each branch.
{
"StartAt":"Build Decider Step Output",
"States":{
"Build Decider Step Output":{
"Type":"Pass",
"Result":{
"firstReq":true,
"secondReq":true,
"thridReq":false
},
"ResultPath":"$.decider",
"Next":"my-parallel"
},
"my-parallel":{
"Type":"Parallel",
"End":true,
"Branches":[
{
"StartAt":"should we run first step?",
"States":{
"should we run first step?":{
"Type":"Choice",
"Choices":[
{
"Variable":"$.decider.firstReq",
"BooleanEquals":true,
"Next":"First Requirement"
}
],
"Default":"First Req Skipped"
},
"First Req Skipped":{
"Type":"Pass",
"End":true
},
"First Requirement":{
"Type":"Pass",
"End":true
}
}
},
{
"StartAt":"should we run second step?",
"States":{
"should we run second step?":{
"Type":"Choice",
"Choices":[
{
"Variable":"$.decider.secondReq",
"BooleanEquals":true,
"Next":"Second Requirement"
}
],
"Default":"Second Req Skipped"
},
"Second Req Skipped":{
"Type":"Pass",
"End":true
},
"Second Requirement":{
"Type":"Pass",
"End":true
}
}
},
{
"StartAt":"should we run thrid step?",
"States":{
"should we run thrid step?":{
"Type":"Choice",
"Choices":[
{
"Variable":"$.decider.thridReq",
"BooleanEquals":true,
"Next":"Third Requirement"
}
],
"Default":"Third Req Skipped"
},
"Third Req Skipped":{
"Type":"Pass",
"End":true
},
"Third Requirement":{
"Type":"Pass",
"End":true
}
}
}
]
}
}
}
I mocked this json in first step output
{
"firstReq":true,
"secondReq":true,
"thridReq":false
}
and resulted in below, skipping third branch.

Related

How to pass parameters to all of the inner workflow tasks inside a Map task?

I am trying to use a Map task inside my State Machine. The design is simple - I have a lambda function that generates a list that the Map state uses. Each task inside the Map task should have access to the list's properties.
Right now I am observing that only the 1st task "ETL" get's each list element as an input but not the second task "WELL CALC".
My Step Machine fails with
{
"error": "States.Runtime",
"cause": "An error occurred while executing the state 'WELL CALC' (entered at the event id #15). The JSONPath '$.basinName' specified for the field 'basin.$' could not be found in the input '{}'"
}
The above error is true, the result of the ETL lambda task returns an empty object. Is there a way I can pass the "current item" of the Map iteration to all of the tasks as input?
This is my State Machine definition:
{
"StartAt": "Generate Basin List",
"States": {
"Generate Basin List": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:451713003466:function:StepFunctionsSample-HelloLam-CheckStockPriceLambda-15TO7SRHAQYQV",
"Next": "Process Basins"
},
"Process Basins": {
"Type": "Map",
"ItemsPath": "$.BasinList",
"End": true,
"Iterator": {
"StartAt": "ETL",
"States": {
"ETL": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:451713003466:function:StepFunctionsSample-HelloLambda0b4d-BuyStockLambda-1WW8B8JQSZ0SY",
"Next": "WELL CALC"
},
"WELL CALC": {
"Type": "Task",
"End": true,
"Parameters": {
"basin.$":"$.basinName",
"phases.$":"$.phases"
},
"Resource": "arn:aws:lambda:us-east-1:451713003466:function:StepFunctionsSample-HelloLambda0b4-SellStockLambda-15I8EE3RWDMS5"
}
}
}
}
}
}
This is the "Generate Basin List" lambda:
exports.lambdaHandler = async (event, context) => {
return {
"BasinList": [
{
"basinName": "ARDMORE_AND_MARIETTA_BASINS",
"phases": "1,4,5,8,9,10,13"
}
]
}
};
It turns out that I can add "ResultPath": null field to the first state inside the Map - ETL which would result in the whole input state being passed as an output to the next state (source - https://states-language.net/spec.html#filters):
...
"ETL": {
"Type": "Task",
"Resource": "arn:aws:lambda:us..."
"Next": "WELL CALC",
"ResultPath": null
},
...

How do I output original input of a state in a step function?

I have a step function with the following definition:
{
"StartAt": "A",
"States": {
"A": {
"Type": "Task",
"Resource": "do something",
"Next": "B"
},
"B": {
"Type": "Task",
"Resource": "do something",
"End": true
}
}
}
The problem is the input for the state B. I need it to be the same as the input for the state A. Currently however the input for the step B is the output of the step A. Taken into account that step A in fact calls a different step function or performs a DynamoDB operation (no lambda involved), there's not much I can do about the output of that step, but the step B still needs to receive the same input as step A originally did. How can I define this?
Set ResultPath: null in state A to discard the result and leave the state unchanged.
You can execute tasks in Parallel, something along the lines:
"A_And_B":{
"Type":"Parallel",
"Branches":[
{
"StartAt":"A",
"States":{
"A":{
"Type":"Task",
"Resource":"do something",
"Next":"B"
}
}
},
{
"StartAt":"B",
"States":{
"B":{
"Type":"Task",
"Resource":"do something",
"End":true
}
}
}
],
"Next":"NextState"
}

How to skip particular set of test cases in a collection in terminal using newman?

I am having a postman collection which consists of request and test cases for each requests. I have two test case for each request. one for validating status code and other for validating response time. I need to execute status code test case frequently and response time test case results occasionally.How to achieve it without modifying the collection for every run and is it achievable in providing any option in terminal?
collection.json
{
"name": "Metadata",
"item": [
{
"name": "info",
"event": [
{
"listen": "test",
"script": {
"id": "32cf67e7-5d42-4231-86fe-e7fffa32c855",
"exec": [
"pm.test(\"Status code is 200\", function () {",
" pm.response.to.have.status(200);",
"});",
"pm.test(\"Response time is less than 300ms\", function () {",
" pm.expect(pm.response.responseTime).to.be.below(300);",
"});"
],
"type": "text/javascript"
}
}
],
"request": {
"auth": {
"type": "bearer",
"bearer": [
{
"key": "token",
"value": "{{tokenAdmin}}",
"type": "string"
}
]
},
"method": "GET",
"header": [],
"url": {
"raw": "{{url}}/api/m0/metadata/info",
"host": [
"{{url}}"
],
"path": [
"api",
"m0",
"metadata",
"info"
]
}
},
"response": []
}
],
"protocolProfileBehavior": {},
"_postman_isSubFolder": true
}
For a very basic flow, you can use moment to check which day it currently is and if that matches the condition, it will run the responseTime test.
let moment = require('moment'),
date = moment().format('dddd');
// Runs on each request
pm.test("Status code is 200", function () {
pm.response.to.have.status(200);
});
// Only runs on a Friday
if (date === 'Friday') {
pm.test("Response time is less than 1000ms", function () {
pm.expect(pm.response.responseTime).to.be.below(1000);
});
}
Moment has lots of different options available to you and might work if you want to only run that check at the end of the sprint or on a given day.

Amazon States Language Errors

I am creating an AWS State machine - I am getting an error:
Here is the site that the error links too: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-amazon-states-language.html
I have been combing through my state machine code for hours, and I can't figure out what is wrong with it. Previously when I had errors, they showed up in the GUI - but for this, it just says there is an error with no indication of where the error is from.
Here is my state machine visualized:
Here is the code for my state machine:
{
"StartAt":"Pass",
"States":{
"Pass":{
"Type":"Pass",
"Next":"Transform 1"
},
"Send Notification 1":{
"Type":"Task",
"Resource":"arn:aws:states:::lambda:invoke",
"Parameters":{
"FunctionName":"arn:aws:lambda:us-east-1:432700302163:function:CheckInterviewStatus:$LATEST",
"Payload":{
"Input.$":"$"
}
},
"Catch":[
{
"ErrorEquals":[
"States.ALL"
],
"Next":"CatchAllFallback"
}
],
"Next":"Interview Completed 1"
},
"Send Notification 2":{
"Type":"Task",
"Resource":"arn:aws:states:::lambda:invoke",
"Parameters":{
"FunctionName":"arn:aws:lambda:us-east-1:432700302163:function:CheckInterviewStatus:$LATEST",
"Payload":{
"Input.$":"$.Payload"
}
},
"Catch":[
{
"ErrorEquals":[
"States.ALL"
],
"Next":"CatchAllFallback"
}
],
"Next":"Interview Completed 2"
},
"CatchAllFallback":{
"Type":"Pass",
"Result":"This is a fallback from any error code",
"End":false
},
"Send Notification 3":{
"Type":"Task",
"Resource":"arn:aws:states:::lambda:invoke",
"Parameters":{
"FunctionName":"arn:aws:lambda:us-east-1:432700302163:function:CheckInterviewStatus:$LATEST",
"Payload":{
"Input.$":"$.Payload"
}
},
"Catch":[
{
"ErrorEquals":[
"States.ALL"
],
"Next":"CatchAllFallback"
}
],
"Next":"Interview Completed 3"
},
"CatchAllFallback":{
"Type":"Pass",
"Result":"This is a fallback from any error code",
"End":false
},
"Send Notification 4":{
"Type":"Task",
"Resource":"arn:aws:states:::lambda:invoke",
"Parameters":{
"FunctionName":"arn:aws:lambda:us-east-1:432700302163:function:CheckInterviewStatus:$LATEST",
"Payload":{
"Input.$":"$.Payload"
}
},
"Catch":[
{
"ErrorEquals":[
"States.ALL"
],
"Next":"CatchAllFallback"
}
],
"Next":"Interview Completed 4"
},
"CatchAllFallback":{
"Type":"Pass",
"Result":"This is a fallback from any error code",
"End":false
},
"CatchAllFallback":{
"Type":"Pass",
"Result":"This is a fallback from any error code",
"End":false
},
"Interview Completed 1":{
"Type":"Choice",
"Choices":[
{
"Variable":"$.Payload.completed",
"BooleanEquals":true,
"Next":"Yes 1"
},
{
"Variable":"$.Payload.completed",
"BooleanEquals":false,
"Next":"No 1"
}
],
"Default":"No 1"
},
"No 1":{
"Type":"Pass",
"Next":"Wait 1"
},
"Yes 1":{
"Type":"Pass",
"End":true
},
"Interview Completed 2":{
"Type":"Choice",
"Choices":[
{
"Variable":"$.Payload.completed",
"BooleanEquals":true,
"Next":"Yes 2"
},
{
"Variable":"$.Payload.completed",
"BooleanEquals":false,
"Next":"No 2"
}
],
"Default":"No 2"
},
"No 2":{
"Type":"Pass",
"Next":"Wait 2"
},
"Yes 2":{
"Type":"Pass",
"End":true
},
"Interview Completed 3":{
"Type":"Choice",
"Choices":[
{
"Variable":"$.Payload.completed",
"BooleanEquals":true,
"Next":"Yes 3"
},
{
"Variable":"$.Payload.completed",
"BooleanEquals":false,
"Next":"No 3"
}
],
"Default":"No 3"
},
"No 3":{
"Type":"Pass",
"Next":"Wait 3"
},
"Yes 3":{
"Type":"Pass",
"End":true
},
"Interview Completed 4":{
"Type":"Choice",
"Choices":[
{
"Variable":"$.Payload.completed",
"BooleanEquals":true,
"Next":"Yes 4"
},
{
"Variable":"$.Payload.completed",
"BooleanEquals":false,
"Next":"No 4"
}
],
"Default":"No 4"
},
"No 4":{
"Type":"Pass",
"End":true
},
"Yes 4":{
"Type":"Pass",
"End":true
},
"Transform 1":{
"Type":"Pass",
"Result":0,
"Next":"Send Notification 1"
},
"Transform 2":{
"Type":"Pass",
"Result":2,
"Next":"Send Notification 2"
},
"Transform 3":{
"Type":"Pass",
"Result":2,
"Next":"Send Notification 3"
},
"Transform 4":{
"Type":"Pass",
"Result":1,
"Next":"Send Notification 4"
},
"Wait 1":{
"Type":"Wait",
"Seconds":1,
"Next":"Transform 2"
},
"Wait 2":{
"Type":"Wait",
"Seconds":1,
"Next":"Transform 3"
},
"Wait 3":{
"Type":"Wait",
"Seconds":1,
"Next":"Transform 4"
}
}
}
A pointer in the right direction would be extremely helpful.
I figured it out.
"CatchAllFallback":{
"Type":"Pass",
"Result":"This is a fallback from any error code",
"End":false
},
Was defined multiple times - which was causing the error.

How reference AWS step function parallel task output?

I have a parallel task in the step function that contains two branches.
The input was:
{
"database": "test",
"userName": "tester",
"userID": "test123",
"adGroup": "testADGroup",
"dbGroup": "ReadGroup"
}
Each branch return a json result like the following
Branch 1 (I used "OutputPath": "$"):
{
"requestType": "GrantAccess",
"DBUser": "exists",
"ADUser": "exists"
}
Branch 2 (I used "ResultPath": "$.approvalStatus"):
{
"database": "test",
"userName": "tester",
"userID": "test123",
"adGroup": "testADGroup",
"dbGroup": "ReadGroup"
"approvalStatus": "Approved"
}
When both the branches complete, the output of the parallel task return:
[
{
"requestType": "GrantAccess",
"DBUser": "exists",
"ADUser": "exists"
},
{
"database": "test",
"userName": "tester",
"userID": "test123",
"adGroup": "testADGroup",
"dbGroup": "ReadGroup"
"approvalStatus": "Approved"
}
]
The next task is a choices,
"Choices": [
{
"Variable": "$.input[1].approvalStatus",
"StringEquals": "Approved",
"Next": "ProcessRequest"
},
{
"Variable": "$.input[1].approvalStatus",
"StringEquals": "Declined",
"Next": "SendDeclineNotification"
}
]
and it is keep giving me the following error:
"cause": "An error occurred while executing the state 'CheckApprovalStatus' (entered at the event id #16). Invalid path '$.input[1].approvalStatus': The choice state's condition path references an invalid value."
So here are my questions,
1) How should I reference it in the choice task to get the approvalStatus value?
2) Is there are anyway I can make the parallel task return in json format instead of an array?
Thanks in advance
I think you should use something like "$[1].approvalStatus" if you don't want to change the ResultPath.