Currently, a wait state in AWS can wait only for a defined set period of time.
let's say my step function checks with API for status if the status is updated it will move ahead or else it will wait again for a set period of time!
I would like to make this waiting period dynamic
i.e. (the backoff rate is set to 2)
1st retry: wait for 3600s
2nd retry: wait for 7200s (3600x2)
3rd retry: wait for 14400s (7200x2)
and so on.
Is there any way I can do this without using any other external computation resource (such as lambda)
Just raise a custom exception (for example: StatusNotUpdated) in your function if the status is not updated, and then you can define the step like this:
"Check API Status": {
"Type": "Task",
"Resource": "arn:aws:states:us-east-1:123456789012:task:check_api_status",
"Next": "Status Updated",
"Retry": [ {
"ErrorEquals": [ "StatusNotUpdated" ],
"IntervalSeconds": 3600,
"BackoffRate": 2.0,
"MaxAttempts": 15
} ],
"Catch": [ {
"ErrorEquals": [ "States.ALL" ],
"Next": "Status Update Failed"
} ]
}
Check here for more info
I was not able to find a inbuilt tool for this
So
I created a custom logic in a library
the library has 2 parts
CDK template housing the lambda/compute service
Service Code housing the exp wait logic code
The approach I took to solve this problem was
when the request comes in the step function I append an object with wait time parameters
These parameters are used by the lambda to calculate the dynamic wait time and update the json path with new wait time value
Related
Let's say I have this state machine in AWS Step Function:
And I had started it with this input:
{
"item1": 1,
"item2": 2,
"item3": 3
}
It's clear for me that Action A is receiving the input payload. But, how can Action C access the state machine input to get the value of item3? Is it possible?
Thanks!!
Typically, the data available in Action C will be dependent on what the result/output of Action B is.
However, if you just care about the original input to the state machine execution, you can set the payload of Action C using the Context Object.
// roughly
"Action C": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"Payload.$": "$$.Execution.Input",
"FunctionName": "<action c lambda>"
},
Check out the AWS documentation for Context Object
I am running a step function with many different step yet I am still stuck on the 2nd step.
The first step is a Java Lambda that gets all the input parameters and does what it needs to do.
The lambda returns null as it doesn't need to return anything.
The next step is a call for API gateway which needs to use one of the parameters in the URL.
However, I see that neither the URL has the needed parameter nor do I actually get the parameters into the step. ("input": null under TaskStateEntered)
The API gateway step looks as follows: (I also tried "Payload.$": "$" instead of the "Input.$": "$")
"API Gateway start": {
"Type": "Task",
"Resource": "arn:aws:states:::apigateway:invoke",
"Parameters": {
"Input.$": "$",
"ApiEndpoint": "aaaaaa.execute-api.aa-aaaa-1.amazonaws.com",
"Method": "GET",
"Headers": {
"Header1": [
"HeaderValue1"
]
},
"Stage": "start",
"Path": "/aaa/aaaa/aaaaa/aaaa/$.scenario",
"QueryParameters": {
"QueryParameter1": [
"QueryParameterValue1"
]
},
"AuthType": "IAM_ROLE"
},
"Next": "aaaaaa"
},
But when my step function gets to this stage it fails and I see the following in the logs:
{
"name": "API Gateway start",
"input": null,
"inputDetails": {
"truncated": false
}
}
And eventually:
{
"error": "States.Runtime",
"cause": "An error occurred while executing the state 'API Gateway start' (entered at the event id #9). Unable to apply Path transformation to null or empty input."
}
What am I missing here? Note that part of the path is a value that I enter at the step function execution. ("Path": "/aaa/aaaa/aaaaa/aaaa/$.scenario")
EDIT:
As requested by #lynkfox, I am adding the lambda definition that comes before the API gateway step:
And to answer the question, yes its standard and I see no input.
"Run tasks": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": "$",
"FunctionName": "arn:aws:lambda:aaaaaa-1:12345678910:function:aaaaaaa-aaa:$LATEST"
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException"
],
"IntervalSeconds": 2,
"MaxAttempts": 6,
"BackoffRate": 2
}
],
"Next": "API Gateway start"
},
So yes, as I commented, I believe the problem is the OutputPath of your lambda task definition. What this is saying is Take whatever comes out of this lambda (which is nothing!) and cut off everything other than the key Payload.
Well you are returning nothing, so this causes nothing to be sent to the next task.
I am assuming your incoming vent already has a key in the Json that is named Payload, so what you want to do is remove the OutputPath from your lambda. It doesn't need to return anything so it doesn't need an Output or Result path.
Next, on your API task, assuming again that your initializing event has a key of Payload, you would have "InputPath": "$.Payload" - if you have your headers or parameters in the initializing json Event then, you can reference those keys in the Parameters section of the definition.
Every AWS Service begins with an Event and ends with an Event. Each Event is a JSON object. (Which I'm sure you know). With State Machines, this continues - the State Machine/Step Function is just the controller for passing Events from one Task to the next.
So any given task can have an InputPath, OutputPath, or Result Path - These three definition parameters can decide what values go into the Task and what are sent onto the Next Task. State machines are, by definition, for maintaining State between Tasks, and these help control that 'State' (and there is pretty much only one 'state' at any given time, the event heading to the next Task(s)
The ResultPath is where, in that overall Event, the task puts the data. If you put ResultPath: "$.MyResult" by itself it appends this key to the incoming event
If you add OutputPath, it ONLY passes that key from the output event of the Task onto the next step in the Step Functions.
These three give you a lot of control.
Want to Take an Event into a Lambda and respond with something completely different - you don't need the incoming data - you combine OutputPath and ResultPath with the same value (and your Lambda needs to respond with a Json Object) then you can replace the event wholesale.
If you have ResultPath of some value and OutputPath: "$." you create a new json object with a single Key that contains the result of your task (the key being the definition set in ResultPath
InputPath allows you to set what goes into the Task. I am not 100% certain but I'm pretty sure it does not remove anything from the next Task in the chain.
More information can be found here but it can get pretty confusing.
My quick guide:
ResultPath by itself if you want to append the data to the event
ResultPath + OutputPath of the same value if you want to cut off the Input and only have the output of the task continue (and it returns a JSON style object)
I am using the AWS CDK to create a state machine that sends a message to a fifo queue and waits for a callback from the lambda worker to continue execution.
I would like the messages that get sent to the fifo queue to have a dynamic MessageGroupId assigned to them so I can control the number of lambda workers processing the messages. The only way I can think of to have a dynamic MessageGroupId is to reference some parameter on the step function input with JsonPath, however I have not come across any documentation about it. My initial tests to use JsonPath to dynamically pass the MessageGroupId failed, simply passing the string "$.MessageGroupId" effectively giving each message the same message group id and thus one lambda worker.
Is it possible to dynamically assign a message group id to a sqs message when sent from a step function?
If so, how?
With the help AWS Support, I managed to do it by either using the Context Object or passing an ID from the initial input and reference it with $.
Here's an example:
{
"Comment": "Generate unique MessageGroupId",
"StartAt": "Start",
"States": {
"Start": {
"Type": "Task",
"TimeoutSeconds": 60,
"Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
"Parameters": {
"QueueUrl": "<YOUR_QUEUE_URL>",
"MessageBody": {
"Input.$": "$",
"TaskToken.$": "$$.Task.Token"
},
"MessageGroupId.$": "$$.Execution.Id"
},
"ResultPath": "$",
"End": true
}
}
}
My problem was that I was trying to MessageGroupId like so:
"MessageGroupId": "$$.Execution.Id"
Where I should have done:
"MessageGroupId.$": "$$.Execution.Id"
Appending .$ would resolve the expression "$$.Execution.Id" instead of putting literally the string "$$.Execution.Id".
Is there a way I can pass a date or a cron schedule as an input to my state function - which is getting called by a cloud watch event? The cloud watch event runs on a cron schedule and I would like to pass that dynamically on a daily basis to the step function
For example:
This gives some static input, but I want to give each day's date as input
resource "aws_cloudwatch_event_target" "target" {
rule = aws_cloudwatch_event_rule.samplerule.id
arn = aws_sfn_state_machine.samplemachine.id
role_arn = aws_iam_role.iam_for_sfn.arn
input = <<EOF
{
"operand1": "3",
"operand2": "5",
"operator": "add"
}
EOF
}
An alternative solution could be to use the global access to the context object, as explained here to get the execution start time of the step functions state machine.
So you can send it through your different states of your state machine like this:
"mystep1": {
"Type": "task",
"Parameters": {
"StartTime.$": "$$.Execution.StartTime"
}
}
Make sure to use the double $ to tell Cloudformation that you're using the global access to the context object.
The input to your Lambda function from a Scheduled Event looks something like this:
{
"id": "53dc4d37-cffa-4f76-80c9-8b7d4a4d2eaa",
"detail-type": "Scheduled Event",
"source": "aws.events",
"account": "123456789012",
"time": "2019-10-08T16:53:06Z",
"region": "us-east-1",
"resources": [ "arn:aws:events:us-east-1:123456789012:rule/MyScheduledRule" ],
"detail": {}
}
Using your choice of programming language, you can extract the time value and convert it from a string to a date resource/object (depending on your language). From that time, you can get the data components you are looking for.
IMPORTANT: All scheduled events use UTC time zone and the minimum precision for schedules is 1 minute: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
The below step function is executed in aws and when there is a missing of a required parameter it cancel the flow and throws States.Runtime Error. This is in catch phase of the step function but it is not catching the error as stated.
Defined Step function is as below,
{
"StartAt": "Log Start Step Function",
"Comment": "Executed with inputs",
"States": {
"Log Start Step Function": {
"Type": "Task",
"Resource": "arn:aws:lambda:eu-west-1:0000000:function:update",
"Parameters": {
"body": {
"itemID.$": "$.itemID",
"functionName.$": "$.stepFunctionName ",
"executionARN.$": "$$.Execution.Id",
"complete": false,
"inprogress": true,
"error": false
}
},
"Catch": [
{
"ErrorEquals": [
"States.Runtime"
],
"ResultPath": "$.taskresult",
"Next": "Log Failed Module"
},
{
"ErrorEquals": [
"States.ALL"
],
"ResultPath": "$.taskresult",
"Next": "Log Failed Module"
}
],
"ResultPath": "$.taskresult",
"Next": "Evaluate Module PA1"
}
}
}
Below is the step function,
And the error thrown is as below,
Runtime error is not executing Log failed module.
{
"ErrorEquals": [
"States.Runtime"
],
"ResultPath": "$.taskresult",
"Next": "Log Failed Module"
},
Is this AWS error or something wrong with the configuration which is done here or is there any other way to validate parameters in AWS Step Functions
From https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html
A States.Runtime error is not retriable, and will always cause the execution to fail. A retry or catch on States.ALL will not catch States.Runtime errors.
Your state machine is expecting the following as input:
"Parameters": {
"body": {
"itemID.$": "$.itemID",
"functionName.$": "$.stepFunctionName ",
"executionARN.$": "$$.Execution.Id",
"complete": false,
"inprogress": true,
"error": false
}
},
You need to pass them when you start a new execution instead of:
{
"Comment": "Insert your JSON here"
}
Which you are currently passing because it comes by default as the input body of a new execution in the AWS Console.
Read more about InputPath and Parameters here: https://docs.aws.amazon.com/step-functions/latest/dg/input-output-inputpath-params.html
I have the same problem.
I am beginning to think that the runtime error happens when the input path is processed, and before the catcher can be initialized. This means that try / catch to test for parameters present in the input is not possible. I also tried ChoiceState, to no avail.
So I think there is no solution but to provide every parameter you refer to in the state machine definition. But the documentation is not clear on this.
This caught me out too. My scenario was setting the output based on the results of S3 ListObjectVersions, with the versions to be deleted in a later task. In this case, $.Versions didn't exist because there was nothing in the bucket so States.Runtime was thrown.
{
"bucket.$": "$.Name",
"objects.$": "$.Versions"
}
To work around this -
I don't transform the result of ListObjectVersions task with ResultSelector. Instead this state simply outputs the unedited result.
I added a Choice state underneath with a rule to check if $.Versions is present.
If it is present, move to a Pass state and transform the input in the exact same was a I was originally transforming the result of the ListObjectVersions task using the ResultSelector (because the Pass state doesn't transform on output, only input).
If it is not present, move to a Success state because there is nothing to delete.
Here is a screen grab of the relevant section, just in case it's helpful to visualise.