Error when invoking lambda from StepFunction with paypload - amazon-web-services

I have a lambda and it's one of the steps of a StepFunction statemachine, the lambda handler looks like this below :
def lambda_handler(event, context):
year = event['year'] #payload
month = event['month'] #payload
example_function(year, month)
This lambda execution succeeded if f I start a run in Lambda itself with payload:
{
"year": "2019",
"month": "06"
}
However if I kick off a stepfunction run, it will fail with error:
[ERROR] KeyError: 'year'
Traceback (most recent call last):
File "/var/task/xxx.py", line 34, in lambda_handler
year = event['year']
My stepfunction definition is:
{
"Comment": "xxxxxxxxx",
"StartAt": "invoke lambda",
"States": {
"invoke lambda" : {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName":"xxxxxxxxxx",
"Payload": {
"Input": {
"year": "2019",
"month": "06"
}
}
}
"End": true
}
}
}
I'm not sure how to debug and solve this issue, might someone be able to help please?

There are two ways to call the lambda function from step function.
First Method, directly by giving the Resource name as lambda Arn and passing the entire input to Parameters. This passes given input as is to Lambda
{
"StartAt":"invoke-lambda",
"States":{
"invoke-lambda":{
"End":true,
"Type":"Task",
"Resource":"arn:aws:lambda:us-east-1:660008888333:function:HelloWorld",
"Parameters":{
"year":"2019",
"month":"06"
}
}
}
}
Second Method using Resource lambda:invoke and Payload with Json body(no need to wrap within Input)
{
"StartAt":"invoke-lambda",
"States":{
"invoke-lambda":{
"End":true,
"Type":"Task",
"Resource":"arn:aws:states:::lambda:invoke",
"Parameters":{
"FunctionName":"arn:aws:lambda:us-east-1:660008888333:function:HelloWorld",
"Payload":{
"year":"2019",
"month":"06"
}
}
}
}
}
If input to Lambda is from either previous step or from input to step function input, we can use "Payload.$": "$" or even "Payload.$": "$.subPath"(to pass part of json)
Also it is recommended to add a Retry for exceptions that are triggered because of Exceptions caused by AWS.
{
"StartAt": "invoke-lambda",
"States": {
"invoke-lambda": {
"End": true,
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException"
],
"IntervalSeconds": 2,
"MaxAttempts": 6,
"BackoffRate": 2
}
],
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-east-1:660008888333:function:HelloWorld",
"Payload.$": "$",
"InvocationType": "RequestResponse"
}
}
}
}

Related

AWS StepFunction: Passing callback token as output from catch

i am building a step function that publishes to sns and waits for a callback. if the state times out, i want the $$.Task.Token to be passed as part of the output to the next state. i've been reading the documentation and looking at posts, but i haven't found anything that seems to to this. is this possible?
my (simplified) state machine definition looks something like the following - i want the Timeout state to have access to the callback token from the SNS Publish state (i am not very picky about structure/naming)
{
"StartAt": "SNS Publish",
"States": {
"SNS Publish": {
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish.waitForTaskToken",
"Parameters": {
"TopicArn": "arn:aws:sns:XXXXXX:XXXXXX:XXXXXX",
"Message.$": "$$.Task.Token"
},
"End": true,
"TimeoutSeconds": 1,
"Catch": [
{
"ErrorEquals": [
"States.Timeout"
],
"ResultPath": "$.error",
"Next": "Timeout"
}
]
},
"Timeout": {
"Type": "Pass",
"End": true,
"Parameters": {
"Identifier.$": "$.TaskToken",
"IsSuccess": false
}
}
}
}

AWS Step function process output from lambda

I have a AWS step function need to call 3 lambda in sequence, but at the end of each call, the step function need to process the response from a lambda, and determine the next lambda call.
So how can the step function process the response from a lambda? Can you show an example please?
Assuming that, you have called a lambda function as the first step of your step function. Based on the response from the lambda, you need to decide on which lambda should be triggered next.
This workaround is pretty straightforward, you can return an attribute (eg: next_state) in the lambda response, create a "Choice" flow, in the step function, and give this next_state attribute as an input.
"Choice" flow is nothing but an if-else condition and you can redirect the next step to the expected lambda.
For example,
and the definition will be something like below,
{
"Comment": "A description of my state machine",
"StartAt": "Lambda Invoke 1",
"States": {
"Lambda Invoke 1": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": "$",
"FunctionName": "<your lambda name>"
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException"
],
"IntervalSeconds": 2,
"MaxAttempts": 6,
"BackoffRate": 2
}
],
"Next": "Choice"
},
"Choice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.next_state",
"StringEquals": "Lambda Invoke 2",
"Next": "Lambda Invoke 2"
},
{
"Variable": "$.next_state",
"StringEquals": "Lambda Invoke 3",
"Next": "Lambda Invoke 3"
}
],
"Default": "Lambda Invoke 3"
},
"Lambda Invoke 2": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": "$",
"FunctionName": "<your lambda name>"
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException"
],
"IntervalSeconds": 2,
"MaxAttempts": 6,
"BackoffRate": 2
}
],
"End": true
},
"Lambda Invoke 3": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": "$",
"FunctionName": "<your lambda name>"
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException"
],
"IntervalSeconds": 2,
"MaxAttempts": 6,
"BackoffRate": 2
}
],
"End": true
}
}
}
There are two ways of catching response of lambda function from step function.
Using add_retry and add_catch to handle any exception from lambda function
eg.
.start(record_ip_task
.add_retry(errors=["States.TaskFailed"],
interval=core.Duration.seconds(2),
max_attempts=2)
.add_catch(errors=["States.ALL"], handler=notify_failure_job)) \
Response value from lambda function such as return '{"Result": True} then the step function job will check that value for next task, eg.
.next(
is_block_succeed
.when(step_fn.Condition.boolean_equals('$.Result', False), notify_failure_job)
.otherwise(send_slack_task)
)
Ref: https://dev.to/vumdao/aws-guardduty-combine-with-security-hub-and-slack-17eh
https://github.com/vumdao/aws-guardduty-to-slack

Map step lambda function with empty event is running a keyerror

I'm writing a step function where lambda A queries a database and creates an array with s3bucket/s3key dictionaries for each one that needs to be processed.
For the processing I have a Map state that takes this s3bucket/s3key and do some processing.
For some reason, the event is coming empty resulting in an error in the Map state execution:
Here is the step function definition
{
"Comment": "A Hello World example of the Amazon States Language using Pass states",
"StartAt": "QueryTable",
"States": {
"QueryTable",": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:...function:query_table:$LATEST"
},
"OutputPath": "$.Payload",
"Next": "ProcessDocuments"
},
"ProcessDocuments": {
"Type": "Map",
"ItemsPath": "$.toprocess",
"MaxConcurrency": 5,
"Iterator": {
"StartAt": "Process",
"States": {
"Process": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:...:...:function:process:$LATEST"
},
"End": true
}
}
},
"End": true
}
}
}
And the error
{
"resourceType": "lambda",
"resource": "invoke",
"error": "KeyError",
"cause": {
"errorMessage": "'s3bucket'",
"errorType": "KeyError",
"stackTrace": [
" File \"/var/task/process.py\", line 260, in lambda_handler\n raise e\n",
" File \"/var/task/process.py\", line 230, in lambda_handler\n s3bucket, s3key = extract_event_data(event)\n",
" File \"/var/task/process.py\", line 125, in extract_event_data\n bucket = record['s3bucket']\n"
]
}
}
I read the ItemsPath and InputPath documentation. Also, used this example to see if I was using it incorrectly but it was no use.

aws step function parallel with input parameters

I am trying to use AWS step functions to create parallel branches of execution.
One of the parallel branches starts another step function invocation, how can we pass input from this parallel branch to next step function execution
{
"Comment": "Parallel Example.",
"StartAt": "FunWithMath",
"States": {
"FunWithMath": {
"Type": "Parallel",
"End": true,
"Branches": [
{
"StartAt": "Add", /// This receives some json object here input {}
"States": {
"Add": {
"Type": "Task", ***//How to pass the received input to the following arn as input?***
"Resource": ""arn:aws:states:::states:startExecution",
Parameters: {
"StateMachineArn": "anotherstepfunctionarnpath"
}
"End": true
}
}
},
{
"StartAt": "Subtract",
"States": {
"Subtract": {
"Type": "Task",
"Resource": "some lambda arn here,
"End": true
}
}
}
]
}
}
}
anotherstepfunctionarnpath :
{
"Comment": "Second state machine",
"StartAt": "stage1",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters":{
"Arguments":{
"Variable1" :"???" / how to access the value of the input passed to here
}
}
}
You can use Input to pass output from one SFN to other one:
First SFN(It will call second SFN)
{
"Comment": "My first SFN",
"StartAt": "First SFN",
"States": {
"First SFN": {
"Type": "Task",
"ResultPath": "$.to_pass",
"Resource": "arn:aws:lambda:us-east-1:807278658150:function:test-lambda",
"Next": "Trigger Next SFN"
},
"Trigger Next SFN": {
"Type": "Task",
"Resource": "arn:aws:states:::states:startExecution",
"Parameters": {
"Input": {
"Comment.$": "$"
},
"StateMachineArn": "arn:aws:states:us-east-1:807278658150:stateMachine:MyStateMachine2"
},
"End": true
}
}
}
Second SFN (MyStateMachine2)
{
"Comment": "A Hello World example of the Amazon States Language using Pass states",
"StartAt": "Hello",
"States": {
"Hello": {
"Type": "Pass",
"Result": "Hello",
"Next": "World"
},
"World": {
"Type": "Pass",
"Result": "World",
"End": true
}
}
}
First SFN's Execution
Second SFN's Execution
Explanation
The Lambda test-lambda is returning:
{
"user": "stackoverflow",
"id": "100"
}
Which is stored in "ResultPath": "$.to_pass" here in to_pass variable. I am passing the same output to next state machine MyStateMachine2 which is done by
"Input": {
"Comment.$": "$"
}
In the next State Machine's execution you see that same data is received as input which was created by first Lambda.
You can read more about it here.

AWS Step-Function: pass a specific value from one AWS lambda to another in step function parallel state

I have the below state machine. The requirement is to have a lambda to query DB and get all the ids. Next I have a parallel state call that calls more than five lambdas at once. Instead of passing all the ids fetched to all the lambdas, I need to pass the respective ids to each lambda.
In the below state language, first call is DB_CALL, lets say it returns {id1, id2, id3, id4, id5, id6}, I want to pass only id1 to First_Lambda and id2 to Second_Lambda etc...
The entire id object should get passed to all lambdas. Please suggest a way to achieve this.
{
"Comment": "Concurrent Lambda calls",
"StartAt": "StarterLambda",
"States": {
"StarterLambda": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:DB_CALL",
"Next": "ParallelCall"
},
"State": {
"ParallelCall": {
"Type": "Parallel",
"End": true,
"Branches": [
{
"StartAt": "First",
"States": {
"First": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:First_Lambda",
"TimeoutSeconds": 120,
"End": true
}
}
},
{
"StartAt": "Second",
"States": {
"Second": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:Second_Lambda",
"Retry": [ {
"ErrorEquals": ["States.TaskFailed"],
"IntervalSeconds": 1,
"MaxAttempts": 2,
"BackoffRate": 2.0
} ],
"End": true
}
}
},
{
"StartAt": "Third",
"States": {
"Third": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:Third_Lambda",
"Catch": [ {
"ErrorEquals": ["States.TaskFailed"],
"Next": "CatchHandler"
} ],
"End": true
},
"CatchHandler": {
"Type": "Pass",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:CATCH_HANDLER",
"End": true
}
}
},
{
"StartAt": "Fourth",
"States": {
"Fourth": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:Fourth_Lambda",
"TimeoutSeconds": 120,
"End": true
}
}
},
{
"StartAt": "Fifth",
"States": {
"Fifth": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:Fifth_Lambda",
"TimeoutSeconds": 120,
"End": true
}
}
},
{
"StartAt": "Sixth",
"States": {
"Sixth": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:Sixth_Lambda",
"TimeoutSeconds": 120,
"End": true
}
}
}
}
]
}
}
}
}
You can use Step Function parameter option.
This would allow you to send specific value or json to next lambda.
"Parameters": {
"toprocess.$": "$.MetaData.CorrelationId"
},
So input to this lambda would be smaller dto than compared to you first lambda. So while returning value from this lambda avoid assigning it back to Step function result.
"OutputPath": "$",
"ResultPath": "$.PartialResutl",
What you are looking for is the Map State. With this state, you pass in the iterator, in your case the path to the ids. The map state will run once for each item in the list. Within the map state, you have a full state machine, so you can call a Lambda or any other state. It has controls to limit how many are running at once if that is needed.