How do I invoke a step function using AWS Lambda ( Python) - amazon-web-services

I am trying to create a simple step function comprising of two lambda functions. The first lambda function takes two numbers as input from the user (say num1=2, num2 =5), and passes on the sum of the two numbers to the second lambda function. The second lambda function multiplies the output of the first lambda function with a constant (say 10).
I want to be able to get the final answer (which is (2+5)*10 = 70) in this case) as the output of an API invocation.
I am able to execute the Step Function successfully (from the AWS Step Function Console). But the output I get when I invoke the API (integrated with the Step Function) is not a number but as follows:
{
"executionArn": "arn:aws:states:ap-south-1:123456789012:execution:Test_Math:xxx",
"startDate": 1560344276.117
}
How do I get the API call to return the answer (which is the number 70 in this case)?
I have gone already gone through the AWS Documentation and AWS API Gateway with Step Function this question but I am not still not clear.
How can I make a new lambda (or use any of the above two) function which invokes/executes this step function to return the answer?
Are there any other ways to return the answer of a step function via an API call?
My question is somewhat similar to this one Api gateway get output results from step function?.
I tried adding a second method in API Gateway which will call the Step Function with the DescribeExecution action but it did not work.
The mapping template (application/json) for the POST Method with StartExecution action is as follows:
#set($inputRoot = $input.path('$'))
#set($data = $util.escapeJavaScript($input.json('$')))
{
"input": "{ \"num1\": $inputRoot.num1, \"num2\": $inputRoot.num2 }",
"stateMachineArn": "arn:aws:states:ap-south 1:998338******:stateMachine:Test_Math"
}
I created a new GET Method with DescribeExecution action with the following Integration Request details:
Integration type: AWS Service
AWS Service: Step Functions
HTTP method: GET
Action: Describe Execution
Content Handling: Passthrough
Mapping template : application/json
#set($inputRoot = $input.path('$'))
#set($data = $util.escapeJavaScript($input.json('$')))
{
"executionArn": "arn:aws:states:ap-south-1:998338321653:execution:Test_Math:3981114a-da51-411d-9533-8571dc976e2d",
"input": "{ \"num1\": $inputRoot.num1, \"num2\": $inputRoot.num2 }"
}
Please let me know what changes do I need to make in the above to be able to return the answer (70) with an API call.
P.S: I want to deploy my machine learning model using step functions and this is just a simple test which I was trying out.

you can use describe_execution method to fetch the final result of your stepfunction. you have to pass the execution arn as an input to this method.
Find more details here, https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/stepfunctions.html#SFN.Client.describe_execution
Below python code will print the output given the execution arn.
import boto3
client = boto3.client('stepfunctions')
executionArn = ''
response = client.describe_execution(
executionArn=executionArn
)
#print(response)
print(response.get('output'))

Related

Getting authorizer context from Step Function executed from API Gateway

I'm trying to get my API Gateway api to:
Run an authorizer
Pass authorizer context to a Step Function execution
Respond to client with Step Function output
I already have #1 and #3 done, but passing the response of the attached authorizer lambda to the step function is proving to be impossible.
I found this page and this page with reference sheets on what interpolation values you can use for your parameter mapping (Create Integration -> Step Function: StartSyncExecution -> Advanced Settings -> Input) but any time I try to use anything related to $context like $context.authorizer.email, API Gateway just responds with an HTTP 400 and gives me this CloudWatch output:
"Unable to resolve property Input from source {\"lambdaName\": \"arn:aws:lambda:us-east-1:xxxxxxx\", \"reqBody\": $request.body.Input, \"authContext\": $context.apiId }. Please make sure that the request to API Gateway contains all the necessary fields specified in request parameters."
These are the JSON objects I've tried using for the Input text box and all of them either give me an errors when trying to save or throw an HTTP 400 and log the above errors when I visit the route:
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": $context.authorizer.email }
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": "$context.authorizer.email" }
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": $context.apiId }
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": $context }
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": $event.requestContext.authorizer.email }
It seems like the only way to have authorization code to work with step functions is to wrap my step function called by API Gateway in another step function that authorizes the request and then invokes the endpoint step function. I've researched this for hours and I'm not getting anywhere. Any help at all is appreciated.
I ended up solving this by using API Gateway v1 and a REST API instead of a HTTP API. For some reason v2's input field currently doesn't work for anything other than $request.body.Input. From there, I hooked up all of my endpoints to a step function that runs the authorization lambda on their Authorization header in the request.
I have a step function that allows me to chain together step function and lambda actions so for most requests I just chain together the authorizer lambda and the endpoint's action (can be lambda or another step function).
The main takeaway here is that if you're using API Gateway and Step Functions, it looks like passing custom-formatted input into your step function isn't very easy to do without using the v1 of API Gateway in a REST API, not an HTTP api. Hopefully this will be fixed in the future.
Another solution could be to use the Mapping Template in API Gateway Integration Request.
Example: Consider the response from the Lambda Authorizer as this:
{
principalId: 'myuser',
context: {
customKey: 'CustomValue'
},
policyDocument: {
Version: '2012-10-17',
Statement: [
{
Action: ['execute-api:Invoke'],
Effect: 'Allow',
Resource:
'arn:aws:execute-api:_region_:111111111111:0a0a0a0a0a/default/POST/my-endpoint'
}
]
}
}
In the Mapping Template for application/json, map whatever property you want in the input field (quote escaped). Each property sent from the Lambda Authorizer in the context field will be available as $context.authorizer.property.
One possible Template Mapping could be:
{
"input": "{\"origin\":\"$input.json('$.requestvalue').replaceAll('\"','')\", \"customPropertyFromLambdaAuthorizer\" : \"$context.authorizer.customKey\" }",
"name": "DemoStateMachineRequest",
"stateMachineArn": "arn:aws:states:_region_:11111111111:stateMachine:MyStateMachine"
}
Reference $context.authorizer.property

AWS Step Function Synchronous Task Token

I have a use case which I want to use Step Functions to solve but I can't find a way to solve this problem. Your help would be greatly appreciated.
The problem goes like this: I have an Amazon API Gateway which has a /start endpoint. a POST to this endpoint should start a data processing session and return a URL to an app which the API client can use to capture some data. Once data capture is complete, some processing takes place before the final response is sent to the API client via a callback.
My thinking, as you can see below, is to generate a task token and send it to the Data Capture Service. Then, when the user data capture is complete, the service can send a request to the Step Function API to say that stage is complete. The problem with this is how can I return the URL to the client from within the Step Function? I don't want to use a callback to do this.
One option is to create the data capture session within the 'Step Function Initiator' Lambda but then how do I provide the Data Capture Service with a task token?
Really, what I need is some mechanism of synchronously returning something (either a URL from that call or the task token from the first stage) from within the Step Function to the Lambda which started the execution. Is this possible? How would you solve this?
In step function initiator lambda, you must be doing start-execution which returns an executionArn
Next, you can loop and call get-execution-history api and task token will be part of the 'capture data' task parameters. Since this is the first step, this really should be done with in couple of seconds, so, we can keep running this loop every second until desired step in step function is initiated and task token can be obtained.
Take this example, i am passing the task token to another step function call from current step function.
{
"StartAt":"ChildTask",
"States":{
"ChildTask":{
"End":true,
"Type":"Task",
"Resource":"arn:aws:states:::states:startExecution.waitForTaskToken",
"Parameters":{
"Input":{
"token.$":"$$.Task.Token",
"foo":"bar"
},
"StateMachineArn":"arn:aws:states:us-east-1:110011001100:stateMachine:ChildStateMachine",
"Name":"MyExecutionName"
}
}
}
}
Get Execution history:
aws stepfunctions get-execution-history --execution-arn arn:aws:states:us-east-1:110011001100:execution:ParentStateMachine:667102b3-b19c-b7ab-b119-9ec6cf23e505
Result:
one of the first few entries in execution history and task token is part of the parameters. we can exit the loop, grab that, send it back to Api Gateway.
{
"timestamp": "2021-03-12T13:56:58.097000-05:00",
"type": "TaskScheduled",
"id": 3,
"previousEventId": 2,
"taskScheduledEventDetails": {
"resourceType": "states",
"resource": "startExecution.waitForTaskToken",
"region": "us-east-1",
"parameters": "{\"Input\":{\"foo\":\"bar\",\"token\":\"o6QVQ9gls.......=\"},\"StateMachineArn\":\"arn:aws:states:us-east-1:110011001100:stateMachine:ChildStateMachine\",\"Name\":\"MyExecutionName\"}"
}
}

How to send multi part key value via http get

I am trying to learn to use AWS Lambda, DynamoDB, and API Gateway.
I am able to set up a run a successful test of the Lambda that would take querystring parameters and returns the data from the database and I know I have the ability to access the Lambda from my API as I have created one to just return hello world. However, I am stuck as to how to send the parameters I need to the Lambda. These work fine from the Lambda test feature within Lambda.
{
"queryStringParameters": {
"TableName": "Exams",
"Key": {
"ExamName": "MyFirstExam"
}
}
}
However I can not seem to create a html query http://apiurl/route?parameters
as I have no idea how to send the Key.
I have tried everything I can think of here is just a sample of what I have tried
http://apiurl/route?TableName=Exams&Key=ExamName&ExamName=MyFirstExam
http://apiurl/route?TableName=Exams&key=ExamName&Keyvalue=MyFirstExam
http://apiurl/route?TableName=Exams&key=ExamName&value=MyFirstExam
http://apiurl/route?TableName=Exams&key=ExamName:MyFirstExam
but nothing has worked so I am wondering how do you send key = ExamName : MyfirstExam so that it is a part of the querystring parameters?
UPDATE:
I found that the rest api gave me more feed back and it looks like I need to use
````http://apiURL/Route?TableName= Exams&Key= { ExamName: MyFirstExam } ```
as this populates the querystring parameters.
"queryStringParameters":{"TableName":"Exams","Key":"{ ExamName: MyFirstExam }"}
However, I am still getting an error. That leads me to believe that I still may not have the format right. The problem is that when I used the same data in the test from within the lambda console it works and returns the data. When I use the querystring above which appears to be the same data as used in the lambda test it throws an error.
Lambda execution failed with status 200 due to customer function error: The provided key element does not match the schema. So my Key is getting there as I get a status 200 and if I leave the key off of the querystring it throws and error that the key is missing but it appears to not be formatted correctly apparently or is it something else that I am missing. It appears to be the same as the format that is working in the Lambda test.
This works from the lambda test
{ "queryStringParameters": { "TableName": "Exams", "Key": { "ExamName": "MyFirstExam" } } }
this does not work from the API test
"queryStringParameters":{"TableName":"Exams","Key":"{ ExamName: MyFirstExam }"}
This is how I am formatting the querystring
http://apiURL/Route?TableName= Exams&Key= { ExamName: MyFirstExam }
How do I format a querystring so that it works from the API gateway?
The easiest and most commonly used method of joining API Gateway and lambda is through AWS_PROXY integration between your API and your function.
With this type of integration:
API Gateway applies a default mapping template to send the entire request to the Lambda function and transforms the output from the Lambda function to HTTP responses.
This means that every query parameter you provide through your API endpoint is going to be delivered directly into lambda function:
This request data includes the request headers, query string parameters, URL path variables, payload, and API configuration data.
The event format your lambda function gets is described here.
I resolved this by having the key part already in the code and just now only need to pass a normal variable ExamName=Exam.
var queryParameters ={
TableName: request.TableName,
Key:{
"ExamName": request.ExamName
},
};```

Api gateway get output results from step function?

I followed tutorial on creating and invoking step functions
I'm getting output in my GET request of api as
{
"executionArn": "arn:aws:states:ap-northeast-1:123456789012:execution:HelloWorld:MyExecution",
"startDate": 1.486772644911E9
}
But, instead of above response I want my step functions output, which is given by end state as below.
{
"name":"Hellow World"
}
How to achieve this?
Update: You can now use Express Step Functions for synchronous requests.
AWS Step Functions are asynchronous and do not immediately return their results. API Gateway methods are synchronous and have a maximum timeout of 29 seconds.
To get the function output from a Step Function, you have to add a second method in API Gateway which will call the Step Function with the DescribeExecution action. The API Gateway client will have to call this periodically (poll) until the returned status is no longer "RUNNING".
Here's the DescribeExecution documentation
Use Express Step Functions instead. This type of Step Functions can be called synchronously. Go to your API Gateway and in the Integration Request section make sure you have the StartSyncExecution action:
After that, go a bit lower in the same page to the Mapping Templates:
and include the following template for the application/json Content-Type:
#set($input = $input.json('$'))
{
"input": "$util.escapeJavaScript($input)",
"stateMachineArn": "arn:aws:states:us-east-1:your_aws_account_id:stateMachine:your_step_machine_name"
}
After that, go back to the Method Execution and go to the Integration Response and then to the Mapping Templates section:
And use the following template to have a custom response from your lambda:
#set ($parsedPayload = $util.parseJson($input.json('$.output')))
$parsedPayload
My testing Step Function is like this:
And my Lambda Function code is:
Deploy your API Gateway stage.
Now, if you go to Postman and send a POST request with any json body, now you have a response like this:
New Synchronous Express Workflows for AWS Step Functions is the answer:
https://aws.amazon.com/blogs/compute/new-synchronous-express-workflows-for-aws-step-functions/
Amazon API Gateway now supports integration with Step Functions StartSyncExecution for HTTP APIs:
https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-api-gateway-supports-integration-with-step-functions-startsyncexecution-http-apis/
First of all the step functions executes asynchronously and API Gateway is only capable of invoking the step function (Starting a flow) only.
If you are waiting for the results of a step function invocation from a web application, you can use AWS IOT WebSockets for this. The steps are as follows.
Setup AWS IOT topic with WebSockets.
Configure the API Gateway and Step functions invocation.
From the Web Frontend subscribe to the IOT Topic as a WebSocket listener.
At the last step (And in error steps) in the Step Functions workflow use AWS SDK to trigger the IOT Topic which will broadcast the results to the Web App running in the browser using WebSockets.
For more details on WebSockets with AWS IOT refer the medium article Receiving AWS IoT messages in your browser using websockets.
Expanding on what #MikeD at AWS says, if you're certain that the Step Function won't exceed the 30 second timeout, you could create a lambda that executes the step function and then blocks as it polls for the result. Once it has the result, it can return it.
It is a better idea to have the first call return immediately with the execution id, and then pass that id into a second call to retrieve the result, once it's finished.

Api Gateway: AWS Subdomain for Lambda Integration

I'm attempting to integrate my lambda function, which must run async because it takes too long, with API gateway. I believe I must, instead of choosing the "Lambda" integration type, choose "AWS Service" and specify Lambda. (e.g. this and this seem to imply that.)
However, I get the message "AWS ARN for integration must contain path or action" when I attempt to set the AWS Subdomain to the ARN of my Lambda function. If I set the subdomain to just the name of my Lambda function, when attempting to deploy I get "AWS ARN for integration contains invalid path".
What is the proper AWS Subdomain for this type of integration?
Note that I could also take the advice of this post and set up a Kinesis stream, but that seems excessive for my simple use case. If that's the proper way to resolve my problem, happy to try that.
Edit: Included screen shot
Edit: Please see comment below for an incomplete resolution.
So it's pretty annoying to set up, but here are two ways:
Set up a regular Lambda integration and then add the InvocationType header described here http://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html. The value should be 'Event'.
This is annoying because the console won't let you add headers when you have a Lambda function as the Integration type. You'll have to use the SDK or the CLI, or use Swagger where you can add the header easily.
Set the whole thing up as an AWS integration in the console (this is what you're doing in the question), just so you can set the InvocationType header in the console
Leave subdomain blank
"Use path override" and set it to /2015-03-31/functions/<FunctionARN>/invocations where <FunctionARN> is the full ARN of your lambda function
HTTP method is POST
Add a static header X-Amz-Invocation-Type with value 'Event'
http://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html
The other option, which I did, was to still use the Lambda configuration and use two lambdas. The first (code below) runs in under a second and returns immediately. But, what it really does is fire off a second lambda (your primary one) that can be long running (up to the 15 minute limit) as an Event. I found this more straightforward.
/**
* Note: Step Functions, which are called out in many answers online, do NOT actually work in this case. The reason
* being that if you use Sequential or even Parallel steps they both require everything to complete before a response
* is sent. That means that this one will execute quickly but Step Functions will still wait on the other one to
* complete, thus defeating the purpose.
*
* #param {Object} event The Event from Lambda
*/
exports.handler = async (event) => {
let params = {
FunctionName: "<YOUR FUNCTION NAME OR ARN>",
InvocationType: "Event", // <--- This is KEY as it tells Lambda to start execution but immediately return / not wait.
Payload: JSON.stringify( event )
};
// we have to wait for it to at least be submitted. Otherwise Lambda runs too fast and will return before
// the Lambda can be submitted to the backend queue for execution
await new Promise((resolve, reject) => {
Lambda.invoke(params, function(err, data) {
if (err) {
reject(err, err.stack);
}
else {
resolve('Lambda invoked: '+data) ;
}
});
});
// Always return 200 not matter what
return {
statusCode : 200,
body: "Event Handled"
};
};