I'm trying to run a StepFunction Step using the SNS integration with the following step definition:
"HandleError": {
Type: 'Task',
Resource: 'arn:aws:states:::sns:publish',
Parameters: {
TopicArn: `arn:aws:sns:region:accountid:topic`,
"Message.$": "States.Format('Message: {}', $.message)",
}
}
message is defined as an input to the step. According to the examples in the intrinsic function documentation and integration documentation, this should be doable. However, it's throwing an error, complaining:
The value for the field 'Message.$' must be a valid JSONPath or a valid intrinsic function call (at /States/HandleError/Parameters)
Did I do something wrong? Can the SNS integration for Step Function use an intrinsic function for the Message body?
Related
I'm trying to get my API Gateway api to:
Run an authorizer
Pass authorizer context to a Step Function execution
Respond to client with Step Function output
I already have #1 and #3 done, but passing the response of the attached authorizer lambda to the step function is proving to be impossible.
I found this page and this page with reference sheets on what interpolation values you can use for your parameter mapping (Create Integration -> Step Function: StartSyncExecution -> Advanced Settings -> Input) but any time I try to use anything related to $context like $context.authorizer.email, API Gateway just responds with an HTTP 400 and gives me this CloudWatch output:
"Unable to resolve property Input from source {\"lambdaName\": \"arn:aws:lambda:us-east-1:xxxxxxx\", \"reqBody\": $request.body.Input, \"authContext\": $context.apiId }. Please make sure that the request to API Gateway contains all the necessary fields specified in request parameters."
These are the JSON objects I've tried using for the Input text box and all of them either give me an errors when trying to save or throw an HTTP 400 and log the above errors when I visit the route:
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": $context.authorizer.email }
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": "$context.authorizer.email" }
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": $context.apiId }
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": $context }
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": $event.requestContext.authorizer.email }
It seems like the only way to have authorization code to work with step functions is to wrap my step function called by API Gateway in another step function that authorizes the request and then invokes the endpoint step function. I've researched this for hours and I'm not getting anywhere. Any help at all is appreciated.
I ended up solving this by using API Gateway v1 and a REST API instead of a HTTP API. For some reason v2's input field currently doesn't work for anything other than $request.body.Input. From there, I hooked up all of my endpoints to a step function that runs the authorization lambda on their Authorization header in the request.
I have a step function that allows me to chain together step function and lambda actions so for most requests I just chain together the authorizer lambda and the endpoint's action (can be lambda or another step function).
The main takeaway here is that if you're using API Gateway and Step Functions, it looks like passing custom-formatted input into your step function isn't very easy to do without using the v1 of API Gateway in a REST API, not an HTTP api. Hopefully this will be fixed in the future.
Another solution could be to use the Mapping Template in API Gateway Integration Request.
Example: Consider the response from the Lambda Authorizer as this:
{
principalId: 'myuser',
context: {
customKey: 'CustomValue'
},
policyDocument: {
Version: '2012-10-17',
Statement: [
{
Action: ['execute-api:Invoke'],
Effect: 'Allow',
Resource:
'arn:aws:execute-api:_region_:111111111111:0a0a0a0a0a/default/POST/my-endpoint'
}
]
}
}
In the Mapping Template for application/json, map whatever property you want in the input field (quote escaped). Each property sent from the Lambda Authorizer in the context field will be available as $context.authorizer.property.
One possible Template Mapping could be:
{
"input": "{\"origin\":\"$input.json('$.requestvalue').replaceAll('\"','')\", \"customPropertyFromLambdaAuthorizer\" : \"$context.authorizer.customKey\" }",
"name": "DemoStateMachineRequest",
"stateMachineArn": "arn:aws:states:_region_:11111111111:stateMachine:MyStateMachine"
}
Reference $context.authorizer.property
I've created an AWS Lambda based Spring Cloud Function with multiple functions and it works fine when I define exactly one "spring_cloud_function_definition" environment variable in the AWS Lambda Configuration panel. These functions' input and output is a Message<?>, so the payload and http headers can be accessed runtime as well.
I'd like to use this AWS Lambda function in different API Gateway methods, so for example:
/item [POST] -> saveItem should be called
/item [DELETE] -> deleteItem should be called
... and so on...
https://cloud.spring.io/spring-cloud-static/spring-cloud-function/3.0.0.RC1/reference/html/spring-cloud-function.html#_function_catalog_and_flexible_function_signatures
I found this documentation on Spring.io which says it's feasible:
If the input argument is of type Message<?>, you can communicate routing instruction by setting one of spring.cloud.function.definition or spring.cloud.function.routing-expression Message headers. For more static cases you can use spring.cloud.function.definition header which allows you to provide the name of a single function (e.g., …definition=foo) or a composition instruction (e.g., …definition=foo|bar|baz)
So the following test event should run in AWS Lambda without any failure:
{
"method": "POST",
"body": "{ "key" : "value" }",
"headers": {
"spring.cloud.function.definition": "saveItem"
}
}
but of course it throws an exception:
No function defined: java.lang.IllegalStateException
java.lang.IllegalStateException: No function defined
at org.springframework.cloud.function.context.AbstractSpringFunctionAdapterInitializer.apply(AbstractSpringFunctionAdapterInitializer.java:187)
at org.springframework.cloud.function.adapter.aws.SpringBootRequestHandler.handleRequest(SpringBootRequestHandler.java:51)
at org.springframework.cloud.function.adapter.aws.SpringBootApiGatewayRequestHandler.handleRequest(SpringBootApiGatewayRequestHandler.java:140)
at org.springframework.cloud.function.adapter.aws.SpringBootApiGatewayRequestHandler.handleRequest(SpringBootApiGatewayRequestHandler.java:43)
I've tried many configurations (application.properties, AWS Lambda env property, header, etc.) but nothing happened:
spring.cloud.function.routing-expression=headers.function_name
setting spring.cloud.function.definition=saveItem
What could be the problem? What is the best configuration in this case? Any help would be appreciated!
Used technologies:
Cloud: AWS
Cloud components: Lambda + API Gateway
AWS Lambda handler class: org.springframework.cloud.function.adapter.aws.SpringBootApiGatewayRequestHandler
AWS Lambda input and output events: APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent
Java: 8
Spring Cloud Function version: 3.0.6.RELEASE
I've updated to v3.1.3 and switched to FunctionInvoker and I've added the function definition to the request template mapping header, and it worked fine.
Anyways, thanks for the help. :)
I am trying to create a simple step function comprising of two lambda functions. The first lambda function takes two numbers as input from the user (say num1=2, num2 =5), and passes on the sum of the two numbers to the second lambda function. The second lambda function multiplies the output of the first lambda function with a constant (say 10).
I want to be able to get the final answer (which is (2+5)*10 = 70) in this case) as the output of an API invocation.
I am able to execute the Step Function successfully (from the AWS Step Function Console). But the output I get when I invoke the API (integrated with the Step Function) is not a number but as follows:
{
"executionArn": "arn:aws:states:ap-south-1:123456789012:execution:Test_Math:xxx",
"startDate": 1560344276.117
}
How do I get the API call to return the answer (which is the number 70 in this case)?
I have gone already gone through the AWS Documentation and AWS API Gateway with Step Function this question but I am not still not clear.
How can I make a new lambda (or use any of the above two) function which invokes/executes this step function to return the answer?
Are there any other ways to return the answer of a step function via an API call?
My question is somewhat similar to this one Api gateway get output results from step function?.
I tried adding a second method in API Gateway which will call the Step Function with the DescribeExecution action but it did not work.
The mapping template (application/json) for the POST Method with StartExecution action is as follows:
#set($inputRoot = $input.path('$'))
#set($data = $util.escapeJavaScript($input.json('$')))
{
"input": "{ \"num1\": $inputRoot.num1, \"num2\": $inputRoot.num2 }",
"stateMachineArn": "arn:aws:states:ap-south 1:998338******:stateMachine:Test_Math"
}
I created a new GET Method with DescribeExecution action with the following Integration Request details:
Integration type: AWS Service
AWS Service: Step Functions
HTTP method: GET
Action: Describe Execution
Content Handling: Passthrough
Mapping template : application/json
#set($inputRoot = $input.path('$'))
#set($data = $util.escapeJavaScript($input.json('$')))
{
"executionArn": "arn:aws:states:ap-south-1:998338321653:execution:Test_Math:3981114a-da51-411d-9533-8571dc976e2d",
"input": "{ \"num1\": $inputRoot.num1, \"num2\": $inputRoot.num2 }"
}
Please let me know what changes do I need to make in the above to be able to return the answer (70) with an API call.
P.S: I want to deploy my machine learning model using step functions and this is just a simple test which I was trying out.
you can use describe_execution method to fetch the final result of your stepfunction. you have to pass the execution arn as an input to this method.
Find more details here, https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/stepfunctions.html#SFN.Client.describe_execution
Below python code will print the output given the execution arn.
import boto3
client = boto3.client('stepfunctions')
executionArn = ''
response = client.describe_execution(
executionArn=executionArn
)
#print(response)
print(response.get('output'))
I enabled streams on my dynamoDB table. As items are modified, a lambda function is triggered. I think I set up everything correctly both on the lambda trigger side, permissions, and dynamodb side. I also ran my lambda function with test data and it succeeded. However, when items are modified in the table, the trigger did not start my lambda function. Instead, I got the following error:
Batch size: 100 Last processing result: PROBLEM: Function call failed
Any idea what's the best way to debug this? I went on CloudWatch logs but there were no logs associated with the trigger/stream.
Thanks.
Edit: Logs for the lambda function (not its dynamodb trigger). The trigger didn't generate any log statements.
START RequestId: 3a08eedc-f0de-11e8-9008-033b48d2cb67 Version: $LATEST
18:16:28
END RequestId: 3a08eedc-f0de-11e8-9008-033b48d2cb67
18:16:28
REPORT RequestId: 3a08eedc-f0de-11e8-9008-033b48d2cb67 Duration: 81.85 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 30 MB
I ran into this issue today.
I debugged it by manually triggering the lambda with the Test button on the top of the main lambda page. It showed the error output trying to run my lambda.
The reason I had an error was the handler parameter as I had a non-standard javascript function name and I forgot to configure that in my lambda.
In my case, my lambda role did not have permissions to write to the SNS and the lambda code was writing to a SNS. So i added a policy to the lambda role giving it permissions to write to any SNS topic.
In my case, the problem came from the stream batch size 100. In the lambda code I was checking the event and I exit if the event doesn't meet the requirement.
In my case, the handler method was not configured correctly for the Lambda function written in Java. The format that I used for setting the handler is as follows
packageName.className::handlerMethod
For example, for my handler class
package com.example;
public class App implements
RequestHandler<DynamodbEvent, String> {
public String handleRequest(DynamodbEvent ddbEvent, Context context) {
the handler should be defined as
com.example.App::handleRequest
This sounds like a possible use-case for Rookout if you need to follow variable values in your live Lambda in a situation where you're not able to generate logs and running it locally isn't going to give you real-world event trigger data.
I'm attempting to integrate my lambda function, which must run async because it takes too long, with API gateway. I believe I must, instead of choosing the "Lambda" integration type, choose "AWS Service" and specify Lambda. (e.g. this and this seem to imply that.)
However, I get the message "AWS ARN for integration must contain path or action" when I attempt to set the AWS Subdomain to the ARN of my Lambda function. If I set the subdomain to just the name of my Lambda function, when attempting to deploy I get "AWS ARN for integration contains invalid path".
What is the proper AWS Subdomain for this type of integration?
Note that I could also take the advice of this post and set up a Kinesis stream, but that seems excessive for my simple use case. If that's the proper way to resolve my problem, happy to try that.
Edit: Included screen shot
Edit: Please see comment below for an incomplete resolution.
So it's pretty annoying to set up, but here are two ways:
Set up a regular Lambda integration and then add the InvocationType header described here http://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html. The value should be 'Event'.
This is annoying because the console won't let you add headers when you have a Lambda function as the Integration type. You'll have to use the SDK or the CLI, or use Swagger where you can add the header easily.
Set the whole thing up as an AWS integration in the console (this is what you're doing in the question), just so you can set the InvocationType header in the console
Leave subdomain blank
"Use path override" and set it to /2015-03-31/functions/<FunctionARN>/invocations where <FunctionARN> is the full ARN of your lambda function
HTTP method is POST
Add a static header X-Amz-Invocation-Type with value 'Event'
http://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html
The other option, which I did, was to still use the Lambda configuration and use two lambdas. The first (code below) runs in under a second and returns immediately. But, what it really does is fire off a second lambda (your primary one) that can be long running (up to the 15 minute limit) as an Event. I found this more straightforward.
/**
* Note: Step Functions, which are called out in many answers online, do NOT actually work in this case. The reason
* being that if you use Sequential or even Parallel steps they both require everything to complete before a response
* is sent. That means that this one will execute quickly but Step Functions will still wait on the other one to
* complete, thus defeating the purpose.
*
* #param {Object} event The Event from Lambda
*/
exports.handler = async (event) => {
let params = {
FunctionName: "<YOUR FUNCTION NAME OR ARN>",
InvocationType: "Event", // <--- This is KEY as it tells Lambda to start execution but immediately return / not wait.
Payload: JSON.stringify( event )
};
// we have to wait for it to at least be submitted. Otherwise Lambda runs too fast and will return before
// the Lambda can be submitted to the backend queue for execution
await new Promise((resolve, reject) => {
Lambda.invoke(params, function(err, data) {
if (err) {
reject(err, err.stack);
}
else {
resolve('Lambda invoked: '+data) ;
}
});
});
// Always return 200 not matter what
return {
statusCode : 200,
body: "Event Handled"
};
};