The api gateway url is triggered/called by a form submit. Now, as this is a synchronous invocation, how do I handle a lambda retries and handle throttles?
Note: I am deploying api gateway along with lambda and use the url generated as webhook for form submit. So basically, it would be a one-way communication.
Flow:
form(payload)-> (api gateway)-> lambda-> lambda-> sqs-> lambda-> dynamoDb
There won't be an automatic Lambda retry when it's invoked from API Gateway afaik. If you are throttled or the request fails (perhaps because of a Lambda function error), your client is responsible for deciding how to recover and whether or not to retry, likely based on the HTTP response code.
Also worth reading:
A Detailed Overview of AWS API Gateway
How AWS Lambda Retry really works
Related
I have a lambda that is invoked by Kinesis Stream. The lambda then calls an API endpoint and depending on the response, attempts to retry the call to the API. If the response is 200, it won't retry. However, if it receives, say a 500 error back from the API, it will retry.
I want to know if this is an anti-pattern when calling REST endpoints from a lambda or not. In my specific case, the Lambda retry should be determined by the response from the API its calling.
Im assuming lambda is like the missing piece of the puzzle for a complete api request. So you create the apigateway and then write the lambda function which bridges the gap between taking a request and returning the output of the lambda function as the http response.
I've successfully followed guides on how to set up an API gateway that triggers AWS lambda to do something, but I still don't really understand what is being done.
How is the function def handler(event, context): being called by the aws apigateway? How does it get triggered and how is the output of handler sent back?
You do not need a Lambda "in the middle". Using Lambda Proxy integration in the API Gateway you can receive the full information about the request (endpoint URL, query parameters, etc) in your targeted Lambda event.
Have a look at the following Tutorial how to setup Lambda Proxy integration with API Gateway.
The tricky thing you should care about is the structure of the response that you will return from your lambda_handler. See the requirements here.
Answering the question of "how this happens"... In short, when an HTTP request comes to your API endpoint it is automatically routed to the mapped Lambda function. Behind the scenes a new container for the Function is spawned and your request comes to the event of the lambda_handler. API Gateway by default also creates a CloudFront distribution in front of itself to serve your requests more efficiently. Once your Lambda returns the response, API Gateway parses it and constructs the HTTP response out of it. The nice thing is that all of this is managed by AWS.
I have a API gateway connected to SQS service, currently it just forward all the incoming requests bodies to SQS by SendMessage action.
I hope at integration request step I can check if the request has a certain field. If so, return a custom response and do not call the SQS service, otherwise forward the request body to SQS as I am doing right now.
I can do this by using a lambda function triggered by API gateway but i am wondering if I can do this without using lambda.
You may achieve that by setting up a request validator on AWS Api Gateway as explained here
I have been experimenting with AWS API Gateway and AWS Lambda to try out a serverless architecture. Have been going through blogs and AWS documentation. Have tried out sample GET/POST. But, I have the following requirement w.r.t tracking user events from my custom application
Events are posted from my application to API end point
I wanted the API to respond back with a custom response (Say {'fine'})
(acknowledging that the request has been received)
After the response is sent, hand over the event payload to a AWS Lambda function
As per the documentation, I understand,
a) I can post events to API end point
b) On GET/POST trigger an AWS Lambda Function
- Respond back from AWS Lambda function to API request
I wanted to change the above and modify it to
a) Post events to API end point
a.0) Respond back acknowledging that request is received [Say {'fine'} ]
b) Trigger AWS Lambda function to process the event payload
Please share across suggestions on how to achieve the same.
Another asynchronous model many customers have used:
Set up an API configured to send requests to Amazon Kinesis. This API could acknowledge the request.
Set up AWS Lambda to consume your Kinesis stream.
This setup has some advantages for high workload APIs as fetches from the Kinesis stream can be batched and don't require a 1-to-1 scaling of both your API Gateway limits and Lambda limits.
Update
To answer your questions about scalability:
Kinesis
Kinesis scales by adding what it calls "shards" to the stream. Each shard handles a portion of your traffic, based on a partition key. Each shard scales up to 1000 rps or 1MBps (see limits). Even with the lower default 25 shards, this would support up to 25,000 rps or 25MBps with an evenly distributed partition key.
API Gateway
API Gateway has a default account level limit of 500 rps, but this can easily be extended by requesting a limit increase. We have customers in production that are using the service at limits above your current suggested scale.
If you want a fast response from the API and not have to wait for the processing of data, you could:
post an event to an API Gateway endpoint
trigger an AWS Lambda Function A
call asynchronously a Lambda Function B using the AWS SDK in the Lambda Function A
Call context.succeed() or context.done() or the callback function in the Lambda Function A so it respond back to API Gateway
the Lambda Function B can process the data while API Gateway already received a response
You should first run some tests to see what type of real world response times you are getting from having your lambda function complete all the logic. If the times are above what you feel are acceptable for your use case, here is another asynchronous solution utilizing an SNS Topic to trigger a secondary Lambda function.
Client Request to API Gateway -> Calls Lambda function A
Lambda A verifies payload and then publishes to SNS Topic X
Lambda A returns {fine} success message -> API Gateway -> client
SNS Topic X triggers Lambda function B
Lambda function B implements given logic
I'm using API Gateway-to-Lambda for a few micro-services but in at least one case the service will take 20-30 seconds to complete so in cases like this I'd like to pass back an immediate response to the client, something like:
status: 200
message: {
progressId: 1234
}
and then allow the Lambda Function to continue on (and periodically updating the "processId" somewhere that is accessible to a client. The problem is that if you call context.succeed(), context.fail(), or context.done() that apparently stops the lambda function from further execution and yet it's the only way I know to flush the stdout buffer back to the API Gateway.
This has led me to a second approach which I haven't yet try to tackle (and for simplicity sake would love to avoid) which involves API Gateway calling a "Responder" Lambda function that then asynchronously fires off the Microservice and then immediately responds to the API Gateway.
I've tried to illustrate these two options in sketch format below. I'd love to hear how anyone's been able to solve this problem.
Currently API Gateway requires that the AWS Lambda integration is synchronous. If you desire asynchronous invocation of your Lambda function, you have 2 options:
Invoking the Lambda asynchrously, either with an AWS integration calling InvokeAsync on Lambda, or using an intermediate service such as SNS or Kinesis to trigger the Lambda function.
You're #2 diagram, using a synchronous Lambda invoke to initiate the asynchronous invoke.
As of Apr/2016 is it is possible to create async Lambda execution through API Gateway by using AWS Service Proxy. See http://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-lambda.html
You can send the X-Amz-Invocation-Type header, it supports async calls through the Event value
You can optionally request asynchronous execution by specifying Event as the InvocationType
http://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html#API_Invoke_RequestSyntax
Also, if you can't send it via your micro-service, you can configure this header to be passed by default through the Method Execution -> Integration Request -> HTTP Headers in your API Gateway Resource
This worked for me on a micro-service -> API Gateway -> Lambda scenario, like the mentioned on the question.