I have been experimenting with AWS API Gateway and AWS Lambda to try out a serverless architecture. Have been going through blogs and AWS documentation. Have tried out sample GET/POST. But, I have the following requirement w.r.t tracking user events from my custom application
Events are posted from my application to API end point
I wanted the API to respond back with a custom response (Say {'fine'})
(acknowledging that the request has been received)
After the response is sent, hand over the event payload to a AWS Lambda function
As per the documentation, I understand,
a) I can post events to API end point
b) On GET/POST trigger an AWS Lambda Function
- Respond back from AWS Lambda function to API request
I wanted to change the above and modify it to
a) Post events to API end point
a.0) Respond back acknowledging that request is received [Say {'fine'} ]
b) Trigger AWS Lambda function to process the event payload
Please share across suggestions on how to achieve the same.
Another asynchronous model many customers have used:
Set up an API configured to send requests to Amazon Kinesis. This API could acknowledge the request.
Set up AWS Lambda to consume your Kinesis stream.
This setup has some advantages for high workload APIs as fetches from the Kinesis stream can be batched and don't require a 1-to-1 scaling of both your API Gateway limits and Lambda limits.
Update
To answer your questions about scalability:
Kinesis
Kinesis scales by adding what it calls "shards" to the stream. Each shard handles a portion of your traffic, based on a partition key. Each shard scales up to 1000 rps or 1MBps (see limits). Even with the lower default 25 shards, this would support up to 25,000 rps or 25MBps with an evenly distributed partition key.
API Gateway
API Gateway has a default account level limit of 500 rps, but this can easily be extended by requesting a limit increase. We have customers in production that are using the service at limits above your current suggested scale.
If you want a fast response from the API and not have to wait for the processing of data, you could:
post an event to an API Gateway endpoint
trigger an AWS Lambda Function A
call asynchronously a Lambda Function B using the AWS SDK in the Lambda Function A
Call context.succeed() or context.done() or the callback function in the Lambda Function A so it respond back to API Gateway
the Lambda Function B can process the data while API Gateway already received a response
You should first run some tests to see what type of real world response times you are getting from having your lambda function complete all the logic. If the times are above what you feel are acceptable for your use case, here is another asynchronous solution utilizing an SNS Topic to trigger a secondary Lambda function.
Client Request to API Gateway -> Calls Lambda function A
Lambda A verifies payload and then publishes to SNS Topic X
Lambda A returns {fine} success message -> API Gateway -> client
SNS Topic X triggers Lambda function B
Lambda function B implements given logic
Related
I am integrating AWS API Gateway with AWS SQS.
What would happen if API Gateway receives a payload > 256KB? Can I divert it to S3 somehow?
In this use case, you probably want to use a lambda proxy integration with API gateway.
So, your API Gateway endpoint sends the payload to a lambda function.
The lambda function works out the size of the payload. If it is <256KB you may want to place his onto the queue in SQS via the lambda.
If it is >256kb then you could write this down to s3 via the lambda, and then place a message into your queue with the s3 details. Your worker can then use the message in the queue to grab the data in s3 and process it from there.
Note there is a 10MB payload limit for API Gateway.
AWS provide the following guide on Lambda Proxy Integration with API Gateway.
https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html
What would be the most efficient way to EMIT a POST requests (a webhook precisely) from AWS on a daily basis ?
Right away, I started by simply creating a cloudWatch rule with an event schedule CRON that would trigger an SNS publication "every day at 18h", then I created an SNS topic "AlertMyWebhook" with all POST endpoints as subscribers of the topic.
But.. SNS requires me to CONFIRM subscription of each endpoints... which I can't by definition, since the https endpoint is not mine (webhook = HOOK into someone ELSE'S WEBapp).
So now I am starting to think crazy stuff like having a dynamoDB table to store all webhooks endpoint URL, coupled with a lambda function to read the table, take each https endpoints, and send a POST request to them...
Frankly speaking: that doesn't make any sense to me.
Is there any way to avoid SNS confirmation ?
If not, how on earth would you do to "trigger a POST every day at 18h" without creating a monolithic-like architecture ?
AWS SNS and Lambda functions are integrated with each other so you can subscribe a Lambda function to your topic. When a message is posted to that topic the subscribed Lambda function is invoked with the Payload(published message).
Using this Payload as input for the Lambda function trigger the POST requests for the endpoints. A good way to do is make all the HTTPS POST endpoints as Environment variables in Lambda. So there is no code change in the Lambda function whenever a new POST endpoint need to be added as the Subscription endpoints. For more of How to integrate AWS SNS and Lambda look here.
Invoking Lambda functions using Amazon SNS notifications
The sample NodeJS code to invoke the POST request
AWS Lambda HTTP POST Request
I currently have a Web hook that's calling AWS API Gateway -> AWS Lambda function proxy. I'd like to make the web hook more responsive and return an early reply while continuing processing in the Lambda.
I went ahead and did this early reply from the Lambda (Node v6.10) but it didn't appear to have improved responsiveness. Is API Gateway somehow waiting for the Lambda to finish executing despite having the response from the callback already?
The other idea is to post an SNS notification from Lambda and have a second Lambda listen and continue processing but would rather avoid that complication if there's a simpler way.
API Gateway currently only supports synchronous invocation (aka InvocationType: RequestResponse) of Lambda functions, so yes, it is waiting for the full response from the Lambda.
To support your use case, you could use SNS or an another intermediary AWS service like Kinesis, SQS, etc. But you could also do it with Lambda alone. Have the first Lambda function trigger a second Lambda function asynchronously with InvocationType: 'Event', this will achieve the effect you desire.
See this post for more details: https://stackoverflow.com/a/31745774/5705481
As it currently stands AWS SNS provides functionality for retrial(Linear, Geometric and Exponential backoff) with HTTP/HTTPS endpoints in case of a 5XX response returned from the endpoint.
Because of this my application architecture changes and I forcefully need to insert a API gateway between my SNS and Lambda so that in case of a failure I can return a 5XX status from the API gateway and utilise the retrial functionality of SNS.
But there is nothing mentioned for retrial mechanism with AWS lambda. Is there any way I can use the SNS retrial facilities for non-HTTP based subscriptions?
Thanks
After a couple of hours of debugging and going through AWS documentation it seems that there is currently no way of getting exponential back of from AWS SNS for anything else apart from HTTP/HTTPS sources.
You can checkout the this.
As quoted in the documentation:
When a user calls the SNS Publish API on a topic that your Lambda
function is subscribed to, Amazon SNS will call Lambda to invoke your
function asynchronously. Lambda will then return a delivery status. If
there was an error calling Lambda, Amazon SNS will retry invoking the
Lambda function up to three times. After three tries, if Amazon SNS
still could not successfully invoke the Lambda function, then Amazon
SNS will send a delivery status failure message to CloudWatch.
Since there is a async invocation of the Lambda SNS will not care what the exit status of the lambda is. Hence, from the point of view of SNS, a successful invocation of the lambda is success enough and will not provide a failure event, hence no customised back off.
For now it seems, adding an HTTP endpoint is the only option.
I'm using API Gateway-to-Lambda for a few micro-services but in at least one case the service will take 20-30 seconds to complete so in cases like this I'd like to pass back an immediate response to the client, something like:
status: 200
message: {
progressId: 1234
}
and then allow the Lambda Function to continue on (and periodically updating the "processId" somewhere that is accessible to a client. The problem is that if you call context.succeed(), context.fail(), or context.done() that apparently stops the lambda function from further execution and yet it's the only way I know to flush the stdout buffer back to the API Gateway.
This has led me to a second approach which I haven't yet try to tackle (and for simplicity sake would love to avoid) which involves API Gateway calling a "Responder" Lambda function that then asynchronously fires off the Microservice and then immediately responds to the API Gateway.
I've tried to illustrate these two options in sketch format below. I'd love to hear how anyone's been able to solve this problem.
Currently API Gateway requires that the AWS Lambda integration is synchronous. If you desire asynchronous invocation of your Lambda function, you have 2 options:
Invoking the Lambda asynchrously, either with an AWS integration calling InvokeAsync on Lambda, or using an intermediate service such as SNS or Kinesis to trigger the Lambda function.
You're #2 diagram, using a synchronous Lambda invoke to initiate the asynchronous invoke.
As of Apr/2016 is it is possible to create async Lambda execution through API Gateway by using AWS Service Proxy. See http://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-lambda.html
You can send the X-Amz-Invocation-Type header, it supports async calls through the Event value
You can optionally request asynchronous execution by specifying Event as the InvocationType
http://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html#API_Invoke_RequestSyntax
Also, if you can't send it via your micro-service, you can configure this header to be passed by default through the Method Execution -> Integration Request -> HTTP Headers in your API Gateway Resource
This worked for me on a micro-service -> API Gateway -> Lambda scenario, like the mentioned on the question.