My use case is such that I'll have an AWS Lambda front ended with API Gateway.
My requirement is that once the Lambda is invoked it should return a 200 OK response back to API Gateway which get forwards this to the caller.
And then the Lambda should start its actual processing of the payload.
The reason for this is that the API Gateway caller service expects a response within 10 seconds else it times out. So I want to give the response before I start with the processing.
Is this possible?
With API Gateway's "Lambda Function" integration type, you can't do this with a single Lambda function -- that interface is specifically designed to be synchronous. The workaround, if you want to use the Lambda Function integration type is for the synchronous Lambda function, invoked by the gateway, to invoke a second, asynchronous, Lambda function through the Lambda API.
However, asynchronous invocations are possible without the workaround, using an AWS Service Proxy integration instead of a Lambda Function integration.
If your API makes only synchronous calls to Lambda functions in the back end, you should use the Lambda Function integration type. [...]
If your API makes asynchronous calls to Lambda functions, you must use the AWS Service Proxy integration type described in this section. The instructions apply to requests for synchronous Lambda function invocations as well. For the asynchronous invocation, you must explicitly add the X-Amz-Invocation-Type:Event header to the integration request.
http://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-lambda.html
Yes, simply create two Lambda functions. The first Lambda function will be called by the API Gateway and will simply invoke the second Lambda function and then immediately return successfully so that the API Gateway can respond with an HTTP 200 to the client. The second Lambda function will then take as long as long as it needs to complete.
If anyone is interested, here is the code you can use to do the two lambdas approach. The code below is the first lambda that you should setup which would then call the second, longer running, lambda. It takes well under a second to execute.
const Lambda = new (require('aws-sdk')).Lambda();
/**
* Note: Step Functions, which are called out in many answers online, do NOT actually work in this case. The reason
* being that if you use Sequential or even Parallel steps they both require everything to complete before a response
* is sent. That means that this one will execute quickly but Step Functions will still wait on the other one to
* complete, thus defeating the purpose.
*
* #param {Object} event The Event from Lambda
*/
exports.handler = async (event) => {
let params = {
FunctionName: "<YOUR FUNCTION NAME OR ARN>",
InvocationType: "Event", // <--- This is KEY as it tells Lambda to start execution but immediately return / not wait.
Payload: JSON.stringify( event )
};
// we have to wait for it to at least be submitted. Otherwise Lambda runs too fast and will return before
// the Lambda can be submitted to the backend queue for execution
await new Promise((resolve, reject) => {
Lambda.invoke(params, function(err, data) {
if (err) {
reject(err, err.stack);
}
else {
resolve('Lambda invoked: '+data) ;
}
});
});
// Always return 200 not matter what
return {
statusCode : 200,
body: "Event Handled"
};
};
Check the answer here on how to set up an Async Invoke to the Lambda function. This will return 200 immediately to the client, but the Lambda will process on it's own asynchronously.
https://stackoverflow.com/a/40982649/5679071
Related
I have created a Parent Lambda function from which I am invoking a child Lambda function. I want to invoke the child function asynchronously so I am using Invocation type as 'Event'. I am using API gateway to call the Parent Lambda function. When I try to send a single or 2-3 requests to the API endpoint, I see the logs in the Parent Lambda function only and not in the child lambda function(i.e. it is not able to invoke the child function). It is not able to invoke child function when I hit fewer requests. However if I hit 10-15 requests at a time then it starts reflecting the logs in both parent and child Lambda function. Whereas the ideal behavior should be even if I invoke it once it should reflect in both the function logs.
var AWS = require('aws-sdk');
AWS.config.region = 'us-west-2';
var lambda = new AWS.Lambda();
exports.handler = async (event, context) => {
const function1 = {
FunctionName: 'child_lambda',
InvocationType: 'Event',
Payload: JSON.stringify(event)
};
lambda.invoke(function1).promise();
}
Here I have mentioned the code snippet I am using to invoke the child Lambda function.
What about adding this in the end ?
return await lambda.invoke(function1).promise();
My function takes about 1-2 min to execute while API gateway has 30 second timeout. To overcome this I followed AWS documentation and enabled InvocationType:Event header. The issue is that I receive 200 response after execution, but how do I receive my output? I can see lambda output in cloudwatch but what happens to it next?
Code used for making request
var xhttp = new XMLHttpRequest();
xhttp.open("POST", "https://my-endpoint.com", true);
xhttp.setRequestHeader("Content-type", "application/json");
xhttp.setRequestHeader("InvocationType", "Event");
xhttp.send("foo");
xhttp.onreadystatechange = function() {
if (this.readyState == 4 && this.status == 200) {
console.log(this.responseText); //no output here
}
};
If I send a synchronous request by removing InvocationType header then I get desired result.
When dealing to asynchronous functions, you have two ways out:
Use a callback url: on your first request, you pass as parameter (or previously configure) a callback url that your function will call when have the result. This is the most efficient way and widely used on many APIs. It's also called web hook
Store your lambda function response anywhere and your api client must pool for results. This is not so efficient because you will need to do multiple requests until the lambda function finishes, increasing your costs.
The flow on the first method is like:
Hey API, do my long duration task, and when ready call me back on https://sample-domain.com/callback-url
API processes the task, and access the given callback url with the result as payload
The client receives the callback request and processes the result as desired.
There's some security concerns here: someone can discover your callback url and try doing requests to fake something or just trying to DDoS attack.
To mitigate this possibility, you can use random urls for the callback, adding something that identifies the original request.
Im building a lambda that calls another lambda in nodejs with aws. The problem is that my lambdas should run synchrony and I cant finish it after I call return callback function. Im using also context.callbackWaitsForEmptyEventLoop == false.
My code gets a list on firebase and executes this list, If lambda timeout reaches 60 secounds or more, it calls another lambda and returns the callback function.
Like that:
return callback(null, {
statusCode: 400,
headers: {'Content-Type': 'text/plain'},
body: JSON.stringify(result)
})
How can I solve this problem?
thanks.
Lambda supports both sync and async invocation type , you can specify invocation type as RequestResponse.
Please refer this documentation
I am wanting to execute a http request inside of a lambda function, invoked by API Gateway. The problem is, the request takes a bit of time to complete (<20 seconds) and don't want the client waiting for a response. In my research on asynchronous requests, I learned that I can pass the X-Amz-Invocation-Type:Event header to make the request execute asynchronously, however this isn't working and the code still "waits" for the http request to complete.
Below is my lambda code:
'use strict';
const https = require('https');
exports.handler = function (event, context, callback) {
let requestUrl;
requestUrl = event.queryStringParameters.url;
https.get(requestUrl, (res) => {
console.log('statusCode:', res.statusCode);
console.log('headers:', res.headers);
res.on('data', (d) => {
process.stdout.write(d);
});
}).on('error', (e) => {
console.error(e);
});
let response = {
"statusCode": 200,
"body": JSON.stringify(event.queryStringParameters)
};
callback(null, response);
};
Any help would be appreciated.
You can use two Lambda functions.
Lambda 1 is triggered by API Gateway then calls Lambda 2 asynchronously (InvocationType = Event) then returns a response to the user.
Lambda 2, once invoked, will trigger the HTTP request.
Whatever you do, don't use two lambda functions.
You can't control how lambda is being called, async or sync. The caller of the lambda decides that. For APIGW, it has decided to call lambda sync.
The possible solutions are one of:
SQS
Step Functions (SF)
SNS
In your API, you call out to one of these services, get back a success, and then immediately return a 202 to your caller.
If you have a high volume of single or double action execution use SQS. If you have potentially long running with complex state logic use SF. If you for someone reason want to ignore my suggestions, use SNS.
Each of these can (and should) call back out to a lambda. In the case that you need to run more than 15 minutes, they can call back out to CodeBuild. Ignore the name of the service, it's just a lambda that supports up to 8 hour runs.
Now, why not use two lambdas (L1, L2)? The answer is simple. Once you respond that your async call was queued (SQS, SF, SNS), to your users (202). They'll expect that it works 100%. But what happens if that L2 lambda fails. It won't retry, it won't continue, and you may never know about it.
That L2 lambda's handler no longer exist, so you don't know the state any more. Further, you could try to add logging to L2 with wrapper try/catch but so many other types of failures could happen. Even if you have that, is CloudWatch down, will you get the log? Possible not, it just isn't a reliable strategy. Sure if you are doing something you don't care about, you can build this arch, but this isn't how real production solutions are built. You want a reliable process. You want to trust that the baton was successfully passed to another service which take care of completing the user's transaction. This is why you want to use one of the three services: SQS, SF, SNS.
I'm attempting to integrate my lambda function, which must run async because it takes too long, with API gateway. I believe I must, instead of choosing the "Lambda" integration type, choose "AWS Service" and specify Lambda. (e.g. this and this seem to imply that.)
However, I get the message "AWS ARN for integration must contain path or action" when I attempt to set the AWS Subdomain to the ARN of my Lambda function. If I set the subdomain to just the name of my Lambda function, when attempting to deploy I get "AWS ARN for integration contains invalid path".
What is the proper AWS Subdomain for this type of integration?
Note that I could also take the advice of this post and set up a Kinesis stream, but that seems excessive for my simple use case. If that's the proper way to resolve my problem, happy to try that.
Edit: Included screen shot
Edit: Please see comment below for an incomplete resolution.
So it's pretty annoying to set up, but here are two ways:
Set up a regular Lambda integration and then add the InvocationType header described here http://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html. The value should be 'Event'.
This is annoying because the console won't let you add headers when you have a Lambda function as the Integration type. You'll have to use the SDK or the CLI, or use Swagger where you can add the header easily.
Set the whole thing up as an AWS integration in the console (this is what you're doing in the question), just so you can set the InvocationType header in the console
Leave subdomain blank
"Use path override" and set it to /2015-03-31/functions/<FunctionARN>/invocations where <FunctionARN> is the full ARN of your lambda function
HTTP method is POST
Add a static header X-Amz-Invocation-Type with value 'Event'
http://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html
The other option, which I did, was to still use the Lambda configuration and use two lambdas. The first (code below) runs in under a second and returns immediately. But, what it really does is fire off a second lambda (your primary one) that can be long running (up to the 15 minute limit) as an Event. I found this more straightforward.
/**
* Note: Step Functions, which are called out in many answers online, do NOT actually work in this case. The reason
* being that if you use Sequential or even Parallel steps they both require everything to complete before a response
* is sent. That means that this one will execute quickly but Step Functions will still wait on the other one to
* complete, thus defeating the purpose.
*
* #param {Object} event The Event from Lambda
*/
exports.handler = async (event) => {
let params = {
FunctionName: "<YOUR FUNCTION NAME OR ARN>",
InvocationType: "Event", // <--- This is KEY as it tells Lambda to start execution but immediately return / not wait.
Payload: JSON.stringify( event )
};
// we have to wait for it to at least be submitted. Otherwise Lambda runs too fast and will return before
// the Lambda can be submitted to the backend queue for execution
await new Promise((resolve, reject) => {
Lambda.invoke(params, function(err, data) {
if (err) {
reject(err, err.stack);
}
else {
resolve('Lambda invoked: '+data) ;
}
});
});
// Always return 200 not matter what
return {
statusCode : 200,
body: "Event Handled"
};
};