After finishing successfully, a Lambda function insists on timing out.
The function's triggering event is s3:ObjectCreated:*.
The function uses MongoDB Atlas and does so according to the optimisation suggestions on https://www.mongodb.com/blog/post/optimizing-aws-lambda-performance-with-mongodb-atlas-and-nodejs, including setting:
context.callbackWaitsForEmptyEventLoop = false; before using the DB.
The function also calls some AWS SDK methods with successfully resolved promises.
After finishing my code successfully and doing everything it set out to do, I get the following in my CloudWatch logs, (both the request's END event and its timeout):
START RequestId: XXX
... my logs...
END RequestId: XXX
REPORT RequestId: XXX Duration: 6001.12 ms Billed Duration: 6000 ms Memory Size: 1024 MB Max Memory Used: 49 MB
XXX Task timed out after 6.00 seconds
The function then repeats itself twice more with the same unfortunate result.
Any immediate suspects? Where should I look?
You need to call callback(null, <any>) in order to end your function handler and tell Lambda that your function executed successfully.
Without that, Lambda will retry the same invocation after a delay and it will again finish but without telling Lambda that it finished successfully.
Related
I have a lambda that is running an API, so every instance of the lambda starts a flask server, which runs until the lambda is killed. When another request is received, a new instance of the lambda is created- starting the flask server again.
Looking at the logs from one of my lambdas, a log stream ran for more than 15 minutes (~20 minutes before a process failed). The timeout is set to 5 min 0 seconds.
From what I understand, one long stream is generated per instance, an instance is killed after the timeout has been reached, and no timeout can be set for more than 15 minutes.
How do I have logs that span for ~20 minutes in the same log stream? Is the timeout the amount of time a lambda can run without generating logs before being killed? Is there a way to limit the time a lambda instance runs before the instance is terminated?
Initial log timestamp: 2022-02-20T11:05:05.970-07:00
Final log timestamp: 2022-02-20T11:25:29.895-07:00
Lambda Timeout: 5min0sec
START, END, REPORT statements -- I have multiple of these per log stream:
2022-02-20T11:05:06.509-07:00
START RequestId: 6807306e-f05f-425b-bd75-985b42794eaa
2022-02-20T11:05:07.203-07:00
END RequestId: 6807306e-f05f-425b-bd75-985b42794eaa
2022-02-20T11:05:07.203-07:00
REPORT RequestId: 6807306e-f05f-425b-bd75-985b42794eaa Duration: 691.58 ms Billed Duration: 2856 ms Memory Size: 128 MB Max Memory Used: 119 MB Init Duration: 2163.68 ms
XRAY TraceId: 1-621282cf-00745c0f030b1d810fad9710 SegmentId: 3effe66c64dc69d4 Sampled: true
2022-02-20T11:05:07.767-07:00
START RequestId: 8c724d99-8640-4634-88b9-1efb774c54a9 Version: $LATEST
2022-02-20T11:05:07.845-07:00
END RequestId: 8c724d99-8640-4634-88b9-1efb774c54a9
2022-02-20T11:05:07.845-07:00
REPORT RequestId: 8c724d99-8640-4634-88b9-1efb774c54a9 Duration: 76.09 ms Billed Duration: 77 ms Memory Size: 128 MB Max Memory Used: 119 MB
XRAY TraceId: 1-621282d3-55a51f9d69552d8905b959b7 SegmentId: 47dc3b533278679f Sampled: true
2022-02-20T11:05:08.182-07:00
START RequestId: ee69e296-af07-49a2-8264-ed9758d03992 Version: $LATEST
2022-02-20T11:05:08.263-07:00
END RequestId: ee69e296-af07-49a2-8264-ed9758d03992
2022-02-20T11:05:08.263-07:00
REPORT RequestId: ee69e296-af07-49a2-8264-ed9758d03992 Duration: 78.26 ms Billed Duration: 79 ms Memory Size: 128 MB Max Memory Used: 119 MB
XRAY TraceId: 1-621282d4-7c8f7ea75ead3f7c166a44a6 SegmentId: 2dcaed42621dbebd Sampled: true
... the final start/end that fails.
2022-02-20T11:25:29.749-07:00
START RequestId: 5521e958-0ffb-4de7-b028-953a21ac2ac9 Version: $LATEST
2022-02-20T11:25:29.895-07:00
END RequestId: 5521e958-0ffb-4de7-b028-953a21ac2ac9
2022-02-20T11:25:29.895-07:00
REPORT RequestId: 5521e958-0ffb-4de7-b028-953a21ac2ac9 Duration: 142.60 ms Billed Duration: 143 ms Memory Size: 128 MB Max Memory Used: 119 MB
XRAY TraceId: 1-62128799-19bd91ab324cf0271e964434 SegmentId: 554bd90b4a8ba1c5 Sampled: true
I think your misconception here is When another request is received, a new instance of the lambda is created- starting the flask server again. This is not necessarily true. According to the documentation:
The first time you invoke your function, AWS Lambda creates an instance of the function and runs its handler method to process the event. When the function returns a response, it stays active and waits to process additional events.
This makes perfect sense, especially for an http api. If a new lambda instance had to spin up every request, database connections, caching layers, etc would all have to be reestablished. This would not be scalable or performant. So AWS is doing you a favor by keeping the lambda instance up and running, even though you only pay for the time it's actually working.
As for the timeout question, the timeout is the max time that the lambda will complete a single job, in your case responding to an API request. Not how long a lambda will stay active.
A single request will be terminated after 15 min, but the instance, where the request was worked on, stays for 4.5 hours.(last time I checked at least)
and it will write every request in the same log stream.
If another request comes in, when the lambda instance is "alive" and not busy, the same instance will be reused, and will write in the same log stream, as the previous call.
Only if no lambda instance is free to work on request, a new instance will be started.
This is the reason, why the "warm up" calls for some bad designed lambdas with long start time work.
I have a simple lambda function which calculates a result asynchronously. I can log the result and it seems to be correct but for some reason the lambda function doesn't return successfully, like I am getting a timeout. If you look at the timestamps you can see that the result is calculated way before the timeout. The weird thing is that it works fine when I am using axios but whenever I use fauna it doesn't work anymore, but it does log the correct result... I have been sitting on this problem for days and have no clue what to do. I am using the serverless framework along with this template.
Response
{
"errorMessage": "2021-03-10T07:11:11.567Z 0180b87e-e01f-4527-8c7e-4c1dd5e3e354 Task timed out after 6.01 seconds"
}
Function Logs
START RequestId: 0180b87e-e01f-4527-8c7e-4c1dd5e3e354 Version: $LATEST
2021-03-10T07:11:05.811Z 0180b87e-e01f-4527-8c7e-4c1dd5e3e354 INFO Sending response: { statusCode: 200, body: '{"result":100}' }
END RequestId: 0180b87e-e01f-4527-8c7e-4c1dd5e3e354
REPORT RequestId: 0180b87e-e01f-4527-8c7e-4c1dd5e3e354 Duration: 6007.06 ms Billed Duration: 6000 ms Memory Size: 256 MB Max Memory Used: 76 MB Init Duration: 205.66 ms
2021-03-10T07:11:11.567Z 0180b87e-e01f-4527-8c7e-4c1dd5e3e354 Task timed out after 6.01 seconds
Any help would be much appreciated!
Found the issue. Within the handler I set the context.callbackWaitsForEmptyEventLoop = false. Alternatively when using middy you can use this middleware
I have create a one function and sent mail to users but it time out
05:48:54
END RequestId: 3508fc6c-29cb-442b-95dd-c963018ca5f3
05:48:54
REPORT RequestId: 3508fc6c-29cb-442b-95dd-c963018ca5f3 Duration: 900083.68 ms Billed Duration: 900000 ms Memory Size: 256 MB Max Memory Used: 43 MB
05:48:54
2019-06-25T05:48:54.532Z 3508fc6c-29cb-442b-95dd-c963018ca5f3 Task timed out after 900.08 seconds
I have set TimeoutInfo 15 min.
Another potential issue might be that you have an active connection, so the aws lambda is not responding until the connection terminates.
callbackWaitsForEmptyEventLoop – Set to false to send the response
right away when the callback executes, instead of waiting for the
Node.js event loop to be empty. If this is false, any outstanding
events continue to run during the next invocation.
If this is the case, you can set callbackWaitsForEmptyEventLoop = false to make callback response immediately by
context.callbackWaitsForEmptyEventLoop = false;
Official doc
Check what is the timeout set in configuration file
Also check for VPC!
"Error handling for a given event source depends on how Lambda is invoked. Amazon CloudWatch Events is configured to invoke a Lambda function asynchronously."
"Asynchronous invocation – Asynchronous events are queued before being used to invoke the Lambda function. If AWS Lambda is unable to fully process the event, it will automatically retry the invocation twice, with delays between retries."
So the retry should happen in this case. Not sure what was wrong with your lambda function , just delete and created again and may works!
Is there a clear way to identify "cold starts"? Either in runtime in the Lambda itself, or via the logs? I know that cold starts are characterized by longer runtimes, which I can actually see, but I'm looking for a clear cut way.
I'm using Node.js if that matters.
Update: There are two good answers below, for two use cases:
- Identifying the cold start as the lambda runs.
- Identifying the cold start from the CloudWatch log.
If you add some initialization code to the top of your NodeJS script, you will be able to tell in the code that it is a cold start, and you will then be able to log that if you want to see it in the logs. For example:
var coldStart = true;
console.log("This line of code exists outside the handler, and only executes on a cold start");
exports.myHandler = function(event, context, callback) {
if (coldStart) {
console.log("First time the handler was called since this function was deployed in this container");
}
coldStart = false;
...
callback(...);
}
Update:
If you only care about seeing cold starts in the logs, Lambda now logs an extra "Init Duration" value in CloudWatch Logs for cold starts.
As an update, AWS now provide visible info on cold starts in the form of "Init Duration" , inside the Report section of a Cloudwatch Log. The calls that do not suffer from a cold start will not contains this information in the log
Duration: 1866.19 ms Billed Duration: 1900 ms Memory Size: 512 MB Max Memory Used: 163 MB Init Duration: 2172.14 ms
If you're looking at CloudWatch logs, each LogGroup for your Lambda function represents a separate container and therefore the first invocation for that LogGroup is your cold start.
I am using aws lambda function to convert uploaded wav file in a bucket to mp3 format and later move file to another bucket. It is working correctly. But there's a problem with triggering. When i upload small wav files,lambda function is called once. But when i upload a large sized wav file, this function is triggered multiple times.
I have googled this issue and found that it is stateless, so it will be called multiple times(not sure this trigger is for multiple upload or a same upload).
https://aws.amazon.com/lambda/faqs/
Is there any method to call this function once for a single upload?
Short version:
Try increasing timeout setting in your lambda function configuration.
Long version:
I guess you are running into the lambda function being timed out here.
S3 events are asynchronous in nature and lambda function listening to S3 events is retried atleast 3 times before that event is rejected. You mentioned your lambda function is executed only once (with no error) during smaller sized upload upon which you do conversion and re-upload. There is a possibility that the time required for conversion and re-upload from your code is greater than the timeout setting of your lambda function.
Therefore, you might want to try increasing the timeout setting in your lambda function configuration.
By the way, one way to confirm that your lambda function is invoked multiple times is to look into cloudwatch logs for the event id (67fe6073-e19c-11e5-1111-6bqw43hkbea3) occurrence -
START RequestId: 67jh48x4-abcd-11e5-1111-6bqw43hkbea3 Version: $LATEST
This event id represents a specific event for which lambda was invoked and should be same for all lambda executions that are responsible for the same S3 event.
Also, you can look for execution time (Duration) in the following log line that marks end of one lambda execution -
REPORT RequestId: 67jh48x4-abcd-11e5-1111-6bqw43hkbea3 Duration: 244.10 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 20 MB
If not a solution, it will at least give you some room to debug in right direction. Let me know how it goes.
Any event Executing Lambda several times is due to retry behavior of Lambda as specified in AWS document.
Your code might raise an exception, time out, or run out of memory. The runtime executing your code might encounter an error and stop. You might run out concurrency and be throttled.
There could be some error in Lambda which makes the client or service invoking the Lambda function to retry.
Use CloudWatch logs to find the error and resolving it could resolve the problem.
I too faced the same problem, in my case it's because of application error, resolving it helped me.
Recently AWS Lambda has new property to change the default Retry nature. Set the Retry attempts to 0 (default 2) under Asynchronous invocation settings.
For some in-depth understanding on this issue, you should look into message delivery guarantees. Then you can implement a solution using the idempotent consumers pattern.
The context object contains information on which request ID you are currently handling. This ID won't change even if the same event fires multiple times. You could save this ID for every time an event triggers and then check that the ID hasn't already been processed before processing a message.
In the Lambda Configuration look for "Asynchronous invocation" there is an option "Retry attempts" that is the maximum number of times to retry when the function returns an error.
Here you can also configure Dead-letter queue service
Multiple retry can also happen due read time out. I fixed with '--cli-read-timeout 0'.
e.g. If you are invoking lambda with aws cli or jenkins execute shell:
aws lambda invoke --cli-read-timeout 0 --invocation-type RequestResponse --function-name ${functionName} --region ${region} --log-type Tail --```payload {""} out --log-type Tail \
I was also facing this issue earlier, try to keep retry count to 0 under 'Asynchronous Invocations'.