I came across with Lambda new feature called "Provisioned Concurrency": End of Cold Start. First invocation will always a cold start as expected later for subsequent request it has to be warm and initialized but in the console I can see "Init Duration" for the second request which was invoked after 10 minutes. As per provisioned concurrency, Lambda has to be warm and no initialization should be done after first invocation.
X-ray details for subsequent calls
Related
I am new to AWS and having some difficulty tracking down and resolving some latency we are seeing on our API. Looking for some help diagnosing and resolving the issue.
Here is what we are seeing:
If an endpoint hasn't been hit recently, then on the first request we see a 1.5-2s delay marked as "Initialization" in the CloudWatch Trace.
I do not believe this is a cold start, because each endpoint is configured to have 1 provisioned concurrency, so we should not get a cold start unless there are 2 simultaneous requests. Also, the billed duration includes this initialization period.
Cold start means when your first request hit to aws lambda it will be prepared container to run your scripts,this will take some time and your request will delay.
When second request hit lambda and lambda and container is already up and runing will be process quickly
https://aws.amazon.com/blogs/compute/operating-lambda-performance-optimization-part-1/
This is the default behavior of cold start, but since you said that you were using provisioned concurrency, that shouldn't happen.
Provisioned concurrency has a delay to activate in the account, you can follow this steps to verify if this lambda used on demand or provisioned concurrency.
AWS Lambda sets an ENV called AWS_LAMBDA_INITIALIZATION_TYPE that contain the values on_demand or provisioned_concurrency.
I have enabled provisioned concurrency (5 units) on one of my lambda functions which is invoked via API Gateway requests.
My impression of this was that it eliminates "cold starts" or "inits" altogether yet when I hit my appropriate Api endpoint after an "idle period" X-Ray & Cloudwatch show me what I would have thought is a cold start?
Init Duration: 2379.57 ms
Subsequent requests don't have this and my total time for the request goes from approx 3.8s to approx 200ms.
My understanding was that provisioning the concurrency would automatically hit the provisioned ones first and thus ALWAYS produce that latter 200ms scenario (assuming we are not exceeding concurrency count). I've created a separate lambda and am the only one using it so I know that is not the issue. I've created an alias, which point to to a version, not $LATEST.
Lambda metrics display "Invocations" "ProvisionedConcurrencyInvocations" with the appropriate colour on the chart which indicated provisioned concurrency.
Why am I getting what is seemingly a cold start still?
If all the above seems OK and my understanding is indeed what should be happening then would it be related to the services used in the lambda itself?
I use S3, SQS and DynamoDB.
Lambda is a c# .net core 3.1 function
Edit #1 -
I Should clarify that the function is actually an asp.net core 3.1 web api serverless and has all the usual attached ConfigureServices/Configure points as well as Controller classes.
I have timed and written to console the times it takes to ConfigureServices/Configure and Constructor of the controller. Times are .6s/0.0s/0.0s at first run.
I have enabled XRay at the SDK level in Startup.cs with:
AWSSDKHandler.RegisterXRayForAllServices();
I can now see the following breakdown in X-Ray:
And after a few (quick) executions I see:
I want to have EVERY single execution sitting at that 163ms and bypass the nearly 4s entirely.
Here's what I know, or think I know.
In AWS Lambda, the first time you call a function is commonly called a "cold start" -- this is akin to starting up your program for the first time.
If you make a second function invocation relatively quickly after your first, this cold start won't happen again. This is colloquially known as a "warm start"
If a function is idle for long enough, the execution environment goes away, and the next request will need to cold start again.
It's also possible to have a single AWS Lambda function with multiple triggers. Here's an example of a single function that's handling both API Gateway requests and SQS messages.
My question: Will AWS Lambda reuse (warm start) an execution environment when different event triggers come in? Or will each event trigger have it's own cold start? Or is this behavior that's not guaranteed by Lambda?
Yes, different triggers will use the same containers since the execution environment is the same for different triggers, the only difference is the event that is passed to your Lambda.
You can verify this by executing your Lambda with two types of triggers (i.e. API Gateway and simply the Test function on the Lambda Console) and looking at the CloudWatch logs. Each Lambda container creates its own Log Stream inside of your Lambda's Log Group. You should see both event logs going to the same Log Stream which means the 2nd event is successfully using the warm container created by the first event.
I was trying to set up lambda with provisioned concurrency. I enabled this feature for the latest version of my lambda function.
After that, i ran this function and watched logs in AWS X-RAY. I saw that my function, still has initialization, but it should become warm with provisioned concurrency.
Without latency, after first start, i ran it twice and it became warm as expected (because it is a default behaviour when lambda became warm after first start without provisioned concurrency).
I waited 15 minutes and was trying to start my lambda again and still it still has initialization time in the logs. It doesn't become warm with provisioned concurrency as expected and always has initialization time.
How can i resolve it?
I have a few AWS Lambda functions, but the troubleshooting is for one of them. this Lambda function is triggered by message queue, read DynamoDB, process, write DynamoDB. it is called up to 10 requests per second and I have set Lambda provision concurrency. Average Lambda duration is 60 ms which I am very happy with. But every day there are around 10 instances which Lambda function duration is more than 1 second up to 3 second timeout.
I put log in my Lambda, during duration spikes, read/write (getitem/putitem) DynamoDB took more than 1 second. Dynamodb is set to on-demend. it is a very simple table, two columns, ID (auto number) and a json string(about 1KB). I have tried Redis, but weird enough, still had spikes. Lambda is not put in VPC. Dynamo connection has been set to http timeout 500, max retry to 2.
Code to read DynamodDB:
Log for Duration:
When using provisioned concurrency, the Lambda service would keep a set number of the underlying containers "warm" so as to minimize start up time. Since you mention that you intermittently face higher execution durations, refer to the below debugging steps which you can do:
Check the "Concurrent Executions" metric for the Lambda function against the "Duration" metric: If the number of instances of the function executing at a particular time is higher than the set provisioned concurrency, then that would imply that s few of these instances had cold starts causing the higher duration.
Enable X-Ray tracing for the Lambda function and also add X-ray instrumentation to your code: This would give a complete idea of which network call takes up too much time and also give you the cold start "init" duration (if any).