I am new to AWS and having some difficulty tracking down and resolving some latency we are seeing on our API. Looking for some help diagnosing and resolving the issue.
Here is what we are seeing:
If an endpoint hasn't been hit recently, then on the first request we see a 1.5-2s delay marked as "Initialization" in the CloudWatch Trace.
I do not believe this is a cold start, because each endpoint is configured to have 1 provisioned concurrency, so we should not get a cold start unless there are 2 simultaneous requests. Also, the billed duration includes this initialization period.
Cold start means when your first request hit to aws lambda it will be prepared container to run your scripts,this will take some time and your request will delay.
When second request hit lambda and lambda and container is already up and runing will be process quickly
https://aws.amazon.com/blogs/compute/operating-lambda-performance-optimization-part-1/
This is the default behavior of cold start, but since you said that you were using provisioned concurrency, that shouldn't happen.
Provisioned concurrency has a delay to activate in the account, you can follow this steps to verify if this lambda used on demand or provisioned concurrency.
AWS Lambda sets an ENV called AWS_LAMBDA_INITIALIZATION_TYPE that contain the values on_demand or provisioned_concurrency.
Related
I have enabled provisioned concurrency (5 units) on one of my lambda functions which is invoked via API Gateway requests.
My impression of this was that it eliminates "cold starts" or "inits" altogether yet when I hit my appropriate Api endpoint after an "idle period" X-Ray & Cloudwatch show me what I would have thought is a cold start?
Init Duration: 2379.57 ms
Subsequent requests don't have this and my total time for the request goes from approx 3.8s to approx 200ms.
My understanding was that provisioning the concurrency would automatically hit the provisioned ones first and thus ALWAYS produce that latter 200ms scenario (assuming we are not exceeding concurrency count). I've created a separate lambda and am the only one using it so I know that is not the issue. I've created an alias, which point to to a version, not $LATEST.
Lambda metrics display "Invocations" "ProvisionedConcurrencyInvocations" with the appropriate colour on the chart which indicated provisioned concurrency.
Why am I getting what is seemingly a cold start still?
If all the above seems OK and my understanding is indeed what should be happening then would it be related to the services used in the lambda itself?
I use S3, SQS and DynamoDB.
Lambda is a c# .net core 3.1 function
Edit #1 -
I Should clarify that the function is actually an asp.net core 3.1 web api serverless and has all the usual attached ConfigureServices/Configure points as well as Controller classes.
I have timed and written to console the times it takes to ConfigureServices/Configure and Constructor of the controller. Times are .6s/0.0s/0.0s at first run.
I have enabled XRay at the SDK level in Startup.cs with:
AWSSDKHandler.RegisterXRayForAllServices();
I can now see the following breakdown in X-Ray:
And after a few (quick) executions I see:
I want to have EVERY single execution sitting at that 163ms and bypass the nearly 4s entirely.
I have a script running on Lambda, I've set the timeout to maximum 15 mins but it's still giving me time out error, there is not much infomation in the lofs, how I can solve this issue and spot what is taking soo much time? I tested the script locally and it's fairly quick.
Here's the error:
{
"errorMessage": "2020-09-10T18:26:53.180Z xxxxxxxxxxxxxxx Task timed out after 900.10 seconds"
}
If you're exceeding the 15 minutes period there are a few things you should check to identify:
Is the Lambda connecting to resources in a VPC? If so is it attached via VPC config, and do the target resources allow inbound access from the Lambda.
Is the Lambda connecting to a public IP but using VPC configuration? If so it will need a NAT attached to allow outbound access.
Are there any long running processes as part of your Lambda?
Once you've ruled these out consider increasing the available resources of your Lambda, perhaps its hitting a cap and is therefore performing slow. Increasing the memory will also increase the available CPU for you.
Adding comments in the code will log to CloudWatch logs, these can help you identify where in the code the slowness starts. This is done by simply calling the general output/debug function of your language i.e. print() in Python or console.log() in NodeJS.
If the function is still expected to last longer than 15 minutes after this you will need to break it down into smaller functions performing logical segments of the operation
A suggested orchestrator for this would be to use a step function to handle the workflow for each stage. If you need shared storage between each Lambda you can make use of EFS to be attached to all of your Lambdas so that they do not need to upload/download between the operations.
Your comment about it connecting to a SQL DB is likely the key. I assume that DB is in AWS in your VPC. This requires particular setup. Check out
https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html
https://docs.aws.amazon.com/lambda/latest/dg/services-rds-tutorial.html
Another thing you can do is enable debug level logging and then look at the details in CloudWatch after trying to run it. You didn't mention which language your lambda uses, so how to do this could be different for the language you use. Here's how it would be done in python:
LOGGER = logging.getLogger()
LOGGER.setLevel(logging.getLevelName('DEBUG'))
Using a SAM template to deploy an apigateway that calls a couple of lambdas.
All lambdas are written in java11 and are connected to a VPC. This adds a lot of delay between calling the function and the code actually starting up (about 30 seconds).
Used
AutoPublishAlias: "latestAlias"
ProvisionedConcurrencyConfig:
ProvisionedConcurrentExecutions:
1
to enable Provisioned Concurrency but i still get long starts of 30 seconds from time to time.
Also tried using ping events via AWS::Events::Rule to call the lambdas every 9 minutes to keep them warn (i recall reading that 15 minutes of inactivity leads to lambda containers being destroyed). Also left Provisioned Concurrency enabled while doing this, and I still see cold starts. I also see new files being created in CloudWatch logs, indicating that a new container has been spun up.
Using xray to validate that i don't have multiple requests at the same time that might trigger the creating of a new container.
Am I missing something in enabling this? From the documentation i understand that "initialization" should not appear in xray once the lambda has been started. Also, AWS Hyperplane was supposed to reduce lambda-VPC integration times to around 1 second (deploying on eu-central-1 which should have Hyperplane for a while now).
response from an api from aws api gateway integrated to a lambda
function takes a lot more time than compared to a regular node project
on aws elasticbeanstalk
is there any way to reduce response time for aws api gateway
There's definitely more information needed to answer this question but from what you've said, your problems may be caused by the cold start time of Lambda functions. An Elastic Beanstalk stack will spin up EC2 instances (which are ready once they're spun up and until they're removed). Lambda will create instances of your handler as needed to address incoming traffic. The first time you call a Lambda, it needs to provision an environment for the function for the first time. Depending on the language used, this can take some time. Successive requests should be faster unless you wait a while (in which case the lambda needs to re-initialize).
So here's more information that would be useful in case this answer is not helpful:
How much slower is Lambda than your Elastic Beanstalk stack?
Is it slower only on the first couple requests or does it continue being slow when you keep requesting?
Is it slow every day or only occasionally?
I'm learning about AWS Lambda and I'm worried about synchronized real-time requests.
The fact the lambda has a "cold start" it doesn't sounds good for handling GET petitions.
Imagine a user is using the application and do a GET HTTP Request to get a Product or a list of Products, if the lambda is sleeping, then it will take 10 seconds to respond, I don't see this as an acceptable response time.
Is it good or bad practice to use AWS Lambda for classic (sync responses) API Rest?
Like most things, I think you should measure before deciding. A lot of AWS customers use Lambda as the back-end for their webapps quite successfully.
There's a lot of discussion out there on Lambda latency, for example:
2017-04 comparing Lambda performance using Node.js, Java, C# or Python
2018-03 Lambda call latency
2019-09 improved VPC networking for AWS Lambda
2019-10 you're thinking about cold starts all wrong
In December 2019, AWS Lambda introduced Provisioned Concurrency, which improves things. See:
2019-12 AWS Lambda announces Provisioned Concurrency
2020-09 AWS Lambda Cold Starts: Solving the Problem
You should measure latency for an environment that's representative of your app and its use.
A few things that are important factors related to request latency:
cold starts => higher latency
request patterns are important factors in cold starts
if you need to deploy in VPC (attachment of ENI => higher cold start latency)
using CloudFront --> API Gateway --> Lambda (more layers => higher latency)
choice of programming language (Java likely highest cold-start latency, Go lowest)
size of Lambda environment (more RAM => more CPU => faster)
Lambda account and concurrency limits
pre-warming strategy
Update 2019-12: see Predictable start-up times with Provisioned Concurrency.
Update 2021-08: see Increasing performance of Java AWS Lambda functions using tiered compilation.
As an AWS Lambda + API Gateway user (with Serverless Framework) I had to deal with this too.
The problem I faced:
Few requests per day per lambda (not enough to keep lambdas warm)
Time critical application (the user is on the phone, waiting for text-to-speech to answer)
How I worked around that:
The idea was to find a way to call the critical lambdas often enough that they don't get cold.
If you use the Serverless Framework, you can use the serverless-plugin-warmup plugin that does exactly that.
If not, you can copy it's behavior by creating a worker that will invoke the lambdas every few minutes to keep them warm. To do this, create a lambda that will invoke your other lambdas and schedule CloudWatch to trigger it every 5 minutes or so. Make sure to call your to-keep-warm lambdas with a custom event.source so you can exit them early without running any actual business code by putting the following code at the very beginning of the function:
if (event.source === 'just-keeping-warm) {
console.log('WarmUP - Lambda is warm!');
return callback(null, 'Lambda is warm!');
}
Depending on the number of lamdas you have to keep warm, this can be a lot of "warming" calls. AWS offers 1.000.000 free lambda calls every month though.
We have used AWS Lambda quite successfully with reasonable and acceptable response times. (REST/JSON based API + AWS Lambda + Dynamo DB Access).
The latency that we measured always had the least amount of time spent in invoking functions and large amount of time in application logic.
There are warm up techniques as mentioned in the above posts.