I have a Lambda function inside a VPC to access ElasticSearch Endpoints. The Lambda has been auto provisioned (Provisioned Concurrency). The issue is, whenever the Lambda function executes, there is an Initialization time. I have set up other Lambdas without VPC setup and with auto provisioning and they do not have the cold start issue.
I read about the extra time that Lambda takes when inside a VPC to initialize the ENIs and the Lambda needs to attach itself to the ENI. But the linked document doesn't mention whether the Lambda has been enabled with auto provisioning.
I wanted to know if the Lambda is inside a VPC, even if it has been enabled with Auto Provisioning, will it still take a considerable amount of Initialization time that can be seen on X-Ray Traces. Also, if this is true, is there a way to reduce this Initialization time by emitting dummy CloudWatch alerts to keep the Lambda warm?
Related
I am new to AWS and experimenting with AWS Lambda and Fargate. I have a long-running process that I have defined as an AWS Fargate containerized task. This task is triggered using ecs.runTask(taskParams, callback) api call from an AWS Lambda function. The Lambda function is triggered by a notification of a file uploaded into an S3 bucket.
You can now configure your AWS Lambda functions to run up to 15
minutes per execution. Previously, the maximum execution time
(timeout) for a Lambda function was 5 minutes. (Source: Amazon)
My question is, does ecs.runTask() run the task asynchronously inside an on-demand container without the lambda function that triggered it waiting for its completion? Does that explain how the lambda function is no longer bound by the task running time? Is this a recommended approach for long-running processes where we don't want an ECS instance just around?
Finally, what is the difference between ecs.runTask() and ecs.startTask() api calls?
asynchronously inside an on-demand container without the lambda function that triggered it waiting for its completion?
Yes. Lambda will just start it.
what is the difference between ecs.runTask() and ecs.startTask() api calls?
startTask can be only used on EC2 launch type and requires you to explicitly choose which EC2 instance to use for your task. Can't be used for Fargate and allows you to launch a single task.
runTask can be use for both EC2 and Fargate launch types. When you use runTask ECS decides where to place your tasks, e.g. which instance. Also, you can run multiple copies of a single task at once.
There are probably more differences, but I think the above are the key ones.
From the docs:
When your function is invoked, Lambda attempts to re-use the execution
environment from a previous invocation if one is available.
When a lambda function is invoked, an instance of the configured runtime will be launched.
Once the execution completes, after what time that instance of lambda function is destroyed?
Suppose,
at t0=0 sec, lambda is invoked and subsequently starts to initialize its runtime and lets say it takes 2sec to launch the instance of its runtime (say nodejs)
at t1=2 sec, lambda function execution started, and
at t2=3 sec, lambda function execution completed,
and at some time t the instance of nodejs runtime will be destroyed on aws backend,
Is there any way the time t or can it be computed or if it is predefined ?
AWS Lambda functions run in ephemeral containers and all those mechanisms are implemented and maintained by AWS itself.
Developer is not supposed to worry about environment. Developer only need to keep in mind that all functions in his app should be stateless and should not be designed to share any data via container itself.
AWS scales the amount of available containers automatically to provide the optimal performance and minimum cold starts during spikes of load to your service.
There is no exact time to live for those containers.
You can read more about reusing in Understanding Container Reuse in AWS Lambda | AWS Compute Blog.
I have a Lambda that I have created following the example given by the aws docs (https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-invoke-lambda-function.html), but I am invoking my Lambda from within a VPC and it seems that the CodePipeline never successfully talks to the Lambda (it gets a timeout and never seems to enter the Lambda as CloudWatch has none of my console.logs); this is despite the fact that I have created a CodePipeline Endpoint from within the VPC and associated the private subnet from which I launch the Lambda out to codepipeline.
I can give the Lambda an API Gateway endpoint and fire it manually just fine from Postman; it takes ~1 second to run. My Cloudwatch logs just have "Task timed out after 20.02 seconds." I'm not sure what else I can try; what else might prevent CodePipeline from talking to the Lambda?
After additional logging, I discovered that I actually had the VPC set up correctly and that the Lambda was being invoked; the Lambda was failing to get to S3 and was hanging on getting objects. I created another Endpoint for S3 for the VPC and was able to move passed the initial issue.
In one of my project, one of the AWS Lambda function (usually called every minute) invoking another AWS Lambda function inside its function ( using AWSLambdaClient lambdaClient;). sometimes lambdaClient on invocation of lambda function (its not frequent say 4 to 5 time in every hour) throwing SubnetIPAddressLimitReachedException :
2016-11-24 14 <---------------------> INFO xyzHandler:395 - Lambda was not able to set up VPC access for the Lambda function because one or more configured subnets has no available IP addresses. (Service: AWSLambda; Status Code: 502; Error Code: SubnetIPAddressLimitReachedException; Request ID: XXXX)
I searched here and here , but I didn't find any clear explaination of this exception ?
When your Lambda function is configured to execute inside your VPC, you specify one or more subnet IDs in which the Lambda function will execute.
The subnets that you specify needs to have enough free IP addresses inside them to accomodate all of the simultaneous executions of your Lambda function.
For example, if you choose one subnet and it is defined as a /24, then you have at most 254 or so IP addresses.
If your Lambda function(s) are called 300 times simultaneously, they're going to need 300 individual IP addresses, which your subnet cannot accomodate. In this case, you will get the SubnetIPAddressLimitReachedException error.
When Lambda functions complete, their resources will be reused. So they will free up the used IP addresses and/or re-use them during subsequent Lambda executions.
There is currently no way to limit the number of simultaneous executions within Lambda itself. I've seen people use other services, such as Kinesis, to limit it.
There are 3 avenues of resolution:
If your Lambda function does not need to execute within your VPC, and/or access resources from within your VPC, move it out of the VPC.
Specify more or different subnet IDs with more available IP addresses.
Modify your Lambda function to not call other Lambda functions. The root Lambda function and the subsequently called Lambda functions will each require an IP address.
Accessing Resources in a VPC
You can set this up when you create a new function. You can also update an existing function so that it has VPC access. You can configure this feature from the Lambda Console or from the CLI. Here’s how you set it up from the Console:
That’s all you need to do! Be sure to read Configuring a Lambda Function to Access Resources in an Amazon VPC in the Lambda documentation if you have any questions.
Resource link:
Access Resources in a VPC from Your Lambda Functions
Accessing the Internet and other AWS Resources in your VPC from AWS
Lambda
I understand that AWS Lambda is supposed to abstract the developer from the infrastructure. However I don't quite understand how scaling would work.
Does it automatically start new containers during high traffic?
AWS Lambda functions can be triggered by many different event sources.
AWS Lambda runs each Lambda function runs as a standalone process in its own environment. There is a default limit of 1000 concurrent Lambda functions.
There is no need to think of Lambda "scaling". Rather, whenever an event source (or your own application) runs a Lambda function, the environment is created, the function is run, and the environment is torn down. When there is nothing that is invoking a Lambda function, it is not running. When 1000 invocations happen, then 1000 Lambda functions run.
It "scales" automatically by running in parallel on AWS infrastructure. You only pay while a function is running, per 100ms. It is the job of AWS to ensure that their back-end infrastructure scales to support the number of Lambda functions being run by all customers in aggregate.
If you whant to change the nubmer of desired instanzes in Auto Scaling Group, you chan use botocore.session
import botocore.session
client = session.create_client('autoscaling')
client.set_desired_capacity(
AutoScalingGroupName='NAME',
DesiredCapacity=X,
HonorCooldown=True|False
)
https://docs.aws.amazon.com/cli/latest/reference/autoscaling/set-desired-capacity.html