I understand that AWS Lambda is supposed to abstract the developer from the infrastructure. However I don't quite understand how scaling would work.
Does it automatically start new containers during high traffic?
AWS Lambda functions can be triggered by many different event sources.
AWS Lambda runs each Lambda function runs as a standalone process in its own environment. There is a default limit of 1000 concurrent Lambda functions.
There is no need to think of Lambda "scaling". Rather, whenever an event source (or your own application) runs a Lambda function, the environment is created, the function is run, and the environment is torn down. When there is nothing that is invoking a Lambda function, it is not running. When 1000 invocations happen, then 1000 Lambda functions run.
It "scales" automatically by running in parallel on AWS infrastructure. You only pay while a function is running, per 100ms. It is the job of AWS to ensure that their back-end infrastructure scales to support the number of Lambda functions being run by all customers in aggregate.
If you whant to change the nubmer of desired instanzes in Auto Scaling Group, you chan use botocore.session
import botocore.session
client = session.create_client('autoscaling')
client.set_desired_capacity(
AutoScalingGroupName='NAME',
DesiredCapacity=X,
HonorCooldown=True|False
)
https://docs.aws.amazon.com/cli/latest/reference/autoscaling/set-desired-capacity.html
Related
I am invoking an AWS Lambda function locally using aws-sam cli command and I have set the Timeout property to 900 seconds but still it shows function timeout error. However, when I was invoking this function in lambda handler in AWS Console these 900 seconds were enough for the inferencing.
Please help me figure out a solution for this issue and what is the maximum limit I can go for Timeout?
AWS Lambda functions (as at July 2021) can only run for a maximum of 15 minutes (which is 900 seconds).
Some people do 'interesting' things like:
Call another Lambda function to continue the work, or
Use AWS Step Functions to orchestrate multiple AWS Lambda functions
However, it would appear that your use-case is Machine Learning, which does not like to have operations stopped in the middle of processing. Therefore, AWS Lambda is not suitable for your use-case.
Instead, I would recommend using Amazon EC2 spot instances, which will likely be lower-cost for your use-case. While spot instances might occasionally be terminated, your use-case can probably handle the need to re-run some processing if this happens.
I am new to AWS and experimenting with AWS Lambda and Fargate. I have a long-running process that I have defined as an AWS Fargate containerized task. This task is triggered using ecs.runTask(taskParams, callback) api call from an AWS Lambda function. The Lambda function is triggered by a notification of a file uploaded into an S3 bucket.
You can now configure your AWS Lambda functions to run up to 15
minutes per execution. Previously, the maximum execution time
(timeout) for a Lambda function was 5 minutes. (Source: Amazon)
My question is, does ecs.runTask() run the task asynchronously inside an on-demand container without the lambda function that triggered it waiting for its completion? Does that explain how the lambda function is no longer bound by the task running time? Is this a recommended approach for long-running processes where we don't want an ECS instance just around?
Finally, what is the difference between ecs.runTask() and ecs.startTask() api calls?
asynchronously inside an on-demand container without the lambda function that triggered it waiting for its completion?
Yes. Lambda will just start it.
what is the difference between ecs.runTask() and ecs.startTask() api calls?
startTask can be only used on EC2 launch type and requires you to explicitly choose which EC2 instance to use for your task. Can't be used for Fargate and allows you to launch a single task.
runTask can be use for both EC2 and Fargate launch types. When you use runTask ECS decides where to place your tasks, e.g. which instance. Also, you can run multiple copies of a single task at once.
There are probably more differences, but I think the above are the key ones.
I have one cloud watch event set per minute which triggers AWS Lambda.I have set concurrent executions of lambda to 10 however it's only triggering a single instance per minute. I want it to run 10 concurrent instances per minute.
Concurrency in Lambda is managed pretty differently from what you expect.
In your case you want a single CloudWatch Event to trigger multiple instances each minute.
However, Concurrency in Lambda is working as follows: think you have CloudWatch Event triggering your Lambda and also other AWS services (e.g. S3 and DynamoDB) which trigger your Lambda. What happens when one of your triggers activate the Lambda is that a Lambda instance is active and is consumed until the Lambda finishes its work/computation. During that period of time, the total concurrency units will be decreased by one. At that very moment if another trigger activates the Lambda, the total concurrency units will be decreased again. And this will happen until your Lambda instances are being executed.
So, in your case there will be always a single event (CloudWatch) triggering a single Lambda instance, causing the system not to trigger multiple instances, as for its operation this is the correct way to work. In other words, you do not want to increase concurrent lambda execution to 10 (or whatever) to reach your goal of running 10 parallel instances per minute.
In order to do so, it's probably better for you to create a Lambda orchestrator which calls multiple instances of your Lambda and then setting the Lambda Concurrency in this last Lambda higher than 10 (if you do not want the Lambda to throttle). This way is also pretty good in order to manage the execution of your multiple instances and to catch errors atomically with a greater error flow control.
You can refer to this article in order to get the Lambda Concurrency behavior. The implementation of Lambda orchestrator to manage the multiple instances execution, instead is pretty straightforward.
From the docs:
When your function is invoked, Lambda attempts to re-use the execution
environment from a previous invocation if one is available.
When a lambda function is invoked, an instance of the configured runtime will be launched.
Once the execution completes, after what time that instance of lambda function is destroyed?
Suppose,
at t0=0 sec, lambda is invoked and subsequently starts to initialize its runtime and lets say it takes 2sec to launch the instance of its runtime (say nodejs)
at t1=2 sec, lambda function execution started, and
at t2=3 sec, lambda function execution completed,
and at some time t the instance of nodejs runtime will be destroyed on aws backend,
Is there any way the time t or can it be computed or if it is predefined ?
AWS Lambda functions run in ephemeral containers and all those mechanisms are implemented and maintained by AWS itself.
Developer is not supposed to worry about environment. Developer only need to keep in mind that all functions in his app should be stateless and should not be designed to share any data via container itself.
AWS scales the amount of available containers automatically to provide the optimal performance and minimum cold starts during spikes of load to your service.
There is no exact time to live for those containers.
You can read more about reusing in Understanding Container Reuse in AWS Lambda | AWS Compute Blog.
So our project was using Hangfire to dynamically schedule tasks but keeping in mind auto scaling of server instances we decided to do away with it. I was looking for cloud native serverless solution and decided to use CloudWatch Events with Lambda. I discovered later on that there is an upper limit on the number of Rules that can be created (100 per account) and that wouldn't scale automatically. So now I'm stuck and any suggestions would be great!
As per CloudWatch Events documentation you can request a limit increase.
100 per region per account. You can request a limit increase. For
instructions, see AWS Service Limits.
Before requesting a limit increase, examine your rules. You may have
multiple rules each matching to very specific events. Consider
broadening their scope by using fewer identifiers in your Event
Patterns in CloudWatch Events. In addition, a rule can invoke several
targets each time it matches an event. Consider adding more targets to
your rules.
If you're trying to create a serverless task scheduler one possible way could be:
CloudWatch Event that triggers a lambda function every minute.
Lambda function reads a DynamoDB table and decide which actions need to be executed at that time.
Lambda function could dispatch the execution to other functions or services.
So I decided to do as Diego suggested, use CloudWatch Events to trigger a Lambda every minute which would query DynamoDB to check for the tasks that need to be executed.
I had some concerns regarding the data that would be fetched from dynamoDb (duplicate items in case of longer than 1 minute of execution), so decided to set the concurrency to 1 for that Lambda.
I also had some concerns regarding executing those tasks directly from that Lambda itself (timeouts and tasks at the end of a long list) so what I'm doing is pushing the tasks to SQS each separately and another Lambda is triggered by the SQS to execute those tasks parallely. So far results look good, I'll keep updating this thread if anything comes up.