I know about the provisioned instance configuration for lambda functions. Is it possible to run multiple instances of a lambda function on a timer basis? I know generally we use CloudWatch Events for this, just not how to specify multiple instances.
To be clear, I want something like: I want 10 instances of my function to run at "2022-02-02 10:10:10".
Some options:
Create 10 identical CloudWatch events
Create a new Lambda that is triggered by your single CloudWatch event. The new Lambda would invokes your worker Lambda function 10 times asynchronously
Create a Step Functions state machine that triggers 10 Lambda invocations, and trigger the step function on a schedule
Related
I have an aws lambda function. When it receives only one trigger, it always succeds. But when it receives more than one trigger, it sometimes throws error. The first trigger always succeds.
Can I configure one aws lambda function receives only one trigger?
can one aws lambda function handle multiple triggers at once?
Yes, Lambda functions can handle multiple triggers at once.
when it receives more than one trigger, it sometimes throws error
This is most probably related to your implementation. Are you doing something different based on the inputs? Is the code behaving differently based on time?
Can I configure one aws lambda function receives only one trigger?
You can limit the concurrency of the Lambda function. If you set it to 1, you can only have one Lambda function running at any given time.
See: Set Concurrency Limits on Individual AWS Lambda Functions
I am new to AWS and experimenting with AWS Lambda and Fargate. I have a long-running process that I have defined as an AWS Fargate containerized task. This task is triggered using ecs.runTask(taskParams, callback) api call from an AWS Lambda function. The Lambda function is triggered by a notification of a file uploaded into an S3 bucket.
You can now configure your AWS Lambda functions to run up to 15
minutes per execution. Previously, the maximum execution time
(timeout) for a Lambda function was 5 minutes. (Source: Amazon)
My question is, does ecs.runTask() run the task asynchronously inside an on-demand container without the lambda function that triggered it waiting for its completion? Does that explain how the lambda function is no longer bound by the task running time? Is this a recommended approach for long-running processes where we don't want an ECS instance just around?
Finally, what is the difference between ecs.runTask() and ecs.startTask() api calls?
asynchronously inside an on-demand container without the lambda function that triggered it waiting for its completion?
Yes. Lambda will just start it.
what is the difference between ecs.runTask() and ecs.startTask() api calls?
startTask can be only used on EC2 launch type and requires you to explicitly choose which EC2 instance to use for your task. Can't be used for Fargate and allows you to launch a single task.
runTask can be use for both EC2 and Fargate launch types. When you use runTask ECS decides where to place your tasks, e.g. which instance. Also, you can run multiple copies of a single task at once.
There are probably more differences, but I think the above are the key ones.
I have a lambda function that accepts a parameter i.e a category_id, pulls some data from an API, and updates the database based on the response.
I have to execute the same lambda function for Multiple Ids after an interval of 1 minute on daily basis.
For example, run lambda for category 1 at 12:00 AM, then run for category 2 at 12:01 AM and so one for 500+ categories.
What could be the best possible solution to achieve this?
This is what I am currently thinking:
Write Lambda using AWS SAM
Add Lambda Layer for Shared Dependencies
Attach Lambda with AWS Cloudwatch Events to run it on schedule
Add Environment Variable for category_id in lambda
Update the SAM template to use the same lambda function again and again but only change will be in the Cron expression schedule and Value of Environment Variable category_id
Problems in the Above Solution:
Number of Lambda functions will increase in the account.
Each Lambda will be attached with a Cloudwatch Event so its number will also increase
There is a quota limit of max 300 Cloudwatch Event per account (though we can request support to increase that limit)
It'll require the use of nested stacks because of the SAM template size limit as well as the number of resources per template which 200 max.
I'll be able to create only 50 Lambda Functions per nested stack, it means the number of nested stacks will also increase because 1 lambda = 4 resources (Lambda + Role + Rule + Event)
Other solutions (not sure if they can be used):
Use of Step Functions
Trigger First Lambda function only using Cron Schedule and Invoke Lambda for the next category using current lambda(only one CloudWatch Event will be required to invoke the function for the first category but time difference will vary i.e next lambda will not execute exactly after one minute).
Use Only One Lambda and One Cloud Watch Schedule Event, Lambda Function will have a list of all category ids and that function will invoke itself recursively by using one category id at a time and removing the use category id from the list (the only problem is lambda will not execute exactly after one minute for next category_id in the list)
Looking forward to hearing about the best solution.
I would suggest using a standard Worker pattern:
Create an Amazon SQS queue
Configure the AWS Lambda function so that it is triggered to run whenever a message is sent to the SQS queue
Trigger a separate process at midnight (eg another Lambda function) that sends the 500 messages to the SQS queue, each with a different category ID
This will cause the Amazon SQS functions to execute. If you only want one of the Lambda functions to be running at any time (with no parallel executions), set the function's Concurrency Limit to 1 so that only one is running at any time. When one function completes, Lambda will automatically grab another message from the queue and start executing. There will be practically no "wasted time" between executions of the function.
Given that you are doing a large amount of processing, an Amazon EC2 instance might be more appropriate.
If the bandwidth requirements are low (eg if it is just making API calls), then a T3a.micro ($0.0094 per Hour) or even T3a.nano instance ($0.0047 per Hour) can be quite cost-effective.
A script running on the instance could process a category, then sleep for 30 seconds, in a big loop. Running 500 categories at one minute each would take about 8 hours. That's under 10c each day!
The instance can then stop or self-terminate when the work is complete. See: Auto-Stop EC2 instances when they finish a task - DEV Community
I have a Scheduled Lambda function (via CloudWatch event rule) which is triggered every minute.
This lambda picks up a request from SQS queue, process the parameters and triggers AWS step functions workflow.
Now, ONLY 1 Lambda function instance is running every minute. How can I trigger multiple (e.g. 10) concurrent Lambda functions like this?
One way I can think of is to create 10 Cloudwatch event rule which runs every 1 minute, but I am not sure if that is the right way of doing it. Also, if I use this way, 10 lambda would be called even if I don't have entries in my SQS queue.
You can use the lambda step function.
Event trigger first function. Then it will call multiple functions parallel.
Some useful links:
https://www.youtube.com/watch?v=c797gM0f_Pc
https://medium.com/soluto-nashville/simplifying-workflows-with-aws-step-functions-57d5fad41e59
since your lambda function fetching data from SQS so you can create event source mapping between lambda and SQS so whenever message published to SQS, your lambda function will invoke concurrently depending on number of messages in queue so you do not need to invoke lamnda from cloudwatch event
I have one cloud watch event set per minute which triggers AWS Lambda.I have set concurrent executions of lambda to 10 however it's only triggering a single instance per minute. I want it to run 10 concurrent instances per minute.
Concurrency in Lambda is managed pretty differently from what you expect.
In your case you want a single CloudWatch Event to trigger multiple instances each minute.
However, Concurrency in Lambda is working as follows: think you have CloudWatch Event triggering your Lambda and also other AWS services (e.g. S3 and DynamoDB) which trigger your Lambda. What happens when one of your triggers activate the Lambda is that a Lambda instance is active and is consumed until the Lambda finishes its work/computation. During that period of time, the total concurrency units will be decreased by one. At that very moment if another trigger activates the Lambda, the total concurrency units will be decreased again. And this will happen until your Lambda instances are being executed.
So, in your case there will be always a single event (CloudWatch) triggering a single Lambda instance, causing the system not to trigger multiple instances, as for its operation this is the correct way to work. In other words, you do not want to increase concurrent lambda execution to 10 (or whatever) to reach your goal of running 10 parallel instances per minute.
In order to do so, it's probably better for you to create a Lambda orchestrator which calls multiple instances of your Lambda and then setting the Lambda Concurrency in this last Lambda higher than 10 (if you do not want the Lambda to throttle). This way is also pretty good in order to manage the execution of your multiple instances and to catch errors atomically with a greater error flow control.
You can refer to this article in order to get the Lambda Concurrency behavior. The implementation of Lambda orchestrator to manage the multiple instances execution, instead is pretty straightforward.