I made a eventbridge evoking every minute to fetch data and save it.
Lmabda that eventbridge evokes consumes around 80mb ~ 90mb and takes around 3000ms ~3500ms (3~3.5 sec)
As you could see, my lambda is triggered at ??:35 sec which I didn't expected
I expected lambda to run every ??:00 like example AWS shows
expected to run at :
05:01:00, 05:02:00, 05:03:00, 05:04:00, ...
reality :
05:01:35, 05:02:35, 05:03:35, 05:04:35, ...
should I start event trigger at ??:00 by my self?
You can't change that, as resolution of EB is 1 minute, not seconds. From docs:
Your scheduled rule runs within that minute, but not on the precise 0th second.
Related
Is there a way aws lambda can wait (block and not run/sleep) on an event for a certain amount of time (say 10 hour) and if event isnt received in the window, lambda timeout and raises error. If not with lambda can it be acheived with other aws tech like eventbridge, step function etc?
Lambda run time is only 15 mins max - you can however have another set of lambda's set up that use the SDK to do what you wish - You can use the SDK to either disable / redirect the endpoint the lambda you wish to sleep is at, you can disable invocations, you can (and this would be my suggestion) have a variable stored in the SSM Parameter store that you update that bypasses all the structure in the lambda itself.
You can then use the SDK to set up a TTL/Timed Event to re-enable the parameter store variable after a given time (through the use of another lambda)
When the delay is less than 30 seconds or so, then you can just wait inside the lambda. For longer delays, you should send an SQS message with a delay, and configure your lambda to process it.
The maximum delay you can get from SQS is only 15 minutes, but you can just pick it up and resend with a new delay if you need longer. It'll be very inexpensive.
I have 2 Lambda Functions and an SQS queue inbetween.
The first Lambda sends the messages to the Queue.
Then second Lambda has a trigger for this Queue with a batch size of 250 and a batch window of 65 seconds.
I expect the second Lambda to be triggered in batches of 250 messages after about every 65 seconds. In the second Lambda I'm calling a 3rd party API that is limited to 250 API calls per minute (I get 250 tokens per minute).
I tested this setup with for 32.000 messages being added to the queue and the second Lambda didn't pick up the messages in batches as expected. At first it got executed for 15k messages and then there were not enough tokens so it did not process those messages.
The 3rd party API is based on a token bucket with a fill rate of 250 per minute and a maximum capacity of 15.000. It managed to process the first 15.000 messages due to the bucket capacity and then didn't have enough capacity to handle the rest.
I don't understand what went wrong.
The misunderstanding is probably related to how Lambda handles scaling.
Whenever there are more events than a single Lambda execution context/instance can handle, Lambda just creates more execution contexts/instances to process these events.
What probably happened is that Lambda saw there are a bunch of messages in the queue and it tries to work on these as fast as possible. It created a Lambda instance to handle the first event and then talked to SQS and asked for more work. When it got the next batch of messages, the first instance was still busy, so it scaled out and created a second one that worked on the second batch in parallel, etc. etc.
That's how you ended up going through your token budget in a few minutes.
You can limit how many functions Lambda is allowed to execute in parallel by using reserved concurrency - here are the docs for reference. If you set the reserved concurrency to 1, there will be no parallelization and only one Lambda is allowed to work on the messages.
This however opens you up to another issue. If that single Lambda takes less than 60 seconds to process the messages, Lambda will call it again with another batch ASAP and you might go over your budget again.
At this point a relatively simple approach would be to make sure that your lambda function always takes about 60 seconds by adding a sleep for the remaining time at the end.
i created an AWS Lambda function, and created a scheduled event rule in Cloudwatch to trigger it every 5 minutes :
Schedule Cron expression : 0/5 * * * ? *
The problem is that when looking at the logs, the Lambda seems to be executed every 1~2 minute.
I checked the CRON expression, it seem correct because AWS shows previews of the next triggers. I also tried with a rate expression, but i still have the same issue.
I tooks a look at cloudwatch metrics :
- The scheduled event seems fine, it triggered once every 10minutes
- The lambda invocation metric show that it is invoked more often
Any help ?
Thanks #JohnRotenstein for the lead,
the issue came from the returned result from the NodeJS promise.
If the JS handler doesn't give back any result inside the returned promise, lambda considers the execution as a failure.
In my case i simply had to return a value (even null) so that the execution completes successfully.
Otherwise it looks like lambda will to re-execute the code 1~2 minutes later
I would like to create a Scheduled Events to run a Lambda to execute an api call every 1 minute (cron-line behaviour).
The caveat to this setup is that; the external api is un-reliable / slow and the api call sometimes could last longer than 1 minute.
So, my question here is; given the setup & scenario - would AWS run another Scheduled Event and execute the lambda before the previous executing finished? I.e. overlap?
If it does; is there a way to configure the scheduled event to not "overlap"?
I did some initial research into this and came across this article:
https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html
It looks like you can set concurrency limits at function level? Is this the way to achieve non-overlapping scheduled lambda executions? i.e. set the function's concurrency limit to 1?
Yes by default it will execute your Lambda function every 1 minute, regardless if the previous invocation has completed or not.
To enforce no more than one running instance of your Lambda function at a time, set the Concurrency setting of your Lambda function to 1.
We are trying to develop a true lambda-based application in which certain tasks need to be performed at schedules of variable frequencies. They are actually polling for data, and at certain times of the day, this polling can be as slow as once every hour, while at other times, it has to be once every second. I have looked at the options for scheduling (e.g. Using AWS Lambda with Scheduled Events and AWS re:Invent 2015 | (CMP407) Lambda as Cron: Scheduling Invocations in AWS Lambda), but it seems that short of spinning up an EC2 instance or a long-running lambda, there's no built-in way of firing up lambdas at a frequency of less than one minute. The lambda rate expression doesn't have a place for seconds. Is there a way to do this without an EC2 instance or long-running lambda? Ideally, something that can be done without incurring additional cost for scheduling.
You theoretically can wire up a high-frequency task-trigger without an EC2 instance or long-running lambda using an AWS Step Function executed by a CloudWatch Events scheduled event (see Emanuele Menga's blog post for an example), but that approach would actually be more expensive than the other two options, according to my current (as of May 2019) estimates:
(assuming 1 year = 31536000 seconds)
Step Function:
$0.0250 per 1000 state transitions
2 state transitions per tick (Wait, Task) + at least 1 per minute (for setup/configuration)
31536000 * (2 + 1/60) * 0.0250 / 1000 = $1589.94/year, or as low as $65.70/year for lower frequency trigger (2 ticks per minute)
Lambda Function:
$0.000000208 per 100ms (for smallest 128mb function)
31536000 * 0.000000208 * 10 = $65.595488/year
EC2 Instance:
t3a.nano is $0.0047 per hour on-demand, or as low as $0.0014 using Spot instances
31536000 * 0.0047 / 3600 = $41.172/year, or $12.264/year using Spot instances
So there will be some scheduling cost, but as you can see the cost is not that much.
Currently lambda functions are invoked, at the very least, every 1 minute from Cloudwatch schedule events.
A solution that might work would be the following:
Setup an EC2 instance and from a program, that you will run as a background job, use the aws sdk to invoke your lambda function. Example:
while True:
invoke lambda function via aws sdk
sleep for x seconds
At this point in time AWS Lambda allows functions to be scheduled to run every 5 minutes with a maximum execution time of 5 minutes.
This means if you want to run an AWS Lambda function at intervals less than every 5 minutes while not using EC2 you can take a two phased approach. Create an AWS Lambda function to run every 5 minutes and have it actually run for the entire 5 minutes calling other AWS Lambda functions asynchronously at the appropriate time intervals.
This approach will cost you more since the first AWS Lambda function that runs for the entire 5 minutes will essentially be running continuously, but you can reduce the cost by having the smallest amount of RAM allocated.
UPDATE
CloudWatch Events now allow for schedules at a frequency of 1 minute. This means you can now schedule a Lambda function to run every minute and have it run for up to a minute to achieve sub-minute accuracy. It is important to note that scheduled events do not fire at the beginning of every minute so achieving exact sub-minute timing is still a bit tricky.
Don't poll with lambda or you are better off using EC2 if you expect to spend more time polling than performing work. Lambda charges by execution time and polling is costly.
You really need an event driven system with Lambda. What determines when you poll?
There is a simple hack though where you could use a setTimeout or setInterval.
i.e:
'use strict';
async function Task() {
console.log('Ran at'+new Date())
}
const TIMEOUT = 30000;
exports.handler = (event, context) => {
return Promise.all([
Task(),
new Promise(function(resolve, reject) {
let timeout = setTimeout(function() {
Task().then(() => {
clearTimeout(timeout)
})
}, TIMEOUT);
})
])
}