i created an AWS Lambda function, and created a scheduled event rule in Cloudwatch to trigger it every 5 minutes :
Schedule Cron expression : 0/5 * * * ? *
The problem is that when looking at the logs, the Lambda seems to be executed every 1~2 minute.
I checked the CRON expression, it seem correct because AWS shows previews of the next triggers. I also tried with a rate expression, but i still have the same issue.
I tooks a look at cloudwatch metrics :
- The scheduled event seems fine, it triggered once every 10minutes
- The lambda invocation metric show that it is invoked more often
Any help ?
Thanks #JohnRotenstein for the lead,
the issue came from the returned result from the NodeJS promise.
If the JS handler doesn't give back any result inside the returned promise, lambda considers the execution as a failure.
In my case i simply had to return a value (even null) so that the execution completes successfully.
Otherwise it looks like lambda will to re-execute the code 1~2 minutes later
Related
I made a eventbridge evoking every minute to fetch data and save it.
Lmabda that eventbridge evokes consumes around 80mb ~ 90mb and takes around 3000ms ~3500ms (3~3.5 sec)
As you could see, my lambda is triggered at ??:35 sec which I didn't expected
I expected lambda to run every ??:00 like example AWS shows
expected to run at :
05:01:00, 05:02:00, 05:03:00, 05:04:00, ...
reality :
05:01:35, 05:02:35, 05:03:35, 05:04:35, ...
should I start event trigger at ??:00 by my self?
You can't change that, as resolution of EB is 1 minute, not seconds. From docs:
Your scheduled rule runs within that minute, but not on the precise 0th second.
I have been trying to address the issue (dropping an event in a queue with “cron") and need to get someone else’s insight on the issue.
Current status:
Currently, I have two lambdas. The thing that I am trying to do is to set the schedule (cloudwatch - every 12PM) to trigger the Lambda A, and the Lambda A triggers Lambda B via Event bridge.
It is all working if I use schedule-expression as "cron(0/5 * * * ? *)" which means it triggers the Lambda A every 5 mins and I can get the end result from Lambda B.
Since the 5 mins interval is working with the whole flow (CloudWatch schedule -> Lambda A -> Lambda B), I can tell all the rules and event bridge setting etc are working.
Problem:
The problem is that when I change the Schedule expression to cron(0 12 * * ? *) from cron(0/5 * * * ? *), it only worked until Lambda A - which means the “cron" triggers the Lambda A at 12PM correctly but the following event to trigger the Lambda B is not working for some reason.
The event on SQS (calling Lambda B) is dropped and put it to Deadletter queue.
I have tried:
I have compared the event(calling Lambda B) detail between using cron 12PM and 5mins interval expressions and no differences found.
I have checked UTC now(let's say this moment is 1am), then I have set cron(5 1 * * ? *), but it only calls the Lambda A.
I have checked the individual Lambdas and event bridge and setting. It worked fine individually.
I have used rate(5mins) instead of cron without any other changes, the Lambda B got correctly triggered.
Questions:
I would like to know what could be the cause of this issue and what could be a way to get proper logs for it.
Please let me know if you have any thought or ideas on this issue.
I have the follow schema (using AWS-IoTCore and AWS-Lambdas) that starts upon some MQTT event. At the end of the main process a flag success is saved in the database. This process takes around 5 min. I would like to call a Lambda function 24hs after the beginning of the process to check if the flag exists in the database. How can I set this timeout or delay in the Lambda function without stepfunctions?
There are two possible paths to "wait 24 hours" until execution of an AWS Lambda function.
Using a 'wait' step in AWS Step Functions
AWS Step Functions is ideal for orchestrating multi-step Lambda functions. There is a Wait state that can be triggered, and the flow could then trigger another Lambda function. Step Functions will track each individual request and you can see each execution and their current state. Sounds good!
Schedule your own check
Use Amazon CloudWatch Events to trigger an AWS Lambda function at regular intervals. The Lambda function could query the database to locate records for processes that did not complete successfully. However, since your code needs to find "non-successful" processes, each process would first need to be added to the database (eg at the start of the 'Process' function).
Alternate approach: Only track failures
The architecture in the above diagram stores 'success' in the database, which is then checked 24 hours later. Based on this diagram, it appears as though the only purpose of the database is to track successful processes.
Instead, I would recommend:
Changing the top line to only store 'failed' processes, and the time of their failure (assuming failure can be detected)
Trigger an AWS Lambda function at regular intervals that will:
Check the database
Retrieve failed processes that are older that 24 hours
Submit them for re-processing
Delete the records from the database, or mark them as 'retried' to avoid future retrievals
Basically, the database is used to track failures, rather than successes. The "wait 24 hours" step is replaced with a regular database check for failed processes.
I'm not sure why the systems wants to wait 24 hours before sending a 'success' email, but I presume that is an intentional part of your design.
You can use CloudWatch Events (or EventBridge) to schedule a Lambda function to run once, at a set time.
To create a CloudWatch Event rule, you can use the PutRule API operation to create the rule and then add the Lambda function as the target using PutTargets. The Lambda function should also allow CloudWatch Events to invoke the function in its function policy.
I also tested this out with the following cron expression: cron(0 0 1 1 ? 2021)
This would trigger the target Lambda function once at Fri, 01 Jan 2021 00:00:00 GMT. Also note, function may be invoked more than once due to CW invoking the function asynchronously. After the Lambda function has been invoked, you can delete the rule from the same function for clean-up.
I would like to create a Scheduled Events to run a Lambda to execute an api call every 1 minute (cron-line behaviour).
The caveat to this setup is that; the external api is un-reliable / slow and the api call sometimes could last longer than 1 minute.
So, my question here is; given the setup & scenario - would AWS run another Scheduled Event and execute the lambda before the previous executing finished? I.e. overlap?
If it does; is there a way to configure the scheduled event to not "overlap"?
I did some initial research into this and came across this article:
https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html
It looks like you can set concurrency limits at function level? Is this the way to achieve non-overlapping scheduled lambda executions? i.e. set the function's concurrency limit to 1?
Yes by default it will execute your Lambda function every 1 minute, regardless if the previous invocation has completed or not.
To enforce no more than one running instance of your Lambda function at a time, set the Concurrency setting of your Lambda function to 1.
I am using aws lambda function to convert uploaded wav file in a bucket to mp3 format and later move file to another bucket. It is working correctly. But there's a problem with triggering. When i upload small wav files,lambda function is called once. But when i upload a large sized wav file, this function is triggered multiple times.
I have googled this issue and found that it is stateless, so it will be called multiple times(not sure this trigger is for multiple upload or a same upload).
https://aws.amazon.com/lambda/faqs/
Is there any method to call this function once for a single upload?
Short version:
Try increasing timeout setting in your lambda function configuration.
Long version:
I guess you are running into the lambda function being timed out here.
S3 events are asynchronous in nature and lambda function listening to S3 events is retried atleast 3 times before that event is rejected. You mentioned your lambda function is executed only once (with no error) during smaller sized upload upon which you do conversion and re-upload. There is a possibility that the time required for conversion and re-upload from your code is greater than the timeout setting of your lambda function.
Therefore, you might want to try increasing the timeout setting in your lambda function configuration.
By the way, one way to confirm that your lambda function is invoked multiple times is to look into cloudwatch logs for the event id (67fe6073-e19c-11e5-1111-6bqw43hkbea3) occurrence -
START RequestId: 67jh48x4-abcd-11e5-1111-6bqw43hkbea3 Version: $LATEST
This event id represents a specific event for which lambda was invoked and should be same for all lambda executions that are responsible for the same S3 event.
Also, you can look for execution time (Duration) in the following log line that marks end of one lambda execution -
REPORT RequestId: 67jh48x4-abcd-11e5-1111-6bqw43hkbea3 Duration: 244.10 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 20 MB
If not a solution, it will at least give you some room to debug in right direction. Let me know how it goes.
Any event Executing Lambda several times is due to retry behavior of Lambda as specified in AWS document.
Your code might raise an exception, time out, or run out of memory. The runtime executing your code might encounter an error and stop. You might run out concurrency and be throttled.
There could be some error in Lambda which makes the client or service invoking the Lambda function to retry.
Use CloudWatch logs to find the error and resolving it could resolve the problem.
I too faced the same problem, in my case it's because of application error, resolving it helped me.
Recently AWS Lambda has new property to change the default Retry nature. Set the Retry attempts to 0 (default 2) under Asynchronous invocation settings.
For some in-depth understanding on this issue, you should look into message delivery guarantees. Then you can implement a solution using the idempotent consumers pattern.
The context object contains information on which request ID you are currently handling. This ID won't change even if the same event fires multiple times. You could save this ID for every time an event triggers and then check that the ID hasn't already been processed before processing a message.
In the Lambda Configuration look for "Asynchronous invocation" there is an option "Retry attempts" that is the maximum number of times to retry when the function returns an error.
Here you can also configure Dead-letter queue service
Multiple retry can also happen due read time out. I fixed with '--cli-read-timeout 0'.
e.g. If you are invoking lambda with aws cli or jenkins execute shell:
aws lambda invoke --cli-read-timeout 0 --invocation-type RequestResponse --function-name ${functionName} --region ${region} --log-type Tail --```payload {""} out --log-type Tail \
I was also facing this issue earlier, try to keep retry count to 0 under 'Asynchronous Invocations'.