I'm experimenting with dynamo db and lambda and am having trouble with the following flow:
Lambda A is triggered by a put to S3 event. It takes the object, an audio file, calculates its duration and writes a record in dynamoDB for each 30 second segment.
Lambda B is triggered by dynamoDB, downloads the file from S3 and operates on the 30 second record defined in the dynamo row.
My trouble is that when I run this flow, function A writes all of the rows required to dynamo, by function B
Does not seem to be triggered for each row in dynamo
Times out after 5 minutes.
Configuration
Function B is set with the highest memory and 5 minute expiration
The trigger is set with a batch size of 1 and starting position latest
Things I've confirmed
When function B is triggered, the download from S3 happens fast. This does not seem to be the blocker
When I trigger function B with a test event it executes perfectly.
When I look at the cloudwatch metrics, function B has a nearly 100% error rate in invocation. I can't tell if this means he function was invoked and had an error or could not be invoked at all.
Has anyone had similar issues? Any idea what to check next?
Thanks
I had the same problem, the solution was to create a VERSION from the Lambda and NOT to use the $LATEST Version, but a 'fixed' one.
It is not possible to use the latest ever-changing version to build a trigger upon.
Place to do that:
Lambda / Functions / YourLambdaName / Qualifiers Dropdown on the page / Switch versions/aliases / Version Tab -> check that you have a version
If not -> Actions / Publish new version
Check for DynamoDB "Stream" is it is enabled on the table.
Checkout this
5 min timeout is default for lambda, you can find this mentioned in forums.
Related
I have a lambda function that generates thumbnails on S3 Put event. It works fine. But I want to handle the case when it accidentally takes longer than reserved time(3 sec).
It's because I'm fetching the lambda generated thumbnail by suffixing '-small.jpg' or '-medium.jpg'. If the timeout happens and the thumbnails are not generated, I must have an alternative image in my bucket.
if you want to increase function timeout you can edit in the general setting of your function. steps and screenshot below will explain how to do it.
Click on the lambda function hyperlink and click on General Configuration.
click on edit [top right pane], and increase the function timeout.
2:
I suspect that creating thumbnails will not take more than 15 minutes at maximum Lambda RAM (and correspondingly max CPU) but if you need to handle that possibility then configure your Lambda function with a DLQ and trigger subsequent processing of failed Lambda functions.
There is a Lambda say my Lambda Name is - XYZ which has a s3 file upload trigger. Now lets say if I upload multiple files say around 100 and the lambda execution starts.
Is there any way I can track if the lambda is running or has processed all the files?
The reason for this is, once the lambda has completed processing all the file, I would want to trigger a step function, so for me to trigger my Step Function I would want to do that only once all the files are processed by my lambda (XYZ).
FYI: There is no current way to track how many files have been uploaded
I think it's not a good design to run a step function state machine after the lambda completes the job without having a perfect logical event.
Because you can for example receive a bunch of files (say 100) completed by the lambda, then using alarms with Cloudwatch once we complete the 100 values of a custom metric, or check value in DynamoDb or number of object in a custom folder, fires a job step function, at the same time you receive another file with +5 sec delay to complete the 101, in this case maybe you miss this file.
If you don't have a special event or a condition to have the completion of the files, then you can work with time scheduling and trigger your Step function with Cloudwatch event ( scheduled event) like every 15 min, check if there is work if not, exit the job.
Otherwise, either include the lambda (file process) in your step function as a step or change your design.
I have a step function with 3 lambdas, the last lambda is basically writing an entry in the dynamo DB with a timestamp, status = "unpaid" (this is updated to "paid" for some automatically based on another workflow), email and closes the execution. Now I want to schedule a reminder on any entry in the DynamoDB which is unpaid & over 7 days, a second reminder if any entry is unpaid over 14 days, a third last reminder on 19th day - sent via email. So the question is:
Is there any way to do this scheduling per Step function execution (that can then monitor that particular entry in ddb for 7, 14, 19 days and send reminders accordingly until the status is "unpaid").
If yes, would it be too much overhead since there could be millions of transactions.
The second way which I was thinking was to build another scheduler lambda sequence: the first lambda basically parsing through the whole ddb searching for entries valid for reminder (either 7, 14, 19). The second lambda getting the list from the first lambda and prepares the reminder based on whether its first, second or third (in loop) & the third Lambda one sending the reminder through SES.
Is there a better or easier way to do this?
I know we can trigger step functions or lambdas through cloud events or we also have crons that we can use but they were not suiting the use case much.
Any help here is appreciated?
DynamoDB does not have functionality for a delayed notification based on logic, you would need to design this flow yourself. Luckily AWS has all the tools you need to perform this.
I believe the best option would probably be to create a CloudWatch Events/EventBridge when the item is written to DynamoDB (either via your application or as a trigger via a Lambda using DynamoDB Streams).
This event would be scheduled for 7 days time, in the 7 days any checks could be performed to validate if it has been paid or not. If it has not been paid you schedule the next event and send out the notification. If it had been paid you would simply exit the Lambda function. This would then continue for the next 2 time periods.
You could then further enhance this by using DynamoDB streams so that in the event of the DynamoDB table being updated a Lambda is triggered to detect whether status has changed from unpaid. If this occurs simply remove the event trigger to prevent it even having to process.
Perhaps I have a misunderstanding of something, but I have a Lambda function that is triggered by new items being added to a DynamoDB table. My trigger is configured like so;
DynamoDB Table Name: My Table
Batch Size: 100
Starting Position: Latest
My function's code parses out any events that are not INSERT, and for the most part, this is functioning well. I am noticing, however, that some of my new records will occasionally not trigger the Lambda function (I update the record with a completed tag when the function has run). I can not find any rhyme or reason as to why, but wondering if I'm missing what the Batch Size is (I want every new record to trigger the function to run, as my users will be publishing individual records to the table).
Is this common behavior or is there more I could share to learn what could be causing this?
I want to be able to set a time to invoke an AWS Lambda function, then have that function be invoked then and only then. For example, I want my Lambda function to run at 9:00pm on December 19th, 2017. I don't want it to repeat, I don't want it to invoke now, just at 9:00pm on the 19th.
I understand that CloudWatch provides Scheduled Events, and I was thinking that when a time to schedule this reminder for is inputted, a CloudWatch Scheduled Events is created to fire in that amount of time from now (so like if you schedule it at 8:22pm to run at 9pm, it’ll be 38 mins), then it invokes the Lambda function at 9pm which then deletes the CloudWatch Scheduled Event. My issue with this is that when a CloudWatch Scheduled Event is created, it executes right then, then at the specified interval.
Any other ideas would be appreciated, as I can't think of another solution. Thanks in advance!
You can schedule lambda event using following syntax:
cron(Minutes Hours Day-of-month Month Day-of-week Year)
Note: All fields are required and time zone is UTC only
Please refer this AWS Documentation for Details.
Thanks
You can use DynamoDB TTL feature to implement this easily, simply do the following:
1- Put item with TTL, the exact time you want to execute or invoke a lambda function.
2- Configure DynamoDB Streams to trigger a lambda function on item's remove event.
Once the item/record is about to expire, your lambda will be invoked. you don't have to delete or cleanup anything as the item in dynamodb is already gone.
NOTE: However the approach is easy to implement and scales very well, but there's one precaution to mention; using DynamoDB TTL as a scheduling mechanism cannot guarantee exact time precision as there might be a delay. The scheduled tasks are executed couple of minutes behind.
You can schedule a step function which can wait until a specific point in time before invoking the lambda with an arbitrary payload.
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-wait-state.html
Something like this
const stepFunctions = new AWS.StepFunctions()
const payload = {
stateMachineArn: process.env.SCHEDULED_LAMBDA_SF_ARN,
name: `${base64.encode(email)}-${base64.encode(timestamp)}`, // Dedupe key
input: JSON.stringify({
timestamp,
lambdaName: 'myLambdaName',
lambdaPayload: {
email,
initiatedBy
},
}),
}
await stepFunctions.startExecution(payload).promise()
I understand its quite late to answer this question. But anyone who wants to use CRON expression to trigger an event(or call an API) only once can use following example:
This event will be triggered only once on January 1, 2025 - 12:00:00 GMT
00 12 01 01 ? 2025
For those who do not have much knowledge of cron syntax:
Minutes Hours DayOfMonth Month DayOfWeek Year
I am using this with AWS Cloudwatch Events and the result looks like this:
Note: I did not have to specify Day of week, since I have given it a fixed date and thats obvious.
Invoking a lambda function via Events is asynchronous invocation option. By using CloudWatchEvent to trigger Lambda function we can use cron job but we are still facing issues as Lambda function triggered multiple times for the same cron schedule.PFB Link:
https://cloudonaut.io/your-lambda-function-might-execute-twice-deal-with-it/
But this needs Dynamo DB to be implemented in your account and then make your Lambda function Idempotent.