I enabled streams on my dynamoDB table. As items are modified, a lambda function is triggered. I think I set up everything correctly both on the lambda trigger side, permissions, and dynamodb side. I also ran my lambda function with test data and it succeeded. However, when items are modified in the table, the trigger did not start my lambda function. Instead, I got the following error:
Batch size: 100 Last processing result: PROBLEM: Function call failed
Any idea what's the best way to debug this? I went on CloudWatch logs but there were no logs associated with the trigger/stream.
Thanks.
Edit: Logs for the lambda function (not its dynamodb trigger). The trigger didn't generate any log statements.
START RequestId: 3a08eedc-f0de-11e8-9008-033b48d2cb67 Version: $LATEST
18:16:28
END RequestId: 3a08eedc-f0de-11e8-9008-033b48d2cb67
18:16:28
REPORT RequestId: 3a08eedc-f0de-11e8-9008-033b48d2cb67 Duration: 81.85 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 30 MB
I ran into this issue today.
I debugged it by manually triggering the lambda with the Test button on the top of the main lambda page. It showed the error output trying to run my lambda.
The reason I had an error was the handler parameter as I had a non-standard javascript function name and I forgot to configure that in my lambda.
In my case, my lambda role did not have permissions to write to the SNS and the lambda code was writing to a SNS. So i added a policy to the lambda role giving it permissions to write to any SNS topic.
In my case, the problem came from the stream batch size 100. In the lambda code I was checking the event and I exit if the event doesn't meet the requirement.
In my case, the handler method was not configured correctly for the Lambda function written in Java. The format that I used for setting the handler is as follows
packageName.className::handlerMethod
For example, for my handler class
package com.example;
public class App implements
RequestHandler<DynamodbEvent, String> {
public String handleRequest(DynamodbEvent ddbEvent, Context context) {
the handler should be defined as
com.example.App::handleRequest
This sounds like a possible use-case for Rookout if you need to follow variable values in your live Lambda in a situation where you're not able to generate logs and running it locally isn't going to give you real-world event trigger data.
Related
I need to ping my EP with some ID of lambda task once lambda finished (pass or crash).
For pass event I can implement it in code, but what aboout crash event? Is there any generic solution in aws for this?
If you catch an exception could send the message otherwise for all unhandled exception/lambda crashes better to rely on creating lambda DLQ and use it for your purpose.
https://aws.amazon.com/about-aws/whats-new/2016/12/aws-lambda-supports-dead-letter-queues/
Extra:
There are step functions https://aws.amazon.com/step-functions/ where there is a way to define if Lambda A is successful execute Lambda B , in case of failure call Lambda C.
If you try to make any kind of custom lambda observability check what is available on the market, there are way to do this without writing extra code inside your lambda.
You can do this using Cloudwatch with either custom metrics for specific errors or just use what they have for 400s or 500s. Then you can have an alarm for that which triggers a Cloudwatch rule (which you could call your crash event)
I have the follow schema (using AWS-IoTCore and AWS-Lambdas) that starts upon some MQTT event. At the end of the main process a flag success is saved in the database. This process takes around 5 min. I would like to call a Lambda function 24hs after the beginning of the process to check if the flag exists in the database. How can I set this timeout or delay in the Lambda function without stepfunctions?
There are two possible paths to "wait 24 hours" until execution of an AWS Lambda function.
Using a 'wait' step in AWS Step Functions
AWS Step Functions is ideal for orchestrating multi-step Lambda functions. There is a Wait state that can be triggered, and the flow could then trigger another Lambda function. Step Functions will track each individual request and you can see each execution and their current state. Sounds good!
Schedule your own check
Use Amazon CloudWatch Events to trigger an AWS Lambda function at regular intervals. The Lambda function could query the database to locate records for processes that did not complete successfully. However, since your code needs to find "non-successful" processes, each process would first need to be added to the database (eg at the start of the 'Process' function).
Alternate approach: Only track failures
The architecture in the above diagram stores 'success' in the database, which is then checked 24 hours later. Based on this diagram, it appears as though the only purpose of the database is to track successful processes.
Instead, I would recommend:
Changing the top line to only store 'failed' processes, and the time of their failure (assuming failure can be detected)
Trigger an AWS Lambda function at regular intervals that will:
Check the database
Retrieve failed processes that are older that 24 hours
Submit them for re-processing
Delete the records from the database, or mark them as 'retried' to avoid future retrievals
Basically, the database is used to track failures, rather than successes. The "wait 24 hours" step is replaced with a regular database check for failed processes.
I'm not sure why the systems wants to wait 24 hours before sending a 'success' email, but I presume that is an intentional part of your design.
You can use CloudWatch Events (or EventBridge) to schedule a Lambda function to run once, at a set time.
To create a CloudWatch Event rule, you can use the PutRule API operation to create the rule and then add the Lambda function as the target using PutTargets. The Lambda function should also allow CloudWatch Events to invoke the function in its function policy.
I also tested this out with the following cron expression: cron(0 0 1 1 ? 2021)
This would trigger the target Lambda function once at Fri, 01 Jan 2021 00:00:00 GMT. Also note, function may be invoked more than once due to CW invoking the function asynchronously. After the Lambda function has been invoked, you can delete the rule from the same function for clean-up.
I have written and saved a lambda function. I see:
Congratulations! Your Lambda function "lambda_name" has been
successfully created. You can now change its code and configuration.
Choose Test to input a test event when you want to test your function.
Now how do I run it? I cannot see a 'run' or 'invoke' button as I would expect
Note
The lambda doesn't accept any arguments (it's extremely simple - for the purposes of this question, please presume it's simply 2 * 2 so when I run it it should not require any inputs and should return 4).
Also note
I can see a tonne of different ways to run the lambda here. I just want the simplest way (preferably a button in the browser)
Sending a test message via the Lambda console will run your Lambda function. The test message that you configure will define what is in the event parameter of your lambda handler function.
Since you are not doing anything with that message, you can send any arbitrary test message and it should work for you. You can just use the default hello world message and give it an arbitrary name.
It should then show you the results: any logs or returned objects right in the AWS Lambda console.
Further reading here
AWS Lambda functions are typically triggered by an event, such as an object being uploaded to Amazon S3 or a message being send to an Amazon SNS topic.
This is because Lambda functions are great at doing a small task very often. Often, Lambda functions only run for a few seconds, or even less than a second! Thus, they are normally triggered in response to something else happening. It's a bit like when somebody rings your phone, which triggers you to answer the phone. You don't normally answer your phone when it isn't ringing.
However, it is also possible to directly invoke an AWS Lambda function using the Invoke() command in the AWS SDK. For convenience, you can also use the AWS Command-Line Interface (CLI) aws lambda invoke command. When directly invoking an AWS Lambda function, you can receive a return value. This is in contrast to situations where a Lambda function is triggered by an event, in which case there is nowhere to 'return' a value since it was not directly invoked.
I need my python lambda to invoke multiple lambda so they can run in parallel. I have it working - kind of. To test it, I invoke the code below for 5 different lambda calls (could be more in the future)... when i look at the Step Function console, I only see 2 lamdbas running concurrently and the other 3 are ignored after the first 2 finished. In researching concurrency issues, I didnt find a limit that is this low (5 should be nothing for AWS!)..
response = lambdaClient.invoke(
FunctionName=lambdaToInvoke <---this var is an ARN,
InvocationType='Event',
Payload=json.dumps(event) <--just passing state
)
All IAM is provisioned...Any thoughts are appreciated!
Usually when I need to do a "fan out" operation like you are describing I don't invoke the lambda functions directly.
Rather I would recommend you setup your 5 "slave" lambda functions to get activated from either SNS or SQS. From your "master" function you can use the SDK to create a SNS alert or SQS message.
What happens when you try to manually invoke the 3 functions that don't run?
Do you have logging setup in CW logs? Can you post?
I am using aws lambda function to convert uploaded wav file in a bucket to mp3 format and later move file to another bucket. It is working correctly. But there's a problem with triggering. When i upload small wav files,lambda function is called once. But when i upload a large sized wav file, this function is triggered multiple times.
I have googled this issue and found that it is stateless, so it will be called multiple times(not sure this trigger is for multiple upload or a same upload).
https://aws.amazon.com/lambda/faqs/
Is there any method to call this function once for a single upload?
Short version:
Try increasing timeout setting in your lambda function configuration.
Long version:
I guess you are running into the lambda function being timed out here.
S3 events are asynchronous in nature and lambda function listening to S3 events is retried atleast 3 times before that event is rejected. You mentioned your lambda function is executed only once (with no error) during smaller sized upload upon which you do conversion and re-upload. There is a possibility that the time required for conversion and re-upload from your code is greater than the timeout setting of your lambda function.
Therefore, you might want to try increasing the timeout setting in your lambda function configuration.
By the way, one way to confirm that your lambda function is invoked multiple times is to look into cloudwatch logs for the event id (67fe6073-e19c-11e5-1111-6bqw43hkbea3) occurrence -
START RequestId: 67jh48x4-abcd-11e5-1111-6bqw43hkbea3 Version: $LATEST
This event id represents a specific event for which lambda was invoked and should be same for all lambda executions that are responsible for the same S3 event.
Also, you can look for execution time (Duration) in the following log line that marks end of one lambda execution -
REPORT RequestId: 67jh48x4-abcd-11e5-1111-6bqw43hkbea3 Duration: 244.10 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 20 MB
If not a solution, it will at least give you some room to debug in right direction. Let me know how it goes.
Any event Executing Lambda several times is due to retry behavior of Lambda as specified in AWS document.
Your code might raise an exception, time out, or run out of memory. The runtime executing your code might encounter an error and stop. You might run out concurrency and be throttled.
There could be some error in Lambda which makes the client or service invoking the Lambda function to retry.
Use CloudWatch logs to find the error and resolving it could resolve the problem.
I too faced the same problem, in my case it's because of application error, resolving it helped me.
Recently AWS Lambda has new property to change the default Retry nature. Set the Retry attempts to 0 (default 2) under Asynchronous invocation settings.
For some in-depth understanding on this issue, you should look into message delivery guarantees. Then you can implement a solution using the idempotent consumers pattern.
The context object contains information on which request ID you are currently handling. This ID won't change even if the same event fires multiple times. You could save this ID for every time an event triggers and then check that the ID hasn't already been processed before processing a message.
In the Lambda Configuration look for "Asynchronous invocation" there is an option "Retry attempts" that is the maximum number of times to retry when the function returns an error.
Here you can also configure Dead-letter queue service
Multiple retry can also happen due read time out. I fixed with '--cli-read-timeout 0'.
e.g. If you are invoking lambda with aws cli or jenkins execute shell:
aws lambda invoke --cli-read-timeout 0 --invocation-type RequestResponse --function-name ${functionName} --region ${region} --log-type Tail --```payload {""} out --log-type Tail \
I was also facing this issue earlier, try to keep retry count to 0 under 'Asynchronous Invocations'.