Called a lambda function once, it's executed twice - amazon-web-services

Question

This is more of a concern than a question, but still, has anyone experienced this before? Does anyone know how to prevent it?
I have a lambda function (L1) which calls a second lambda function (L2) all written in NodeJs (runtime: Node.Js 8.10, and aws-sdk should be v2.488.0 - but I'm just pasting that from the documentation). The short story is that L1 is supposed to call L2, and when it does L2 is executed twice! I discovered this by writing logs to CloudWatch and I could see one L1 log and two L2 logs.
Here's a simplified version of L1 and L2.
L1:
const AWS = require('aws-sdk');
const lambda = new AWS.Lambda();
module.exports = {
handler: async (event, context, callback) => {
const payload: { rnd: Math.random() };
const lambdaParams = {
FunctionName: 'L2',
Qualifier: `dev`,
Payload: JSON.stringify(payload),
};
console.log(`L1 calling: ${JSON.stringify(payload)}`);
return await lambda.invoke(lambdaParams).promise();
},
};
L2:
module.exports = {
handler: async (event, context, callback) => {
console.log(`L2 called: ${JSON.stringify(event)}`);
},
};
In CloudWatch I can see one L1 calling {"rnd": 0.012072353149807702} and two L2 called: {"rnd": 0.012072353149807702}!
BTW, this does not happen all the time. This is part of a step function process which was going to call L1 10k times. My code is written in a way that if L2 is executed twice (per one call), it will fail the whole process (because L2 inserts a record to DB only if it does not exist and fails if it does). So far, I managed to log this behaviour three times. All of them processing the same 10k items, facing the issue at a different iteration each time.
Does anyone have the same experience? Or even better, knows how to make sure one call leads to exactly one execution?

Answer 1

Your lambda function must be idempotent, because it can be called twice in different situations.
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-idempotent/
https://cloudonaut.io/your-lambda-function-might-execute-twice-deal-with-it/

Answer 2

With 10K lambda invokes it must be experiencing a failure and doing a retry.
From the documentation:
Asynchronous Invocation – Lambda retries function errors twice. If the
function doesn't have enough capacity to handle all incoming requests,
events may wait in the queue for hours or days to be sent to the
function. You can configure a dead-letter queue on the function to
capture events that were not successfully processed. For more
information, see Asynchronous Invocation.
If this is what is a happening and you setup the dead letter queue you'll be able to isolate the failure event.
You can also use CloudWatch Logs Insights to easily and quickly search for errors messages of the lambda. Once you select the log group this query should help you get started. Just change the time window.
fields #timestamp, #message
| filter #message like /(?i)(Exception|error|fail|5\d\d)/
| sort #timestamp desc
| limit 20

Answer 3

One case that may cause this is that in your L2 lambda you didn't return anything, which will lead the L1 lambda (the caller) to think there is an error with L2 and so the Retry mechanism is triggered. Try to return something in L2, even simply an "OK".

Answer 4

In my case, it happened that when calling my second lambda, there was a try that was catching an exception with traceback, this triggered the lambda to retry the call, normally without traceback this does not happen, but when commenting the module it stopped happening.
Also within the try it had a condition that yes or yes it could fail, since it had to query for a resource with boto3, so if it existed there was no problem, but since there was no traceback it forced the general failure, not capturing it as an exception.