Why AWS Lambda function invoked multiple times for a single event? - amazon-web-services

Question

I am trying to create AWS Lambda function that does following process.
Receive S3 "Put" event
Get fileA from S3
Get fileB from S3 that invoked lambda
Launch just one EC2 instance
Create tags for the new EC2 instance
Problem: Multiple(5) instances are launched unexpectedly.
An instance is successfully created, but 4 other instances are also launched. 5 instances in total are launched.
Logs
In the Log Streams for this function, I found 4 Streams for this invocation. Each Stream doesn't show any errors or exceptions, but it seems that the function is executed repeatedly.
Trial
I guessed that the function has been timed out and then re-run.
Then, I changed Timeout from 5s to 60s and put a file on S3.
It somehow effected. Only 2 Log Streams appeared, first one shows that the function has been executed just once, second shows the function has been executed twice. Number of launched instances is 3.
However, I have no idea why multiple(3) instances are launched.
Any comments are welcome!
Thank you in advance :-)
My Lambda function
My Lambda function is following. (It's simplified to hide credential informations but it doesn't lose its basic structure)
var AWS = require('aws-sdk');
function composeParams(data, config){
var block_device_name = "/dev/xvdb";
var security_groups = [
"MyGroupName"
];
var key_name = 'mykey';
var security_group_ids = [
"sg-xxxxxxx"
];
var subnet_id = "subnet-xxxxxxx";
// Configurations for a new EC2 instance
var params = {
ImageId: 'ami-22d27b22', /* required */
MaxCount: 1, /* required */
MinCount: 1, /* required */
KeyName: key_name,
SecurityGroupIds: security_group_ids,
InstanceType: data.instance_type,
BlockDeviceMappings: [
{
DeviceName: block_device_name,
Ebs: {
DeleteOnTermination: true,
Encrypted: true,
VolumeSize: data.volume_size,
VolumeType: 'gp2'
}
}
],
Monitoring: {
Enabled: false /* required */
},
SubnetId: subnet_id,
UserData: new Buffer(config).toString('base64'),
DisableApiTermination: false,
InstanceInitiatedShutdownBehavior: 'stop',
DryRun: data.dry_run,
EbsOptimized: false
};
return params;
}
exports.handler = function(event, context) {
// Get the object from the event
var s3 = new AWS.S3({ apiVersion: '2006-03-01' });
var bucket = event.Records[0].s3.bucket.name;
var key = event.Records[0].s3.object.key;
// Get fileA
var paramsA = {
Bucket: bucket,
Key: key
};
s3.getObject(paramsA, function(err, data) {
if (err) {
console.log(err);
} else {
var dataA = JSON.parse(String(data.Body));
// Get fileB
var paramsB = {
Bucket: bucket,
Key: 'config/config.yml'
};
s3.getObject(paramsB, function(err, data) {
if (err) {
console.log(err, err.stack);
} else {
var config = data.Body;
/* Some process */
// Launch EC2 Instance
var ec2 = new AWS.EC2({ region: REGION, apiVersion: '2015-04-15' });
var params = composeParams(dataA, config);
ec2.runInstances(params, function(err, data) {
if (err) {
console.log(err, err.stack);
} else {
console.log(data);
// Create tags for instance
for (var i=0; i<data.Instances.length; i++){
var instance = data.Instances[i];
var params = {
Resources: [ /* required */
instance.InstanceId
],
Tags: [ /* required */
{
Key: 'Name',
Value: instance_id
},
{
Key: 'userID',
Value: dataA.user_id
}
],
DryRun: dataA.dry_run
};
ec2.createTags(params, function(err, data) {
if (err) {
console.log(err, err.stack);
} else {
console.log("Tags created.");
console.log(data);
}
});
}
}
});
}
});
}
});
};

Answer 1

Solved.
Adding context.succeed(message); to the last part of the nested callback prevents the repeated execution of the function.
ec2.createTags(params, function(err, data) {
if (err) {
console.log(err, err.stack);
context.fail('Failed');
} else {
console.log("Tags created.");
console.log(data);
context.succeed('Completed');
}
});

Answer 2

Check in cloudwatch event that context.aws_request_id value for each invokation. If it is
same than it is retry because aws function got some error raised.
make your lambda idempotent
different than it is because of connection timeout from your aws
lambda client. check aws client configuration request timeout and
connect timeout values.

Answer 3

I was having the same problem with the newer runtime (Node.JS v4.3). Call
context.callbackWaitsForEmptyEventLoop = false;
before calling
callback(...)

Answer 4

Maximum Event Age
When a function returns an error before execution, Lambda returns the event to the queue and attempts to run the function again for up to 6 hours by default. With Maximum Event Age, you can configure the lifetime of an event in the queue from 60 seconds to 6 hours. This allows you to remove any unwanted events based on the event age.
Maximum Retry Attempts
When a function returns an error after execution, Lambda attempts to run it two more times by default. With Maximum Retry Attempts, you can customize the maximum number of retries from 0 to 2. This gives you the option to continue processing new events with fewer or no retries.
under the Configuration > Asynchronous invocation > Retry Attempts
you can set it to 0-2
Source:
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-max-retry-attempts-event-age-asynchronous-invocations/