AWS Lambda "Process exited before completing request" - amazon-web-services

I am trying to call a DynamoDB client method and get one item from the DynamoDB table. I am using AWS Lambda. However, I keep getting the message:
"Process exited before completing request."
I have increased the timeout just to make sure, but the processing time is less than the timeout. Any advice?
console.log('Loading event');
var AWS = require('aws-sdk');
var dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
exports.handler = function(event, context) {
dynamodb.listTables(function(err, data) {
});
var params = {
"TableName": "User",
"Key":
{"User Id" : {"S":event.objectId}
},
"AttributesToGet" : ["First Name","Last Name", "Latitude", "Longitude"],
"ConsistentRead" : true
}
dynamodb.getItem(params, function(response,result) {
response.on('data', function(chunk){
console.log(""+chunk);
console.log("test1")
context.done(result);
});
result.on('ready', function(data){
console.log("test2")
console.log("Error:" + data.error);
console.log("ConsumedCapacityUnits:" + data.ConsumedCapacityUnits);
context.done('Error',data);
// ...
});
});
};

Take a look at your memory consumption (included in last log line). I got the same message when I assigned too little memory to my lambda function.

The message "Process exited before completing request" means that the Javascript function exited before calling context.done (or context.succeed, etc.). Usually, this means that there is some error in your code.
I'm not a Javascript expert (at all) so there may be more elegant ways to find the error but my approach has been to put a bunch of console.log messages in my code, run it, and then look at the logs. I can usually zero in on the offending line and, if I look at it long enough, I can usually figure out my mistake.
I see you have some logging already. What are you seeing in the output?

I have used callback, instead of context.
More recent examples on aws website use callback instead of context.
To complete request, either of the below must be called:
callback(error); // This is used when there is an error
// or
callback(null, data); // This is used when there is a success
// 'data' will contain success result, like some JSON object
When lambda execution completes the request,
failing to call one of the above callbacks,
you will see below error:
"Process exited before completing request."

Error in your code. Remove the last }); and don't use context it is there for backward compatibility, use callbacks on node.js 4.3 and 6.1 runtime.

Maybe you are not following aws lamda standard of using the function
check this Golang code.
package main
import "github.com/aws/aws-lambda-go/lambda"
func main() {
lambda.Start(yourFunction)
}
func yourFunction(){
// do your stuff
}

Check your lamda memory usage, for me this error was occurred because of lambda was using 201 MB memory, which was greater than allowed 200 MB of memory for its execution.
First verify your code and if it is ok, increase memory allotment to this lambda from configuration > General Configuration > Edit > Increase memory

Related

Strange issues calling one Lambda function from another Lamda function

I have an AWS Lambda Function which is called using API gateway. This has a default timeout of 30 seconds.
The processing I need to do takes longer than 30 seconds so I have a second Lambda which is called by the first like this.
using (var client = new AmazonLambdaClient(RegionEndpoint.EUWest2))
{
var lambdaRequest = new InvokeRequest
{
FunctionName = "****LambdaFunction",
InvocationType = "Event",
Payload = JsonConvert.SerializeObject(callProcessingRequest)
};
client.InvokeAsync(lambdaRequest);
}
I've got a couple of issues with it.
First is the documentation says that InvocationType="Event" should call the Lambda async, but that doesn't seem to be the case, it takes about 10 seconds to run that invoke call.
The second more urgent issue is i'm getting intermittant errors logged like this.
1. Lambda encountered an UnobservedTaskException via 'TaskScheduler.UnobservedTaskException' event:
2. A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Signature expired: 20230119T093505Z is now earlier than 20230119T093648Z (20230119T094148Z - 5 min.))

how can I prevent dynamoDB Stream handler from infinitely processing a record when I use batchItemFailures

I have a dynamoDB stream which is triggering a lambda handler that looks like this:
let failedRequestId: string
await asyncForEachSerial(event.Records, async (record) => {
try {
await handle(record.dynamodb.OldImage, record.dynamodb.NewImage, record, context)
return true
} catch (e) {
failedRequestId = record.dynamodb.SequenceNumber
}
return false //break;
})
return {
batchItemFailures:[ { itemIdentifier: failedRequestId } ]
}
I have my lambda set up with a DestinationConfig.onFailure pointing to a DLQ I configured in SQS. The idea behind the handler is to process a batch of events and interrupt at the first failure. Then it reports the most recent failure in 'batchItemFailures' which tells the stream to continue at that record next try. (I pulled the idea from this article)
My current issue is that if there is a genuine failure of my handle() function on one of those records, then my exit code will trigger that record as my checkpoint for the next handler call. However the dlq condition doesn't ever trigger and I end up processing that record over and over again. I should also note that I am trying to avoid reprocessing records multiple times since handle() is not idempotent.
How can I elegantly handle errors while maintaining batching, but without triggering my handle() function more than once for well-behaved stream records?
I'm not sure if you have found the answer you were looking for. I'll respond in case someone else come across this issue.
There are 2 other parameters you'd want to use to avoid that issue. Quoting documentation (https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html):
Retry attempts – The maximum number of times that Lambda retries when the function returns an error. This doesn't apply to service errors or throttles where the batch didn't reach the function.
Maximum age of record – The maximum age of a record that Lambda sends to your function.
Basically, you'll have to specify how many time the failures should be retried and how far back in the events Lambda should be looking at.

Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded at Object.exports.createStatusError - GCP

I am trying to create a google cloud task from one of my Google Cloud Functions. This function gets triggered when a new object is added to one of my Cloud Storage buckets.
I followed the instructions given here to create my App Engine (App Engine Quickstart Guide)
Then in my Cloud Function, I added the following code to create a cloud task (as described here - Creating App Engine Tasks)
However, there is something wrong with my task or App Engine call (not sure what).
I am getting the following errors every now and then. Sometimes it works and sometimes it does not.
{ Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded at Object.exports.createStatusError (/srv/node_modules/grpc/src/common.js:91:15) at Object.onReceiveStatus (/srv/node_modules/grpc/src/client_interceptors.js:1204:28) at InterceptingListener._callNext (/srv/node_modules/grpc/src/client_interceptors.js:568:42) at InterceptingListener.onReceiveStatus (/srv/node_modules/grpc/src/client_interceptors.js:618:8) at callback (/srv/node_modules/grpc/src/client_interceptors.js:845:24) code: 4, metadata: Metadata { _internal_repr: {} }, details: 'Deadline Exceeded' }
Do let me know if you need more information and I will add them to this question here.
I had the same problem with firestore, trying to write one doc at time; I solve it by returning the total promise. That is because cloud function needs to know when is convenient to terminate the function but if you do not return anything maybe cause this error.
My example:
data.forEach( d => {
reports.doc(_date).collection('data').doc(`${d.Id}`).set(d);
})
This was the problem with me, I was writing document 1 by 1 but I wasn't returning the promise. So I solve it doing this:
const _datarwt = [];
data.forEach( d => {
_datarwt.push( reports.doc(_date).collection('data').doc(`${d.Id}`).set(d) );
})
const _dataloaded = await Promise.all( _datarwt );
I save the returned promise in an array and await for all the promises. That solved it for me. Hope been helpful.

Aws Lambda call to external api after callback

I have a lambda function that sends http call to a API(Let's say 'A'). After getting response from 'A' Immediately return the stuff's to the caller i.e., (callback(null, success)) within 10secs. Then save the Data fetched from API 'A' to My External API(Let's Say 'B').
I tried like below but Lambda waits until event loop is empty(It is waiting for the response from second http call).
I doesn't want to set the eventLoopWaitEmpty to false since it freezes the eventloop and Execute next time when invoked.
request.get({url: endpointUrlA},
function (errorA, responseA, bodyA) {
callback(null, "success");
request.post({url: endpointUrlB,
body: bodyA,
json: true}, function(errorB, responseB, bodyB){
//Doesn't want to wait for this response
});
/* Also tried the callback(null, "success"); here too
});
Anybody have any thoughts on How can I implement this? Thanks!
PS - Btw I read the Previous similar questions doesn't seems to clear with those.
This seems like a good candidate for breaking up this lambda into two lambdas with some support code.
First lambda recieves request to 'A' and places a message onto SQS. It then returns to the caller the success status.
A separate process monitors the SQS queue and invokes a second Lambda on it when a message becomes available.
This has several benefits.
Firstly, you no longer have a long-running lambda waiting for a second system that may be down to return.
Secondly, you're doing things asynchronously in the background.
Take a look at this blog post for an overview of how this could work in practice.

AWS Lambda Function lifetime

Consider the following AWS Lambda function:
var i = 0;
exports.handler = function (event, context) {
context.succeed(++i);
};
Executing this function multiple times, I end up with an output similar like the following:
> 0
> 1
> 2
> 0
> 1
> 0
> 3
> 2
> 4
> 1
> 2
As you can see, it seems like there are 3 singletons of the script, and I am randomly ending up in one of them when I execute the function.
Is this an expected behaviour? I couldn't find any related information on the documentation.
I'm asking this because I intend to connect to MySQL and keep a connection pool:
var MySQL = require('mysql');
var connectionPool = MySQL.createPool({
connectionLimit: 10,
host: '*****',
user: '*****',
pass: '*****',
database: '*****'
});
function logError (err, callback) {
console.error(err);
callback('Unable to perform operation');
}
exports.handler = function (event, context) {
connectionPool.getConnection(function (err, connection) {
err && logError(err, context.fail);
connection.query('CALL someSP(?)', [event.user_id], function (err, data) {
err && logError(err, context.fail);
context.succeed(data[0]);
connection.release();
});
});
};
The connection pool needs to be disposed using connectionPool.end() but where shall I execute this?
If I add it at the end of the script (after the handler), then the connection pool will be closed immediately when the lambda function first executes.
If I dispose the connection pool inside the handler, then the connection pool will be closed for future requests.
Furthermore, should I dispose it? If I don't dispose it, the connections will be kepts in the pool and in memory, but as you have seen in the first code sample, AWS keeps ~ 3 singletons of my module, that would mean that I'd end up with 3 different connection pools, with 10 connections each.
Unless I am badly misunderstanding your question, this is well documented and expected behavior for lambda. See here: https://aws.amazon.com/lambda/faqs/
Lambda spins up instances of your container to match the usage patterns of your lambda function. If it is not being used at the moment, then it will spin it down, if it is being used heavily, then more containers will be created. You should never depend on persistent state in a lambda function. It is ok to use state if it is for the lifecycle of your function, or you are optimizing something.
As far as I know, you can not control the number of function instances in memory at any given time, so if you are worried about using up your mysql connections, you should design accordingly.
From the documentation:
"AWS Lambda can start as many copies of your function as needed without lengthy deployment and configuration delays. There are no fundamental limits to scaling a function. AWS Lambda will dynamically allocate capacity to match the rate of incoming events."
As applies directly to your mysql question, I would always return your connection to the pool when you are finished using it. Then I would do some calculations on how many concurrent requests you expect to have and plan accordingly with your mysql server configuration.