I have an AWS Lambda Function which is called using API gateway. This has a default timeout of 30 seconds.
The processing I need to do takes longer than 30 seconds so I have a second Lambda which is called by the first like this.
using (var client = new AmazonLambdaClient(RegionEndpoint.EUWest2))
{
var lambdaRequest = new InvokeRequest
{
FunctionName = "****LambdaFunction",
InvocationType = "Event",
Payload = JsonConvert.SerializeObject(callProcessingRequest)
};
client.InvokeAsync(lambdaRequest);
}
I've got a couple of issues with it.
First is the documentation says that InvocationType="Event" should call the Lambda async, but that doesn't seem to be the case, it takes about 10 seconds to run that invoke call.
The second more urgent issue is i'm getting intermittant errors logged like this.
1. Lambda encountered an UnobservedTaskException via 'TaskScheduler.UnobservedTaskException' event:
2. A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Signature expired: 20230119T093505Z is now earlier than 20230119T093648Z (20230119T094148Z - 5 min.))
Related
I have a dynamoDB stream which is triggering a lambda handler that looks like this:
let failedRequestId: string
await asyncForEachSerial(event.Records, async (record) => {
try {
await handle(record.dynamodb.OldImage, record.dynamodb.NewImage, record, context)
return true
} catch (e) {
failedRequestId = record.dynamodb.SequenceNumber
}
return false //break;
})
return {
batchItemFailures:[ { itemIdentifier: failedRequestId } ]
}
I have my lambda set up with a DestinationConfig.onFailure pointing to a DLQ I configured in SQS. The idea behind the handler is to process a batch of events and interrupt at the first failure. Then it reports the most recent failure in 'batchItemFailures' which tells the stream to continue at that record next try. (I pulled the idea from this article)
My current issue is that if there is a genuine failure of my handle() function on one of those records, then my exit code will trigger that record as my checkpoint for the next handler call. However the dlq condition doesn't ever trigger and I end up processing that record over and over again. I should also note that I am trying to avoid reprocessing records multiple times since handle() is not idempotent.
How can I elegantly handle errors while maintaining batching, but without triggering my handle() function more than once for well-behaved stream records?
I'm not sure if you have found the answer you were looking for. I'll respond in case someone else come across this issue.
There are 2 other parameters you'd want to use to avoid that issue. Quoting documentation (https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html):
Retry attempts – The maximum number of times that Lambda retries when the function returns an error. This doesn't apply to service errors or throttles where the batch didn't reach the function.
Maximum age of record – The maximum age of a record that Lambda sends to your function.
Basically, you'll have to specify how many time the failures should be retried and how far back in the events Lambda should be looking at.
This is more of a concern than a question, but still, has anyone experienced this before? Does anyone know how to prevent it?
I have a lambda function (L1) which calls a second lambda function (L2) all written in NodeJs (runtime: Node.Js 8.10, and aws-sdk should be v2.488.0 - but I'm just pasting that from the documentation). The short story is that L1 is supposed to call L2, and when it does L2 is executed twice! I discovered this by writing logs to CloudWatch and I could see one L1 log and two L2 logs.
Here's a simplified version of L1 and L2.
L1:
const AWS = require('aws-sdk');
const lambda = new AWS.Lambda();
module.exports = {
handler: async (event, context, callback) => {
const payload: { rnd: Math.random() };
const lambdaParams = {
FunctionName: 'L2',
Qualifier: `dev`,
Payload: JSON.stringify(payload),
};
console.log(`L1 calling: ${JSON.stringify(payload)}`);
return await lambda.invoke(lambdaParams).promise();
},
};
L2:
module.exports = {
handler: async (event, context, callback) => {
console.log(`L2 called: ${JSON.stringify(event)}`);
},
};
In CloudWatch I can see one L1 calling {"rnd": 0.012072353149807702} and two L2 called: {"rnd": 0.012072353149807702}!
BTW, this does not happen all the time. This is part of a step function process which was going to call L1 10k times. My code is written in a way that if L2 is executed twice (per one call), it will fail the whole process (because L2 inserts a record to DB only if it does not exist and fails if it does). So far, I managed to log this behaviour three times. All of them processing the same 10k items, facing the issue at a different iteration each time.
Does anyone have the same experience? Or even better, knows how to make sure one call leads to exactly one execution?
Your lambda function must be idempotent, because it can be called twice in different situations.
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-idempotent/
https://cloudonaut.io/your-lambda-function-might-execute-twice-deal-with-it/
With 10K lambda invokes it must be experiencing a failure and doing a retry.
From the documentation:
Asynchronous Invocation – Lambda retries function errors twice. If the
function doesn't have enough capacity to handle all incoming requests,
events may wait in the queue for hours or days to be sent to the
function. You can configure a dead-letter queue on the function to
capture events that were not successfully processed. For more
information, see Asynchronous Invocation.
If this is what is a happening and you setup the dead letter queue you'll be able to isolate the failure event.
You can also use CloudWatch Logs Insights to easily and quickly search for errors messages of the lambda. Once you select the log group this query should help you get started. Just change the time window.
fields #timestamp, #message
| filter #message like /(?i)(Exception|error|fail|5\d\d)/
| sort #timestamp desc
| limit 20
One case that may cause this is that in your L2 lambda you didn't return anything, which will lead the L1 lambda (the caller) to think there is an error with L2 and so the Retry mechanism is triggered. Try to return something in L2, even simply an "OK".
In my case, it happened that when calling my second lambda, there was a try that was catching an exception with traceback, this triggered the lambda to retry the call, normally without traceback this does not happen, but when commenting the module it stopped happening.
Also within the try it had a condition that yes or yes it could fail, since it had to query for a resource with boto3, so if it existed there was no problem, but since there was no traceback it forced the general failure, not capturing it as an exception.
I need to call my lambda - A from lambda - B. I have done the required code but need few clarifications.
AWSLambda lambdaClient = AWSLambdaClientBuilder.standard().withRegion(region)
.withCredentials(new DefaultAWSCredentialsProviderChain()).build();
InvokeRequest request = new InvokeRequest().withClientContext(clientContext).withFunctionName(functionName)
.withQualifier(alias).withPayload(payload).withInvocationType(InvocationType.Event);
InvokeResult response = lambdaClient.invoke(request);
I have a N number of table names which I need to pass from lambda B to lambda A one by one so that it can do the needful work on that DDB table.
The problem is withPayload takes a JSON payload which is passed to the lambda function.
I will pass the payload to lambda-B but then the code is calling lambdaClient.invoke(request) which will have all the table names and it will call our lambda-A. But handler function in the lambda-A expects a single table name.
I am not sure how to do this.
Also do i need to run this in a loop so that every time it takes new payload value and then calls lambdaClient.invoke(request) or does it happen automatically?
You should create pyload with a single table name. In this case, you will have many payloads so you can have array of them and process them in a loop.
For Example:
InvokeRequest request = null;
InvokeResult response = null;
for(String payload : payloads){
request = new InvokeRequest().withClientContext(clientContext).withFunctionName(functionName)
.withQualifier(alias).withPayload(payload).withInvocationType(InvocationType.Event);
response = lambdaClient.invoke(request);
}
In which case you will get that payload? Lamda normally runs on some event and a lambda cannot run more than 15 minutes(For more than 15 minutes you have to contact the AWS Support Team).
I have a lambda function that sends http call to a API(Let's say 'A'). After getting response from 'A' Immediately return the stuff's to the caller i.e., (callback(null, success)) within 10secs. Then save the Data fetched from API 'A' to My External API(Let's Say 'B').
I tried like below but Lambda waits until event loop is empty(It is waiting for the response from second http call).
I doesn't want to set the eventLoopWaitEmpty to false since it freezes the eventloop and Execute next time when invoked.
request.get({url: endpointUrlA},
function (errorA, responseA, bodyA) {
callback(null, "success");
request.post({url: endpointUrlB,
body: bodyA,
json: true}, function(errorB, responseB, bodyB){
//Doesn't want to wait for this response
});
/* Also tried the callback(null, "success"); here too
});
Anybody have any thoughts on How can I implement this? Thanks!
PS - Btw I read the Previous similar questions doesn't seems to clear with those.
This seems like a good candidate for breaking up this lambda into two lambdas with some support code.
First lambda recieves request to 'A' and places a message onto SQS. It then returns to the caller the success status.
A separate process monitors the SQS queue and invokes a second Lambda on it when a message becomes available.
This has several benefits.
Firstly, you no longer have a long-running lambda waiting for a second system that may be down to return.
Secondly, you're doing things asynchronously in the background.
Take a look at this blog post for an overview of how this could work in practice.
I am trying to call a DynamoDB client method and get one item from the DynamoDB table. I am using AWS Lambda. However, I keep getting the message:
"Process exited before completing request."
I have increased the timeout just to make sure, but the processing time is less than the timeout. Any advice?
console.log('Loading event');
var AWS = require('aws-sdk');
var dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
exports.handler = function(event, context) {
dynamodb.listTables(function(err, data) {
});
var params = {
"TableName": "User",
"Key":
{"User Id" : {"S":event.objectId}
},
"AttributesToGet" : ["First Name","Last Name", "Latitude", "Longitude"],
"ConsistentRead" : true
}
dynamodb.getItem(params, function(response,result) {
response.on('data', function(chunk){
console.log(""+chunk);
console.log("test1")
context.done(result);
});
result.on('ready', function(data){
console.log("test2")
console.log("Error:" + data.error);
console.log("ConsumedCapacityUnits:" + data.ConsumedCapacityUnits);
context.done('Error',data);
// ...
});
});
};
Take a look at your memory consumption (included in last log line). I got the same message when I assigned too little memory to my lambda function.
The message "Process exited before completing request" means that the Javascript function exited before calling context.done (or context.succeed, etc.). Usually, this means that there is some error in your code.
I'm not a Javascript expert (at all) so there may be more elegant ways to find the error but my approach has been to put a bunch of console.log messages in my code, run it, and then look at the logs. I can usually zero in on the offending line and, if I look at it long enough, I can usually figure out my mistake.
I see you have some logging already. What are you seeing in the output?
I have used callback, instead of context.
More recent examples on aws website use callback instead of context.
To complete request, either of the below must be called:
callback(error); // This is used when there is an error
// or
callback(null, data); // This is used when there is a success
// 'data' will contain success result, like some JSON object
When lambda execution completes the request,
failing to call one of the above callbacks,
you will see below error:
"Process exited before completing request."
Error in your code. Remove the last }); and don't use context it is there for backward compatibility, use callbacks on node.js 4.3 and 6.1 runtime.
Maybe you are not following aws lamda standard of using the function
check this Golang code.
package main
import "github.com/aws/aws-lambda-go/lambda"
func main() {
lambda.Start(yourFunction)
}
func yourFunction(){
// do your stuff
}
Check your lamda memory usage, for me this error was occurred because of lambda was using 201 MB memory, which was greater than allowed 200 MB of memory for its execution.
First verify your code and if it is ok, increase memory allotment to this lambda from configuration > General Configuration > Edit > Increase memory