AWS Lambda timeout when another (long) Lambda is invoked - amazon-web-services

I am using Step Functions that link together multiple Lambdas. In one of these lambdas I invoke another lambda. Both lambdas have 15 min timeout. However, after some mins the primary lambda that invokes the secondary lambda times out (the secondary lambda happily continues its work).
Error
com.amazonaws.SdkClientException
Cause
{"errorMessage": "Unable to execute HTTP request: Read timed out",
"errorType": "com.amazonaws.SdkClientException",
"stackTrace": [
cause": {
"errorMessage": "Read timed out",
"errorType": "java.net.SocketTimeoutException",
"stackTrace": [
"java.net.SocketInputStream.socketRead0(Native Method)",
"java.net.SocketInputStream.socketRead(SocketInputStream.java:116)",
"java.net.SocketInputStream.read(SocketInputStream.java:171)",
"java.net.SocketInputStream.read(SocketInputStream.java:141)",
"sun.security.ssl.InputRecord.readFully(InputRecord.java:465)",
"sun.security.ssl.InputRecord.read(InputRecord.java:503)",
"sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)",
"sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)",
"sun.security.ssl.AppInputStream.read(AppInputStream.java:105)",
This is the code, as you can see I try to raise the timeout using 4 different commands.
// (2) Instantiate AWSLambdaClientBuilder to build the Lambda client
AWSLambdaClientBuilder builder =
AWSLambdaClientBuilder.standard().withRegion(region);
// (3) Build the client, which will ultimately invoke
the function
AWSLambda client = builder.build();
// (4) Create an InvokeRequest with required parameters
InvokeRequest req = new
InvokeRequest().withFunctionName(random_arn).withPayload(jsonString);
// (5) Invoke the function and capture response
int timeout = (15*60*1000);
req.setSdkClientExecutionTimeout(timeout);
req.setSdkRequestTimeout(timeout);
req.withSdkClientExecutionTimeout(timeout);
req.withSdkRequestTimeout(timeout);
InvokeResult result = client.invoke(req);
Any idea how to tackle this timeout?

There is more than one timeout you might be interested when working with AWS Lambda. The one you are referring to is the function execution timeout - the timeout people usually mean while talking about Lambda. However, there are also other timeouts you want to configure. Take a look at AWS CLI docs:
https://docs.aws.amazon.com/cli/latest/reference/index.html
Here you have --cli-read-timeout and --cli-connect-timeout. If I'd create a Lambda function with say 5 mins function execution timeout, and invoke it using AWS CLI using RequestResponse (synchronous) invocation type, if it happens it takes more than 1 min to execute (default value for --cli-read-timeout), AWS Lambda will fail this execution and trigger automatic re-try, up to a max number of retries. From what you are describing, I believe you are facing that kind of trouble. Set all the timeouts and try again.
A few more links you may find helpful:
AWS Lambda using Java - setting the configuration connection/socket timeouts
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html#setSocketTimeout-int-

As you don't explicitly specify the invocation type, you invoke your second AWS Lambda function from your first AWS Lambda synchronously. So while your second AWS Lambda function is running, the first one will keep running as well, waiting for the response of the second function. That's why your first AWS Lambda function will time out if the sum of the run time of the first and the second AWS Lambda function is higher than the configured maximum timeout of the first AWS Lambda function (15 minutes in your case).
Invoking an AWS Lambda function synchronously from another AWS Lambda function is bad practice as you might run in timeout issues as you experience right now and spend money for running the first AWS Lambda function while the second AWS Lambda function is running.
What you could do is to invoke the second AWS Lambda function asynchronously, by setting the proper invocation type for your InvokeRequest:
InvokeRequest req = new InvokeRequest().withFunctionName(random_arn)
.withPayload(jsonString)
.withInvocationType(InvocationType.Event);
That of course only works if you don't depend on the output of the second AWS Lambda function to finish your first function.
The probably better option, as you're already using AWS Step functions, is to use a Step Functions state machine to orchestrate the invoking of the second AWS Lambda function, as described in the following tutorial: https://aws.amazon.com/de/getting-started/tutorials/create-a-serverless-workflow-step-functions-lambda/
Depending on how your AWS Lambda functions it might make sense to split the first AWS Lambda function, which is currently invoking the second AWS Lambda function into two parts running before and after the invoked Lambda function.

Related

can one aws lambda function handle multiple triggers at once?

I have an aws lambda function. When it receives only one trigger, it always succeds. But when it receives more than one trigger, it sometimes throws error. The first trigger always succeds.
Can I configure one aws lambda function receives only one trigger?
can one aws lambda function handle multiple triggers at once?
Yes, Lambda functions can handle multiple triggers at once.
when it receives more than one trigger, it sometimes throws error
This is most probably related to your implementation. Are you doing something different based on the inputs? Is the code behaving differently based on time?
Can I configure one aws lambda function receives only one trigger?
You can limit the concurrency of the Lambda function. If you set it to 1, you can only have one Lambda function running at any given time.
See: Set Concurrency Limits on Individual AWS Lambda Functions

AWS Lambda Functions: Will Different Triggers Reuse an Exection Enviornment?

Here's what I know, or think I know.
In AWS Lambda, the first time you call a function is commonly called a "cold start" -- this is akin to starting up your program for the first time.
If you make a second function invocation relatively quickly after your first, this cold start won't happen again. This is colloquially known as a "warm start"
If a function is idle for long enough, the execution environment goes away, and the next request will need to cold start again.
It's also possible to have a single AWS Lambda function with multiple triggers. Here's an example of a single function that's handling both API Gateway requests and SQS messages.
My question: Will AWS Lambda reuse (warm start) an execution environment when different event triggers come in? Or will each event trigger have it's own cold start? Or is this behavior that's not guaranteed by Lambda?
Yes, different triggers will use the same containers since the execution environment is the same for different triggers, the only difference is the event that is passed to your Lambda.
You can verify this by executing your Lambda with two types of triggers (i.e. API Gateway and simply the Test function on the Lambda Console) and looking at the CloudWatch logs. Each Lambda container creates its own Log Stream inside of your Lambda's Log Group. You should see both event logs going to the same Log Stream which means the 2nd event is successfully using the warm container created by the first event.

Invoking AWS Step Function from API Gateway and returning response?

From my understanding, API Gateway can trigger the Step Function. To kick off our state machine execution we need to send a post request as input.
The issue I am coming across is that API gateway will not wait for the execution to finish so it is no way of allowing the step function to respond back to the request:
Below is an image to outline this.
If this is in fact true, is there a way to call a lambda from within my lambda?
That is true, any invocation of Step Functions will immediately return a success response with an execution ARN that can be used to track the execution status.
If this is in fact true, is there a way to call a lambda from within
my lambda?
Yes absolutely, via the AWS SDK for the programming language you are using you can call the Lambda invoke() method. Although if you have one Lambda waiting on another Lambda to finish executing, be aware that you are going to be doubling the AWS Lambda charges you incur.

How many times does AWS Lambda fuctions retry to be invoked asynchronously?

I'm learning AWS Lambda, a bit confused about this one, in official FaQ it says:
Q: What happens if my Lambda function fails during processing an event?
On failure, Lambda functions being invoked synchronously will respond with an exception. Lambda functions being invoked asynchronously are retried at least 3 times. Events from Amazon Kinesis streams and Amazon DynamoDB streams are retried until the Lambda function succeeds or the data expires. Kinesis and DynamoDB Streams retain data for a minimum of 24 hours.
However, in the Lambda console, it can only be selected as 0, 1, 2 (shown in the figure below)
I think 3 times you mention contains the first try.
If you test, you can find lambda retry twice by default on asynchronous-invocation(for me, s3 event notification).
In addition, other the two references are saying like the below
reference1
Lambda attempts to run it two more times by default.
reference2
Asynchronous invocation – Lambda retries function errors twice.

How to run one lambda function after another has finished

I have an aws lambda function which is running daily at same time as cron job and is generating cloudwatch logs. I have another lambda function that takes those cloudwatch logs and move it to S3. So I want that when my first lambda function finishes execution, the logs lambda function starts and pushes the logs to S3 bucket. Kindly suggest how I can achieve this.
You can invoke a Lambda function from another Lambda function through the AWS SDK. So your first function should call the second function when it is finished. Make sure to select the InvocationType "Event" when invoking the second function and do not add any callbacks to avoid having the functions run in parallel and paying twice.