I have an aws lambda function which is running daily at same time as cron job and is generating cloudwatch logs. I have another lambda function that takes those cloudwatch logs and move it to S3. So I want that when my first lambda function finishes execution, the logs lambda function starts and pushes the logs to S3 bucket. Kindly suggest how I can achieve this.
You can invoke a Lambda function from another Lambda function through the AWS SDK. So your first function should call the second function when it is finished. Make sure to select the InvocationType "Event" when invoking the second function and do not add any callbacks to avoid having the functions run in parallel and paying twice.
Related
Here's what I know, or think I know.
In AWS Lambda, the first time you call a function is commonly called a "cold start" -- this is akin to starting up your program for the first time.
If you make a second function invocation relatively quickly after your first, this cold start won't happen again. This is colloquially known as a "warm start"
If a function is idle for long enough, the execution environment goes away, and the next request will need to cold start again.
It's also possible to have a single AWS Lambda function with multiple triggers. Here's an example of a single function that's handling both API Gateway requests and SQS messages.
My question: Will AWS Lambda reuse (warm start) an execution environment when different event triggers come in? Or will each event trigger have it's own cold start? Or is this behavior that's not guaranteed by Lambda?
Yes, different triggers will use the same containers since the execution environment is the same for different triggers, the only difference is the event that is passed to your Lambda.
You can verify this by executing your Lambda with two types of triggers (i.e. API Gateway and simply the Test function on the Lambda Console) and looking at the CloudWatch logs. Each Lambda container creates its own Log Stream inside of your Lambda's Log Group. You should see both event logs going to the same Log Stream which means the 2nd event is successfully using the warm container created by the first event.
I have a Scheduled Lambda function (via CloudWatch event rule) which is triggered every minute.
This lambda picks up a request from SQS queue, process the parameters and triggers AWS step functions workflow.
Now, ONLY 1 Lambda function instance is running every minute. How can I trigger multiple (e.g. 10) concurrent Lambda functions like this?
One way I can think of is to create 10 Cloudwatch event rule which runs every 1 minute, but I am not sure if that is the right way of doing it. Also, if I use this way, 10 lambda would be called even if I don't have entries in my SQS queue.
You can use the lambda step function.
Event trigger first function. Then it will call multiple functions parallel.
Some useful links:
https://www.youtube.com/watch?v=c797gM0f_Pc
https://medium.com/soluto-nashville/simplifying-workflows-with-aws-step-functions-57d5fad41e59
since your lambda function fetching data from SQS so you can create event source mapping between lambda and SQS so whenever message published to SQS, your lambda function will invoke concurrently depending on number of messages in queue so you do not need to invoke lamnda from cloudwatch event
I have an API Gateway with Lambdas behind, for some of the endpoints I want to schedule an execution in the future, to run once, for example the REST call was made at T time, I want that lambda to schedule an execution ONCE at T+20min.
The only solution I found to achieve this is to use boto3 and Cloudwatch to setup a cron at the moment the REST call was made, send an event with the payload, then when the delayed lambda runs, it removes the rule.
I found this very heavy, is there any other way to achieve such pattern ?
Edit: It is NOT A RECURRING Lambda, just to run ONCE.
One option is to use AWS Step Functions to trigger the AWS Lambda function after a given delay.
Step Functions has a Wait state that can schedule or delay execution, so you can can implement a fairly simple Step Functions state machine that puts a delay in front of calling a Lambda function. No database required!
For an example of the concept (slightly different, but close enough), see:
Using AWS Step Functions To Schedule Or Delay SNS Message Publication - Alestic.com
Task Timer - AWS Step Functions
I am using Step Functions that link together multiple Lambdas. In one of these lambdas I invoke another lambda. Both lambdas have 15 min timeout. However, after some mins the primary lambda that invokes the secondary lambda times out (the secondary lambda happily continues its work).
Error
com.amazonaws.SdkClientException
Cause
{"errorMessage": "Unable to execute HTTP request: Read timed out",
"errorType": "com.amazonaws.SdkClientException",
"stackTrace": [
cause": {
"errorMessage": "Read timed out",
"errorType": "java.net.SocketTimeoutException",
"stackTrace": [
"java.net.SocketInputStream.socketRead0(Native Method)",
"java.net.SocketInputStream.socketRead(SocketInputStream.java:116)",
"java.net.SocketInputStream.read(SocketInputStream.java:171)",
"java.net.SocketInputStream.read(SocketInputStream.java:141)",
"sun.security.ssl.InputRecord.readFully(InputRecord.java:465)",
"sun.security.ssl.InputRecord.read(InputRecord.java:503)",
"sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)",
"sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)",
"sun.security.ssl.AppInputStream.read(AppInputStream.java:105)",
This is the code, as you can see I try to raise the timeout using 4 different commands.
// (2) Instantiate AWSLambdaClientBuilder to build the Lambda client
AWSLambdaClientBuilder builder =
AWSLambdaClientBuilder.standard().withRegion(region);
// (3) Build the client, which will ultimately invoke
the function
AWSLambda client = builder.build();
// (4) Create an InvokeRequest with required parameters
InvokeRequest req = new
InvokeRequest().withFunctionName(random_arn).withPayload(jsonString);
// (5) Invoke the function and capture response
int timeout = (15*60*1000);
req.setSdkClientExecutionTimeout(timeout);
req.setSdkRequestTimeout(timeout);
req.withSdkClientExecutionTimeout(timeout);
req.withSdkRequestTimeout(timeout);
InvokeResult result = client.invoke(req);
Any idea how to tackle this timeout?
There is more than one timeout you might be interested when working with AWS Lambda. The one you are referring to is the function execution timeout - the timeout people usually mean while talking about Lambda. However, there are also other timeouts you want to configure. Take a look at AWS CLI docs:
https://docs.aws.amazon.com/cli/latest/reference/index.html
Here you have --cli-read-timeout and --cli-connect-timeout. If I'd create a Lambda function with say 5 mins function execution timeout, and invoke it using AWS CLI using RequestResponse (synchronous) invocation type, if it happens it takes more than 1 min to execute (default value for --cli-read-timeout), AWS Lambda will fail this execution and trigger automatic re-try, up to a max number of retries. From what you are describing, I believe you are facing that kind of trouble. Set all the timeouts and try again.
A few more links you may find helpful:
AWS Lambda using Java - setting the configuration connection/socket timeouts
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html#setSocketTimeout-int-
As you don't explicitly specify the invocation type, you invoke your second AWS Lambda function from your first AWS Lambda synchronously. So while your second AWS Lambda function is running, the first one will keep running as well, waiting for the response of the second function. That's why your first AWS Lambda function will time out if the sum of the run time of the first and the second AWS Lambda function is higher than the configured maximum timeout of the first AWS Lambda function (15 minutes in your case).
Invoking an AWS Lambda function synchronously from another AWS Lambda function is bad practice as you might run in timeout issues as you experience right now and spend money for running the first AWS Lambda function while the second AWS Lambda function is running.
What you could do is to invoke the second AWS Lambda function asynchronously, by setting the proper invocation type for your InvokeRequest:
InvokeRequest req = new InvokeRequest().withFunctionName(random_arn)
.withPayload(jsonString)
.withInvocationType(InvocationType.Event);
That of course only works if you don't depend on the output of the second AWS Lambda function to finish your first function.
The probably better option, as you're already using AWS Step functions, is to use a Step Functions state machine to orchestrate the invoking of the second AWS Lambda function, as described in the following tutorial: https://aws.amazon.com/de/getting-started/tutorials/create-a-serverless-workflow-step-functions-lambda/
Depending on how your AWS Lambda functions it might make sense to split the first AWS Lambda function, which is currently invoking the second AWS Lambda function into two parts running before and after the invoked Lambda function.
Currently I have a use case that a cloud watch rule will trigger a step function every 5 minutes. I want to have a logic to skip starting another execution if there is one execution already running in step function.
Any way to do that?
Instead of having your CloudWatch event rule trigger the Step Function directly, you could have it trigger a Lambda function. The Lambda function could check if there are any Step Function executions in the RUNNING state, via the ListExecutions API. If not, the Lambda function could start a new execution via the StartExecution API.