How can we invoke multiple AWS Lambda functions one after the other ?For example if an AWS Lambda chain consists of 8 separate lambda functions and each simulate a 1 sec processing event and then invoke the next function in the chain.
I wouldn't recommend using direct invoke to launch your functions. Instead you should consider creating an SNS Topic and subscribing your Lambda functions to this topic. Once a message is published to your topic, all functions will fire at the same time. This solution is also easily scalable.
See more information at official documentation Invoking Lambda functions using Amazon SNS notifications
With python:
from boto3 import client as botoClient
import json
lambdas = botoClient("lambda")
def lambda_handler(event, context):
response1 = lambdas.invoke(FunctionName="myLambda1", InvocationType="RequestResponse", Payload=json.dumps(event));
response2 = lambdas.invoke(FunctionName="myLambda2", InvocationType="RequestResponse", Payload=json.dumps(event));
A simple way to do it is to use the AWS sdk to invoke the lambda function.
The solution would look different depending on what sdk you use. If using the Node sdk I would suggest promisifying the sdk with a Promise library like for example Bluebird.
The code would look something like:
const Promise = require('bluebird');
const AWS = require('aws-sdk');
const lambda = Promise.promisifyAll(new AWS.Lambda({ apiVersion: '2015-03-31' }));
lambda.invokeAsync({FunctionName: 'FirstLambdaFunction'})
.then(() => {
// handle successful response from first lambda
return lambda.invokeAsync({FunctionName: 'SecondLambdaFunction'});
})
.then(() => lambda.invokeAsync({FunctionName: 'ThirdLambdaFunction'}))
.catch(err => {
// Handle error response
);
The reason why I like this approach is that you own the context of all the lambdas and can decide to do whatever you like with the different responses.
Just call the next Lambda function at the end of each function?
Use http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda_20141111.html#invokeAsync-property if you are using Node.js/JavaScript.
Related
I am new to AWS and cloud technology in general. So, please bear with me if the use case below is a trivial one.
Well, I have a table in Amazon DynamoDB which I am exporting to Amazon S3 using exportTableToPointInTime API (ExportsToS3) on a scheduled basis everyday at 6 AM. It is being done using an AWS Lambda function in this way -
const AWS = require("aws-sdk");
exports.handler = async (event) => {
const dynamodb = new AWS.DynamoDB({ apiVersion: '2012-08-10' });
const tableParams = {
S3Bucket: '<s3-bucket-name>',
TableArn: '<DynamoDB-Table-ARN>',
ExportFormat: 'DYNAMODB_JSON'
};
await dynamodb.exportTableToPointInTime(tableParams).promise();
};
The CFT template of the AWS Lambda function takes care of creating lambda roles and policies, etc. along with scheduling using Cloudwatch events. This setup works and the table is exported to the target Amazon S3 bucket everyday at the scheduled time.
Now, the next thing I want is that after the export to Amazon S3 is complete, I should be able to invoke an another lambda function and pass the export status to that lambda function which does some processing with it.
The problem I am facing is that the above lambda function finishes execution almost immediately with the exportTableToPointInTime call returning status as IN_PROGRESS.
I tried capturing the response of the above call like -
const exportResponse = await dynamodb.exportTableToPointInTime(tableParams).promise();
console.log(exportResponse);
Output of this is -
{
"ExportDescription": {
"ExportArn": "****",
"ExportStatus": "IN_PROGRESS",
"StartTime": "2021-09-20T16:51:52.147000+05:30",
"TableArn": "****",
"TableId": "****",
"ExportTime": "2021-09-20T16:51:52.147000+05:30",
"ClientToken": "****",
"S3Bucket": "****",
"S3SseAlgorithm": "AES256",
"ExportFormat": "DYNAMODB_JSON"
}
}
I am just obfuscating some values in the log with ****
As can be seen, the exportTableToPointInTime API call does not wait for the table to be exported completely. If it would have, it would have returned ExportStatus as either COMPLETED or FAILED.
Is there a way I can design the above use case to achieve my requirement - invoking an another lambda function only when the export is actually complete?
As of now, I have tried a brute force way to do it and which works but it definitely seems to be inefficient as it puts in a sleep there and also the lambda function is running for the entire duration of the export leading to cost impacts.
exports.handler = async (event) => {
const dynamodb = new AWS.DynamoDB({ apiVersion: '2012-08-10' });
const tableParams = {
S3Bucket: '<s3-bucket-name>',
TableArn: '<DynamoDB-Table-ARN>',
ExportFormat: 'DYNAMODB_JSON'
};
const exportResponse = await dynamodb.exportTableToPointInTime(tableParams).promise();
const exportArn = exportResponse.ExportDescription.ExportArn;
let exportStatus = exportResponse.ExportDescription.ExportStatus;
const sleep = (waitTimeInMs) => new Promise(resolve => setTimeout(resolve, waitTimeInMs));
do {
await sleep(60000); //waiting every 1 min and then calling listExports API
const listExports = await dynamodb.listExports().promise();
const filteredExports = listExports.ExportSummaries.filter(e => e.ExportArn == exportArn);
const currentExport = filteredExports[0];
exportStatus = currentExport.ExportStatus;
}
while (exportStatus == 'IN_PROGRESS');
var lambda = new AWS.Lambda();
var paramsForInvocation = {
FunctionName: 'another-lambda-function',
InvocationType: 'Event',
Payload: JSON.stringify({ 'ExportStatus': exportStatus })
};
await lambda.invoke(paramsForInvocation).promise();
};
What can be done to better it or the above solution is okay?
Thanks!!
One option to achieve this is to define a waiter in order to wait till a "Completed" status is returned from exportTableToPointInTime.
As far I can see there are a few default Waiters for DynamoDB already present, but there is not one for the export, so you'll need to write your own (you can use those already present as an example).
A good post describing how to use and write a waiter could be found here.
This way if the export takes less than 15 minutes you'll be able to catch it within the Lambda limits without the need of a secondary lambda.
If it takes longer than that, you'll need to decouple it, where you have multiple options as suggested by #Schepo and #wahmd:
using an S3 event on the other end
Using AWS EventBridge
Using SNS
combinations of the above.
Context: we want to export the DynamoDB table content into an S3 bucket and trigger a lambda when the export is complete.
In CloudTrail there's an ExportTableToPointInTime event that is sent when the export is started, but no event for when the export is finished.
A way to trigger a lambda once the export is completed is by creating an S3 trigger using this configuration:
In particular:
The creation event type is a complete multi-upload (others do not seem to work, not sure why).
I think the prefix can be omitted, but it's useful. It's composed of:
The first part is the table name, content.
The second part, AWSDynamoDB, is set automatically by the export tool.
This is the most important part. The last files created once the export is complete are manifest-summary.json and manifest-summary.md5. We must set the suffix as one of these files.
For an await call, you are missing "async" keyword on handler.
Change
exports.handler = (event) => {
to
exports.handler = async event => {
Since this is an await call, you need 'async' keyword with it.
Let me know if it fixed your issue.
Also, I suspect you don't need .promise() as it might be already returning promise. Anyways, please try with & without it incase it still doesn't work.
After dynamoDB await call, You can invoke another lambda. It would make sure that your lambda is invoked after dynamoDb export call is completed.
To invoke second lambda,
you can use aws sdk invoke package.
putEvent api using eventBridge.
Later option is better as it decouples both lambdas & also, first lambda does not have to wait until the seconds invocation is completed. (reduces lambda time, hence reduces cost)
I have a Lambda that requires messages to be sent to another Lambda to perform some action. In my particular case it is passing a message to a Lambda in order for it to perform HTTP requests and refresh cache entries.
Currently I am relying on the AWS SDK to send an SQS message. The mechanics of this are working fine. The concern that I have is that the SQS send method call takes around 50ms on average to complete. Considering I'm in a Lambda, I am unable to perform this in the background and expect for it to complete before the Lambda returns and is frozen.
This is further compounded if I need to make multiple SQS send calls, which is particularly bad as the Lambda is responsible for responding to low-latency HTTP requests.
Are there any alternatives in AWS for communicating between Lambdas that does not require a synchronous API call, and that exhibits more of a fire and forget and asynchronous behavior?
Though there are several approaches to trigger one lambda from another, (in my experience) one of the fastest methods would be to directly trigger the ultimate lambda's ARN.
Did you try invoking one Lambda from the other using AWS SDKs?
(for e.g. in Python using Boto3, I achieved it like this).
See below, the parameter InvocationType = 'Event' helps in invoking target Lambda asynchronously.
Below code takes 2 parameters (name, which can be either your target Lambda function's name or its ARN, params is a JSON object with input parameters you would want to pass as input). Try it out!
import boto3, json
def invoke_lambda(name, params):
lambda_client = boto3.client('lambda')
params_bytes = json.dumps(params).encode()
try:
response = lambda_client.invoke(FunctionName = name,
InvocationType = 'Event',
LogType = 'Tail',
Payload = params_bytes)
except ClientError as e:
print(e)
return None
return response
Hope it helps!
For more, refer to Lambda's Invoke Event on Boto3 docs.
Alternatively, you can use Lambda's Async Invoke as well.
It's difficult to give exact answers without knowing what language are you writing the Lambda function in. To at least make "warm" function invocations faster I would make sure you are creating the SQS client outside of the Lambda event handler so it can reuse the connection. The AWS SDK should use an HTTP connection pool so it doesn't have to re-establish a connection and go through the SSL handshake and all that every time you make an SQS request, as long as you reuse the SQS client.
If that's still not fast enough, I would have the Lambda function handling the HTTP request pass off the "background" work to another Lambda function, via an asynchronous call. Then the first Lambda function can return an HTTP response, while the second Lambda function continues to do work.
You might also try to use Lambda Destinations depending on you use case. With this you don't need to put things in a queue manually.
https://aws.amazon.com/blogs/compute/introducing-aws-lambda-destinations/
But it limits your flexibility. From my point of view chaining lambdas directly is an antipattern and if you would need that, go for step functions
My use case is such that I'll have an AWS Lambda front ended with API Gateway.
My requirement is that once the Lambda is invoked it should return a 200 OK response back to API Gateway which get forwards this to the caller.
And then the Lambda should start its actual processing of the payload.
The reason for this is that the API Gateway caller service expects a response within 10 seconds else it times out. So I want to give the response before I start with the processing.
Is this possible?
With API Gateway's "Lambda Function" integration type, you can't do this with a single Lambda function -- that interface is specifically designed to be synchronous. The workaround, if you want to use the Lambda Function integration type is for the synchronous Lambda function, invoked by the gateway, to invoke a second, asynchronous, Lambda function through the Lambda API.
However, asynchronous invocations are possible without the workaround, using an AWS Service Proxy integration instead of a Lambda Function integration.
If your API makes only synchronous calls to Lambda functions in the back end, you should use the Lambda Function integration type. [...]
If your API makes asynchronous calls to Lambda functions, you must use the AWS Service Proxy integration type described in this section. The instructions apply to requests for synchronous Lambda function invocations as well. For the asynchronous invocation, you must explicitly add the X-Amz-Invocation-Type:Event header to the integration request.
http://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-lambda.html
Yes, simply create two Lambda functions. The first Lambda function will be called by the API Gateway and will simply invoke the second Lambda function and then immediately return successfully so that the API Gateway can respond with an HTTP 200 to the client. The second Lambda function will then take as long as long as it needs to complete.
If anyone is interested, here is the code you can use to do the two lambdas approach. The code below is the first lambda that you should setup which would then call the second, longer running, lambda. It takes well under a second to execute.
const Lambda = new (require('aws-sdk')).Lambda();
/**
* Note: Step Functions, which are called out in many answers online, do NOT actually work in this case. The reason
* being that if you use Sequential or even Parallel steps they both require everything to complete before a response
* is sent. That means that this one will execute quickly but Step Functions will still wait on the other one to
* complete, thus defeating the purpose.
*
* #param {Object} event The Event from Lambda
*/
exports.handler = async (event) => {
let params = {
FunctionName: "<YOUR FUNCTION NAME OR ARN>",
InvocationType: "Event", // <--- This is KEY as it tells Lambda to start execution but immediately return / not wait.
Payload: JSON.stringify( event )
};
// we have to wait for it to at least be submitted. Otherwise Lambda runs too fast and will return before
// the Lambda can be submitted to the backend queue for execution
await new Promise((resolve, reject) => {
Lambda.invoke(params, function(err, data) {
if (err) {
reject(err, err.stack);
}
else {
resolve('Lambda invoked: '+data) ;
}
});
});
// Always return 200 not matter what
return {
statusCode : 200,
body: "Event Handled"
};
};
Check the answer here on how to set up an Async Invoke to the Lambda function. This will return 200 immediately to the client, but the Lambda will process on it's own asynchronously.
https://stackoverflow.com/a/40982649/5679071
Does anyone know the current and correct way to invoke Amazon AWS Lambda functions asynchronously instead of synchronously?
The InvokeAsync API in the AWS Java SDK is still available but marked as deprecated and they suggest you use use the Invoke API. I can't figure out why they would be forcing us to using sync. I have a web frontend that dispatches some batch jobs. I can't expect the frontend to keep a connection open for several minutes while it waits for the response (which is actually e-mailed to them after about 4-5 minutes of processing).
Ideally I'm trying to figure out how to do this with their API Endpoints rather than the Java SDK because the environment (GAE) that I'm running my backend in doesn't support AWS's use of HttpClient.
I'm looking at the latest API docs here, and it looks like only AWSLambdaAsyncClient.invokeAsyncAsync() is deprecated. The AWSLambdaAsyncClient.invokeAsync() method is not marked as deprecated. It looks like they are just doing some code cleanup by removing the need for the InvokeAsyncRequest and InvokeAsyncResult classes and the extra invokeAsyncAsync() methods.
You should be able to use the AWSLambdaAsyncClient.invokeAsync() method which uses InvokeRequest and returns InvokeResult. You might have to set the InvocationType on the InvokeRequest to InvocationType.Event. It's not clear if that's needed if you are using the Async client.
Regarding your second question about calling Lambda functions asynchronously without using the SDK, I would look into using API Gateway as a service proxy. This is the recommended way to expose Lambda functions for asynchronous calls.
The below code can be used to invoke the Lambda asynchronously from another Lambda
AWSLambdaAsyncClient client = new AWSLambdaAsyncClient();
client.withRegion(Regions.fromName(region));
InvokeRequest request = new InvokeRequest();
request.setInvocationType("Event");
request.withFunctionName(functionName).withPayload(payload);
InvokeResult invoke = client.invoke(request);
Approach given in the accepted answer is now deprecated. Answer given by the user #dassum is the approach to be followed, but the answer lacks a bit of explanation.
When creating the InvokeRequest, set the InvocationType as "Event" for asynchronous invocation and "RequestResponse" for synchronous invocation.
AWSLambda lambda = //create your lambda client here
lambda.invoke(new InvokeRequest()
.withFunctionName(LAMBDA_FUNCTION_NAME)
.withInvocationType(InvocationType.Event) //asynchronous
.withPayload(payload))
Reference to docs:
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/lambda/AWSLambda.html#invoke-com.amazonaws.services.lambda.model.InvokeRequest-
You can try the following:
YourCustomRequestBean request = new YourCustomRequestBean();
request.setData1(data1);
request.setData2(data2);
request.setData3(data3);
ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setSocketTimeout(xxxx);
clientConfiguration.setRequestTimeout(xxxx);
ObjectMapper objectMapper = new ObjectMapper();
String jsonStr = objectMapper.writeValueAsString(request);
AWSLambdaAsync awsLambdaAsync =
AWSLambdaAsyncClientBuilder.standard().withRegion(<mention your region here>).withClientConfiguration(clientConfiguration).build();
InvokeRequest invokeRequest = new InvokeRequest()
.withFunctionName(lambda function urn)
.withPayload(jsonStr);
awsLambdaAsync.invokeAsync(invokeRequest);
I am publishing a String message as message payload using SNS notification from Raspberry Pi using Python program and I want to pass that message payload to a Lambda function.
I have configured the requirement in the SNS console on AWS i.e., I have created a topic and added the lambda function to its subscribers.
Now, I want to get that message payload in the lambda function. But I can't find any method that can help me do that. For example, something like getMessage or something similar to that.
So my questions are: Since I have configured the publishing and subscription on AWS, can I assume that the clients are connected and if I publish a message I should be getting that at the subscriber's end which is my lambda function here?
Also, what's the technique in which I can get the message payload in my lambda function?
I am adding the below as per cjwfuller's suggestion.
Below I have written down the method for publishing in Python
client_boto = boto3.client('sns', aws_access_key_id='###',
aws_secret_access_key='###', region_name='us-west-2')
REGION = 'us-west-2'
TOPIC = 'arn:aws:sns:us-west-2:***:topic_name'
MSG = ntpath.basename(f_string)
SUBJECT_boto = 'File Name'
pub =client_boto.publish(TopicArn = TOPIC, Message = MSG,
Subject=SUBJECT_boto)
I am writing the subscribing code in Java.
Since my lambda func is already subscribed to it on AWS console,
should my Java program include the subscription again or is there a
way to get the msg payload directly.
Which language are you writing the function in? JavaScript as an example:
exports.myHandler = function(event, context, callback) {
console.log("value1 = " + event.key1);
console.log("value2 = " + event.key2);
// ...
}
Source: http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html
It's useful testing the sort of stuff in the AWS Lambda console before writing all the code.
Since I have configured the publishing and subscription on AWS, can I assume that the clients are connected
Clients aren't really "connected", they're simply subscribed to a topic.
publish a message I should be getting that at the subscriber's end which is my lambda function here?
Sounds like you're doing the right sort of thing - posting example code will help us come up with more precise answers.
On searching, I have found the class for SNSEvent which is, https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-events/src/main/java/com/amazonaws/services/lambda/runtime/events/SNSEvent.java
This Class contains all the methods related to and needed to get the message payload.
The Lambda function handler in Java goes something like this,
example;
public void handleRequest(SNSEvent input, Context context){
String this_takes_message=input.getRecords().get(0).getSNS().getMessage();
}