In our system we have a flow of Lambdas calling other Lambdas based on results they get.
This flow will probably be refactor to use the AWS step function in the future but will probably stay as it is for the next month or two.
Recently a new lambda has been introduced, and we're encountering a problem:
if we test our Lambda function 1 locally, it properly calls function 2 on AWS.
However, after deploying the lambda and running the same test, function doesn't get called (there's no invocation in the monitoring chart)
here's a code example of Function 1:
'use strict';
const AWS = require("aws-sdk");
const lambda = new AWS.Lambda({ region: "eu-west-1" });
const myHandler = async (event, _, callback) => {
console.log(event)
return doSomething()
.then(something)
.then(somethingElse)
.catch(handleError)
function doSomething() {
return fetch('someurl.com');
}
function something(r) {
if(r.status === 204) {
let lambdaRes = callLambda( "LambdaB", { foo: bar } );
console.log(lambdaRes)
}
return r
}
somethingElse(r) {
if(r.status === 500) {
throw new Error(`bar`);
}
return r.json()
}
async function callLambda(fnName, payload ) {
let params = {
FunctionName: fnName,
InvocationType: 'Event',
Payload: JSON.stringify(payload)
}
return await lambda.invoke(params);
}
function handleError(e) {
if(e.message === 'bar') {
let lambdaRes = callLambda( "LambdaC", { bar: 'foo' } );
console.log(lambdaRes)
}
return callback(null, { message: `Completed with error`});
}
exports.handler = myHandler;
We tried to check if the problem was caused by authorization and tested by providing ALL authorization possible to function 1 but with no results.
Any suggestion?
UPDATE:
Both Function use the same
VPC: vpc-hashnumberVpc,
Subnets: [ subnet-hashnumber1 , subnet-hashnumber2 , subnet-hashnumber3 ]
Security Group: sg-hashnumberSg
When the first Lambda function calls lambda.invoke() it is not somehow connecting to the other Lambda function inside the VPC. At that point there is no running instance of the other Lambda function, only the definition of it exists. What is happening is lambda.invoke() makes a call to the AWS API which is on the public Internet, requesting that AWS create a new invocation of the second function.
Lambda functions running inside a VPC do not get public IP addresses assigned to them, even if they are in a public subnet. The only way to give a Lambda function inside a VPC access to the Internet is to place the Lambda function in a private subnet with a route to a NAT Gateway.
Alternatively, you can configure an AWS Lambda VPC Endpoint, which gives resources inside your VPC access to call the AWS Lambda API without going over the Internet.
So in order for a Lambda function running inside a VPC to make AWS API calls to invoke other Lambda functions, your options are to add a NAT Gateway to the VPC, or add an AWS Lambda VPC Endpoint to the VPC.
So if function 1 is able to call function 2 locally, I think the problem is with networking/AWS setup, here are a few things you can do :
Set up Log/Print statements at every level, so you can pinpoint which line is returning an error.
Is the function timing out? If so check the timeout and Memory Allocated to the function, if the function is memory intensive it will not complete.
Last but very important , check your network/Subnets, if you have configured lambda to be running on multiple subnets, choose one of them and try, look at the VPC, perhaps the Lambda is not attached to any VPC or check if the attached vpc is able to connect to the other lambda, have a look at Security Groups as well.
Most of my lambda issues are resolved with this, all the best.
Related
I'm creating a Square Client object like this:
const squareClient = new Client({
environment: Environment.Sandbox,
accessToken:
"The_correct_sandbox_token_goes_here",
});
const paymentsApi = squareClient.paymentsApi;
and calling the createPayment method from within a lambda function with a body like the one below:
{
"sourceId": "cnon:CBASEHY1uZmmlYRYagaqS7yd9Zo",
"amountMoney": {
"amount": "12500",
"currency": "USD"
},
"locationId": "Location_ID_here",
"idempotencyKey": "6a36e49c-914d-4934-bc34-c183ba0a08c5"
}
This works fine on my local machine (using serverless offline), but when deployed to AWS, the call to createPayment times out after six seconds. Is there something extra that needs to be done to call createPayment from a lambda function?
The timeout you experienced might be due to the function being attached to a VPC.
If an AWS Lambda function is configured to use a VPC, then the function only has access to the Internet if the following have been configured:
Lambda function is connected to a private subnet
A NAT Gateway or NAT Instance is running in a public subnet in the same VPC
A Route Table on the private subnet directs Internet-bound traffic to the NAT
If the Lambda function does not require access to any resources in the VPC, do not attach the function to a VPC. This will automatically grant direct access to the Internet.
Alternatively, you might try increasing the timeout of the Lambda function in case the external service simply needed more time.
I am trying to invoke an AWS Lambda function from another one. But after adding VPC with Lambda I receive this error:
send request failed caused by: Post "https://lambda.ap-southeast-1.amazonaws.com/2015-03-31/functions/seconda_lambda/invocations"
Here is the function:
_invoke_copier_lambda(map[string]string{
"parameter1" : "param_val",
})
func _invoke_copier_lambda(params map[string]string) error {
// Marshal the map into a JSON string.
bytes__data, err := json.Marshal(params)
if err != nil {
return err
}
jsonStr := string(bytes__data)
svci := sLambda.New(session.New())
input := &sLambda.InvokeInput{
FunctionName: aws.String(config.CopierLambdaName),
Payload: []byte(jsonStr),
}
_, err = svci.Invoke(input)
return err
}
What have I missed?
The API endpoint for AWS Lambda lives on the Internet. Therefore, invoking a Lambda function requires access to the Internet.
It appears that your Lambda functions are in a public subnet. This means that they do not have access to the Internet. To be able to communicate with the Internet an AWS Lambda function must either:
Be configured to not use a VPC, OR
Be configured to connect to a private subnet in a VPC, and the VPC has an NAT Gateway or NAT Instance in a public subnet. Also, the Route Table for the private subnet(s) point to the NAT Gateway/NAT Instance.
Lambda Destinations
There is actually an alternative way for one Lambda function to trigger another Lambda function by using the Lambda Destinations feature. When the first Lambda function completes, it can trigger another Lambda function as a Destination. This does not require Internet access because it is handled by the AWS Lambda service itself (rather than calling an API). Please note that Destinations is only available if the first function is triggered asynchronously.
See:
Introducing AWS Lambda Destinations | AWS Compute Blog
Trek10 | Lambda Destinations: What We Learned the Hard Way
I am running a Node(12.x) Lambda in AWS. The purpose of this lambda is to interact with Cloudformation stacks, and I'm doing that via the aws-sdk. When testing this lambda locally using lambda-local, it executes successfully and the stack can be seen in CREATING state in AWS console.
However, when I push and run this lambda in AWS, it fails after 15 seconds, and I get this error:
{"errorType":"TimeoutError","errorMessage":"Socket timed out without establishing a connection","code":"TimeoutError","message":"Socket timed out without establishing a connection","time":"2020-06-29T03:10:27.668Z","region":"us-east-1","hostname":"cloudformation.us-east-1.amazonaws.com","retryable":true,"stack":["TimeoutError: Socket timed out without establishing a connection"," at Timeout.connectTimeout [as _onTimeout] (/var/task/node_modules/aws-sdk/lib/http/node.js:69:15)"," at listOnTimeout (internal/timers.js:549:17)"," at processTimers (internal/timers.js:492:7)"]}
This lead me to investigate the lambda timeout and the possible configuration changes I could make found in https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/ and https://aws.amazon.com/premiumsupport/knowledge-center/lambda-vpc-troubleshoot-timeout/ but nothing worked.
I found a couple of similar issues such as AWS Lambda: Task timed out which include possible suggestions such as lambda timeout and lambda memory issues, but Ive set mine to 30 seconds and the logs show max memory used is 88MB out of possible 128MB, but I tried with an increase anyway, and no luck.
The curious part is that it fails without establishing a connection to hostname cloudformation.us-east-1.amazonaws.com. How is that possible when the role assigned to the lambda has full Cloudformation privileges? I'm completely out of ideas so any help would be greatly appreciated. Heres my code:
TEST EVENT:
{
"stackName": "mySuccessfulStack",
"app": "test"
}
Function my handler calls (createStack):
const AWS = require('aws-sdk');
const templates = {
"test": {
TemplateURL: "https://<bucket>.s3.amazonaws.com/<path_to_file>/test.template",
Capabilities: ["CAPABILITY_IAM"],
Parameters: {
"HostingBucket": "test-hosting-bucket"
}
}
}
async function createStack(event) {
AWS.config.update({
maxRetries: 2,
httpOptions: {
timeout: 30000,
connectTimeout: 5000
}
});
const cloudformation = new AWS.CloudFormation();
const { app, stackName } = event;
let stackParams = templates[app];
stackParams['StackName'] = app + "-" + stackName;
let formattedTemplateParams = [];
for (let [key, value] of Object.entries(stackParams.Parameters)) {
formattedTemplateParams.push({"ParameterKey":key, "ParameterValue": value})
}
stackParams['Parameters'] = formattedTemplateParams;
const result = await cloudformation.createStack(stackParams).promise();
return result;
}
Lambda function in a VPC does not public IP address nor internet access. From docs:
Connect your function to private subnets to access private resources. If your function needs internet access, use NAT. Connecting a function to a public subnet does not give it internet access or a public IP address.
There are two common solutions for that:
place lambda function in a private subnet and setup NAT gateway in public subnet. Then set route table from private subnet to the NAT device. This will enable the lambda to access the internet and subsequently CloudFormation service.
setup a VPC interface endpoint for CloudFormation. This will allow your lambda function in private subnet to access CloudFormation without the internet.
Want to connect one AWS lambda which is in VPC to another lambda which is not in VPC but in same region and account.
Code of caller lambda "lambda_1":
#Override
public String handleRequest(final Object input, final Context context) {
logger.log(context.getFunctionName() + " invoked");
final AWSLambda client = AWSLambdaClientBuilder.standard().withRegion(Regions.EU_WEST_2).build();
final InvokeRequest request = new InvokeRequest();
request.withFunctionName("lambda_2").withPayload("JSON data as String").withInvocationType(InvocationType.RequestResponse);
logger.log("Lambda is about to invoke");
final InvokeResult response = client.invoke(request);
logger.log(context.getFunctionName() + " returned");
return input.toString();
}
AWS java sdk for lambda is used through maven dependency
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-lambda</artifactId>
<version>1.11.681</version>
</dependency>
Allow: lambda:InvokeFunction is given to lambda_1 for lambda_2
Code of callee lambda "lambda_2":
#Override
public String handleRequest(final Object input, final Context context) {
logger.log(context.getFunctionName() + " invoked");
logger.log(context.getFunctionName() + " returned");
return input.toString();
}
Logs get created for lambda_1 till Lambda is about to invoke line, execution doesn't go further and times up.
Enough time and memory has been set for lambda_1, lambda_2 as 2 min 198 MB and 1 min 128 MB respectively.
But Unable to call lambda_2 and even not getting any kind of error at runtime, kindly help thanks
Timeout with no further log output sounds like a typical networking-issue, I'd start looking there.
To reach the Lambda service, the calling Lambda (and with that your VPC) needs access to the Internet, have you got an Internet Gateway or NAT Interface/Gateway in the VPC?
Alternatively, instead of directly invoking the Lambda, you could go over SNS, that would allow you to add an SNS Interface endpoint in your VPC, if connecting your VPC to the Internet is not an option.
I am just attempting to call describeEndpoints of mediaconvert SDK but seems like it times out, why could that be. I already gave my Lambda function admin access. I set timeout to 30s which should be more than sufficient but it still fails
const AWS = require('aws-sdk');
const util = require('util');
async function test() {
let mediaconvert = new AWS.MediaConvert();
const describeEndpoints = util
.promisify(mediaconvert.describeEndpoints)
.bind(mediaconvert);
return await describeEndpoints()
}
Have you launched lambda in vpc ? if so, check if it has nat gateway , lambda doesn't work with subnet with route igw.
Use this instead of describing then:
const mediaConvert = new AWS.MediaConvert(
{ endpoint: 'MEDIACONVERT REGIONAL API ENDPOINT', });