I am trying to invoke an AWS Lambda function from another one. But after adding VPC with Lambda I receive this error:
send request failed caused by: Post "https://lambda.ap-southeast-1.amazonaws.com/2015-03-31/functions/seconda_lambda/invocations"
Here is the function:
_invoke_copier_lambda(map[string]string{
"parameter1" : "param_val",
})
func _invoke_copier_lambda(params map[string]string) error {
// Marshal the map into a JSON string.
bytes__data, err := json.Marshal(params)
if err != nil {
return err
}
jsonStr := string(bytes__data)
svci := sLambda.New(session.New())
input := &sLambda.InvokeInput{
FunctionName: aws.String(config.CopierLambdaName),
Payload: []byte(jsonStr),
}
_, err = svci.Invoke(input)
return err
}
What have I missed?
The API endpoint for AWS Lambda lives on the Internet. Therefore, invoking a Lambda function requires access to the Internet.
It appears that your Lambda functions are in a public subnet. This means that they do not have access to the Internet. To be able to communicate with the Internet an AWS Lambda function must either:
Be configured to not use a VPC, OR
Be configured to connect to a private subnet in a VPC, and the VPC has an NAT Gateway or NAT Instance in a public subnet. Also, the Route Table for the private subnet(s) point to the NAT Gateway/NAT Instance.
Lambda Destinations
There is actually an alternative way for one Lambda function to trigger another Lambda function by using the Lambda Destinations feature. When the first Lambda function completes, it can trigger another Lambda function as a Destination. This does not require Internet access because it is handled by the AWS Lambda service itself (rather than calling an API). Please note that Destinations is only available if the first function is triggered asynchronously.
See:
Introducing AWS Lambda Destinations | AWS Compute Blog
Trek10 | Lambda Destinations: What We Learned the Hard Way
Related
I'm creating a Square Client object like this:
const squareClient = new Client({
environment: Environment.Sandbox,
accessToken:
"The_correct_sandbox_token_goes_here",
});
const paymentsApi = squareClient.paymentsApi;
and calling the createPayment method from within a lambda function with a body like the one below:
{
"sourceId": "cnon:CBASEHY1uZmmlYRYagaqS7yd9Zo",
"amountMoney": {
"amount": "12500",
"currency": "USD"
},
"locationId": "Location_ID_here",
"idempotencyKey": "6a36e49c-914d-4934-bc34-c183ba0a08c5"
}
This works fine on my local machine (using serverless offline), but when deployed to AWS, the call to createPayment times out after six seconds. Is there something extra that needs to be done to call createPayment from a lambda function?
The timeout you experienced might be due to the function being attached to a VPC.
If an AWS Lambda function is configured to use a VPC, then the function only has access to the Internet if the following have been configured:
Lambda function is connected to a private subnet
A NAT Gateway or NAT Instance is running in a public subnet in the same VPC
A Route Table on the private subnet directs Internet-bound traffic to the NAT
If the Lambda function does not require access to any resources in the VPC, do not attach the function to a VPC. This will automatically grant direct access to the Internet.
Alternatively, you might try increasing the timeout of the Lambda function in case the external service simply needed more time.
In our system we have a flow of Lambdas calling other Lambdas based on results they get.
This flow will probably be refactor to use the AWS step function in the future but will probably stay as it is for the next month or two.
Recently a new lambda has been introduced, and we're encountering a problem:
if we test our Lambda function 1 locally, it properly calls function 2 on AWS.
However, after deploying the lambda and running the same test, function doesn't get called (there's no invocation in the monitoring chart)
here's a code example of Function 1:
'use strict';
const AWS = require("aws-sdk");
const lambda = new AWS.Lambda({ region: "eu-west-1" });
const myHandler = async (event, _, callback) => {
console.log(event)
return doSomething()
.then(something)
.then(somethingElse)
.catch(handleError)
function doSomething() {
return fetch('someurl.com');
}
function something(r) {
if(r.status === 204) {
let lambdaRes = callLambda( "LambdaB", { foo: bar } );
console.log(lambdaRes)
}
return r
}
somethingElse(r) {
if(r.status === 500) {
throw new Error(`bar`);
}
return r.json()
}
async function callLambda(fnName, payload ) {
let params = {
FunctionName: fnName,
InvocationType: 'Event',
Payload: JSON.stringify(payload)
}
return await lambda.invoke(params);
}
function handleError(e) {
if(e.message === 'bar') {
let lambdaRes = callLambda( "LambdaC", { bar: 'foo' } );
console.log(lambdaRes)
}
return callback(null, { message: `Completed with error`});
}
exports.handler = myHandler;
We tried to check if the problem was caused by authorization and tested by providing ALL authorization possible to function 1 but with no results.
Any suggestion?
UPDATE:
Both Function use the same
VPC: vpc-hashnumberVpc,
Subnets: [ subnet-hashnumber1 , subnet-hashnumber2 , subnet-hashnumber3 ]
Security Group: sg-hashnumberSg
When the first Lambda function calls lambda.invoke() it is not somehow connecting to the other Lambda function inside the VPC. At that point there is no running instance of the other Lambda function, only the definition of it exists. What is happening is lambda.invoke() makes a call to the AWS API which is on the public Internet, requesting that AWS create a new invocation of the second function.
Lambda functions running inside a VPC do not get public IP addresses assigned to them, even if they are in a public subnet. The only way to give a Lambda function inside a VPC access to the Internet is to place the Lambda function in a private subnet with a route to a NAT Gateway.
Alternatively, you can configure an AWS Lambda VPC Endpoint, which gives resources inside your VPC access to call the AWS Lambda API without going over the Internet.
So in order for a Lambda function running inside a VPC to make AWS API calls to invoke other Lambda functions, your options are to add a NAT Gateway to the VPC, or add an AWS Lambda VPC Endpoint to the VPC.
So if function 1 is able to call function 2 locally, I think the problem is with networking/AWS setup, here are a few things you can do :
Set up Log/Print statements at every level, so you can pinpoint which line is returning an error.
Is the function timing out? If so check the timeout and Memory Allocated to the function, if the function is memory intensive it will not complete.
Last but very important , check your network/Subnets, if you have configured lambda to be running on multiple subnets, choose one of them and try, look at the VPC, perhaps the Lambda is not attached to any VPC or check if the attached vpc is able to connect to the other lambda, have a look at Security Groups as well.
Most of my lambda issues are resolved with this, all the best.
is it possible to invoke a lambda from another lambda through its vpc endpoint (not through its name)?
I can't invoke the lambda directly because the calling lambda is in a VPC and the lambda i want to call is not in the VPC.
It seems the AWS-SDK in the caller lambda needs an internet access to invoke another lambda(default behaviour) and in the VPC i don't have one (i can't add a NAT Gateway or a Nat Instance).
if the lambdas are in the same vpc i would use this code:
AmazonLambdaConfig lambdaConfig = new AmazonLambdaConfig() { RegionEndpoint = RegionEndpoint.EUWest1 };
AmazonLambdaClient lCli = new AmazonLambdaClient("****APIKEY******", "****SECRET******", lambdaConfig); //se presente la variabile di ambiente AWS_PROFILE prende si logga con quelle credenziali
Amazon.Lambda.Model.InvokeRequest lambdaRequest = new Amazon.Lambda.Model.InvokeRequest()
{
FunctionName = "notification-prd-notifyses",
InvocationType = InvocationType.RequestResponse,
Payload = myJobj.ToString()
};
Amazon.Lambda.Model.InvokeResponse response = lCli.InvokeAsync(lambdaRequest).Result;
so, the question is: can i invoke a lambda referring to its VPC endpoint i created (henche through its DNS name)?
i can't use/create an apigateway as well, or associate a vpc endpoint to it, so the question is related only to the direct invocation lambda to lambda. is it possible?
I am working on an task which involves Lambda function running inside VPC.
This function is supposed to push messages to SQS and lambda execution role has policies : AWSLambdaSQSQueueExecutionRole and AWSLambdaVPCAccessExecutionRole added.
Lambda functions :
# Create SQS client
sqs = boto3.client('sqs')
queue_url = 'https://sqs.ap-east-1a.amazonaws.com/073x08xx43xx37/xyz-queue'
# Send message to SQS queue
response = sqs.send_message(
QueueUrl=queue_url,
DelaySeconds=10,
MessageAttributes={
'Title': {
'DataType': 'String',
'StringValue': 'Tes1'
},
'Author': {
'DataType': 'String',
'StringValue': 'Test2'
},
'WeeksOn': {
'DataType': 'Number',
'StringValue': '1'
}
},
MessageBody=(
'Testing'
)
)
print(response['MessageId'])
On testing the execution result is as :
{
"errorMessage": "2020-07-24T12:12:15.924Z f8e794fc-59ba-43bd-8fee-57f417fa50c9 Task timed out after 3.00 seconds"
}
I increased the Timeout from Basic Settings to 5 seconds & 10
seconds as well. But the error kept coming.
If anyone has faced similar issue in past or is having an idea how to get this resolved, Please help me out.
Thanks you in advance.
When an AWS Lambda function is configured to use an Amazon VPC, it connects to a nominated subnet of the VPC. This allows the Lambda function to communicate with other resources inside the VPC. However, it cannot communicate with the Internet. This is a problem because the Amazon SQS public endpoint lives on the Internet and the function is timing-out because it is unable to reach the Internet.
Thus, you have 3 options:
Option 1: Do not connect to a VPC
If your Lambda function does not need to communicate with a resource in the VPC (such as the simple function you have provided above), simply do not connect it to the VPC. When a Lambda function is not connected to a VPC, it can communicate with the Internet and the Amazon SQS public endpoint.
Option 2: Use a VPC Endpoint
A VPC Endpoint provides a means of accessing an AWS service without going via the Internet. You would configure a VPC endpoint for Amazon SQS. Then, when the Lambda function wishes to connect with the SQS queue, it can access SQS via the endpoint rather than via the Internet. This is normally a good option if the Lambda function needs to communicate with other resources in the VPC.
Option 3: Use a NAT Gateway
If the Lambda function is configured to use a private subnet, it will be able to access the Internet if a NAT Gateway has been provisioned in a public subnet and the Route Table for the private subnet points to the NAT Gateway. This involves extra expense and is only worthwhile if there is an additional need for a NAT Gateway.
If you're using the boto3 python library in a lambda in a VPC, and it's failing to connect to an sqs queue through a vpc endpoint, you must set the endpoint_url when creating the sqs client. Issue 1900 describes the background behind this.
The solution looks like this (for an sqs vpc endpoint in us-east-1):
sqs_client = boto3.client('sqs',
endpoint_url='https://sqs.us-east-1.amazonaws.com')
Then call send_message or send_message_batch as normal.
You need to place your lambda inside your VPC then set up a VPC endpoint for SQS or NAT gateway, When you add your lambda function to a subnet, make sure you ONLY add it to the private subnets, otherwise nothing will work.
Reference
https://docs.aws.amazon.com/lambda/latest/dg/vpc.html
https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/
I am pretty convinced that you cannot call an SQS queue from within a VPC using Lambda using an SQS endpoint. I'd consider it a bug, but maybe the Lambda team did this for a reason. In any case, You will get a message timeout. I cooked up a simple test Lambda
import json
import boto3
import socket
def lambda_handler(event, context):
print('lambda-test SQS...')
sqsDomain='sqs.us-west-2.amazonaws.com'
addr1 = socket.gethostbyname(sqsDomain)
print('%s=%s' %(sqsDomain, addr1))
print('Creating sqs client...')
sqs = boto3.client('sqs')
print('Sending Test Message...')
response = sqs.send_message(
QueueUrl='https://sqs.us-west-2.amazonaws.com/1234567890/testq.fifo',
MessageBody='Test SQS Lambda!',
MessageGroupId='test')
print('SQS send response: %s' % response)
return {
'statusCode': 200,
'body': json.dumps(response)
}
I created a VPC, subnet, etc per - Configuring a Lambda function to access resources in a VPC. The EC2 instance in this example has no problem invoking SQS through the private endpoint from the CLI per this tutorial.
If I drop my simple Lambda above into the same VPC and subnet, with SQS publishing permissions etc. and invoke the test function it will properly resolve the IP address of the SQS endpoint within the subnet, but the call will timeout (making sure your Lambda timeout is more than 60 seconds to let boto fail). Enabling boto debug logging further confirms that the IP is resolved correctly and the HTTP request to SQS times out.
I didn't try this with a non-FIFO queue but as the HTTP call is failing on connection request this shouldn't matter. It's got to be a routing issue from the Lambda as the EC2 in the same subnet works.
I modified my simple Lambda and added an SNS endpoint and did the same test which worked. The issue issue appears to be specific to SQS best I can tell.
import json
import boto3
import socket
def testSqs():
print('lambda-test SQS...')
sqsDomain='sqs.us-west-2.amazonaws.com'
addr1 = socket.gethostbyname(sqsDomain)
print('%s=%s' %(sqsDomain, addr1))
print('Creating sqs client...')
sqs = boto3.client('sqs')
print('Sending Test Message...')
response = sqs.send_message(
QueueUrl='https://sqs.us-west-2.amazonaws.com/1234567890/testq.fifo',
MessageBody='Test SQS Lambda!',
MessageGroupId='test')
print('SQS send response: %s' % response)
return {
'statusCode': 200,
'body': json.dumps(response)
}
def testSns():
print('lambda-test SNS...')
print('Creating sns client...')
sns = boto3.client('sns')
print('Sending Test Message...')
response = sns.publish(
TopicArn='arn:aws:sns:us-west-2:1234567890:lambda-test',
Message='Test SQS Lambda!'
)
print('SNS send response: %s' % response)
return {
'statusCode': 200,
'body': json.dumps(response)
}
def lambda_handler(event, context):
#return testSqs()
return testSns()
I think your only options are NAT (per John above), bounce your calls off a local EC2 (NAT will be simpler, cheaper, and more reliable), or use a Lambda proxy outside the VPC. Which someone else suggested in a similar post. You could also subscribe an SQS queue to an SNS topic (I prototyped this and it works) and route it out that way too, but that just seems silly unless you absolutely have to have SQS for some obscure reason.
I switched to SNS. I was just hoping to get some more experience with SQS. Hopefully somebody can prove me wrong, but I call it a bug.
I am running a Node(12.x) Lambda in AWS. The purpose of this lambda is to interact with Cloudformation stacks, and I'm doing that via the aws-sdk. When testing this lambda locally using lambda-local, it executes successfully and the stack can be seen in CREATING state in AWS console.
However, when I push and run this lambda in AWS, it fails after 15 seconds, and I get this error:
{"errorType":"TimeoutError","errorMessage":"Socket timed out without establishing a connection","code":"TimeoutError","message":"Socket timed out without establishing a connection","time":"2020-06-29T03:10:27.668Z","region":"us-east-1","hostname":"cloudformation.us-east-1.amazonaws.com","retryable":true,"stack":["TimeoutError: Socket timed out without establishing a connection"," at Timeout.connectTimeout [as _onTimeout] (/var/task/node_modules/aws-sdk/lib/http/node.js:69:15)"," at listOnTimeout (internal/timers.js:549:17)"," at processTimers (internal/timers.js:492:7)"]}
This lead me to investigate the lambda timeout and the possible configuration changes I could make found in https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/ and https://aws.amazon.com/premiumsupport/knowledge-center/lambda-vpc-troubleshoot-timeout/ but nothing worked.
I found a couple of similar issues such as AWS Lambda: Task timed out which include possible suggestions such as lambda timeout and lambda memory issues, but Ive set mine to 30 seconds and the logs show max memory used is 88MB out of possible 128MB, but I tried with an increase anyway, and no luck.
The curious part is that it fails without establishing a connection to hostname cloudformation.us-east-1.amazonaws.com. How is that possible when the role assigned to the lambda has full Cloudformation privileges? I'm completely out of ideas so any help would be greatly appreciated. Heres my code:
TEST EVENT:
{
"stackName": "mySuccessfulStack",
"app": "test"
}
Function my handler calls (createStack):
const AWS = require('aws-sdk');
const templates = {
"test": {
TemplateURL: "https://<bucket>.s3.amazonaws.com/<path_to_file>/test.template",
Capabilities: ["CAPABILITY_IAM"],
Parameters: {
"HostingBucket": "test-hosting-bucket"
}
}
}
async function createStack(event) {
AWS.config.update({
maxRetries: 2,
httpOptions: {
timeout: 30000,
connectTimeout: 5000
}
});
const cloudformation = new AWS.CloudFormation();
const { app, stackName } = event;
let stackParams = templates[app];
stackParams['StackName'] = app + "-" + stackName;
let formattedTemplateParams = [];
for (let [key, value] of Object.entries(stackParams.Parameters)) {
formattedTemplateParams.push({"ParameterKey":key, "ParameterValue": value})
}
stackParams['Parameters'] = formattedTemplateParams;
const result = await cloudformation.createStack(stackParams).promise();
return result;
}
Lambda function in a VPC does not public IP address nor internet access. From docs:
Connect your function to private subnets to access private resources. If your function needs internet access, use NAT. Connecting a function to a public subnet does not give it internet access or a public IP address.
There are two common solutions for that:
place lambda function in a private subnet and setup NAT gateway in public subnet. Then set route table from private subnet to the NAT device. This will enable the lambda to access the internet and subsequently CloudFormation service.
setup a VPC interface endpoint for CloudFormation. This will allow your lambda function in private subnet to access CloudFormation without the internet.