I have a Lambda function that I run every hour using a cloudwatch scheduled event.
When triggered the function makes an HTTP request to a service on the internet and then, having received the response, performs some SQL queries inside our VPC before making a couple more HTTP requests to the same service.
Because of the SQL database, the Lambda must run in our VPC. So in order to connect out to the internet, I've included our NAT gateway subnet among the Lambda's VPC subnets.
I've tried destroying and creating this Lambda function several times with the same result each time:
It runs as expected for a few/several hours. And then, after that, it never succeeds again because the attempt to make a TCP connection for the first HTTP request reliably fails. Only after I create a whole new Lambda function with the same code and configuration is it able to connect to the internet.
How can this be?
I've included our NAT gateway subnet among the Lambda's VPC subnets.
That is definitely a misconfiguration.
Lambda functions that need Internet access need to be on subnets whose default route points to a NAT Gateway -- none of which can also be a subnet that has an actual NAT Gateway on it. A NAT Gateway never provides NAT services to any instance on the same subnet as the gateway itself.
Related
I have a Lambda function deployed into a public subnet in a VPC and I’m trying to connect to a Lambda function outside of a VPC and I’m running into connectivity issues.
I believe the security group settings and IAM policies will permit the connection, but I'm not sure if there's an issue with connecting to an out-of-VPC Lambda from an in-VPC one.
Is there a tool in AWS Console, AWS CLI or anywhere else that I can use to troubleshoot where the connection is failing? I’ve used the Reachability Analyzer before but it only works on a handful of resource types like EC2 instances.
I've tried invoking the out-of-VPC Lambda from inside my in-VPC Lambda, but the request doesn't work and I don't see any helpful information about what happened. I tried running the Reachability Analyzer, but it doesn't allow you to test if Lambda functions are reachable.
I was expecting the request to work, but I'm not sure if I need to configure a VPC interface endpoint because I'm connecting from an in-VPC Lambda to an out-of-VPC Lambda.
I’m new to networking and would appreciate any help.
I have a Lambda function deployed into a public subnet in a VPC and I’m trying to connect to a Lambda function outside of a VPC and I’m running into connectivity issues.
The Lambda function in the VPC never gets a public IP assigned to it. So it can't connect to anything outside of the VPC. It can't use the Internet Gateway attached to the public subnet because it doesn't have a public IP.
By "connect to a Lambda function outside of a VPC" what you are really doing is connecting to the AWS API outside of the VPC. You never "connect" to a Lambda function, because Lambda functions aren't running and just sitting around idle waiting for your request. Lambda functions don't really exist until a request comes in to the AWS Lambda Invoke API, at which point AWS spins up an instance of the Lambda function and passes it the invocation payload.
To fix this connectivity issue, you either need to create an AWS Lambda VPC Endpoint in your VPC, to handle requests to the Lambda API originating in your VPC. Or you need to move the VPC Lambda function to a private subnet, with a route to a NAT Gateway. Lambda functions in private subnets can access things outside the VPC by having their requests routed through the NAT Gateway.
I was expecting the request to work, but I'm not sure if I need to configure a VPC interface endpoint because I'm connecting from an in-VPC Lambda to an out-of-VPC Lambda.
That's not how VPC Interface Endpoints work. The entire purpose of VPC Interface Endpoints is to allow a resource inside your VPC to access part of the AWS API that exists outside the VPC. A VPC Interface Endpoint will absolutely allow your Lambda function running in the VPC to access the Lambda Invoke API, in order to trigger an execution of your out-of-VPC Lambda function.
I have a lambda function which has the following logic in the handler:
log.info("about to get caller identity..")
caller_identity = boto3.client("sts").get_caller_identity()
log.info(caller_identity)
When I run this lambda function, it times out with the following error:
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://sts.amazonaws.com/"
Why is my lambda function not able to reach STS service?
Thanks!
This was the result of the Lambda being associated to a VPC in a private subnet with no way to communicate to the internet.
It is important that when using VPC configuration the Lambda is located in a subnet with the means to communicate with the internet such as a NAT. Without this your Lambda cannot communicate to the internet.
If you're trying to reach an AWS service you can check whether a VPC Endpoint is supported for the service to remove the need for internet connectivity.
I premise I'm not a systemist nor network engineer.
I'm trying to invoke a lambda function from another lambda function in the same vpc. My network configuration is:
1 vpc
1 public subnet and 1 private subnet
2 route tables
1 internet gateway
1 security group
My lambda:
is attached to both subnets and to sec group;
connects to db and retrieves data;
invokes lambda function to send push notification.
But when the first one tries to invoke the second aws returns timeout exception.
My idea is that the first one "can't see" the second.
How can I solve the problem?
Thanks
This is actually going to be as a result of the first Lambda (which is configured to use a VPC) has no outbound internet connectivity.
Ensure that you have either a NAT Gateway or a NAT instance that are attached to the route table(s) of the associated subnets.
If your function needs internet access, use network address translation (NAT). Connecting a function to a public subnet doesn't give it internet access or a public IP address.
You will need to remove the attachment to the public subnet as a Lambda cannot be assigned a public IP, therefore it cannot use an Internet Gateway.
If the first Lambda function is invoked asynchronously, it can specify a Destination for sending an event at the completion of execution.
The destination can be another AWS Lambda function. This invocation is triggered by the AWS service and does not require Internet access from the first Lambda function (and is therefore cheaper than using a NAT Gateway).
See: Configuring destinations for asynchronous invocation
(I haven't tried it myself, but it should work!)
When a Lambda is attached to one (or more) VPC subnets, the post call to data exchange api times out. And when the Lambda is detached from all subnets, then this post call succeeds. This is happening consistently in golang Lambda environment.
In my use case, I am accessing Redis from Lambda, and Redis is accessible only from within the VPC.
Error message:
error=RequestError: send request failed
caused by: Post https://dataexchange.us-east-1.amazonaws.com/v1/data-sets: dial tcp 52.85.148.96:443: i/o timeout
An AWS Lambda function running in a VPC will never be assigned a public IP address. So in order for the Lambda function to access resources that exist outside the VPC, such as the AWS Data Exchange service, the VPC will need to be configured with a NAT Gateway that provides Internet access to the private subnet(s) the Lambda function is deployed to.
I am looking to assign a static IP to my Lambda which is being invoked via the API gateway. This is required because, the downstream system that i invoke from this lambda accepts web requests only from a Whitelisted IP.
I am successful in achieving this via the VPC that i associate with my lambda. But VPC introduces a bad cold-start time which sometime ranges 12-16seconds. So i am looking for a way to prevent this cold start from the VPC, but at the same time assign a static IP to the lambda.
You will need to:
Create a VPC with an Internet Gateway, a public subnet and a private subnet
Attach the AWS Lambda function to the private subnet
Launch a NAT Gateway in the public subnet and update the Route Table of the private subnet to use the NAT Gateway
The NAT Gateway will use an Elastic IP address (which is a static IP address). All traffic from the Lambda function to the Internet will come from this IP address, which can be used in the whitelist.
You might think that this is a bit of overkill for simply attaching a static IP address, but multiple Lambda function can run in parallel and they could run in multiple Availability Zones. Sending all traffic through the NAT Gateway is the only way to ensure they all have the same IP address. (Or, to be more specific, one IP address per AZ in which the NAT Gateway is launched.)
You can't assign a public/static IP to any Lambda function.
Your only good option is to deploy into a VPC with an Internet Gateway and configure routing from the Lambda's subnet through a NAT which has an Elastic IP. Then your target host can whitelist the Elastic IP.
Also see:
Public IP address for AWS API Gateway & Lambda (no VPC) - Stack Overflow
AWS Lambda functions with a static IP – Matthew Leak – Medium
I agree with the answer by John for having static IP whitelisting part. However, it won't resolve your cold start problem because lambda,if ideal, actually takes a small time to start. So I would recommend you also create a Cloudwatch event to hit lambda periodically to resolve this or write a simple code(either in lambda or somewhere else) which sends an empty request periodically so that cold start problem is resolved. You can view the improvement in X-Ray. This is an overhead but one time process.
To eliminate cold starts, you can use a service like https://lambdawarmer.com to keep a desired number of lambda instances always warm.
It basically uses a bunch of servers to periodically and exactly concurrently hit an endpoint on your lambda to keep a certain number of lambdas always warm.