I premise I'm not a systemist nor network engineer.
I'm trying to invoke a lambda function from another lambda function in the same vpc. My network configuration is:
1 vpc
1 public subnet and 1 private subnet
2 route tables
1 internet gateway
1 security group
My lambda:
is attached to both subnets and to sec group;
connects to db and retrieves data;
invokes lambda function to send push notification.
But when the first one tries to invoke the second aws returns timeout exception.
My idea is that the first one "can't see" the second.
How can I solve the problem?
Thanks
This is actually going to be as a result of the first Lambda (which is configured to use a VPC) has no outbound internet connectivity.
Ensure that you have either a NAT Gateway or a NAT instance that are attached to the route table(s) of the associated subnets.
If your function needs internet access, use network address translation (NAT). Connecting a function to a public subnet doesn't give it internet access or a public IP address.
You will need to remove the attachment to the public subnet as a Lambda cannot be assigned a public IP, therefore it cannot use an Internet Gateway.
If the first Lambda function is invoked asynchronously, it can specify a Destination for sending an event at the completion of execution.
The destination can be another AWS Lambda function. This invocation is triggered by the AWS service and does not require Internet access from the first Lambda function (and is therefore cheaper than using a NAT Gateway).
See: Configuring destinations for asynchronous invocation
(I haven't tried it myself, but it should work!)
Related
I've followed the tutorial here to create a VPC with public and private subnets.
Then I set up an AWS lambda function inside the public subnet to test if it could connect to the outside internet.
Here's my lambda function written in python3
import requests
def lambda_handler(event, context):
r = requests.get('http://www.google.com')
print(r)
The function above failed to fetch the content of http://www.google.com when I set it inside the public subnet in a VPC.
Here's the error message:
"errorMessage": "HTTPConnectionPool(host='www.google.com', port=80):
Max retries exceeded with url: / (Caused by
NewConnectionError(': Failed to establish a new connection: [Errno 110]
Connection timed out',))", "errorType": "ConnectionError",
I don't understand why.
The route table of the public subnet looks like this:
The GET request to http://www.google.com should match igw-XXXXXXXXX target. Why can't the internet-gateway(igw) deliver the request to http://www.google.com and get back the website content?
This article says that I must set the lambda function inside the private subnet in order to have internet access.
If your Lambda function needs to access private VPC resources (for
example, an Amazon RDS DB instance or Amazon EC2 instance), you must
associate the function with a VPC. If your function also requires
internet access (for example, to reach a public AWS service endpoint),
your function must use a NAT gateway or instance.
But it doesn't explain why I can't set the lambda function inside the public subnet.
Lambda functions connected to a VPC public subnet cannot typically access the internet.
To access the internet from a public subnet you need a public IP or you need to route via a NAT that itself has a public IP. You also need an Internet Gateway (IGW). However:
Lambda functions do not, and cannot, have public IP addresses, and
the default route target in a VPC public subnet is the IGW, not a NAT
So, because the Lambda function only has a private IP and its traffic is routed to the IGW rather than to a NAT, all packets to the internet from the Lambda function will be dropped at the IGW.
Should I Configure my Lambda Function for VPC Access?
If your Lambda function does not need to reach private resources inside your VPC (e.g. an RDS database or Elasticsearch cluster) then do not configure the Lambda function to connect to the VPC.
If your Lambda function does need to reach private resources inside your VPC, then configure the Lambda function to connect to private subnets (and only private subnets).
NAT or Not?
If the Lambda function only needs access to resources in the VPC (e.g. an RDS database in a private subnet) then you don't need to route through NAT.
If the Lambda function only needs access to resources in the VPC and access to AWS services that are all available via private VPC Endpoint then you don't need to route through NAT. Use VPC Endpoints.
If your Lambda function needs to reach endpoints on the internet then ensure a default route from the Lambda function's private subnets to a NAT instance or NAT Gateway in a public subnet. And configure an IGW, if needed, without which internet access is not possible.
Be aware that NAT gateway charges per hour and per GB processed so it's worth understanding how to reduce data transfer costs for NAT gateway.
Best Practices
When configuring Lambda functions for VPC access, it is an HA best practice to configure multiple (private) subnets across different Availability Zones (AZs).
Intermittent Connectivity
Be sure that all the subnets you configure for your Lambda function are private subnets. It is a common mistake to configure, for example, 1 private subnet and 1 public subnet. This will result in your Lambda function working OK sometimes and failing at other times without any obvious cause.
For example, the Lambda function may succeed 5 times in a row, and then fail with a timeout (being unable to access some internet resource or AWS service). This happens because the first launch was in a private subnet, launches 2-5 reused the same Lambda function execution environment in the same private subnet (the so-called "warm start"), and then launch 6 was a "cold start" where the AWS Lambda service deployed the Lambda function in a public subnet where the Lambda function has no route to the internet.
You can make a lambda function access the public internet from within your VPC, you just need to make sure you really need it.
For accessing resources external to AWS such as Google API (like OP's example) you do need a Public IP. For other cases like RDS or S3 you don't need Public IP, you can use a VPC Endpoint, so communication between your Lambda and the desired AWS Service doesn't leave AWS network.
By default some AWS Services are indeed reached via public internet, but it doesn't have to be.
[EDIT]
Someone was concerned about scalability in the comments, but they missed this from AWS Docs:
"Multiple Lambda functions can share a network interface, if the functions share the same subnet and security group"
Also, you must have a Public IP for reaching Public Internet, whether you're using Lambda, EC2, ECS, even if you use a NAT Gateway it needs an Elastic Public IP if you want to reach the public internet through it.
Solution
To do that, you need to assign Elastic Public IPs to the Network Interfaces for each subnet linked to your lambda. First let's figure which subnets and security groups are linked to your lambda:
Next, go to EC2 Service, find the Public IPs menu under Network & Security. Allocate one IP for each subnet (in the example above there are two subnets).
Go to Network Interfaces menu, find the network interfaces attached to your lambda (same subnet and security group).
Associate the Public IPs in the actions menu for each one:
That's it, now your Lambda can reach out to public internet.
I have a lambda function that calls the parameter store to retrieve a credential. The code is as follows:
import boto3
ssm = boto3.client('ssm')
parameter = ssm.get_parameter(Name='credentials', WithDecryption=True)
print(parameter['Parameter']['Value'])
I have given AmazonSSMFullAccess to the lambda role. The lambda has a VPC which later I'll use it to connect to a Redshift database without public access. The inbound and outbound rules are as follows:
There is a post AWS Lambda cannot connect to Parameter Store which mentions that if the lambda requires VPC, then add a NAT gateway.
In the lambda subnet route table: , there seems to be already a route that goes to the internet?
But I am still getting lambda time-out errors :(
there seems to be already a route that goes to the internet?
Sadly, it does not. It seems you placed your lambda in a public subnet with route to internet gateway (IGW). However, you have to use private subnet with a route to NAT gateway. IGW and NAT are two different things. Have a look at this AWS guide how to make it work:
How do I give internet access to a Lambda function that's connected to an Amazon VPC?
Alternatively, you can setup VPC interface endpoint for Paramter store. Then you don't need internet access for your Lambda function.
I am looking to assign a static IP to my Lambda which is being invoked via the API gateway. This is required because, the downstream system that i invoke from this lambda accepts web requests only from a Whitelisted IP.
I am successful in achieving this via the VPC that i associate with my lambda. But VPC introduces a bad cold-start time which sometime ranges 12-16seconds. So i am looking for a way to prevent this cold start from the VPC, but at the same time assign a static IP to the lambda.
You will need to:
Create a VPC with an Internet Gateway, a public subnet and a private subnet
Attach the AWS Lambda function to the private subnet
Launch a NAT Gateway in the public subnet and update the Route Table of the private subnet to use the NAT Gateway
The NAT Gateway will use an Elastic IP address (which is a static IP address). All traffic from the Lambda function to the Internet will come from this IP address, which can be used in the whitelist.
You might think that this is a bit of overkill for simply attaching a static IP address, but multiple Lambda function can run in parallel and they could run in multiple Availability Zones. Sending all traffic through the NAT Gateway is the only way to ensure they all have the same IP address. (Or, to be more specific, one IP address per AZ in which the NAT Gateway is launched.)
You can't assign a public/static IP to any Lambda function.
Your only good option is to deploy into a VPC with an Internet Gateway and configure routing from the Lambda's subnet through a NAT which has an Elastic IP. Then your target host can whitelist the Elastic IP.
Also see:
Public IP address for AWS API Gateway & Lambda (no VPC) - Stack Overflow
AWS Lambda functions with a static IP – Matthew Leak – Medium
I agree with the answer by John for having static IP whitelisting part. However, it won't resolve your cold start problem because lambda,if ideal, actually takes a small time to start. So I would recommend you also create a Cloudwatch event to hit lambda periodically to resolve this or write a simple code(either in lambda or somewhere else) which sends an empty request periodically so that cold start problem is resolved. You can view the improvement in X-Ray. This is an overhead but one time process.
To eliminate cold starts, you can use a service like https://lambdawarmer.com to keep a desired number of lambda instances always warm.
It basically uses a bunch of servers to periodically and exactly concurrently hit an endpoint on your lambda to keep a certain number of lambdas always warm.
I've followed the tutorial here to create a VPC with public and private subnets.
Then I set up an AWS lambda function inside the public subnet to test if it could connect to the outside internet.
Here's my lambda function written in python3
import requests
def lambda_handler(event, context):
r = requests.get('http://www.google.com')
print(r)
The function above failed to fetch the content of http://www.google.com when I set it inside the public subnet in a VPC.
Here's the error message:
"errorMessage": "HTTPConnectionPool(host='www.google.com', port=80):
Max retries exceeded with url: / (Caused by
NewConnectionError(': Failed to establish a new connection: [Errno 110]
Connection timed out',))", "errorType": "ConnectionError",
I don't understand why.
The route table of the public subnet looks like this:
The GET request to http://www.google.com should match igw-XXXXXXXXX target. Why can't the internet-gateway(igw) deliver the request to http://www.google.com and get back the website content?
This article says that I must set the lambda function inside the private subnet in order to have internet access.
If your Lambda function needs to access private VPC resources (for
example, an Amazon RDS DB instance or Amazon EC2 instance), you must
associate the function with a VPC. If your function also requires
internet access (for example, to reach a public AWS service endpoint),
your function must use a NAT gateway or instance.
But it doesn't explain why I can't set the lambda function inside the public subnet.
Lambda functions connected to a VPC public subnet cannot typically access the internet.
To access the internet from a public subnet you need a public IP or you need to route via a NAT that itself has a public IP. You also need an Internet Gateway (IGW). However:
Lambda functions do not, and cannot, have public IP addresses, and
the default route target in a VPC public subnet is the IGW, not a NAT
So, because the Lambda function only has a private IP and its traffic is routed to the IGW rather than to a NAT, all packets to the internet from the Lambda function will be dropped at the IGW.
Should I Configure my Lambda Function for VPC Access?
If your Lambda function does not need to reach private resources inside your VPC (e.g. an RDS database or Elasticsearch cluster) then do not configure the Lambda function to connect to the VPC.
If your Lambda function does need to reach private resources inside your VPC, then configure the Lambda function to connect to private subnets (and only private subnets).
NAT or Not?
If the Lambda function only needs access to resources in the VPC (e.g. an RDS database in a private subnet) then you don't need to route through NAT.
If the Lambda function only needs access to resources in the VPC and access to AWS services that are all available via private VPC Endpoint then you don't need to route through NAT. Use VPC Endpoints.
If your Lambda function needs to reach endpoints on the internet then ensure a default route from the Lambda function's private subnets to a NAT instance or NAT Gateway in a public subnet. And configure an IGW, if needed, without which internet access is not possible.
Be aware that NAT gateway charges per hour and per GB processed so it's worth understanding how to reduce data transfer costs for NAT gateway.
Best Practices
When configuring Lambda functions for VPC access, it is an HA best practice to configure multiple (private) subnets across different Availability Zones (AZs).
Intermittent Connectivity
Be sure that all the subnets you configure for your Lambda function are private subnets. It is a common mistake to configure, for example, 1 private subnet and 1 public subnet. This will result in your Lambda function working OK sometimes and failing at other times without any obvious cause.
For example, the Lambda function may succeed 5 times in a row, and then fail with a timeout (being unable to access some internet resource or AWS service). This happens because the first launch was in a private subnet, launches 2-5 reused the same Lambda function execution environment in the same private subnet (the so-called "warm start"), and then launch 6 was a "cold start" where the AWS Lambda service deployed the Lambda function in a public subnet where the Lambda function has no route to the internet.
You can make a lambda function access the public internet from within your VPC. Solution A is the actual answer, Solution B is a more elegant alternative solution.
Solution A - Lambda in VPC + Public IP associated with ENI
For accessing resources external to AWS such as Google API (like OP's example) you do need a Public IP. For other cases like RDS or S3 you don't need a Public IP, you can use a VPC Endpoint, so communication between your Lambda and the desired AWS Service doesn't leave AWS network.
By default some AWS Services are indeed reached via public internet, but it doesn't have to be.
Now if you want an actual external resource (e.g. google), you need to assign Elastic Public IPs to the Network Interfaces for each subnet linked to your lambda. First let's figure which subnets and security groups are linked to your lambda:
Next, go to EC2 Service, find the Public IPs menu under Network & Security. Allocate one IP for each subnet (in the example above there are two subnets).
Go to Network Interfaces menu, find the network interfaces attached to your lambda (same subnet and security group).
Associate the Public IPs in the actions menu for each one:
That's it, now your Lambda can reach out to public internet.
[EDIT]
Someone was concerned about Solution's A scalability issues saying each lambda instance has a new network interface but they missed this from AWS Docs:
"Multiple Lambda functions can share a network interface, if the functions share the same subnet and security group"
So whatever scalability issues you may face has nothing to do with this solution and how Lambda uses ENI, you'd face the same issues using EC2, ECS, EKS, not just Lambda.
Solution B - Decompose into multiple Lambdas
Requiring access both to external resources and VPC resources would seem like too much responsibility for a single function. You may want to rethink your design and decompose your single lambda function into at least two lambda functions:
Lambda A goes to external resources (e.g. Google API), fetches whatever data you need, add to SQS. No need to attach to VPC, no need to manually associate Elastic Public IP to ENI.
Lambda B processes the message from SQS, stores results to a storage (db, s3, efs, another queue, etc). This one lives within your VPC, and don't need external access.
This way seems more scalable, more secure, each individual lambda is less complex and more maintainable, the architecture looks better overall.
Of course life is not always rainbows and butterflies, so Solution A is good and scalable enough, but improving the architecture is even better.
I have a Lambda function that I run every hour using a cloudwatch scheduled event.
When triggered the function makes an HTTP request to a service on the internet and then, having received the response, performs some SQL queries inside our VPC before making a couple more HTTP requests to the same service.
Because of the SQL database, the Lambda must run in our VPC. So in order to connect out to the internet, I've included our NAT gateway subnet among the Lambda's VPC subnets.
I've tried destroying and creating this Lambda function several times with the same result each time:
It runs as expected for a few/several hours. And then, after that, it never succeeds again because the attempt to make a TCP connection for the first HTTP request reliably fails. Only after I create a whole new Lambda function with the same code and configuration is it able to connect to the internet.
How can this be?
I've included our NAT gateway subnet among the Lambda's VPC subnets.
That is definitely a misconfiguration.
Lambda functions that need Internet access need to be on subnets whose default route points to a NAT Gateway -- none of which can also be a subnet that has an actual NAT Gateway on it. A NAT Gateway never provides NAT services to any instance on the same subnet as the gateway itself.