I am connecting different AWS sources (RDS, Elasticache, EFS) in my lambda function. My AWS resources are configured in public subnets including my Lambda function. My Lambda also needs to access the internet but when we configure the lambda function in a public subnet typically it can't get access to the internet.
Now I am thinking to shift my lambda function to a private subnet so, it can access the internet using a NAT gateway. My only concern is, would our lambda function (in the private subnet) be able to access other AWS resources (RDS, Elastichche, EFS) that are in the public subnet?
Or there is any other good approach that suits my scenario?
One more thing, I can't move my (RDS, Elasticache, and EFS) to a private subnet.
would our lambda function (in the private subnet) be able to access other AWS resources (RDS, Elastichche, EFS) that are in the public subnet?
Yes, if they are in the save VPC. So that's how it is usually done. Lambda in a private vpc able to connect internet through NAT. Since other resources are in the save VPC (or peered), lambda will use private IP addresses to connect to them.
Related
I've followed the tutorial here to create a VPC with public and private subnets.
Then I set up an AWS lambda function inside the public subnet to test if it could connect to the outside internet.
Here's my lambda function written in python3
import requests
def lambda_handler(event, context):
r = requests.get('http://www.google.com')
print(r)
The function above failed to fetch the content of http://www.google.com when I set it inside the public subnet in a VPC.
Here's the error message:
"errorMessage": "HTTPConnectionPool(host='www.google.com', port=80):
Max retries exceeded with url: / (Caused by
NewConnectionError(': Failed to establish a new connection: [Errno 110]
Connection timed out',))", "errorType": "ConnectionError",
I don't understand why.
The route table of the public subnet looks like this:
The GET request to http://www.google.com should match igw-XXXXXXXXX target. Why can't the internet-gateway(igw) deliver the request to http://www.google.com and get back the website content?
This article says that I must set the lambda function inside the private subnet in order to have internet access.
If your Lambda function needs to access private VPC resources (for
example, an Amazon RDS DB instance or Amazon EC2 instance), you must
associate the function with a VPC. If your function also requires
internet access (for example, to reach a public AWS service endpoint),
your function must use a NAT gateway or instance.
But it doesn't explain why I can't set the lambda function inside the public subnet.
Lambda functions connected to a VPC public subnet cannot typically access the internet.
To access the internet from a public subnet you need a public IP or you need to route via a NAT that itself has a public IP. You also need an Internet Gateway (IGW). However:
Lambda functions do not, and cannot, have public IP addresses, and
the default route target in a VPC public subnet is the IGW, not a NAT
So, because the Lambda function only has a private IP and its traffic is routed to the IGW rather than to a NAT, all packets to the internet from the Lambda function will be dropped at the IGW.
Should I Configure my Lambda Function for VPC Access?
If your Lambda function does not need to reach private resources inside your VPC (e.g. an RDS database or Elasticsearch cluster) then do not configure the Lambda function to connect to the VPC.
If your Lambda function does need to reach private resources inside your VPC, then configure the Lambda function to connect to private subnets (and only private subnets).
NAT or Not?
If the Lambda function only needs access to resources in the VPC (e.g. an RDS database in a private subnet) then you don't need to route through NAT.
If the Lambda function only needs access to resources in the VPC and access to AWS services that are all available via private VPC Endpoint then you don't need to route through NAT. Use VPC Endpoints.
If your Lambda function needs to reach endpoints on the internet then ensure a default route from the Lambda function's private subnets to a NAT instance or NAT Gateway in a public subnet. And configure an IGW, if needed, without which internet access is not possible.
Be aware that NAT gateway charges per hour and per GB processed so it's worth understanding how to reduce data transfer costs for NAT gateway.
Best Practices
When configuring Lambda functions for VPC access, it is an HA best practice to configure multiple (private) subnets across different Availability Zones (AZs).
Intermittent Connectivity
Be sure that all the subnets you configure for your Lambda function are private subnets. It is a common mistake to configure, for example, 1 private subnet and 1 public subnet. This will result in your Lambda function working OK sometimes and failing at other times without any obvious cause.
For example, the Lambda function may succeed 5 times in a row, and then fail with a timeout (being unable to access some internet resource or AWS service). This happens because the first launch was in a private subnet, launches 2-5 reused the same Lambda function execution environment in the same private subnet (the so-called "warm start"), and then launch 6 was a "cold start" where the AWS Lambda service deployed the Lambda function in a public subnet where the Lambda function has no route to the internet.
You can make a lambda function access the public internet from within your VPC, you just need to make sure you really need it.
For accessing resources external to AWS such as Google API (like OP's example) you do need a Public IP. For other cases like RDS or S3 you don't need Public IP, you can use a VPC Endpoint, so communication between your Lambda and the desired AWS Service doesn't leave AWS network.
By default some AWS Services are indeed reached via public internet, but it doesn't have to be.
[EDIT]
Someone was concerned about scalability in the comments, but they missed this from AWS Docs:
"Multiple Lambda functions can share a network interface, if the functions share the same subnet and security group"
Also, you must have a Public IP for reaching Public Internet, whether you're using Lambda, EC2, ECS, even if you use a NAT Gateway it needs an Elastic Public IP if you want to reach the public internet through it.
Solution
To do that, you need to assign Elastic Public IPs to the Network Interfaces for each subnet linked to your lambda. First let's figure which subnets and security groups are linked to your lambda:
Next, go to EC2 Service, find the Public IPs menu under Network & Security. Allocate one IP for each subnet (in the example above there are two subnets).
Go to Network Interfaces menu, find the network interfaces attached to your lambda (same subnet and security group).
Associate the Public IPs in the actions menu for each one:
That's it, now your Lambda can reach out to public internet.
I have a VPC containing 2 Lambda functions A & B:
A is on 2 public subnets
B is on 2 private subnets with a RDS Database instance
The VPC itself has Internet access via the NAT instance.
I need a 3rd party API to communicate with B, but B is in a private subnet. Now I was wondering whether API Gateway solves this problem or whether it requires more work.
Thanks in advance
If you want the 3rd-party to invoke the Lambda function and pass data to it, then AWS API Gateway is correct for the task. That is exactly what API Gateway is designed to do.
See: Using AWS Lambda with Amazon API Gateway - AWS Lambda
Alternatively, you could use an Elastic Load Balancer and have it invoke the Lambda function.
From: Using AWS Lambda with an Application Load Balancer - AWS Lambda
You also mention that you have an AWS Lambda "on 2 public subnets". AWS Lambda functions should not be configured to connect to public subnets. They should either be configured to use "No VPC" (in which case they can directly access the Internet), or they should be connected to private subnets (and can use a NAT Gateway or NAT Instance to access the Internet if required).
See: Configuring a Lambda function to access resources in a VPC - AWS Lambda:
Connect your function to private subnets to access private resources. If your function needs internet access, use network address translation (NAT). Connecting a function to a public subnet doesn't give it internet access or a public IP address.
As the title suggests, I placed my Lambda function in a private subnet and now It cannot access or timeout when scanning the DB. Prior to this, it could access and scan the DB. What should I do?
Your DynamoDB resources are not in your VPC. Since you've configured your Lambda functions to connect to your VPC, you need to setup a NAT Gateway or NAT Instance to allow your private resources to access the internet. As the docs state:
AWS Lambda uses the VPC information you provide to set up ENIs that
allow your Lambda function to access VPC resources. Each ENI is
assigned a private IP address from the IP address range within the
Subnets you specify, but is not assigned any public IP addresses.
Therefore, if your Lambda function requires Internet access (for
example, to access AWS services that don't have VPC endpoints), you
can configure a NAT instance inside your VPC or you can use the Amazon
VPC NAT gateway. You cannot use an Internet gateway attached to your
VPC, since that requires the ENI to have public IP addresses.
AWS Lambda Doc
Validate the following:
The route table for the Lambda has a NAT Gateway for internet traffic that resides in public subnet.
DynamoDB Gateway endpoint exists? Check its policy to ensure that it is not limited to specific sources
Outbound access is allowed via the security group and NACL
I've followed the tutorial here to create a VPC with public and private subnets.
Then I set up an AWS lambda function inside the public subnet to test if it could connect to the outside internet.
Here's my lambda function written in python3
import requests
def lambda_handler(event, context):
r = requests.get('http://www.google.com')
print(r)
The function above failed to fetch the content of http://www.google.com when I set it inside the public subnet in a VPC.
Here's the error message:
"errorMessage": "HTTPConnectionPool(host='www.google.com', port=80):
Max retries exceeded with url: / (Caused by
NewConnectionError(': Failed to establish a new connection: [Errno 110]
Connection timed out',))", "errorType": "ConnectionError",
I don't understand why.
The route table of the public subnet looks like this:
The GET request to http://www.google.com should match igw-XXXXXXXXX target. Why can't the internet-gateway(igw) deliver the request to http://www.google.com and get back the website content?
This article says that I must set the lambda function inside the private subnet in order to have internet access.
If your Lambda function needs to access private VPC resources (for
example, an Amazon RDS DB instance or Amazon EC2 instance), you must
associate the function with a VPC. If your function also requires
internet access (for example, to reach a public AWS service endpoint),
your function must use a NAT gateway or instance.
But it doesn't explain why I can't set the lambda function inside the public subnet.
Lambda functions connected to a VPC public subnet cannot typically access the internet.
To access the internet from a public subnet you need a public IP or you need to route via a NAT that itself has a public IP. You also need an Internet Gateway (IGW). However:
Lambda functions do not, and cannot, have public IP addresses, and
the default route target in a VPC public subnet is the IGW, not a NAT
So, because the Lambda function only has a private IP and its traffic is routed to the IGW rather than to a NAT, all packets to the internet from the Lambda function will be dropped at the IGW.
Should I Configure my Lambda Function for VPC Access?
If your Lambda function does not need to reach private resources inside your VPC (e.g. an RDS database or Elasticsearch cluster) then do not configure the Lambda function to connect to the VPC.
If your Lambda function does need to reach private resources inside your VPC, then configure the Lambda function to connect to private subnets (and only private subnets).
NAT or Not?
If the Lambda function only needs access to resources in the VPC (e.g. an RDS database in a private subnet) then you don't need to route through NAT.
If the Lambda function only needs access to resources in the VPC and access to AWS services that are all available via private VPC Endpoint then you don't need to route through NAT. Use VPC Endpoints.
If your Lambda function needs to reach endpoints on the internet then ensure a default route from the Lambda function's private subnets to a NAT instance or NAT Gateway in a public subnet. And configure an IGW, if needed, without which internet access is not possible.
Be aware that NAT gateway charges per hour and per GB processed so it's worth understanding how to reduce data transfer costs for NAT gateway.
Best Practices
When configuring Lambda functions for VPC access, it is an HA best practice to configure multiple (private) subnets across different Availability Zones (AZs).
Intermittent Connectivity
Be sure that all the subnets you configure for your Lambda function are private subnets. It is a common mistake to configure, for example, 1 private subnet and 1 public subnet. This will result in your Lambda function working OK sometimes and failing at other times without any obvious cause.
For example, the Lambda function may succeed 5 times in a row, and then fail with a timeout (being unable to access some internet resource or AWS service). This happens because the first launch was in a private subnet, launches 2-5 reused the same Lambda function execution environment in the same private subnet (the so-called "warm start"), and then launch 6 was a "cold start" where the AWS Lambda service deployed the Lambda function in a public subnet where the Lambda function has no route to the internet.
You can make a lambda function access the public internet from within your VPC. Solution A is the actual answer, Solution B is a more elegant alternative solution.
Solution A - Lambda in VPC + Public IP associated with ENI
For accessing resources external to AWS such as Google API (like OP's example) you do need a Public IP. For other cases like RDS or S3 you don't need a Public IP, you can use a VPC Endpoint, so communication between your Lambda and the desired AWS Service doesn't leave AWS network.
By default some AWS Services are indeed reached via public internet, but it doesn't have to be.
Now if you want an actual external resource (e.g. google), you need to assign Elastic Public IPs to the Network Interfaces for each subnet linked to your lambda. First let's figure which subnets and security groups are linked to your lambda:
Next, go to EC2 Service, find the Public IPs menu under Network & Security. Allocate one IP for each subnet (in the example above there are two subnets).
Go to Network Interfaces menu, find the network interfaces attached to your lambda (same subnet and security group).
Associate the Public IPs in the actions menu for each one:
That's it, now your Lambda can reach out to public internet.
[EDIT]
Someone was concerned about Solution's A scalability issues saying each lambda instance has a new network interface but they missed this from AWS Docs:
"Multiple Lambda functions can share a network interface, if the functions share the same subnet and security group"
So whatever scalability issues you may face has nothing to do with this solution and how Lambda uses ENI, you'd face the same issues using EC2, ECS, EKS, not just Lambda.
Solution B - Decompose into multiple Lambdas
Requiring access both to external resources and VPC resources would seem like too much responsibility for a single function. You may want to rethink your design and decompose your single lambda function into at least two lambda functions:
Lambda A goes to external resources (e.g. Google API), fetches whatever data you need, add to SQS. No need to attach to VPC, no need to manually associate Elastic Public IP to ENI.
Lambda B processes the message from SQS, stores results to a storage (db, s3, efs, another queue, etc). This one lives within your VPC, and don't need external access.
This way seems more scalable, more secure, each individual lambda is less complex and more maintainable, the architecture looks better overall.
Of course life is not always rainbows and butterflies, so Solution A is good and scalable enough, but improving the architecture is even better.
I have been working on integrating an Amazon Lambda function with connection to a RDS for the mySQL DB and an external API. To access the API, there needed to be an internet gateway and then security groups that allowed connection from 0.0.0.0/0.
I have a a public subnet and private subnet. The public subnet routes to the internet gateway but the private subnet routes to a NAT.
This lead me to think that if I ran the Lambda function with the Public subnet, it would connect to the internet. However, every time it timed out. But, when I ran the lambda function from within the private subnet, it worked! So it the NAT seems to work since that is what the private sunet was associated with, but just using the internet gateway does not work.
Does anyone have any explanation for this?
For an Amazon Lambda function to connect to the Internet, ONE of the following is required:
The Lambda function is not connected to a VPC, OR
The Lambda function is connected to a private subnet and there is a NAT Gateway/NAT Instance configured, OR
The Lambda function is connected to a public subnet and an Elastic IP address is assigned to the Elastic Network Interface (ENI) being used by the Lambda function in the subnet
Merely connecting a Lambda function to a public subnet (without an EIP) will not provide Internet access.