Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 2 years ago.
Improve this question
I am trying to make an outbound API request to a third-party service from within a Lambda function, but the function always times out without any error.
This previously happened when trying to perform a s3.putObject operation within a different function (still within the same VPC / subnets), and I managed to get around that by adding an Endpoint with a service name com.amazonaws.us-east-1.s3 and connecting it to the route table that is associated with the VPC that this Lambda function resides in.
Within the Lambda dashboard inside Network box -> Security Groups section, I see this warning:
When you enable VPC, your Lambda function will lose default internet
access. If you require external internet access for your function,
ensure that your security group allows outbound connections and that
your VPC has a NAT gateway.
I believe that this security group allows outbound connections, based off of the Outbound rules table right underneath:
For that second requirement, I can confirm this VPC has a NAT gateway, because on the VPC Dashboard, within NAT Gateways tab, the one that appears there has a VCP associated with it, and that VPC is the same one hosting the Lambda function.
I followed a guide to create a Flow Log to monitor traffic in and out of the VPC, hoping to see that those outbound requests are indeed rejected. However, after doing so and inspecting the CloudWatch logs, all of the records end in either ACCEPT OK or NODATA.
How can I grant internet access to my VPC Lambda function? is the guide I originally tried to follow, but I got stuck on step 4 under To create a public or private subnet:
From the Change to: drop-down menu, choose an appropriate route table:
For a private subnet, the default route should point to a NAT gateway
or NAT instance:
Destination: 0.0.0.0/0
Target: nat-… (or eni-…)
For a public subnet,
the default route should point to an internet gateway:
Destination: 0.0.0.0/0
Target: igw-…
For all four of the subnets within this VPC, clicking the drop-down to the right of Change to: only showed one option, the one already selected, rtb-xxxxxxxx. After clicking on the link to that route table, and clicking the Routes tab next to Summary, I see this:
What might I be doing wrong that is blocking the Lambda function's access to the Internet?
For Lambda to have access to the internet via VPC it should be in the Private Subnet with NAT Gateway attached.
As per your screenshots, route table attached to subnet has igw-xxxxxxx attached making your current subnet a public subnet.
So to make things work, you can do following:
Attach NAT Gateway instead of igw-xxxxxx in route table of your current subnet
OR
Find ENI attached to your Lambda and attach Elastic IP if you want to have internet access.
OR
As per #John Rotenstein if your Lambda doesnt need VPC recources you can move Lambda out of VPC
I developed a step by step tutorial with explicit screenshots about it:
Part I
Create a lambda function in AWS
Create an API Gateway to redirect all the requests to your AWS Lambda Function
https://medium.com/#shontauro/how-can-i-turn-my-restful-api-into-a-serverless-application-and-deploy-it-to-aws-lambda-step-by-ec7ce38e18be
Part II
Configure a VPC for your lambda function
Grant internet access to the lambda function in a safety way
Deploy the Node.js Restful API into your Lambda Function
https://medium.com/#shontauro/how-can-i-turn-my-restful-api-into-a-serverless-application-and-deploy-it-to-aws-lambda-step-by-8ff6cc97780f
One thing you may be doing wrong (which is exactly what I did wrong but got a hint about from #Ramratan Gupta above) is to put the NAT gateway in the private subnet and not in the public.
The NAT gateway provides your private IP Lambda function with internet connectivity and so it sort of stands to reason the NAT gateway needs to have internet access itself and be in the Public network.
To summarise:
You have to create at least two subnets: one public and one private. Put both in the same VPC.
What makes a subnet public or private is the 0.0.0.0/0 route.
You need to have both an internet gateway (igw) and a NAT gateway (ngw).
To create the NAT gateway, you need an Elastic IP.
Also, and here's what I did wrong, you need to create the NAT gateway in the PUBLIC network.
The igw goes in the same subnet that the NAT gateway is: the public one.
You need to have two routes: one with the 0.0.0.0/0 route referencing the igw and the other referencing the ngw.
You then need to associate the NAT-route containing route table with the private subnet and the igw-route containing route table with the public subnet. Please note that until you do this association, there is no difference between the subnet that would make one public and one private. You make one private and the other public by how you associate the subnets with the route tables.
Finally, you need to connect your Lambda function to the VPC and also to the private subnet only. This way, when your lambda tries to reach the internet, it goes to the private subnet, looks for the 0.0.0.0/0 route, which directs it to the NAT gateway that is in the public subnet. The NAT gateway then contacts the internet using the igw (because they are in the same subnet) and upon receiving a response, returns it to your Lambda function.
Please also note that there are some permissions your Lambda function needs (that AWS will warn you about) in order to connect to a VPC.
Setting up AWS Lambda Internet access in VPC mode is tricky as no proper AWS documentation is provaded on AWS side, I did read this aritical many times https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/ but did not wached YouTube Video. After so much of R&D I can summrise as below steps are need for set up AWS Lambda in VPC mode
- Route Tables two router
- One for EC2 - EC2_ROUTER
- Another for AWS Lambda - AWS_Lambda_ROUTER
- Create an Internet Gateway
- Add 0.0.0.0/0 rule in EC2_ROUTER for target igw-xxxxx
- This will create public subnet and give INTERNET access for EC2
- Create a NATGateway with public subnet which has igw-xxxxx as target
- Create a private subnet which and attach to AWS_Lambda_ROUTER, with target 0.0.0.0/0 as nat-xxxxxxx
- Select VPC for AWS Lambda
- Select a/all private subnet which is private having NAT Gateway for all traffic 0.0.0.0/0
all done, go ahead and enjoy
Note
NATGateway is required when access to public API other than AWS
There is AWS Endpoints for internal AWS services, and one can attach AWS endpoints in AWS_Lambda_ROUTER by creating
Internet access in AWS Lambda -> Private Subnet with NATGateway -> NATGateway is in public subnet with Internet Gateway which access to AWS Lambda.
PS
I have no Idea why AWS Lambda can not have INTERNET access via InternetGateway why it needs NATGateway
Related
I've followed the tutorial here to create a VPC with public and private subnets.
Then I set up an AWS lambda function inside the public subnet to test if it could connect to the outside internet.
Here's my lambda function written in python3
import requests
def lambda_handler(event, context):
r = requests.get('http://www.google.com')
print(r)
The function above failed to fetch the content of http://www.google.com when I set it inside the public subnet in a VPC.
Here's the error message:
"errorMessage": "HTTPConnectionPool(host='www.google.com', port=80):
Max retries exceeded with url: / (Caused by
NewConnectionError(': Failed to establish a new connection: [Errno 110]
Connection timed out',))", "errorType": "ConnectionError",
I don't understand why.
The route table of the public subnet looks like this:
The GET request to http://www.google.com should match igw-XXXXXXXXX target. Why can't the internet-gateway(igw) deliver the request to http://www.google.com and get back the website content?
This article says that I must set the lambda function inside the private subnet in order to have internet access.
If your Lambda function needs to access private VPC resources (for
example, an Amazon RDS DB instance or Amazon EC2 instance), you must
associate the function with a VPC. If your function also requires
internet access (for example, to reach a public AWS service endpoint),
your function must use a NAT gateway or instance.
But it doesn't explain why I can't set the lambda function inside the public subnet.
Lambda functions connected to a VPC public subnet cannot typically access the internet.
To access the internet from a public subnet you need a public IP or you need to route via a NAT that itself has a public IP. You also need an Internet Gateway (IGW). However:
Lambda functions do not, and cannot, have public IP addresses, and
the default route target in a VPC public subnet is the IGW, not a NAT
So, because the Lambda function only has a private IP and its traffic is routed to the IGW rather than to a NAT, all packets to the internet from the Lambda function will be dropped at the IGW.
Should I Configure my Lambda Function for VPC Access?
If your Lambda function does not need to reach private resources inside your VPC (e.g. an RDS database or Elasticsearch cluster) then do not configure the Lambda function to connect to the VPC.
If your Lambda function does need to reach private resources inside your VPC, then configure the Lambda function to connect to private subnets (and only private subnets).
NAT or Not?
If the Lambda function only needs access to resources in the VPC (e.g. an RDS database in a private subnet) then you don't need to route through NAT.
If the Lambda function only needs access to resources in the VPC and access to AWS services that are all available via private VPC Endpoint then you don't need to route through NAT. Use VPC Endpoints.
If your Lambda function needs to reach endpoints on the internet then ensure a default route from the Lambda function's private subnets to a NAT instance or NAT Gateway in a public subnet. And configure an IGW, if needed, without which internet access is not possible.
Be aware that NAT gateway charges per hour and per GB processed so it's worth understanding how to reduce data transfer costs for NAT gateway.
Best Practices
When configuring Lambda functions for VPC access, it is an HA best practice to configure multiple (private) subnets across different Availability Zones (AZs).
Intermittent Connectivity
Be sure that all the subnets you configure for your Lambda function are private subnets. It is a common mistake to configure, for example, 1 private subnet and 1 public subnet. This will result in your Lambda function working OK sometimes and failing at other times without any obvious cause.
For example, the Lambda function may succeed 5 times in a row, and then fail with a timeout (being unable to access some internet resource or AWS service). This happens because the first launch was in a private subnet, launches 2-5 reused the same Lambda function execution environment in the same private subnet (the so-called "warm start"), and then launch 6 was a "cold start" where the AWS Lambda service deployed the Lambda function in a public subnet where the Lambda function has no route to the internet.
You can make a lambda function access the public internet from within your VPC, you just need to make sure you really need it.
For accessing resources external to AWS such as Google API (like OP's example) you do need a Public IP. For other cases like RDS or S3 you don't need Public IP, you can use a VPC Endpoint, so communication between your Lambda and the desired AWS Service doesn't leave AWS network.
By default some AWS Services are indeed reached via public internet, but it doesn't have to be.
[EDIT]
Someone was concerned about scalability in the comments, but they missed this from AWS Docs:
"Multiple Lambda functions can share a network interface, if the functions share the same subnet and security group"
Also, you must have a Public IP for reaching Public Internet, whether you're using Lambda, EC2, ECS, even if you use a NAT Gateway it needs an Elastic Public IP if you want to reach the public internet through it.
Solution
To do that, you need to assign Elastic Public IPs to the Network Interfaces for each subnet linked to your lambda. First let's figure which subnets and security groups are linked to your lambda:
Next, go to EC2 Service, find the Public IPs menu under Network & Security. Allocate one IP for each subnet (in the example above there are two subnets).
Go to Network Interfaces menu, find the network interfaces attached to your lambda (same subnet and security group).
Associate the Public IPs in the actions menu for each one:
That's it, now your Lambda can reach out to public internet.
I have a Lambda function (NodeJS) which consume an API that need to be accesed from certains IP, so team under the API ask me "please, provide us your IP to add it to the white list", but I'm not sure where or how to configure that static IP on AWS, where should I look?
Since #maurice show you the way, I will try to add some insight though I had to build a similar infrastructure like yours:
Third party provider requests us a static IP to connect to their API and our code was in a Lambda function (NodeJS).
In this case we did not want the lambdas to be reachable from outside, so we built two Subnets, one private and another one public. Lambdas go inside the private, which route Table 0.0.0.0/0 uses a NAT attached to the public subnet to manage the traffic that needs to connect to the Internet. Let me show you a few screenshots:
The Lambda must be in a VPC subnet with a NAT-Gateway.
The private subnet route table, which shows a NAT attachment:
The NAT belongs to the public subnet, and has an elastic IP attached (this is the key):
The public subnet has the following route table (everything is managed by the IGW):
This is something that is currently running in production environment and works flawless
You can't provide a static IP in a Lambda function directly.
You could deploy the Lambda function in a private subnet of a VPC that uses a NAT-Gateway for internet connectivity.
This NAT-Gateway will have an Elastic-IP, which you can tell your provider
(For high-availability you should have one NAT gateway per region).
I have a lambda function that simply does an http.get to http://www.google.com. If I don't have the function behind a VPC, it works fine. The trouble happens when I put it in my VPC.
I know you need to set up an Internet Gateway. I did this. My two subnets are attached to route tables that route 0.0.0.0/0 to this Internet Gateway. Shouldn't that be all I need?
The function still hangs regardless of the Internet Gateway's association. The subnet's security groups allows All Traffic out of 0.0.0.0/0".
According to Grant Internet Access to a VPC Lambda Function that is everything I should need to do.
Edit:
Adding full list of VPC components to be clear.
Created a new VPC (vpc-09*)
Created a new subnet (subnet-05*) point to my new:
route table, (rtb-0b). I see subnet-05* under Subnet Associations. Under Routes, I see Destination 0.0.0.0/0 linked to the Target of
a new NAT Gateway (nat-08*). This NAT Gateway has an Elastic IP Address and a Private IP Address. It resides in the correct Subnet. The status is Available.
Additionally, I created a new Security Group for the Lambda function. This contains one Outbound Rules for "All traffic" with Destination 0.0.0.0/0
As far as I can tell, I've done absolutely everything in that AWS Documentation link to provide my Lambda with internet access. Yet, it still hangs forever when trying to make a request to the outside internet.
You're almost there. The link that you've provided address your issue directly:
If your function also requires internet access (for example, to reach a public AWS service endpoint), your function must use a NAT gateway or instance
You're missing this:
Your VPC should contain a NAT gateway or instance in a public subnet.
This means that without a NAT, your Lambdas won't be able to access the internet - even though "they are" in a public subnet. This is how lambda fundamentally works in VPCs.
The exact same link that you provided instructs you on how to create this NAT Gateway alongside your VPCs and Lambdas.
Complementing the answer - on why you would need a NAT Gateway in this scenario - is due to:
... you can use a network address translation (NAT) gateway to enable instances in a private subnet to connect to the internet or other AWS services, but prevent the internet from initiating a connection with those instances...
Extracted from aws docs
Keep in mind: If you need your lambdas to access only the internet - and not any other resource in the same VPC - I recommend to make them non-VPC and then they'll have internet access out of the box - and you won't pay for the cost of NATs.
I have a AWS Lambda function which:
checks a Redis Elasticache instance,
if the item is not found in the cache, goes to Google Places API service.
The Redis instance is in a private subnet; so, to fetch it, I added the VPC and the subnet in which the instance resides. I also specified the security group which allows all the outbound traffic. The Network ACL is the default one which is supposed to all the inbound and the outbound traffic.
When adding VPC to Lambda function like that via the console, it prompts:
When you enable VPC, your Lambda function will lose default internet access. If you require external internet access for your function, ensure that your security group allows outbound connections and that your VPC has a NAT gateway.
So, in the Route Table of the private subnet, I added a NAT gateway too. However, at the point where the Google Places API service call is made from the Lambda function it is always doomed to result in timeout.
In short, I doubt that the NAT gateway properly allows internet access of the Lambda function. How can I check what goes wrong with it?
Do NAT Gateways log the calls or the call attempts being tried through it somehow in CloudWatch etc.?
I want to elaborate on the answer from #vahdet. I was losing my mind trying to reconcile how the NAT Gateway was supposed to be in the public and private subnets simultaneously. It seemed like the official AWS documentation here was wrong, but of course it's not. There is a very subtle detail that myself and others have missed.
The NAT Gateway has to be "hot-wired" across two different subnets simultaneously in order to bridge private addresses to a public one that is internet facing.
First, I tried to put the NAT Gateway in the same route table as the IGW, but of course that doesn't work because you can't have two identical routes (0.0.0.0/0) with different targets.
The guide was saying to put the NAT Gateway in the route table for the Private Network, which I did, but that didn't seem to work.
The critical detail I was missing was that the NAT Gateway has to be created in a public subnet. The documentation actually says this, but it seems confusing because we are later told to put the route for NAT Gateway in the private table.
Both things are true. Create the NAT Gateway in the public subnet and then only add a route table entry in the private route table.
The documentation tells you to create the following network resources in the VPC:
two new subnets
two new route tables
one new NAT Gateway
I already had a route table and some subnets, so I tried to only add one new subnet and one new route table and this is where I got into trouble. It really was better to create two of each as documented.
Here's what it the subnets look like for me:
subnet-public 10.8.9.0/24 us-east-1a
subnet-private 10.8.8.0/24 us-east-1a
Then create the NAT Gateway in subnet-public.
It will be pending for a couple of minutes, which is important, because it must go to available status before it can be referenced in a route table entry.
Here are the route tables:
route-table-public
10.8.0.0/16 local
0.0.0.0/0 igw-xyz
subnet-association: subnet-public
route-table-private
10.8.0.0/16 local
0.0.0.0/0 nat-abc
subnet-association: subnet-private
Do you see what happened there? It's really subtle. The NAT Gateway is cross-wired. It "lives" in the public subnet it was created in, but all traffic in the private subnet gets routed to it.
If you create the NAT Gateway in the private subnet like I did at first, then the NAT Gateway is just as isolated as everything else in the private subnet, and has no way to route traffic out to the internet. It must be created in the public subnet to have internet access, because it must have an IP address inside the public subnet. My NAT Gateway got an internal IP of 10.8.9.127 and an external IP in the 54.X.X.X range.
By making the NAT Gateway the 0.0.0.0/0 route in the private routing table, we are telling all non-10.8.0.0/16 traffic to go straight to the NAT Gateway, even though it isn't actually inside the private subnet.
The VPC itself knows how to bridge traffic across its own subnets, and is able to send the 10.8.8.X traffic to the NAT Gateway's 10.8.9.X IP. It then acts as a bridge, and translates all of that traffic across it's internal IP to its external IP. Because it is in a public subnet that is in a route table with an internet gateway, the external IP has a clear path to the internet.
Now my private VPC lambda in subnet-private has verified internet access through the NAT Gateway.
The following steps are required
An IAM role with full VPC permission assigned to your lambda function.
VPC with public and private subnet
while creating a NAT Gateway
a)the subnet has to be public subnet
b)Elastic IP creat a new one or allocate one
Create the route table and add another route with target as our NAT gateway we created above.
And your lambda should be happy now
The problem for my case turned out to the fact that, I had created the NAT Gateway in the private subnet.
Make sure you place the NAT Gateways in the public subnet.
By the way, there are metrics but no direct logging records available in CloudWatch for NAT Gateway.
I am trying to setup a nice and secure VPC for my lambda and RDS work. Essentially, I want my lambda to hit a site, get some data, and shove it into a database.
In isolation the parts all work. However the second I go to harden everything it all falls apart. Here is what I do:
Disable "Publically Available" from the RDS instance
Change the RDS instance to only accept connections from inside the VPC using the security group
Associate the lambda with a VPC (this kills the internet access)
Following this tutorial I created a NAT gateway, deleted the internet gateway from the VPC subnet, and replaced it with the NAT. Now, as expected, nothing can talk inbound, but things can talk outbound.
At this point I knew I needed a bastion instance, so I fired up an EC2 instance.
The EC2 instance is set to the same subnet the RDS and Lambda are on, and unfortunately this means that I have a problem - the NAT gateway is currently soaking up all the traffic via 0.0.0.0/0, which means there's no room for the internet gateway. Without the internet gateway I (obviously) can't SSH into my bastion instance so I can jump to access my RDS database.
How can I configure this all correctly? My guess is that I need to split the subnet up somehow and make a private and public subnet, the public having the bastion and internet gateway in it. However, I'm not sure how this will all work so the bastion instance can still properly jump to the RDS.
I'm really quite new to setting up AWS services so I'm hoping I didn't mess anything up long the way.
Following this tutorial I created a NAT gateway, deleted the internet
gateway from the VPC subnet, and replaced it with the NAT. Now, as
expected, nothing can talk inbound, but things can talk outbound.
Short Answer
The short answer is you shouldn't have "Killed the Internet Gateway"; thats not a step in the link you provided :) Leave the internet gateway as is in your current subnet. You're going to need a public subnet and the one that was routing 0.0.0.0/0 to IGW is an example of one you can could use.
The work involved is placing your NAT gateway in the Public Subnet, placing your bastion host in the public subnet, placing your lambda function in the private subnet, routing traffic in the private subnet to the NAT gateway in the public subnet, and providing your lambda function with access to your security group by putting it in its own lambda security group and "white listing" the lambda security group in the inbound rules for the security group protecting your database.
Background
Below I have an expanded answer providing background as to public/private subnets, granting internet access to private subnets, and allowing lambda access through security groups. If you don't feel like reading the background then jump to very end where I give a bullet point summary of the steps you'll need.
Public Subnet
A public subnet is one in which traffic originating outside your VPC, or destined for a target outside your VPC (internet), is routed through an internet gateway (IGW). AWS gives you initial default public subnets configured this way; you can identify them in the console by looking at their route table and seeing that under "destination" you find "0.0.0.0/0" targeting an IGW. This means a public subnet is more of a design pattern for "internet accessible" subnet made possible by simply configuring its default route to point to an IGW. If you wish to create a new public subnet you can create a new route table as well that point internet traffic at an IGW and link that route table to your new subnet. This is fairly easy in the console.
Private Subnet
A private subnet is a subnet with no IGW and not directly reachable from the internet, meaning you cannot connect to a public IP address of a system on a private subnet. With the exception of the AWS pre-configured default subnets, this is how new subnets your manually create are setup, as black boxes till you specify otherwise.
Granting Internet Access to Private Subnet
When you want things in your private subnet to be able to reach out to external internet services you can do this by using an intermediary known as a NAT gateway. Configure a route table the same as in the public subnet with the only difference being traffic destined for 0.0.0.0/0(Internet) you target for a NAT gateway sitting inside the public subnet. That last part is critical. Your NAT gateway needs to be in the public subnet but your private subnet is using it as the target for external traffic.
Security Group Access for Lambda
One simple way to allow your lambda function through your security group/firewall is to create a security group just for your lambda function and configure the security group protecting your RDS so that it allows traffic from the lambda security group.
In other words, in security group settings you don't have to specify just IP addresses as sources, you can specify other security groups and this is a pretty neat way of grouping items without having to know their IP address. Your lambda functions can run in the "Lambda Security Group" and anything protected by a security group that you want them to access can be configured to accept traffic from the "Lambda Security Group". Just make sure you actually associate your lambda function with the lambda security group as well as place it in the private subnet.
Lambda VPC Steps in a Nutshetll
Create a new NAT gateway and place it in the public subnet. This
point is important, the NAT gateway goes in the public subnet ( a
subnet whose route table routes 0.0.0.0/0 to an IGW)
Create a new subnet, you can call it Private-Lambda-Subnet. Unlike
the
default pre-configured subnets AWS gives you, this new subnet is
immediately private out of the box.
Create a new route table and link it to your Private-Lambda-Subnet
In the new route table for your private subnet add an entry that
routes 0.0.0.0/0 to a target of the NAT gateway. This is how your
private subnet will indirectly access the internet, by forwarding
traffic to the NAT which will then forward it to the IGW.
Your bastion host and anything else you want to be be publicly
accessible will need to be in the public subnet. This is probably
where you already have your RDS instances, which is fine if they are
firewalled/security group protected.
Create a new security group for your lambda function(s). You can
call it LambdaSecurityGroup.
Configure the inbound rules of your RDS guarding security group to
allow traffic from the LambdaSecurityGroup. This is possible because
you can use other security groups as sources in the firewall
settings, not just ip addresses.
You need a public subnet (default route is the Internet Gateway) and a private subnet (default route is the NAT Gateway). The NAT Gateway, itself, goes on the public subnet, so that it can access the Internet on behalf of the other subnets for which it is providing services. The bastion also goes on the public subnet, but Lambda and RDS go on the private subnet.
Anything can talk to anything on any subnet within a VPC as long as security groups allow it (and Network ACLs, but don't change these unless you have a specific reason to -- if you aren't sure, then the default settings are sufficient).