I have an application running in an EC2 instance in a private subnet (to add extra security), receiving traffic directly from an internet-facing NLB that is associated to the public subnet.
I have also configured a NAT Gateway in the public subnet so that the private EC2 instance can download whatever needed from the Internet.
I have just come out to the next conclusion:
If I request from the Internet: http://index.html
The private EC2 Instance uses the NAT Gateway to send back the HTTP response, and therefore you are charged for that NAT processing.
https://aws.amazon.com/vpc/pricing/
"Data processing charges apply for each Gigabyte processed through the NAT gateway regardless of the traffic’s source or destination"
The Route Table associated to the Private Subnet (where the web/app server is located) has { - local ; 0.0.0.0/0 - NATGateway} If I remove the 0.0.0.0/0 entry, then the HTTP requests to the server do not work. And if I remove the NAT Gateway I get the same issue.
Additionally, when I have the NAT Gateway and the route table to use it from the Private Subnet, I can also see traffic in the monitoring tab of the NAT Gateway when I do a simple HTTP request from the Internet - http:///index.html
Has someone faced the same issue? Is my understanding correct?
Is there any workaround to avoid this? I can just think of the following:
Move the application to a Web Tier (in a public subnet) so that the EC2 Instance has a public IP and therefore it does not need the NAT Gateway to respond to every HTTP request
Create a Web Tier in addition to the Application Tier, so that all traffic goes from the NLB <-> Web Tier <-> App Tier
Create a NAT Instance instead of a NAT Gateway so that you are not charged for that NAT processing.
Thanks!
So, it turns out there is an strange behaviour (bug?) in NLB that requires a default route to something for return traffic to work.
Scenario:
NLB in Public Subnet
NLB Target Group pointing to an instance in a Private Subnet
Sending traffic to the NLB will not give a response.
Add:
NAT Gateway
Add a route in the Private Route Table to point to the NAT Gateway
This works.
However, it doesn't actually use the NAT Gateway!
Instead of adding a NAT Gateway, you can instead create a default route to another instance, for example:
0.0.0.0/0 -> another instance
Then, connecting to the NLB works!
Behind the scenes, the VPC network detects that traffic coming out of the instance is in response to a request that came via the NLB, and traffic will be routed out the NLB.
HOWEVER, it only works if there is a valid default route in the route table. The route isn't used -- the network overrides it, so it doesn't matter what it points to. However, it must point to a valid resource. It doesn't work if it points to an ENI that isn't attached to anything. In my tests, it also doesn't seem to work if you point it back to the same instance. But, I pointed it to another instance and it worked just fine.
AWS Support agrees that this is not the best behaviour, but it's how things have to be configured for the moment.
Bottom line: It does not require a NAT Gateway. It just needs a valid default route that will be ignored when routing return traffic.
Related
We have 10 instances which we deployed the app using the AWS ECS and ELB
Due to security reasons the API allows request only through specific IP whitelisted IP addresses.
So we are planning to pass the request through the proxy
How to route an API request go through a proxy
We are using nginx
Any specific way to route an API request go through a proxy will be helful
You won't need NGINX as a proxy for this use-case, I'd propose to consider looking into using AWS NAT Gateways. NAT Gateway is a highly available AWS managed service that makes it easy to connect to the Internet from instances within a private subnet in an Amazon Virtual Private Cloud (Amazon VPC). Its the perfect place to provide a static IP to all your subnet's outbound traffic.
In order to provide a NAT Gateway with static IP (Elastic IP) for your cluster's outbound traffic. This will allow your different tasks running inside your ECS cluster's private subnets to look like a single requesting entity from an outsider's POV (in your case, the 3rd party API is the outsider). To achieve this, you will have to:
Create 2 route tables (1 for private subnets, 1 for public subnets)
Internet gateway on the public subnet
Elastic IP address
Create a NAT Gateway and attach the elastic IP to it (This will be the IP whitelisted to the 3rd party API)
Ensure that all your tasks are running inside private subnets of the VPC
Add a rule in your route table for your private subnets that redirects outbound 0.0.0.0/0 to the NAT Gateway.
Add a rule in your route table for your public subnets that redirects outbound traffic 0.0.0.0/0 to the internet gateway.
You should consider using NAT Gateway instead. I am assuming you already would have all your containers in a VPC, so you can create a new NAT Gateway within this VPC itself.
You can refer to articles attached below to do this:
https://docs.aws.amazon.com/appstream2/latest/developerguide/add-nat-gateway-existing-vpc.html
https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html
Note: NAT Gateways have price associated with them.
If needed, you can use the elastic IP provided by NAT Gateways on your lambdas as well.
We have Typical 3 tier architecture having Web, App and DB.
Can we make EC2 instances in the web tier as Private? and allow incoming traffic only through ALB?
AFAIK we can apply an SG only allowing connections from the SG of the ALB. But What if our Private EC2 instance has to return response back to the client? How it'll be routed through ALB as ALB is mostly used for managing incoming traffic.
Also for outgoing traffic can we configure something like Private EC2 instance -> ALB -> Internet? If yes then how? So, is there any way for private EC2 instances to communicate to internet without assigning them public IP?
But What if our Private EC2 instance has to return response back to the client? How it'll be routed through ALB as ALB is mostly used for managing incoming traffic.
You don't have to do anything special, assuming your ALB and instances are correctly set. Any request coming to ALB, will be able to return to the client event if instances are in private subnet without any internet connectivity.
However, your instance won't be able to initiate internet connections by themselves. So if the instances don't need internet to operate, you don't need NAT. Otherwise, it is required.
Keeping your instances in a private subnet is best practice, even for the web layer.
For inbound traffic you would add a load balancer into your public subnets then allow HTTP/HTTPS ingress on the webs security group only from the load balancer. You can either do this through adding the subnet ranges into the web servers security group, or reference the security group the load balancer has assigned to it instead.
For outbound internet traffic in a private subnet you will need to create either a NAT Gateway or NAT instance within a public subnet, and then add a route for 0.0.0.0/0 for the private subnet to route traffic to the NAT. Additionally if you want IPv6 traffic you would create an egress only internet gateway with a route of ::/0.
I would like to create a lambda(vpc) which would access resources in vpc and make a request to services(REST API) via public application load balancer. I found out that vpc end point is better solution than creating a nat gateway.
I have created a vpc endpoint for elasticloadbalancing(by following steps at https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#create-interface-endpoint) and given full access in the policy. I could not find how to access it from the lambda, what would be the URL to make the request?
Edit:
Thanks to John for the info that vpc endpoint is used to connect to ELB API. So Vpc endpoint would not solve our issue.
We have our infra in vpc which includes database(accessible within vpc only) and application servers running behind the ELB. For certain tasks we want to run lambda which will read database(for this reason lambda has to be inside vpc) and make API calls to our application using ELB. Since ELB is accessible from public dns only, lambda is not able to connect to ELB.
I have read that setting up NAT gateway is a solution. Are there other is simpler ways?
Yes, a NAT Gateway would allow the traffic from a private subnet to go out of the VPC and come back in to the Load Balancer's public IP addresses (via its Public DNS Name).
Alternatively, you could create an additional Internal Load Balancer that could accept traffic from within the VPC and send it to the Amazon EC2 instances.
I have a AWS Lambda function which:
checks a Redis Elasticache instance,
if the item is not found in the cache, goes to Google Places API service.
The Redis instance is in a private subnet; so, to fetch it, I added the VPC and the subnet in which the instance resides. I also specified the security group which allows all the outbound traffic. The Network ACL is the default one which is supposed to all the inbound and the outbound traffic.
When adding VPC to Lambda function like that via the console, it prompts:
When you enable VPC, your Lambda function will lose default internet access. If you require external internet access for your function, ensure that your security group allows outbound connections and that your VPC has a NAT gateway.
So, in the Route Table of the private subnet, I added a NAT gateway too. However, at the point where the Google Places API service call is made from the Lambda function it is always doomed to result in timeout.
In short, I doubt that the NAT gateway properly allows internet access of the Lambda function. How can I check what goes wrong with it?
Do NAT Gateways log the calls or the call attempts being tried through it somehow in CloudWatch etc.?
I want to elaborate on the answer from #vahdet. I was losing my mind trying to reconcile how the NAT Gateway was supposed to be in the public and private subnets simultaneously. It seemed like the official AWS documentation here was wrong, but of course it's not. There is a very subtle detail that myself and others have missed.
The NAT Gateway has to be "hot-wired" across two different subnets simultaneously in order to bridge private addresses to a public one that is internet facing.
First, I tried to put the NAT Gateway in the same route table as the IGW, but of course that doesn't work because you can't have two identical routes (0.0.0.0/0) with different targets.
The guide was saying to put the NAT Gateway in the route table for the Private Network, which I did, but that didn't seem to work.
The critical detail I was missing was that the NAT Gateway has to be created in a public subnet. The documentation actually says this, but it seems confusing because we are later told to put the route for NAT Gateway in the private table.
Both things are true. Create the NAT Gateway in the public subnet and then only add a route table entry in the private route table.
The documentation tells you to create the following network resources in the VPC:
two new subnets
two new route tables
one new NAT Gateway
I already had a route table and some subnets, so I tried to only add one new subnet and one new route table and this is where I got into trouble. It really was better to create two of each as documented.
Here's what it the subnets look like for me:
subnet-public 10.8.9.0/24 us-east-1a
subnet-private 10.8.8.0/24 us-east-1a
Then create the NAT Gateway in subnet-public.
It will be pending for a couple of minutes, which is important, because it must go to available status before it can be referenced in a route table entry.
Here are the route tables:
route-table-public
10.8.0.0/16 local
0.0.0.0/0 igw-xyz
subnet-association: subnet-public
route-table-private
10.8.0.0/16 local
0.0.0.0/0 nat-abc
subnet-association: subnet-private
Do you see what happened there? It's really subtle. The NAT Gateway is cross-wired. It "lives" in the public subnet it was created in, but all traffic in the private subnet gets routed to it.
If you create the NAT Gateway in the private subnet like I did at first, then the NAT Gateway is just as isolated as everything else in the private subnet, and has no way to route traffic out to the internet. It must be created in the public subnet to have internet access, because it must have an IP address inside the public subnet. My NAT Gateway got an internal IP of 10.8.9.127 and an external IP in the 54.X.X.X range.
By making the NAT Gateway the 0.0.0.0/0 route in the private routing table, we are telling all non-10.8.0.0/16 traffic to go straight to the NAT Gateway, even though it isn't actually inside the private subnet.
The VPC itself knows how to bridge traffic across its own subnets, and is able to send the 10.8.8.X traffic to the NAT Gateway's 10.8.9.X IP. It then acts as a bridge, and translates all of that traffic across it's internal IP to its external IP. Because it is in a public subnet that is in a route table with an internet gateway, the external IP has a clear path to the internet.
Now my private VPC lambda in subnet-private has verified internet access through the NAT Gateway.
The following steps are required
An IAM role with full VPC permission assigned to your lambda function.
VPC with public and private subnet
while creating a NAT Gateway
a)the subnet has to be public subnet
b)Elastic IP creat a new one or allocate one
Create the route table and add another route with target as our NAT gateway we created above.
And your lambda should be happy now
The problem for my case turned out to the fact that, I had created the NAT Gateway in the private subnet.
Make sure you place the NAT Gateways in the public subnet.
By the way, there are metrics but no direct logging records available in CloudWatch for NAT Gateway.
Our aim is to get our Elastic Beanstalk setups to route traffic through a NAT gateway as we require for certain traffic connecting to API's which require IP whitelisting. Rather than make modifications to the current setup, I have created a separate/isolated VPC & EC2 instance to familiarise and test the setup. However I am yet to get the setup working as desired.
Here is the setup
VPC (vpc-77049811) with CIDR of 10.0.0.0/16
Internet gateway (igw-4d4b212a) assigned to mentioned VPC
Subnet (subnet-096d8a53) with CIDR of 10.0.1.0/24
NAT Gateway (nat-00bb49204627de7e6) attached to mentioned subnet and assigned Elastic IP
Route table attached to mentioned VPC and associate with mentioned subnet
1x EC2 Instance assigned to VPC and its own Elastic IP and Disabled Source/Destination Check
Route Table Setup
10.0.0.0/16 local
0.0.0.0/0 igw-4d4b212a
With the above setup, and am able to log into the server and make a curl request to get the servers public IP address (curl icanhazip.com). As soon as I add a rule to the route table for the url's resolved IP's to route through the NAT gateway though, I am unable to ping or request the curl request as it will timeout.
Rules added to route table which do not work
45.63.64.111/32 nat-00bb49204627de7e6
144.202.71.30/32 nat-00bb49204627de7e6
Not sure if I've overlooked something here or maybe I have misunderstood the concept and use cases for the NAT gateway?
This is public IP 45.63.64.111. You need IGW to reach to this traffic.
You either do that by directly redirecting your traffic to IGW
OR
You do that by directing to NAT then further directing that traffic to IGW
Directing to IGW part is missing.
Nat gateway is used for EC2 in private subnets (which does not have IGW attached to it). In scenario above, EC2 is in public subnet so ideally it does not need NAT.
Here is what I would do to use NAT-
1. Place EC2 in private subent. and have a Route table where all outgoing traffic to nat-gateway.
2. Nat-gateway which is in public subnet will forward your traffic to IGW.
It seems like you have misunderstood the purpose of a NAT.
Its purpose is to provide outbound internet access to instances in a private subnet without allowing any inbound connectivity - i.e. a subnet where the routing table does NOT have an entry for:
0.0.0.0/0 igw-4d4b212a
If you want to restrict access from your EC2 instance to specific IP addresses, put your NAT in the public subnet, create a private subnet, and put your instance in the private subnet. Then add the two routes to the route table associated with the private subnet:
45.63.64.111/32 nat-00bb49204627de7e6
144.202.71.30/32 nat-00bb49204627de7e6
If you simply want to restrict access of your EC2 instance to a couple of IP addresses, you can only create routes for those addresses:
45.63.64.111/32 igw-4d4b212a
144.202.71.30/32 igw-4d4b212a
Be aware that with this last option, your instance can be reached from the internet if you have rules open in your security groups.