We have a Lambda function in an account and we would like to access an EC2 instance (via HTTP) in another VPC which has a public IP attached. I was wondering what would be the best way to perform this communication. I am new to Lambda and I just got to know of the VPC lambdas. Which CIDRs do I need to open on the Security Group on the EC2 instance? Can I have a specific set of public IPs being picked in the source VPC - this way I can whitelist that range in the SG?
Does VPC peering seem like an overhead for this case or the only possible solution?
It all Depends on your requirements.However, Peering those VPCs is the best way to have the traffic remaining in your trusted Private Subnets which are located in the Internal Trust Boundaries which satisfies Security best practices (Threat/Security Models and Cloud/Network Architecture).
If there is an enterprise policy with strict rules that communication shall not be routed via DMZ/Public Internet and is a must to keep things within the Trusted Boundary of Internal Routes, I don't see any other choice but to go with the VPC peering : https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/peer-with-vpc-in-another-account.html
However, if requirements are less strict and you can use Public Internet to forward your Traffic out via DMZ, it is possible to achieve this with out sacrificing too much of Security (assuming that your EC2 with public IP on other account is providing service over SSL/TLS where your lambda can communicate with it over an Encrypted Communication Channel while validating the EC2's certificate).
This could be achieved by having Lambda associated with an Internal Subnet of your VPC to talk to the EC2 of other account with Public IP. https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html
Yes, You still can have your Lambda remaining inside the Internal Subnet. But you need a NAT GATEWAY and update the Routing Table for Lambda's Internal Subnet to point to the NAT GATEWAY (which should be assigned with an EIP) i.e. it will subsequently point to your INTERNET GATEWAY. By this you will make sure that your LAMBDA which, located in your private subnet of VPC can talk with outside i.e. with the EC2 instance, located in another account's with a Public IP. Therefore you can whitelist one IP in the SecurityGroups of your EC2 in Other Account which is the EIP of your NATGATEWAY which Lambda or any other Internal Components in that subnet will use to find their way out to the Internet.
Related
When I configure an AWS Gateway VPC endpoint, a route table entry is created that points to the Gateway. Here, Gateway can be thought of performing the routing to AWS service (over private network).
However, for an AWS Inteface VPC endpoint, all that is visible is a Network interface that has a private IP address of the subnet. By default, a private IP can send traffic within the subnet or entire VPC provided Security Group and NACL allows the traffic. & it appears in this case there is no Route table entry to a Gateway or a Router for allowing traffic outside VPC.
How / Where is the interface routing the traffic to i.e. How does traffic leave the customer VPC?
Of course I understand that the traffic finally reaches the intended AWS service over private network but here I am trying to find out where is the Gateway or Router? Does AWS hide this implementation?
I cannot get my head around the fact that a simple Network Interface can accept traffic and route it to a service all by itself i.e. performing routing by itself? Clearly, in this case the traffic appears not flowing through the VPC router or another Gateway device.
I am aware this might be an AWS confidential implementation but any thoughts / idea on how they might have designed this feature?
It doesn't provide routing at all, by default a VPC interface endpoint when created will create an ENI per subnet in the VPC for you. It will also provide you a DNS name per each AZ and a global name that you can use within your applications.
In addition it supports the ability to have the AWS service domain name for the VPC interface endpoint be resolvable to the private IPs of the endpoint. As long as your VPC has DNS enabled it will first check the VPC private DNS resolver and then resolve it to the private IP rather than the public one.
This is done by adding an additional private hosted zone to your VPC which resolves service domains in your region such as ec2.us-east-1.amazonaws.com.
From the AWS side this is just an ENI created in your AWS VPC that is connected to one of AWS internal VPCs. It's actually possible to implement this for your own services too to share with another organisations VPCs, this is implemented using AWS PrivateLink.
For more information take a look at the Private DNS for interface endpoints page.
I have a lambda function that simply does an http.get to http://www.google.com. If I don't have the function behind a VPC, it works fine. The trouble happens when I put it in my VPC.
I know you need to set up an Internet Gateway. I did this. My two subnets are attached to route tables that route 0.0.0.0/0 to this Internet Gateway. Shouldn't that be all I need?
The function still hangs regardless of the Internet Gateway's association. The subnet's security groups allows All Traffic out of 0.0.0.0/0".
According to Grant Internet Access to a VPC Lambda Function that is everything I should need to do.
Edit:
Adding full list of VPC components to be clear.
Created a new VPC (vpc-09*)
Created a new subnet (subnet-05*) point to my new:
route table, (rtb-0b). I see subnet-05* under Subnet Associations. Under Routes, I see Destination 0.0.0.0/0 linked to the Target of
a new NAT Gateway (nat-08*). This NAT Gateway has an Elastic IP Address and a Private IP Address. It resides in the correct Subnet. The status is Available.
Additionally, I created a new Security Group for the Lambda function. This contains one Outbound Rules for "All traffic" with Destination 0.0.0.0/0
As far as I can tell, I've done absolutely everything in that AWS Documentation link to provide my Lambda with internet access. Yet, it still hangs forever when trying to make a request to the outside internet.
You're almost there. The link that you've provided address your issue directly:
If your function also requires internet access (for example, to reach a public AWS service endpoint), your function must use a NAT gateway or instance
You're missing this:
Your VPC should contain a NAT gateway or instance in a public subnet.
This means that without a NAT, your Lambdas won't be able to access the internet - even though "they are" in a public subnet. This is how lambda fundamentally works in VPCs.
The exact same link that you provided instructs you on how to create this NAT Gateway alongside your VPCs and Lambdas.
Complementing the answer - on why you would need a NAT Gateway in this scenario - is due to:
... you can use a network address translation (NAT) gateway to enable instances in a private subnet to connect to the internet or other AWS services, but prevent the internet from initiating a connection with those instances...
Extracted from aws docs
Keep in mind: If you need your lambdas to access only the internet - and not any other resource in the same VPC - I recommend to make them non-VPC and then they'll have internet access out of the box - and you won't pay for the cost of NATs.
I've followed the tutorial here to create a VPC with public and private subnets.
Then I set up an AWS lambda function inside the public subnet to test if it could connect to the outside internet.
Here's my lambda function written in python3
import requests
def lambda_handler(event, context):
r = requests.get('http://www.google.com')
print(r)
The function above failed to fetch the content of http://www.google.com when I set it inside the public subnet in a VPC.
Here's the error message:
"errorMessage": "HTTPConnectionPool(host='www.google.com', port=80):
Max retries exceeded with url: / (Caused by
NewConnectionError(': Failed to establish a new connection: [Errno 110]
Connection timed out',))", "errorType": "ConnectionError",
I don't understand why.
The route table of the public subnet looks like this:
The GET request to http://www.google.com should match igw-XXXXXXXXX target. Why can't the internet-gateway(igw) deliver the request to http://www.google.com and get back the website content?
This article says that I must set the lambda function inside the private subnet in order to have internet access.
If your Lambda function needs to access private VPC resources (for
example, an Amazon RDS DB instance or Amazon EC2 instance), you must
associate the function with a VPC. If your function also requires
internet access (for example, to reach a public AWS service endpoint),
your function must use a NAT gateway or instance.
But it doesn't explain why I can't set the lambda function inside the public subnet.
Lambda functions connected to a VPC public subnet cannot typically access the internet.
To access the internet from a public subnet you need a public IP or you need to route via a NAT that itself has a public IP. You also need an Internet Gateway (IGW). However:
Lambda functions do not, and cannot, have public IP addresses, and
the default route target in a VPC public subnet is the IGW, not a NAT
So, because the Lambda function only has a private IP and its traffic is routed to the IGW rather than to a NAT, all packets to the internet from the Lambda function will be dropped at the IGW.
Should I Configure my Lambda Function for VPC Access?
If your Lambda function does not need to reach private resources inside your VPC (e.g. an RDS database or Elasticsearch cluster) then do not configure the Lambda function to connect to the VPC.
If your Lambda function does need to reach private resources inside your VPC, then configure the Lambda function to connect to private subnets (and only private subnets).
NAT or Not?
If the Lambda function only needs access to resources in the VPC (e.g. an RDS database in a private subnet) then you don't need to route through NAT.
If the Lambda function only needs access to resources in the VPC and access to AWS services that are all available via private VPC Endpoint then you don't need to route through NAT. Use VPC Endpoints.
If your Lambda function needs to reach endpoints on the internet then ensure a default route from the Lambda function's private subnets to a NAT instance or NAT Gateway in a public subnet. And configure an IGW, if needed, without which internet access is not possible.
Be aware that NAT gateway charges per hour and per GB processed so it's worth understanding how to reduce data transfer costs for NAT gateway.
Best Practices
When configuring Lambda functions for VPC access, it is an HA best practice to configure multiple (private) subnets across different Availability Zones (AZs).
Intermittent Connectivity
Be sure that all the subnets you configure for your Lambda function are private subnets. It is a common mistake to configure, for example, 1 private subnet and 1 public subnet. This will result in your Lambda function working OK sometimes and failing at other times without any obvious cause.
For example, the Lambda function may succeed 5 times in a row, and then fail with a timeout (being unable to access some internet resource or AWS service). This happens because the first launch was in a private subnet, launches 2-5 reused the same Lambda function execution environment in the same private subnet (the so-called "warm start"), and then launch 6 was a "cold start" where the AWS Lambda service deployed the Lambda function in a public subnet where the Lambda function has no route to the internet.
You can make a lambda function access the public internet from within your VPC. Solution A is the actual answer, Solution B is a more elegant alternative solution.
Solution A - Lambda in VPC + Public IP associated with ENI
For accessing resources external to AWS such as Google API (like OP's example) you do need a Public IP. For other cases like RDS or S3 you don't need a Public IP, you can use a VPC Endpoint, so communication between your Lambda and the desired AWS Service doesn't leave AWS network.
By default some AWS Services are indeed reached via public internet, but it doesn't have to be.
Now if you want an actual external resource (e.g. google), you need to assign Elastic Public IPs to the Network Interfaces for each subnet linked to your lambda. First let's figure which subnets and security groups are linked to your lambda:
Next, go to EC2 Service, find the Public IPs menu under Network & Security. Allocate one IP for each subnet (in the example above there are two subnets).
Go to Network Interfaces menu, find the network interfaces attached to your lambda (same subnet and security group).
Associate the Public IPs in the actions menu for each one:
That's it, now your Lambda can reach out to public internet.
[EDIT]
Someone was concerned about Solution's A scalability issues saying each lambda instance has a new network interface but they missed this from AWS Docs:
"Multiple Lambda functions can share a network interface, if the functions share the same subnet and security group"
So whatever scalability issues you may face has nothing to do with this solution and how Lambda uses ENI, you'd face the same issues using EC2, ECS, EKS, not just Lambda.
Solution B - Decompose into multiple Lambdas
Requiring access both to external resources and VPC resources would seem like too much responsibility for a single function. You may want to rethink your design and decompose your single lambda function into at least two lambda functions:
Lambda A goes to external resources (e.g. Google API), fetches whatever data you need, add to SQS. No need to attach to VPC, no need to manually associate Elastic Public IP to ENI.
Lambda B processes the message from SQS, stores results to a storage (db, s3, efs, another queue, etc). This one lives within your VPC, and don't need external access.
This way seems more scalable, more secure, each individual lambda is less complex and more maintainable, the architecture looks better overall.
Of course life is not always rainbows and butterflies, so Solution A is good and scalable enough, but improving the architecture is even better.
I am trying to setup a nice and secure VPC for my lambda and RDS work. Essentially, I want my lambda to hit a site, get some data, and shove it into a database.
In isolation the parts all work. However the second I go to harden everything it all falls apart. Here is what I do:
Disable "Publically Available" from the RDS instance
Change the RDS instance to only accept connections from inside the VPC using the security group
Associate the lambda with a VPC (this kills the internet access)
Following this tutorial I created a NAT gateway, deleted the internet gateway from the VPC subnet, and replaced it with the NAT. Now, as expected, nothing can talk inbound, but things can talk outbound.
At this point I knew I needed a bastion instance, so I fired up an EC2 instance.
The EC2 instance is set to the same subnet the RDS and Lambda are on, and unfortunately this means that I have a problem - the NAT gateway is currently soaking up all the traffic via 0.0.0.0/0, which means there's no room for the internet gateway. Without the internet gateway I (obviously) can't SSH into my bastion instance so I can jump to access my RDS database.
How can I configure this all correctly? My guess is that I need to split the subnet up somehow and make a private and public subnet, the public having the bastion and internet gateway in it. However, I'm not sure how this will all work so the bastion instance can still properly jump to the RDS.
I'm really quite new to setting up AWS services so I'm hoping I didn't mess anything up long the way.
Following this tutorial I created a NAT gateway, deleted the internet
gateway from the VPC subnet, and replaced it with the NAT. Now, as
expected, nothing can talk inbound, but things can talk outbound.
Short Answer
The short answer is you shouldn't have "Killed the Internet Gateway"; thats not a step in the link you provided :) Leave the internet gateway as is in your current subnet. You're going to need a public subnet and the one that was routing 0.0.0.0/0 to IGW is an example of one you can could use.
The work involved is placing your NAT gateway in the Public Subnet, placing your bastion host in the public subnet, placing your lambda function in the private subnet, routing traffic in the private subnet to the NAT gateway in the public subnet, and providing your lambda function with access to your security group by putting it in its own lambda security group and "white listing" the lambda security group in the inbound rules for the security group protecting your database.
Background
Below I have an expanded answer providing background as to public/private subnets, granting internet access to private subnets, and allowing lambda access through security groups. If you don't feel like reading the background then jump to very end where I give a bullet point summary of the steps you'll need.
Public Subnet
A public subnet is one in which traffic originating outside your VPC, or destined for a target outside your VPC (internet), is routed through an internet gateway (IGW). AWS gives you initial default public subnets configured this way; you can identify them in the console by looking at their route table and seeing that under "destination" you find "0.0.0.0/0" targeting an IGW. This means a public subnet is more of a design pattern for "internet accessible" subnet made possible by simply configuring its default route to point to an IGW. If you wish to create a new public subnet you can create a new route table as well that point internet traffic at an IGW and link that route table to your new subnet. This is fairly easy in the console.
Private Subnet
A private subnet is a subnet with no IGW and not directly reachable from the internet, meaning you cannot connect to a public IP address of a system on a private subnet. With the exception of the AWS pre-configured default subnets, this is how new subnets your manually create are setup, as black boxes till you specify otherwise.
Granting Internet Access to Private Subnet
When you want things in your private subnet to be able to reach out to external internet services you can do this by using an intermediary known as a NAT gateway. Configure a route table the same as in the public subnet with the only difference being traffic destined for 0.0.0.0/0(Internet) you target for a NAT gateway sitting inside the public subnet. That last part is critical. Your NAT gateway needs to be in the public subnet but your private subnet is using it as the target for external traffic.
Security Group Access for Lambda
One simple way to allow your lambda function through your security group/firewall is to create a security group just for your lambda function and configure the security group protecting your RDS so that it allows traffic from the lambda security group.
In other words, in security group settings you don't have to specify just IP addresses as sources, you can specify other security groups and this is a pretty neat way of grouping items without having to know their IP address. Your lambda functions can run in the "Lambda Security Group" and anything protected by a security group that you want them to access can be configured to accept traffic from the "Lambda Security Group". Just make sure you actually associate your lambda function with the lambda security group as well as place it in the private subnet.
Lambda VPC Steps in a Nutshetll
Create a new NAT gateway and place it in the public subnet. This
point is important, the NAT gateway goes in the public subnet ( a
subnet whose route table routes 0.0.0.0/0 to an IGW)
Create a new subnet, you can call it Private-Lambda-Subnet. Unlike
the
default pre-configured subnets AWS gives you, this new subnet is
immediately private out of the box.
Create a new route table and link it to your Private-Lambda-Subnet
In the new route table for your private subnet add an entry that
routes 0.0.0.0/0 to a target of the NAT gateway. This is how your
private subnet will indirectly access the internet, by forwarding
traffic to the NAT which will then forward it to the IGW.
Your bastion host and anything else you want to be be publicly
accessible will need to be in the public subnet. This is probably
where you already have your RDS instances, which is fine if they are
firewalled/security group protected.
Create a new security group for your lambda function(s). You can
call it LambdaSecurityGroup.
Configure the inbound rules of your RDS guarding security group to
allow traffic from the LambdaSecurityGroup. This is possible because
you can use other security groups as sources in the firewall
settings, not just ip addresses.
You need a public subnet (default route is the Internet Gateway) and a private subnet (default route is the NAT Gateway). The NAT Gateway, itself, goes on the public subnet, so that it can access the Internet on behalf of the other subnets for which it is providing services. The bastion also goes on the public subnet, but Lambda and RDS go on the private subnet.
Anything can talk to anything on any subnet within a VPC as long as security groups allow it (and Network ACLs, but don't change these unless you have a specific reason to -- if you aren't sure, then the default settings are sufficient).
I started to mess around with aws a couple days ago and I have a question about VPCs.
I'm considering two options to make "private" a subnet and I want to know your opinion:
OPTION 1:
A subnet with an ig attached (a public subnet really) with an ACL to only allow traffic from the VPC.
Pros: cheaper!
Cons: public ip for each instance (is this a real problem?), different public Ip for each instance (a kind of messy stuff for ip's whitelists)
OPTION 2:
A subnet without an ig attached using a Nat gateway.
Pros: more secure?, same public Ip for each instance.
Cons: more expensive.
I consider the Option 2 the best technical solution, only a bit more expensive. But, do you consider the option 1 a bad practice? Has more pitfalls that I'm not seeing?
It appears that your requirements are:
Run some Amazon EC2 instances
Keep them "private" from the Internet (no direct access from the Internet to the instances)
Allow the instances to access the Internet (eg to download updates, or to communicate with AWS API endpoints)
Your Option 1 is a public subnet, made private by modifying the Network ACLs to allow traffic only within the subnet. You are giving public IP addresses to the instances, presumably so they can access the Internet. This will not work. The Network ACLs that limit access to the VPC will also block traffic from the instances to the Internet.
Your Option 2 is a private subnet, with instances accessing the Internet via a NAT gateway. There is no need to assign public IP addresses to these instances, since they are in a private subnet. Yes, this will work but, as you note, there is the extra expense of the NAT Gateway (or a NAT instance).
An alternative is to use a Public Subnet with Security Groups:
Use a public subnet and give instances auto-assigned public IP addresses. This will allow them to access the Internet. (Note: This is different to an Elastic IP Address, which is limited to 5 per region per AWS account.)
Use a Security Group on each instance to block Inbound access. The instances will still be able to initiate outbound access to the Internet (and responses will come back because Security Groups are stateful)
See documentation: Amazon EC2 Security Groups for Linux Instances
Of course, since the instances now block inbound traffic, you will need a way to connect to them. This is typically done by launching a Bastion Server (Jump Box) in your public subnet. You can connect to the Bastion Server via public IP address, then connect to the private servers via their private IP address. To allow this, configure the Security Group on the private instances to permit inbound traffic from the IP range of the VPC, or from the Security Group associated with the Bastion Server.