I'm trying to create an AWS Cloud9 environment to access a DocumentDB server for direct access. However, when creating the Cloud9 environment it takes forever then dies with an error regarding an inability to connect to the functional backing EC2 instance. My VPC has 2 public and private subnets and no matter which subnet I use all of them fail with the same error. I have a web server running on the same VPC under one of the private subnets that can be accessed from the open internet perfectly fine and access my DocumentDB database. I have two NAT gateways, one hooked to each of my public subnets. I also have route tables set up to access the open internet. Is there something that could be obviously affecting the Cloud9 environment's ability to connect to it's EC2 instance?
It turns out that my private subnet route tables were misconfigured; they were forwarding 0.0.0.0/0 to an internet gateway, not to my NAT gateway.
Related
Using Terraform to setup a VPC with two EC2s in private subnets. The setup needs to SSH to the EC2s to install package updates from the Internet and install the application software. To do this there is an IGW and a NAT-GW in a public subnet. Both EC2s can access the Internet at this point as both private subnets are routing to the NAT-GW. Terraform and SSH to the private subnets is done via Client VPN.
One of the EC2s is going to host a web service so a Classic mode Load Balancer is added and configured to target the web server EC2. Using Classic mode because I can't find a way to make Terraform build Application mode LBs. The Load Balancer requires the instance to be using a subnet that routes to the IGW, so it is changed from routing to the NAT-GW, to the IGW. At this point, the Load Balancer comes online with the EC2 responding and public Internet can access the web service using the DNS supplied End Point for the LB.
But now the web server EC2 can no longer access the Internet itself. I can't curl google.com or get package updates.
I would like to find a way to let the EC2 access the Internet from behind the LB and not use CloudFront at this time.
I would like to keep the EC2 in a private subnet because a public subnet causes the EC2 to have a public IP address, and I don't want that.
Looking for a way to make LB work without switching subnets, as that would make the EC web service unavailable when doing updates.
Not wanting any iptables or firewalld tricks. I would really like an AWS solution that is disto agnostic.
A few points/clarifications about the problems you're facing:
Instances on a public subnet do not need a NAT Gateway. They can initiate outbound requests to the internet via IGW. NGW is for allowing outbound IPv4 connections from instances in private subnets.
The load balancer itself needs to be on a public subnet. The instances that the LB will route to do not. They can be in the same subnet or different subnets, public or private, as long as traffic is allowed through security groups.
You can create instances without a public IP, on a public subnet. However, they won't be able to receive or send traffic to the internet.
Terraform supports ALBs. The resource is aws_lb with load_balancer_type set to "application" (this is the default option).
That said, the public-private configuration you want is entirely possible.
Your ALB and NAT Gateway need to be on the public subnet, and EC2 instances on the private subnet.
The private subnet's route table needs to have a route to the NGW, to facilitate outbound connections.
EC2 instances' security group needs to allow traffic from the ALB's security group.
It sounds like you got steps 1 and 2 working, so the connection from ALB to EC2 is what you have to work on. See the documentation page here as well - https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenario2.html
I am currently working on a ETL tool at work (python & bash scripts managed with Airflow basically) and I am asking myself wether I should put my EC2 instance which will run the ETL in a public or private subnet. My instance should have acces to the internet to retrieve data (basically ssh through on-premises instances we have) and should also be able to be accesses through SSH.
However, I don't know if allowing outbound connection to the internet and restrict inbound connection to SSH is enough about security or if I should put the instance in a private subnet and tweaking things to be able to connect to it.
Your ETL instance should be in a private subnet behind a NAT instance.
NAT gateway will give your EC2 private network internet connectivity but still ensure that your EC2 instances are not accessible from the internet. So in order to allow access to internet it has to route traffic through public network which has a Internet gateway attached.
You should put your EC2 instance in a private subnet to prevent hackers from gaining access and stealing your data.
You can learn how to setup NAT gateway here
https://aws.amazon.com/premiumsupport/knowledge-center/nat-gateway-vpc-private-subnet/
As the main purpose of your instance is not to deliver a public service, it'd be more secure in a private subnet and go through a NAT gateway to fetch data from the Internet.
That being said a NAT gateway is expensive, so a common pattern is to use a public subnet/ through an Internet gateway with a deny rule for any incoming traffic. If you don't want your instance exposed to ddos, don't even open ssh and use AWS systems manager to ssh your instance.
Is there anyone who can explain me that situation below ?
I connected to Ec2 intance in private subnet via Bastion Host.Then I created a vpc gateway endpoint to reach s3. I assigned the required role to ec2 and the connection to s3 is done. My question is, how can Ec2 instance in private subnet download something from the internet?
A VPC endpoint is used to access a certain service, in this case S3, over the AWS network instead of over the internet.
If you want your instance to be able to talk to the internet, you'll need to configure a NAT gateway in a public subnet, and you'll need to configure the route table of the private subnet to forward traffic to the NAT gateway.
A simple solution would be:
Put everything in a Public Subnet
Assign a Public IP address to the Amazon EC2 instance
Use Security Groups to secure access on the Amazon EC2 instance (that is, only allow Inbound connections from the Bastion)
The effective result is the same as using a Private Subnet, but the EC2 instance will be able to establish Outbound connections to the Internet (but not Inbound, so it remains secure).
I am trying to run the command “aws lambda list-functions” from an ec2 inside my private subnet that I connected to using a bastion host. The lambda has been deployed to the same VPC and private subnet, but when I try to run the command it times out. They are in the same region, and I have been able to connect to some of my other services within this ec2 but can’t connect to my lambda service endpoint.
I tried connecting from an ec2 inside a public subnet and that worked fine, but can't connect from ec2 inside the private subnet.
You have two options for that:
Setup NAT gateway in a public subnet along with route tables entries directly internet traffic in a private subnet to the NAT.
Setup VPC interface endpoint for lambda so that you can connect to the lambda service without needing internet connectivity.
I've created an EC2 instance inside a public subnet (so that I can access it from my home network) and I have created some Lambda's inside the private subnets of my VPC.
My 1st lambda can freely access the internet (through a NAT Gateway) and do its job. Which is to fetch a file from the internet and upload it to S3, once per day.
My 2nd lambda is supposed to retrieve this file from S3 (which it does without issue) read the file & then upload the data to MySQL running on the EC2 instance. It is unable to connect to the Database (using either the EC2's public or private IP's) and the Cloudwatch logs show that the session times out, making me think this is a networking issue.
I have a 3rd lambda that will also need to interact with the EC2/DB instance.
My security group allows for all incoming traffic from my home network IP, the NAT gateway & the VPC IP range. All outbound traffic is allowed.
I appreciate its not usual to have an EC2/DB set up this way, in a public subnet, but its my preference to interact it with it this way from home using Sequel Pro.
However, is my set up even possible? Eg can my private subnet lambdas interact with a public subnet ec2 instance? if so, does anybody have any ideas how I can make this happen?
It appears that your situation is:
An Amazon EC2 instance running in a public subnet, with MySQL
The EC2 instance has a Security Group allowing all incoming traffic from your home network IP, the NAT gateway and the VPC IP range
An AWS Lambda function connected to a private subnet of the same VPC
A NAT Gateway allowing private subnets to connect to the Internet
The Lambda function is unable to connect with the MySQL database running on the EC2 instance
The normal security configuration for this scenario would be:
A Security Group on the Lambda function (Lambda-SG) that allows all Outbound access (no Inbound required)
A Security Group on the EC2 instance (EC2-SG) that allows inbound access from Lambda-SG on port 3306, plus whatever inbound permissions you want for accessing your instance via SSH, etc.
Given that your Security Group includes "the VPC IP range", this should be sufficient to permit inbound access.
The Lambda function should reference the EC2 instance via its private IP address to keep traffic within the VPC. By default, all subnets within a VPC can communicate with each other unless the Network ACLs have been modified (and they should generally be left at default values).
This means that the only remaining explanation would be that the MySQL database is not accepting traffic from the VPC IP range. (I'm not a MySQL person, but I know that PostgreSQL requires incoming IP ranges to be defined, so this might be true for MySQL too.)
To diagnose what might be happening in your network, I recommend:
Launch another Amazon EC2 instance in the public subnet
Connect to the new instance and try to connect to the MySQL database via the private IP address
If that works, repeat the process but from an EC2 instance in the private subnet. To use this you will need to connect to the 'public' EC2 instance, and from there connect to the 'private' EC2 instance. Then, try and connect to MySQL from that private instance.
These steps will progressively let you identify where the network problem might lie. Let us know what you find!