I am trying to reach two private subnet EC2 instances from ELB in public subnet. But the instances are always showing as unhealthy. I tried lot of options but nothing worked.
Here is the configuration I tried:
1.Created VPC
DNS resolution: Enabled
DNS hostnames: Disabled
Created 2 Public subnets and 2 Private Subnets. The only difference between these two is the Route table.
Auto-assign public IPv4 address is set to "No" in Public Subnet
I have added Internet gateway as a route in Route table of public subnet.
Public subnet - Route Table
Private subnet - Route table
Created Two EC2 instances in Private subnet . Userdata set to apache webserver
The Security group of the instances
Created an ALB in Public subnet
ALB Security group config is :
Have also added default security group in ALB:
Target group is showing unhealthy for the private EC2 instances
Any help on pointing out why this fails?
Either the EC2 instances are not receiving the request, or they are receiving the request but aren't responding with a 'healthy' response.
Test that the instances are responding to requests
You could launch an EC2 instance in the public subnet, SSH to it and then send requests directly to the EC2 instances to see if they respond (eg curl http://private-ip-address).
If they respond correctly, then the instances themselves are fine and the problem lies with the Load Balancer.
Check the Load Balancer security group
You have shown the Inbound rules for the Load Balancer, but you should also check that the Outbound rules permit traffic to exit the Load Balancer and go to the EC2 instances.
First and foremost - check the apache logs and make sure you see incoming health check samples.
If you don't see them at all - you have a network connectivity issue.
If you do see them - make sure they return 200 code and that it matches the target group's health check configuration.
Related
This is a follow up to my previous question. I'm successfully able to access the public IP over fargate. However, after trying to attach Application load balancer, I cannot access application over public DNS. The registered target always show unhealthy status with 502 error.
Sharing my configuration settings-
VPC
Route Table Public Subnet
Route Table public subnet config
Route Table private subnet
Route Table private subnet config
ACL subnet associations
ACL inbound rules
ACL outbound rules
security inbound rules
security outbound rules
internet gateway
nat gateway
target group
target group health status
task details
service
ALB settings
ALB listeners
Target group with 8081 port
Based on the comments.
There were two issues identified:
Incorrect port on the target group. It was 80 instead of 8081.
Healthy threshold timeout was too short (5 seconds). Increasing it seemed to solved the issue.
If I create an ELB and try to attach the private subnet instances, my health check fails: OutOfService.
Question 1 : Can I get a internal / private IP(not IP but dns name) for Load Balancing. i.e not accessible to the internet?
Question 2 : If I have a public dnsname for my Application Load Balancer. How do I attach EC2 instances that are in my private subnet without an Elastic IP(aren't internet accessible).
I am looking for the best approach. Should we have-
ELB --> public subnet EC2 instances (proxy configuration- */* [private_ip]:[port]/* ) ---> Service from Private Subnet/EC2 instance with health checks here.
The traditional architecture is:
Elastic Load Balancer in public subnet
Amazon EC2 instances in private subnet
Security group on Load Balancer permitting port 80 & 443 from 0.0.0.0/0
Security group on instances permitting port 80 from the Load Balancer security group
An Amazon Route 53 Hosted Zone with a CNAME record set pointing to the DNS Name of the Load Balancer
If your instances are failing the Load Balancer health check, check the following:
The instances should have a security group permitting inbound access from the Load Balancer
The Load Balancer health check should be configured with a path to a web page to use for the health check
The instances should have a functioning web server that is responding to the health checks
If you are using an ELB, I would recommend using Auto Scaling Group to put instances in various AZ/subnets.
Look at this tutorial. The benefit of having an ASG that you can optionally also have scale in/our policies.
The archiceture that you have described ELB (Assuming this is public) -> public EC2 instances (as Proxies) -> Private EC2 instances.
Rather you can have your Private EC2 instances fronted with a ELB. Not sure if you really need the public EC2 instances.
If your health check is failing you need to ensure that:
a. You are creating the ELB in the right VPC.
b. The ELB -> EC2 communication is setup correctly by ensuring you have the right Security Groups in place on the EC2 instances.
A highly recommended way would be to have the EC2 instances SG rules setup in such way that they accept traffic only from the SG of the ELB.
Our AWS instances are created in a public subnet availability zone and are not able to connect to the internet and SSH. So all the resources are created on the public subnet and one AZ.
I have developed CF nginx template with single VPC and two public subnets, butsecond public subnet instances are unable to connect network and SSH even though I'm giving public IP of the instance in the browser it is not working.
The main issue is instances which are launching in the second public subnet are unable to connect internet system logs are:
Contact the upstream for the repository and get them to fix the
problem
Reconfigure the base URL/etc.
Disable the repository, so yum won't use it by default
Looking at the scenario generally in order to enable access to or from the Internet for instances in a VPC subnet, you must do the following:
Attach an Internet gateway to your VPC.
Ensure that your subnet's route table points to the Internet gateway.
Ensure that instances in your subnet have a globally unique IP
address (public IPv4 address, Elastic IP address, or IPv6 address).
Ensure that your network access control and security group rules
allow the relevant traffic to flow to and from your instance.
To use an Internet gateway, your subnet's route table must contain a route that directs Internet-bound traffic to the Internet gateway. You can scope the route to all destinations not explicitly known to the route table (0.0.0.0/0 for IPv4 or ::/0 for IPv6).
Kindly Refer this AWS Documentation and see what you are missing , as you must have skipped one of the above mentioned things.
I have an ECS cluster consisting of 2 instances in different AZ. One of the many services I run is a SMTP relay. I want to use a Network Load Balancer in front of this service to easily configure other applications to use the relay.
Once I got everything in place, I faced the following issue:
If the container is present on instance 'A' only instance 'B' is able to access it and vice versa, otherwise it times out. So the Network Load Balancer seems to prevent access to a service that lives on the same instance.
Is there something I'm missing here? Is anyone aware of this and have a workaround?
Update:
When scaling the service to 2 instances it started to work. I now tend to believe it's related to the Availability Zones.
I experienced a similar issue.
Here is my setup:
A VPC spread over 3 AZ.
3 public subnets (one in each AZ)
1 instance in a public subnet in AZ-a
3 private subnets (one in each AZ)
1 NLB spread over the 3 private subnets.
A cluster of ECS instances. 1 instance in each private subnet. (instance-a in AZ-a, instance-b in AZ-b, instance-c in AZ-c)
A service running on each instance ; in total 3 healthy services spread over the 3 private subnets registered to the NLB.
A route 53 Alias record to map "myservice.example.com" to the NLB DNS name.
Below the tests executed:
Query initiated from an instance in the private subnet."
Test1: From instance-a (in AZ-a), query "myservice.example.com".
Result1: The query hits the NLB on one of its private IP. If the IP is in the same subnet as instance-a, the query will time-out. If the IP is in a different subnet, the query will succeed.
Test2: Same as Test1 but query from instance-b (in AZ-b).
Result2: The query hits the NLB on one of its private IP. If the IP is in the same subnet as instance-b, the query will time-out. If the IP is in a different subnet, the query will succeed.
Similar result with a query initiated from instance-c.
Query initiated from an instance in a public subnet AZ-a
Test3: From the instance in public subnet in AZ-a, query "myservice.example.com".
Result3: The query hits the NLB on one of its private IP. The query always succeeds, regardless of which private IP was hit.
Query initiated from an extra instance (instance-a2) in private subnet AZ-a
Test4: I have launched an additional instance (instance-a2) in the private subnet in AZ-a. Then, from instance-a2, query to "myservice.example.com". IMPORTANT: This instance does not run any service an therefore can never be selected by the NLB to route any request.
Result4: The query succeeds all the time! Even when hitting a target that is in the private subnet A (same subnet as instance-a2).
Conclusions:
With Test1 and Test2, I could experience the same issue as Laurent Jalber Simard when querying from an instance that was hosting the target service.
Per as Test3, the issue does not seem to come from requests coming from the same AZ as the target service.
With Test4, it appears that the issue cannot be reproduced if the query comes from an instance that is different from the instance hosting the target service ; even if they are in the same subnet.
Therefore, my conclusion so far is that the NLB will timeout if the source ip of the request and the destination ip of the target selected by the NLB is the same.
I couldn't find this issue/limitation documented in AWS NLB docs and so far nothing comes up in a Google search.
Is there anybody outhere reaching to the same conclusion?
Solution
If you would like to keep containers on the same instance and use NLB you need to use "awsvpc" networkMode in your task definition and change target group type to "ip"(not by instance ID).
Explanation
NLB doesn't support hairpinning of requests. When you register targets by instance ID, the source IP addresses of clients are preserved. When you try to connect to the NLB from the backend a loopback is created and this is not allowed by the NLB as the source and destination address is the same and the connection times out. If an instance is a client of an internal load balancer that is registered by instance ID, the connection succeeds only if the request is routed to a different instance.
Some extra info: https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/
I'm putting the instances behind the aws loadbalancer, I have configured the routable and attached the IGW to it, created the loadbalancer and added this instance to the aws loadbalancer every things work well, the endpoint URL of the AWS loadbalancer able to load the HTTP pages
Now i have removed the IGW from the route table and tested it again, the AWS loadbalalncer endpoint URL not able to load the page, but the instace status shows in AWS loadbalancer as inService
Why the IGW is required when loadbalancer is configured over private subnet, it technically Mean it's a public subnet, which is blocking me to create a NAT inatance
A subnet without a default route pointing to the igw-xxxxxxxx Internet Gateway object is, by definition, a private subnet. If you remove the igw from a public subnet, you now have a private subnet.
Placing an Internet-facing load balancer (ELB) in such a private subnet is incorrect.
It sounds as though you are making a commonly-made -- but incorrect -- assumption that the ELB should be configured in the same subnets as the instances behind it. This is also incorrect.
Provision the ELB in public subnets, without regard to the subnets the instances behind it were placed in.
In summary:
Internet-facing ELB requires a public subnet for placement.
NAT instance requires a public subnet for placement.
The instances that use these services (NAT and ELB) belong in different -- private -- subnets, different subnets than the ELB and NAT instances.
ELB and NAT can be placed together in the same subnets, or separately, as long as the subnets are public (have the IGW as their default route) and are in the same availability zones.
I believe you cannot do anything without the IGW attached to the routing table where the subnet is attached to.
Another way to do this is to spin off a NAT instance(can be found in AWS marketplace) in the public subnet, add it to your private route table where your original instance is on (0.0.0.0/0 - instanceid) all the traffic will be routed through the NAT instance.
Here mainly we have to see two things, provided subnet for the elasticloadbalancer is public or private.
Every VPC should have one IGW to connect to the public, every trafic should go through the IGW only. If VPC connected to the IGW, IGW distribute internet to the all instance which are in that VPC, if route table changed to the particular rt, internet traffic can send to only that instance. Here in ELB instance is in service because both are in the same VPC and can communicate each other means checks the status. This IGW will work main role when we are using the NAT.
Always we provide ip range for the IGW or rt as 0.0.0.0/0, it represents as public.
The following link will explain more : http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html
This small explanation might be helpful for someone.
Let me cover your two questions
the AWS loadbalalncer endpoint URL not able to load the page, but the instace status shows in AWS loadbalancer as inService
This is the default behaviour of load balancer , since internally load balancer and your instance may be in the same VPC so they are able to communicate that is why loadbalancer is showing inService status.
Second question Why the IGW is required when loadbalancer is configured over private subnet, it technically Mean it's a public subnet, which is blocking me to create a NAT inatance
You need IGW if you want to access any resources especially EC2 , Loadbalancer from internet. however if you put your load balancer in private subnet it means IGW is not associated with the subnet having loadbalancer in it and hence this load balancer is not accessible outside your VPC that is reason that you were not able to load your page.
NAT instance is usually used when you want your private subnet instances should be able to initiate request over internet and has nothing to do with normal load balancing setup unless and untill you want dont want to install any updates from internet to your instance.
You are trying to access webpage publicly and removing the route entry IGW from the loadbalncer subnet.
The Subnet without IGW will become Private, Hence you can't access it.
First, a subnet with route table that route traffic via the internet gateway (IGW) is a public subnet. An IGW is required because the subnets created in AWS VPC are internal IPs and as internal IPs are not routable via the internet, traffic to and fro EC2 instances that belongs to an internal IP needs a way to complete these request. This is where an IGW comes into play. The IGW allows your EC2 instance to make outbound request to the internet and allows other user/client to make inbound request to your ec2 instance.
A public subnet are group of IPs (called subnets) in your VPC that allows internet traffic to and fro your ec2 instance. A subnet without an Internet gateway is a private subnet. As you already guess no traffic is allowed in or out.
That said, instances in VPC which are in Private subnet still need to initiate an outbound request to the internet to download software or perform update. In this case you have to create and attach a NAT gateway or NAT instance to the private subnet. NAT Gateway and NAT instance only allow outbound traffic to the internet but not the the other way round. In some cases you might want your production EC2 to be in the private subnet and ELB in public subnet for security reason.
ELB usually belong to the public subnet so it reachable from the internet as is your case as well.
TO answer some of your question - when you deleted the IGW from the route table, your ELB automatically becomes a private subnet as such your web page stops loading.
Also, you could still see the ec2 instance behind the ELB as InService even when you deleted the IGW because the ELB and EC2 instance can communicate via the internal IP as they are in same network or VPC.
The ELB needs a route to the internet in order to send you the response over internet.. As simple as that.
Configure your ELB in public subnet, regardless of where your instances are present.
Basically there are two types of load balancers.
1)Internal
2) External
Internal load balancers are those which are launched in a private subnet which will be accessible only internally by the instances on same vpc of the internal elb
External load balancers are which are accessible over the internet which should be launched in a subnet which has internet gateway attached to it and which has route table configured properly to route the requests.
If you attach an internet gateway to a subnet it becomes a public subnet.Also if you create a load-balancer which you need to be accessible from internet it should be a External load balancer and aws will not allow it launch in a private subnet.The instances are showing in service because its communicating internally using private ip-address.