My ELB health check fails all the time but cannot figure it why (502 bad gateway).
I have a cluster (ECS) with a service that runs at least one task (Fargate) which is a Node API listening on port 3000 & 3001 (3000 for http & 3001 for https since I cannot use port below 1024).
I have an Elastic Load Balancer (application) that is listening on port 80. It forwards the trafic on a target group with protocol port 3000.
This target group has as target type: ip address since I use fargate and not EC2 for my tasks.
So when a task is turning on, I correctly see the private IP of the task registering into the target group.
My health route is server_ip_address/health and it returns a classic 200 status code. This route works well because I tried it directly from the public ip address of the task (quickly before it stopped because of the health check failing) and it returns a 200. I also tried it through the ELB dns name (so my-elb.eu-west-1.elb.amazonaws.com/health) and it worked well as well so I don't understand why the health check fail.
Anyone know what I missed ?
In the screenshot of your targets in the target group it is showing the port as 80, this means that the load balancer (and health check) will be attempting to connect to the Fargate container on port 80.
You mentioned that it should be served from port 3000, therefore you will need to ensure that the target group is listening on port 3000 instead. Once this is in place, assuming that the security group of the host allows inbound access the 502 error should go away.
To be clear the listener port is what port the client connects to, whereas the target port is the port the load balancer connects to your target on.
Related
I am trying to host a service on Fargate that exposes a TCP port.
Even this simple example that exposes HTTP on port 80 never becomes healthy on Fargate.
var loadBalancedFargateService = NetworkLoadBalancedFargateService.Builder.create(this, "ServiceSample")
.cluster(fargateCluster)
.publicLoadBalancer(true)
.memoryLimitMiB(1024)
.cpu(512)
.listenerPort(80)
.taskImageOptions(NetworkLoadBalancedTaskImageOptions.builder()
.image(ContainerImage.fromRegistry("amazon/amazon-ecs-sample"))
.containerPort(80)
.build())
.build();
The error I get is:
service dev-shopapi-redis-ServiceSampleService16E525F0-ASe7w3oUlGf9 port 80 is unhealthy in target-group dev-sh-Servi-EFOUJ7LG0YPP due to (reason Health checks failed).
My intention is to expose another service with a TCP protocol and this is a simplified version that exposes HTTP.
What I am doing wrong?
Try these troubleshooting steps:
If your container is mapped to port 80, confirm that your container security group allows inbound traffic on port 80 for the
load balancer.
Confirm that the ping port value for your load balancer health is configured correctly. If this port isn't configured correctly, then
your load balancer could de-register the container from itself.
Define a minimum health check grace period.
This instructs the service scheduler to ignore Elastic Load Balancing
health checks for a pre-defined time period after a task has been
instantiated.
Monitor the CPU and memory metrics of the service. For example, high CPU can make your application unresponsive and result in a 502
error.
Check your application logs for application errors.
Check if the ping port and the health check path are configured correctly.
Unlike the ApplicationLoadBalancedFargateService, the NetworkLoadBalancedFargateService does not automatically configure the container port.
So just add the following in the CDK:
loadBalancedEcsService.getService().getConnections().allowFromAnyIpv4( Port.tcp(80) );//80 since the container is listening on port 80
Source: https://aws.amazon.com/premiumsupport/knowledge-center/ecs-fargate-health-check-failures/
I create ECS service and it runs 1 ecs instance and I can see the instance is registered as a target of the load balancer.
Now I trigger a Auto Scaling Group (by just incrementing desired instance count) to launch a new instance.
The instance is launched and added to the ECS cluster. (I can see it on ECS instances tab)
But the instance is not added to the ALB target. (I expect to see 2 instances in the following image, but I only see 1)
I can edit AutoScalingGroup 's target group like the following
Then I see the following .
But the health check fails. It seems the 80 port is not reachable.
Although I have port 80 open for public in the security group for the instance. (Also, instance created from ecs service uses dynamic port mapping but instance created by ALS does not)
So AutoScalingGroup can launch new instance but my load balancer never gives traffic to the new instance.
I did try https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-unhealthy-checks-ecs/?nc1=h_ls and it shows I can connect to port 80 from host to the docker container by something like curl -v http://${IPADDR}/health.
So it must be the case that there's something wrong with host port 80 (load balancer can't connect to it).
But it is also the case the security group setting is not wrong, because the working instance and this non working instance is using the same SG.
Edit
Because I used dynamic mapping, my webserver is running on some random port.
As you can see the instance started by ecs service has registered itself to target group with random port.
However instance started by ALB has registered itself to target group with port 80.
The instance will not be added to the target group if it's not healthy. So you need to fix the health check first.
From your first instance, your mapped port is 32769 so I assume if this is the same target group and if it is the same application then the port in new instance should be 32769.
When you curl the IP endpoint curl -I -v http://${IPADDR}/health. is the HTTP status code was 200, if it is 200 then it should be healthy if it's not 200 then update the backend http-status code or you can update health check HTTP status code.
I assume that you are also running ECS in both instances, so ECS create target group against each ECS services, are you running some mix services that you need target group in AS group? if you are running dynamic port then remove the health check path to traffic port.
Now if we look the offical possible causes for 502 bad Gateway
Dynamic port mapping is a feature of container instance in Amazon Elastic Container Service (Amazon ECS)
Dynamic port mapping with an Application Load Balancer makes it easier
to run multiple tasks on the same Amazon ECS service on an Amazon ECS
cluster.
With the Classic Load Balancer, you must statically map port numbers
on a container instance. The Classic Load Balancer does not allow you
to run multiple copies of a task on the same instance because the
ports conflict. An Application Load Balancer uses dynamic port mapping
so that you can run multiple tasks from a single service on the same
container instance.
Your created target group will not work with dynamic port, you have to bind the target group with ECS services.
dynamic-port-mapping-ecs
HTTP 502: Bad Gateway
Possible causes:
The load balancer received a TCP RST from the target when attempting to establish a connection.
The load balancer received an unexpected response from the target, such as "ICMP Destination unreachable (Host unreachable)", when attempting to establish a connection. Check whether traffic is allowed from the load balancer subnets to the targets on the target port.
The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target. Check whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer.
The target response is malformed or contains HTTP headers that are not valid.
The load balancer encountered an SSL handshake error or SSL handshake timeout (10 seconds) when connecting to a target.
The deregistration delay period elapsed for a request being handled by a target that was deregistered. Increase the delay period so that lengthy operations can complete.
http-502-issues
It seems you know the root cause, which is that port 80 is failing the health check and thats why it is never added to ALB. Here is what you can try
First, check that your service is listening on port 80 on the new host. You can use command like netcat
nv -v localhost 80
Once you know that the service is listening, the recommended way to allow your ALB to connect to your host is to add a Security group inbound rule for your instance to allow traffic from your ALB security group on port 80
I have a problem with my deployment in ECS.
I try to deploy 4 instances of 2 docker images on 2 EC2 instances with an ALB in front.
So in my tasks definitions, I use the dynamic port mapping (2 Nginx on container port 80).
This creates a trouble in the Health check of my target group.
In fact, for each instance, I have a health check on the dynamics port (that is ok) and on the container port (80).
So the dynamic port says, it's ok. And the container port, logically says unhealthy ...
(Like in my screenshot)
So can you help me ton find why I have that type of error (this error make my server terminated each 5 minutes....)
Thanks in advance for your help :D
So to me it looks like you aren't completely using dynamic port mapping. For dynamic port mapping you have
Client -> ALB (port 80) -> EC2 host (dynamic port) -> container (dynamic port) -> nginx (port 80)
None of your healthchecks should be hitting port 80 since the only thing that uses port 80 is external connection into your application and nginx (but it is mapped to a different port). For ALB healthchecks all you really need is a path to hit and the port will default to the port that it connects on.
See the host port mapping in this doc: http://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_PortMapping.html
ALB Health Check Docs: http://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html
I found the solution with the AWS support.
So there are two problems here:
To disable the health which kill the EC2 instance, go to the auto scaling group and switch health check to "EC2" type
To remove the health check on port 80, go to the auto scaling group, and under "Target groups" section, removes the target groups managed by ECS
I'm trying to setup an ALB which listens on port 443, load balancing to ECS Docker containers on random ports, lets say I have 2 container instances of the same task definition, listening on port 30000 and 30001.
When I try to create a target group in the AWS EC2 Management console, there's a "port" input field with 1-65535 range.
What number should I put there?
And when I try to create a new service in the AWS EC2 Container Service console, together with a new target group to connect to a existing ALB, there's no input field for a target group "port".
After it's created, navigating to the EC2 console, the new target group has port "80".
Do I have to listen on port 80?
But the health check happens against the "traffic port", which is the container port, 30000 and 30001, so what's the point?
Turns out, when combined with ECS, the target group's port doesn't mean anything. You don't need to listen on that port.
I ran into this situation myself at work. I noticed the target group port and the port of the registered instance were different. I've typically set them up to be the same thing so wondered what this was all about which led me to this thread. I couldn't find a good answer on AWS docs, but found this in the Terraform docs for aws_lb_target_group resource:
port - (Optional, Forces new resource) The port on which targets receive traffic, unless overridden when registering a specific target.
So, I guess it's just the default port used unless you override it. Makes sense.
I think what he's referring to is the health checks. If your ELB is listening on port 443 but your target group is set for port 80, then every health check for the target group will attempt a request on port 80 and get redirected to port 443 by the load balancer. This results in a 301 code, which is considered unhealthy. Only 200 codes are supposed to be considered healthy. At that point you either have all unhealthy targets all the time or you add 301 to the list of healthy codes which defeats the whole point in health checks because it will always return a 301 for port 80. You might as well just match the ports.
By default, a load balancer routes requests to its targets using the protocol and port number that you specified when you created the target group. Alternatively, you can override the port used for routing traffic to a target when you register it with the target group.
The port in the target group is used in conjunction with auto-scaling groups and if you ever plan to use those you want to use the right port from the start. Why? Because you can not change it after the target group has been created and auto-scaling will simply not work if you set the port wrong.
I have an Amazon Elastic Load Balancer which has a health checker. It attempts to connect to my Logstash instance running on some_ec2_instance:5000.
The ELB health check attempts to open a tcp connection at some_ec2_instance:5000. However, it never passes this health test. I can manually connect to the the ec2 instance and check that Logstash is running and it is indeed operational. I can also telnet localhost 5000 into it without any problems.
In addition my security group allows input/outputs on port 5000 so I don't think that is a problem.
Does anyone have suggestions for how to enable the ELB to pass the health check? is there a /ping path or a plugin which will enable access to such path?
Assuming you are running elasticsearch and logstash on the same hosts, open port 9200 to your ELB and use a http health check on that port.
Running ELB health checks against port 5000 (logstash itself) overwhelms the port.