I'm trying to set up Load balancing to my cluster that I created via container engine but I keep getting 502 Server Error.
Here is what I do.
I make sure that my service really runs taking this ip in kubernetes dashboard
I have only one instance group. So this belongs cluster
80 port is open for every instance in Firewall rules
I created a load balancer with this configuration. As you can see I set the only instance group I got, also I set up health check. The health check is http endoint /api/ping/ that returns HTTP 200 with body "pong". But I keep getting 502 ERROR. I understand it happens because health check doesn't pass. Apperently it's because when load balancer is sending request it's sending directly to instances, instead of to docker containers. If I go by ip of vm isntance where I host my cluster I will get nothing as well.
I want to do it so I could switch traffic between different clusters If I decide to create a new one, also I would be able to balance https traffic.
Maybe it will help someone so I will post the answer.
I created my pods via UI kubernetes using their consructor-form. When I used this form ports were not exposed to the host machine(Even I marked them as External ports). I wrote an yaml config and exposed ports there to host machine. After this balancer started working.
I used this block to expose 80 port to host machine
ports:
- containerPort: 80
hostPort: 80
protocol: TCP
Related
I am trying to host a service on Fargate that exposes a TCP port.
Even this simple example that exposes HTTP on port 80 never becomes healthy on Fargate.
var loadBalancedFargateService = NetworkLoadBalancedFargateService.Builder.create(this, "ServiceSample")
.cluster(fargateCluster)
.publicLoadBalancer(true)
.memoryLimitMiB(1024)
.cpu(512)
.listenerPort(80)
.taskImageOptions(NetworkLoadBalancedTaskImageOptions.builder()
.image(ContainerImage.fromRegistry("amazon/amazon-ecs-sample"))
.containerPort(80)
.build())
.build();
The error I get is:
service dev-shopapi-redis-ServiceSampleService16E525F0-ASe7w3oUlGf9 port 80 is unhealthy in target-group dev-sh-Servi-EFOUJ7LG0YPP due to (reason Health checks failed).
My intention is to expose another service with a TCP protocol and this is a simplified version that exposes HTTP.
What I am doing wrong?
Try these troubleshooting steps:
If your container is mapped to port 80, confirm that your container security group allows inbound traffic on port 80 for the
load balancer.
Confirm that the ping port value for your load balancer health is configured correctly. If this port isn't configured correctly, then
your load balancer could de-register the container from itself.
Define a minimum health check grace period.
This instructs the service scheduler to ignore Elastic Load Balancing
health checks for a pre-defined time period after a task has been
instantiated.
Monitor the CPU and memory metrics of the service. For example, high CPU can make your application unresponsive and result in a 502
error.
Check your application logs for application errors.
Check if the ping port and the health check path are configured correctly.
Unlike the ApplicationLoadBalancedFargateService, the NetworkLoadBalancedFargateService does not automatically configure the container port.
So just add the following in the CDK:
loadBalancedEcsService.getService().getConnections().allowFromAnyIpv4( Port.tcp(80) );//80 since the container is listening on port 80
Source: https://aws.amazon.com/premiumsupport/knowledge-center/ecs-fargate-health-check-failures/
We have a Docker Swarm cluster that utilizes Traefik to take care of routing requests to the correct service in the cluster. Out of the box, this works properly. However, when an AWS Application Load Balancer was integrated into the infrastructure (to prepare for AWS Web Application Firewall implementation), services became unreachable. See below:
The idea is that ALB requests go through one of the nodes and then goes to Traefik (port 80) and it routes to the correct Docker container.
Security group misconfiguration is not the issue because when running Nginx as a stand-alone container (docker run -p 80:80 nginx) and accessible through port 80, the ALB is able to access the service successfully.
At first we thought it was an IPv6 issue but disabling IPv6 in the host did not solve anything.
Update: Changing the configuration of Traefik like the one below made it work. I'm not quite sure why this is needed when using ALB.
ports:
- target: 80
published: 80
protocol: tcp
mode: host
hey #jackeblagare do you mind share your entire traefik configuration? we are trying to have the same setup
Route53 entry which point to the ALB DNS + Docker Swarm cluster + traefik / services but it seems that we need to change something because with this current setup the traefik dont receive any traffic and we got time out
if we use NLB with just TCP Listener/HC's work perfect.
I dont think that we need to open any ports because as test purposes we are using a security group that has all ports open ( i know that's not recommended but it's on a sandbox environment )
thanks,
I create ECS service and it runs 1 ecs instance and I can see the instance is registered as a target of the load balancer.
Now I trigger a Auto Scaling Group (by just incrementing desired instance count) to launch a new instance.
The instance is launched and added to the ECS cluster. (I can see it on ECS instances tab)
But the instance is not added to the ALB target. (I expect to see 2 instances in the following image, but I only see 1)
I can edit AutoScalingGroup 's target group like the following
Then I see the following .
But the health check fails. It seems the 80 port is not reachable.
Although I have port 80 open for public in the security group for the instance. (Also, instance created from ecs service uses dynamic port mapping but instance created by ALS does not)
So AutoScalingGroup can launch new instance but my load balancer never gives traffic to the new instance.
I did try https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-unhealthy-checks-ecs/?nc1=h_ls and it shows I can connect to port 80 from host to the docker container by something like curl -v http://${IPADDR}/health.
So it must be the case that there's something wrong with host port 80 (load balancer can't connect to it).
But it is also the case the security group setting is not wrong, because the working instance and this non working instance is using the same SG.
Edit
Because I used dynamic mapping, my webserver is running on some random port.
As you can see the instance started by ecs service has registered itself to target group with random port.
However instance started by ALB has registered itself to target group with port 80.
The instance will not be added to the target group if it's not healthy. So you need to fix the health check first.
From your first instance, your mapped port is 32769 so I assume if this is the same target group and if it is the same application then the port in new instance should be 32769.
When you curl the IP endpoint curl -I -v http://${IPADDR}/health. is the HTTP status code was 200, if it is 200 then it should be healthy if it's not 200 then update the backend http-status code or you can update health check HTTP status code.
I assume that you are also running ECS in both instances, so ECS create target group against each ECS services, are you running some mix services that you need target group in AS group? if you are running dynamic port then remove the health check path to traffic port.
Now if we look the offical possible causes for 502 bad Gateway
Dynamic port mapping is a feature of container instance in Amazon Elastic Container Service (Amazon ECS)
Dynamic port mapping with an Application Load Balancer makes it easier
to run multiple tasks on the same Amazon ECS service on an Amazon ECS
cluster.
With the Classic Load Balancer, you must statically map port numbers
on a container instance. The Classic Load Balancer does not allow you
to run multiple copies of a task on the same instance because the
ports conflict. An Application Load Balancer uses dynamic port mapping
so that you can run multiple tasks from a single service on the same
container instance.
Your created target group will not work with dynamic port, you have to bind the target group with ECS services.
dynamic-port-mapping-ecs
HTTP 502: Bad Gateway
Possible causes:
The load balancer received a TCP RST from the target when attempting to establish a connection.
The load balancer received an unexpected response from the target, such as "ICMP Destination unreachable (Host unreachable)", when attempting to establish a connection. Check whether traffic is allowed from the load balancer subnets to the targets on the target port.
The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target. Check whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer.
The target response is malformed or contains HTTP headers that are not valid.
The load balancer encountered an SSL handshake error or SSL handshake timeout (10 seconds) when connecting to a target.
The deregistration delay period elapsed for a request being handled by a target that was deregistered. Increase the delay period so that lengthy operations can complete.
http-502-issues
It seems you know the root cause, which is that port 80 is failing the health check and thats why it is never added to ALB. Here is what you can try
First, check that your service is listening on port 80 on the new host. You can use command like netcat
nv -v localhost 80
Once you know that the service is listening, the recommended way to allow your ALB to connect to your host is to add a Security group inbound rule for your instance to allow traffic from your ALB security group on port 80
I have a EC2 cluster with just one EC2 instance, where two services are running:
api1, listening at port 8080
api2, listening at port 9090
If I make requests against EC2 instance and those ports, both APIs work fine.
Now, I want to create a load balancer so I can make requests against http://{load_balancer_ip}/api1 and http://{load_balancer_ip}/api2, but I'm not able to.
I have created two target groups, both with just one instance (the only one I have)
TargetGroup1: Port 8080 and the EC2 instance registered on port 8080
TargetGroup2: Port 9090 and the EC2 instance registered on port 9090
Then, I have created a load balancer with one listener on port 80 and these two path rules:
When /api1, forward to TargetGroup1
When /api2, forward to TargetGroup2
When I make requests against http://{load_balancer_ip}/api1 or http://{load_balancer_ip}/api2 nothing happens; I don't get any response.
What am I missing?
Ok, I found what's happening thanks to this question's first comment:
AWS Application Load Balancer (ALB) path based routing not functioning as expected
Load balancer is not rewriting the URL and my APIs are listening at /, but load balancer is redirecting all the path /api1.
Solved!
(I couldn't mark it as duplicated because question above does not have any accepted answer)
I'm trying to get a single instance of beanstalk up and running. I don't see anyway to forward 443 to 80 without there being a loadbalancer. How do I get this to work? I can no longer access the application when it has been converted to a single instance.
When using single instance the requests will hit the environments EC2 instances immediately since there is no load balancer in between. This means that the web server must be listening to 443 if you have any intention of providing a responses to requests on that port.
I'm assuming you want HTTPS enabled on this single instance environment, this can be configured using the Resources key in your environment configuration as explained here: Configuring Your Application to Terminate HTTPS Connections at the Instance