Recently started using Network Load Balancer which listens on port 80 and forwards traffic to my target group. My autoscaling group is configured to add any new targets to this target group.
However, my application on the target EC2 instances runs on port 8001, not 80. So my targets should register under port 8001 in the target group. The auto-scaling configuration doesn't seem to support that. All new instances created by auto scaling are added as targets with port 80 and there is no way to auto specify which port that should be used instead (8001 for me).
Any ideas how to implement this?
The port definition in the target group is the port definition you're looking for. The port in the target group is the port on which the targets receive traffic. The port on the listener is the port on which the load balancer listens for requests.
So you should set port 80 on the listener and port 8001 on the target group.
What kind of application are you using (web server, application server, ...)? Maybe ALB would be more suitable for you as it works on layer 7 of OSI model, therefore it is able to proccess HTTP headers, for example.
Back to your question; To be able to forward traffic to your EC2 instances, that runs application on port 8001, you have to set port on your target group to 8001. Auto-scaling group knows nothing about what application is running on EC2 it provisions, nor about ports that are used by that application.
So the final flow is like:
LB listens on port 80 and forwards traffic to target group on port 8001. This target group then sends traffic to its targets (your EC2 instances) on port 8001.
I was crying over this for hours and the answers here gave me a clue and finally, I found out what the heck is going on!
The story of ports can get complicated, but let's clarify them.
Four Ports in the story!
It's crucial to know that you are dealing with 4 ports here! So let's name them one by one.
(I am using ECS, but the same applies to anything else that manages your code to ELB.)
P1: LB's "Listening" Port
The port where the Load Balancer receives traffic. Usually 80 or 443.
P2: TG's Port
The port where the Target Group is set to work on. It's baked into the Target Group at the time of creation of TG (as jpact mentioned) and shows in the description. You cannot change it later.
You, however, can set the health-check of the Target Group on a different port, but it doesn't help you have a working set-up.
P3: Container's "exposed" port
This is the port that the container is giving out and expects to receive traffic.
P4: The application's/host's port
This is the port your code (say a Node.js app) is actually listening to.
Want no pain? set P2 = P3
The thing is P2 (TG's) and P3 (container's) can differ. In fact, you will not even face a challenge if they are set to arbitrarily different numbers, initially.
Application <> LB (e.g. in ECS)
When you register a Service's container to a TG, no one asks about ports and it can work well. You just say attach this container to that TG and it automatically pick's the "container's" port (P3) here.
And then if you go to your TG's page you will see it is on the container's port (P3) which is a different port that the TG's (P2), but it works well and who cares!
Here comes Auto Scaling!
The headache begins when you add Auto Scaling with healthcheck!
ASG knows it should create instances on TG. However, it needs to assign a port between EC2 instance and TG. And clearly, ASG doesn't know what container (P3) is going to be in it, so by default it picks TG's port (P2). And this is where the craziness happens!
TL;DR
Set TG's port == Container's port. They can be anything you want, but make them match.
Below the container and to to your app (P4) you can have a mapping. Above TG (on LB) you have another port mapping (P1). But these P2 and P3 MUST match!
Go ahead, create new TG's with the container's port (P3), wire them to above (ELB) and below (Services) and hopefully it will work well!
PS. Apparently, you cannot change a Service's TG! So new Service as well... :)
Related
I have an app hosted on ECS (ec2) network that listens on port 4000 (HTTP) and it's connected to ALB by dynamic port mapping - here everything is working correctly.
ECS service has ALB as a load balancer
But the same app accepts TCP connection (from GPS devices) port 5010. When I try to create a Network load balancer and in the target group specify port 5010 connection doesn't work.
Security groups are ok, ec2 instance SG allows all TCP from all clients (ipv4 and ipv6)
ECS Task (bridge network) port mapping looks like this:
0:5010
0:4000
Of course, if I create a target group with the id that dynamic port mapping gave me, everything works fine, but that's not the case. I want o use a dynamic port feature
Maybe someone knows how to solve that problem?
Thanks in advance for helping me!
My ELB health check fails all the time but cannot figure it why (502 bad gateway).
I have a cluster (ECS) with a service that runs at least one task (Fargate) which is a Node API listening on port 3000 & 3001 (3000 for http & 3001 for https since I cannot use port below 1024).
I have an Elastic Load Balancer (application) that is listening on port 80. It forwards the trafic on a target group with protocol port 3000.
This target group has as target type: ip address since I use fargate and not EC2 for my tasks.
So when a task is turning on, I correctly see the private IP of the task registering into the target group.
My health route is server_ip_address/health and it returns a classic 200 status code. This route works well because I tried it directly from the public ip address of the task (quickly before it stopped because of the health check failing) and it returns a 200. I also tried it through the ELB dns name (so my-elb.eu-west-1.elb.amazonaws.com/health) and it worked well as well so I don't understand why the health check fail.
Anyone know what I missed ?
In the screenshot of your targets in the target group it is showing the port as 80, this means that the load balancer (and health check) will be attempting to connect to the Fargate container on port 80.
You mentioned that it should be served from port 3000, therefore you will need to ensure that the target group is listening on port 3000 instead. Once this is in place, assuming that the security group of the host allows inbound access the 502 error should go away.
To be clear the listener port is what port the client connects to, whereas the target port is the port the load balancer connects to your target on.
I have en ELB with multiple EC2 instances registered in target groups. I am using port a php application which is running properly. It has SSL.
I want to use port 8000 for my node application. What I would like to do is I want to forward my-elb-address:8000 to any-ec2-ip:8000. So when i access the domain attached to ELB witjh port 8000 it would forward that to ec2 with port 8000. How can I accomplish this? Is their any other way of ELB listening and forwarding multiple ports?
I have added listener for port 80,443 and 8000 in my ELB. Please help
Classic ELB
Using the "classic" ELB you can define custom rules for forwarding the ports in the AWS dashboard:
Mind that the requests will be forwarded to all the available instances, which means in the example above (supposing php is running on the 80, node.js on the 8000) all the instances must have both the services running. If the services are instead on different instances you will need two different load balancers, one per port.
Application ELB
Another option is to use an "application" ELB (ALB).
This option will allow to have single load balancer with fine-grained rules that will allow, for each protocol, to forward the request to a set of instances.
create a "default" ALB
add a new target group (see entry under the Load Balancing section in the sidebar) listening on your custom port
register the instances running your node.js application (right click on the target group)
bind the target group to the listeners of your ALB
Another solution could be, specifying path-based rules, to use only one port (443) and forward only the requests under /to_nodejs to the port 8000.
I have a problem with my deployment in ECS.
I try to deploy 4 instances of 2 docker images on 2 EC2 instances with an ALB in front.
So in my tasks definitions, I use the dynamic port mapping (2 Nginx on container port 80).
This creates a trouble in the Health check of my target group.
In fact, for each instance, I have a health check on the dynamics port (that is ok) and on the container port (80).
So the dynamic port says, it's ok. And the container port, logically says unhealthy ...
(Like in my screenshot)
So can you help me ton find why I have that type of error (this error make my server terminated each 5 minutes....)
Thanks in advance for your help :D
So to me it looks like you aren't completely using dynamic port mapping. For dynamic port mapping you have
Client -> ALB (port 80) -> EC2 host (dynamic port) -> container (dynamic port) -> nginx (port 80)
None of your healthchecks should be hitting port 80 since the only thing that uses port 80 is external connection into your application and nginx (but it is mapped to a different port). For ALB healthchecks all you really need is a path to hit and the port will default to the port that it connects on.
See the host port mapping in this doc: http://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_PortMapping.html
ALB Health Check Docs: http://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html
I found the solution with the AWS support.
So there are two problems here:
To disable the health which kill the EC2 instance, go to the auto scaling group and switch health check to "EC2" type
To remove the health check on port 80, go to the auto scaling group, and under "Target groups" section, removes the target groups managed by ECS
I'm trying to setup an ALB which listens on port 443, load balancing to ECS Docker containers on random ports, lets say I have 2 container instances of the same task definition, listening on port 30000 and 30001.
When I try to create a target group in the AWS EC2 Management console, there's a "port" input field with 1-65535 range.
What number should I put there?
And when I try to create a new service in the AWS EC2 Container Service console, together with a new target group to connect to a existing ALB, there's no input field for a target group "port".
After it's created, navigating to the EC2 console, the new target group has port "80".
Do I have to listen on port 80?
But the health check happens against the "traffic port", which is the container port, 30000 and 30001, so what's the point?
Turns out, when combined with ECS, the target group's port doesn't mean anything. You don't need to listen on that port.
I ran into this situation myself at work. I noticed the target group port and the port of the registered instance were different. I've typically set them up to be the same thing so wondered what this was all about which led me to this thread. I couldn't find a good answer on AWS docs, but found this in the Terraform docs for aws_lb_target_group resource:
port - (Optional, Forces new resource) The port on which targets receive traffic, unless overridden when registering a specific target.
So, I guess it's just the default port used unless you override it. Makes sense.
I think what he's referring to is the health checks. If your ELB is listening on port 443 but your target group is set for port 80, then every health check for the target group will attempt a request on port 80 and get redirected to port 443 by the load balancer. This results in a 301 code, which is considered unhealthy. Only 200 codes are supposed to be considered healthy. At that point you either have all unhealthy targets all the time or you add 301 to the list of healthy codes which defeats the whole point in health checks because it will always return a 301 for port 80. You might as well just match the ports.
By default, a load balancer routes requests to its targets using the protocol and port number that you specified when you created the target group. Alternatively, you can override the port used for routing traffic to a target when you register it with the target group.
The port in the target group is used in conjunction with auto-scaling groups and if you ever plan to use those you want to use the right port from the start. Why? Because you can not change it after the target group has been created and auto-scaling will simply not work if you set the port wrong.