I've set up a TCP network load balancer, as described here: https://cloud.google.com/load-balancing/docs/network. I need to balance traffic from anywhere on the internet to my backend VMs, running a custom application listening to a non-standard TCP port.
Everything seems to work initially, but after about 10 seconds the connected clients are disconnected, reporting the error "An existing connection was forcibly closed by the remote host.". For debugging I allow my backend VMs to have public IPs and when connecting to any of them directly, bypassing the load balancer, everything works and there's no disconnect.
As I understand it, this load balancer setup I'm using should be pass through: Once the backend VM is selected, the TCP connection should essentially be with the back end VM and the load balancer no longer involved. The backend VMs are certainly not terminating the connection forcibly - as far as the backends are concerned, the connection persists after the client disconnect and time out later. The timeout settings described for other google cloud load balancers don't seem to apply to External TCP/UDP Network Load Balancing.
What am I missing?
TCP/UDP network load balancers are pass-through load balancers and do not proxy connections to your backend instances, so your backends receive the original client request. The network load balancer doesn't do any Transport Layer Security (TLS) offloading or proxying. Traffic is directly routed to your VMs.
Confirm that your network load balancer is set up correctly using these
steps.
Ensure that server software running on your backend VMs is listening on the IP address of the load balancer's forwarding rule.
Make sure you’ve configured firewall rules using source IP ranges for Network load balancing health checks.
Additionally, you can capture tcpdump to narrow down your issue, which may provide information to specific resource.
Related
We have configured our website server with network load balancing. When we tried to ping our domain name using terminal all ping lost.
I tried to figure it out and have no clue how to configure NLB to listen ping from terminal.
You need to create one/ multiple listeners in case of NLB and route them to specific target for serving the intended requests
Network traffic that does not match a configured listener is classified as unintended traffic. ICMP requests other than Type 3 (unreachable) are also considered unintended traffic. Network Load Balancers drop unintended traffic without forwarding it to any targets.
Source : https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-listeners.html
We have several servers behind an AWS network load balancer.
The used algorithm for balancing trafic is the "round robin" describe as below :
"With Network Load Balancers, the load balancer node that receives the connection uses the following process:
Selects a target from the target group for the default rule using a flow hash algorithm. It bases the algorithm on:
The protocol
The source IP address and source port
The destination IP address and destination port
The TCP sequence number
Routes each individual TCP connection to a single target for the life of the connection.
The TCP connections from a client have different source ports and sequence numbers, and can be routed to different targets."
Due to the persistance of connections, servers load may be unbalanced and can cause problems.
How to configure the network load balancer to route new connections on the server that have the less load ?
ALBs now support Least Outstanding Request routing. NLB does not appear to support this (yet?)
Is there any possibility of adapting your LB strategy to ALBs from NLBs?
I am currently hosting a game service with Google Game Servers (https://cloud.google.com/game-servers). This is basically running Agones in GKE. This works great for the most part except when it comes to iPv6.
I am struggling to find any suggestions on how to get this kind of a setup to be iPv6 compatible. It seems like this should be the answer https://cloud.google.com/load-balancing/docs/ipv6 but Agones is setup to run servers across a port range as it spins up and shuts down servers and it seems I need to have a specific port and non-persistent connections to a specific machine to use googles Load balancer solution.
For reference, this is a NodeJS backend relying on socket.io communication.
Any suggestions would be appreciated.
As already stated in comments Google Cloud VPCs do not support IPv6 connectivity:
Google Cloud VPCs do not support IPv6. A few public facing services
such as HTTPS Load Balancers do support IPv6 but that will not help
you with internal services. – John Hanley Sep 29 at 12:23
If your stack requires IPv6 connectivity, unfortunatelly you won't be able to deploy it on Google Kubernetes Engine at the time being as it is subject to the same rules as any other Compute resource on GCP and uses the same VPC network.
As you can read in the official VPC specifications:
VPC networks only support IPv4
unicast traffic. They do not
support
broadcast,
multicast, or IPv6 traffic
within the network; VMs in the VPC network can only send to IPv4 destinations and only receive traffic from IPv4 sources. However, it
is possible to create an IPv6 address for a global load
balancer.
As to Global Loadbalancer (that do support IPv6) here is all the information you need:
Google Cloud supports IPv6 clients with HTTP(S) Load
Balancing, SSL
Proxy Load
Balancing, and
TCP Proxy Load
Balancing. The load
balancer accepts IPv6 connections from your users, and then proxies
those connections to your backends.
You can configure both IPv4 and IPv6 external addresses for the
following:
external HTTP(S) load balancers
SSL proxy load balancers
TCP proxy load balancers
Protocols and port ranges it supported by each of them are listed in their individual specifications (all links available above).
SSL Proxy Load Balancing:
This does not affect SSL proxy load balancers. External forwarding
rules, which are used in the definition of an SSL load balancer, can
only reference TCP ports 25, 43, 110, 143, 195, 443, 465, 587, 700,
993, 995, 1883, 3389, 5222, 5432, 5671, 5672, 5900, 5901, 6379, 8085,
8099, 9092, 9200, and 9300. Traffic with a different TCP destination
port is not forwarded to the load balancer's backend.
TCP Proxy Load Balancing:
TCP Proxy Load Balancing is intended for TCP traffic on specific
well-known ports, such as port 25 for Simple Mail Transfer Protocol
(SMTP). For more information, see Port
specifications.
For client traffic that is encrypted on these same ports, use SSL
Proxy Load
Balancing.
with one caveat:
Note: TCP Proxy Load Balancing doesn't support TCP ports 80 or 8080. For HTTP traffic, use HTTP(S) Load Balancing.
When it comes to External HTTP(S) Load Balancing its name speaks for itself.
So if you rather need to use some arbitrary port ranges as you mentioned, the answer is: no, unfortunatelly you can't do that using Google Cloud Load Balancing solutions
I read the aws documentation inside the elastic beanstalk program where aws is responsible for scaling the servers and auto managing it. In the same documentation there is an option for changing and configuring the load balancer. In my case I want to change it to balance the requests that come to the servers on the IP network layer (L3), but it says that only HTTP and TCP can be listened and balanced.
I am developing a chat application backend that need to be developed with scaling in considerations. how can I configure the load balancer to listen on L3 ?
the chat application in order to work it must make the tcp connection with the server not the load balancer so that's why I must load the packets on the IP layer to the server so the server can establish a tcp connection with the app ( if I am wrong and I can do it on the tcp layer tell me ).
If I can't, does that give me another option or I will just be forced on using ec2 and handle all the system management overhead myself and create my own load balancer ?
ELB Classic operates at either Layer 4 or Layer 7. Those are the options.
the chat application in order to work it must make the tcp connection with the server not the load balancer so that's why I must load the packets on the IP layer to the server so the server can establish a tcp connection with the app.
You're actually incorrect about this. If you need to know the client's source IP address, you can enable the Proxy Protocol on your ELB, and support this in your server code.
When the ELB establishes each new connection to the instance, with the Proxy Protocol enabled, the ELB emits a single line preamble containing the 5-way tuple describing the external connection, which your application can interpret. Then it opens up the L4 connection's payload streams and is transparent for the remainder of the connection.
http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html
I'm getting gateway time-outs when trying to use a port specifically for websockets using an Application Load Balancer inside an Elastic Beanstalk environment.
The web application and websocket server is held within a Docker container, the application runs fine however wss://domain.com:8080 will just time out.
Here is the Load balancer listeners, using the SSL cert for wss.
The target group it points to is accepting 'Protocol' of HTTP (I've tried HTTPS) and forwards to 8080 onto an EC2 instance. Or.. It should be. (Doesn't appear to be an option for TCP on Application Load Balancers).
I've had a look over the Application Load Balancer logs and it looks like the it reaches the target group, but times out between it's connection to the EC2 instance, and I'm stumped on why.
All AWS Security Groups have been opened on all traffic for the time being, I've checked the host and found that the port is open and being listened to by Nginx which will route to the correct port to the docker container:
docker ps also shows me:
And once inside the container I can see that the port is being listened to by the Websocket server:
So it can't be the EC2 instance itself, can it? Is there an issue routing websockets via ports in an ALB?
-- Edit --
Current SG of the ALB:
The EC2 instance SG:
Accepted answer here seems to be "open Security Groups for EC2 (web server) and ALB inbound & outbound communication on required ports since websockets need two way communication."
This is incorrect and the reason why it solved the problem is coincidental.
Let me explain:
"Websockets needs two way communication..." - Sure but the TCP sessions is only ever opened from one way - from the client.
You don't have to allow any outbound connections from the EC2 instance (web server) in order to use web sockets.
Of course the ALB needs to be able to do TCP connections to the EC2 instance. But not to the client. Why? Well the ALB is accepting TCP connections (usually on port 80 and 443). It is setting up a TCP session that was initiated by the client. It is then trying to set up a new TCP session to the web server behind the ALB. This should be done on the port that you decided to have the web server listening on. The Security Group around the ALB needs to be able to do outbound connections on this port to the web server. This is the reason why "open up everything" worked. It has nothing to do with "two way communication".
You could use any ports of course but you don't need to use any other ports than 80 & 443 (such as 8080) on both the Load Balancer or the EC2.
Websockets need two way communication, make sure security groups attached to all resources (EC2 & ALB) allow both inbound & outbound communication on required ports.