Why would AWS ALB disconnect when using Socket.io - amazon-web-services

Question

I'm trying to get an ALB/Node.js/socket.io solution working in its simplest form and I'm running into an issue where the handshake disconnects. At the moment, I am intentionally using only one node in the TargetGroup to eliminate variables related to node switching and session stickiness for now.
When connecting directly to the node via my NAT instance, it works fine. The disconnect only happens when going thru the ALB.
Here is what I have set up:
ALB with Listener HTTP 80 -> 8081 (no SSL)
2 AZs, both with routes to internet (as required for ALBs)
One socket.io EC2 node in one of the AZs
Path Pattern for /socket.io/* to socket.io target group (with my one node in it)
Default pattern is also socket.io target group
Stickiness Enabled (should not need for one node, but did it anyway)
Here is what I see in the socket.io node client:
Thu, 22 Dec 2016 20:59:26 GMT socket.io-client:manager opening ws://52.72.198.58
Thu, 22 Dec 2016 20:59:26 GMT engine.io-client:socket creating transport "websocket"
Thu, 22 Dec 2016 20:59:26 GMT engine.io-client:socket setting transport websocket
Thu, 22 Dec 2016 20:59:26 GMT socket.io-client:manager connect attempt will timeout after 20000
Thu, 22 Dec 2016 20:59:26 GMT engine.io-client:socket socket close with reason: "transport close"
And here is what I see on the socket.io node server:
Thu, 22 Dec 2016 20:59:26 GMT socket.io:socket joined room U_qmSv_7gvP_JOFsAAAL
Thu, 22 Dec 2016 20:59:26 GMT socket.io:client client close with reason transport close
Thu, 22 Dec 2016 20:59:26 GMT socket.io:socket closing socket - reason transport close
When I go thru my NAT to the same socket.io ec2 node, it all works with no transport closes.
So somehow the ALB is closing the connection immediately during a successful handshake.
Since it works via the NAT, I think the socket.io node and client are ok. And since I see the DEBUG entries in node, I know the ALB is able to reach the socket.io node ok. And since I only have one single socket.io node, there should be no issues with sessions and stickiness.
What could be contributing to the immediate disconnect when using ALB?
EDIT : I have also found that if the socket.io client making the request to the ELB is on an EC2 node, then it works. This implies something in the network path between the client and the ELB. I've yet to find a case where this works other than when the client is on an EC2. It works everywhere via the NAT, just not via the ELB.

Answer 1

After lots of trial and error, I was able to determine this was due to a specific port range (80-83) in my case for the port the ALB/ELB is listening on. While the HTTP portion of the handshake works, the second TCP upgrade phase disconnects.
There were no restrictions in the VPC related to this port range, so the issue is in the network between my client and the ELB.
In conclusion, the issue is not anything in AWS or how I had set up resources, it lies elsewhere outside AWS. If I find the exact cause I will post a comment back to this answer.

Answer 2

socket = io.connect("https://mywebsite/myroom",{'reconnect':true});
Increased the HeartbeatTimeout and the closeTimeout on initialization
socket.on('connect', function(){
socket.socket.heartbeatTimeout = 500000;
socket.socket.closeTimeout = 500000;
socket.on('disconnect', function() {
socketConnectTimeInterval = setInterval(function () {
socket.socket.reconnect();
if(socket.socket.connected) {
clearInterval(socketConnectTimeInterval);
console.log('reconnected');
location.reload();
}
}, 0);
});
});
Also increase the Idle Timeout on the Load Balancer in AWS
Hopefully that should prevent the timeout issue !