AWS Load Balanced socket.io server, running behind a Proxy on NGINX - amazon-web-services

Question

I'll be candid - I know very little about websockets, my config file was setup by a developer we had years ago, and even then I dont think his configuration was correct. All the HTTP(S) REST API responses work, but the websockets features are having issues.
I am getting an error from the web console in production, everything works on local:
WebSocket connection to 'wss://my.website.com/socket.io/?EIO=3&transport=websocket&sid=bE54hfVzKjRVT5H1AR1i' failed:
I have a little bit of a difficult environment where my socket.io server is running
In our AWS environment I have a Linux server running NGINX, because this sits behind a public facing AWS load balancer (public facing listens via 443 with SSL) sending traffic down through port 80 to the actual box, all the certificates are handled at the AWS load balancer rather than on box.
as a secondary issue, is there a reason my old developer would add keepalive 64 on a websocket server? the verbiage seems to be odd when websockets are supposed to stay up for long periods of time?
I know my issue has to be in the NGINX config, I dont suspect my issue is in my load balancer but I dont know where to start
from my local machine, I have tried to change the url in my config to ws:// and wss:// with no success
upstream my_nodejs_upstream {
server 127.0.0.1:8080;
keepalive 64;
}
server {
listen 80;
server_name my_nodejs_server;
root /home/www/application;
location / {
server_tokens off;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_max_temp_file_size 0;
proxy_pass http://my_nodejs_upstream/;
proxy_redirect off;
proxy_read_timeout 240s;
}
}
Nodejs
const express = require('express')
const app = express()
const server = require('http').Server(app)
const io = require('socket.io')(server) // v2.0.4
io.on('connection', (socket) => {
// ...
})
// ...
app.use((req, res, next) => {
res.io = io
next()
})
app.use('/', require('./routes'))
const port = process.env.PORT
server.listen(port, () => {
// ...
})
client
const socket = io.connect(process.env.SOCKET_SERVER)
socket.on('connect', () => {
socket.emit('...', ...)
})
socket.on('...', data => {
// ...
})
access.log -> filtered for 400 errors
10.3.0.225 - - [29/Jul/2021:18:43:26 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=QNqTyFYHniVsHwRVATA- HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:26 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=eL9XCFhOpUc1x8v5ATA_ HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:26 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=sUeUwuQ0WZZ6srppATBB HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:26 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=RFmviPReFWc6jj0JATBA HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:26 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=h_-JSjypuJkd1VB9ATBC HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:26 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=aGtQf1Ei2YFt-mfZATBD HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:27 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=B3uR9cdYoZS28pJ2ATAh HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
10.3.0.45 - - [29/Jul/2021:18:43:27 +0000] "POST /socket.io/?EIO=3&transport=polling&t=Nhpb0Yf&sid=ok8WIn_zywc-5LhOAS_a HTTP/1.1" 400 52 "https://my.ui-server.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:27 +0000] "POST /socket.io/?EIO=3&transport=polling&t=Nhpb0Yf.0&sid=EJYczgk3g6MkfXWRAS_b HTTP/1.1" 400 52 "https://my.ui-server.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:29 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=jFE3Zhm59giR0_tYATBE HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:30 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=MK_Cxs3O2miJqc8EATBF HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "POST /socket.io/?EIO=3&transport=polling&t=NhpayDE&sid=dweMe6kGyqK-Yt0qATAZ HTTP/1.1" 400 52 "https://my.ui-server.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "GET /socket.io/?EIO=3&transport=polling&t=NhpayDE.0&sid=dweMe6kGyqK-Yt0qATAZ HTTP/1.1" 400 52 "https://my.ui-server.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "POST /socket.io/?EIO=3&transport=polling&t=Nhpb2XP&sid=375bccUv_aKZr9saATAW HTTP/1.1" 400 52 "https://my.ui-server.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "POST /socket.io/?EIO=3&transport=polling&t=Nhpb2XV&sid=cnTj1b2cB9G_SldMATAY HTTP/1.1" 400 52 "https://my.ui-server.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "POST /socket.io/?EIO=3&transport=polling&t=Nhpb2XW&sid=8MEo84boodpm5OPyAS_n HTTP/1.1" 400 52 "https://my.ui-server.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "POST /socket.io/?EIO=3&transport=polling&t=Nhpb2Xx&sid=7d0LBCT9jvM-Q9PoATAU HTTP/1.1" 400 52 "https://my.ui-server.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "POST /socket.io/?EIO=3&transport=polling&t=Nhpb2Y3&sid=DpMuhNMh9jFvu7neATAV HTTP/1.1" 400 52 "https://my.ui-server.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "POST /socket.io/?EIO=3&transport=polling&t=Nhpb2YF&sid=_Ndcj-_BAej_YUjNATAX HTTP/1.1" 400 52 "https://my.ui-server.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "POST /socket.io/?EIO=3&transport=polling&t=Nhpb3Y7&sid=dweMe6kGyqK-Yt0qATAZ HTTP/1.1" 400 52 "https://my.ui-server.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=ULibUHaBWhNSu7hdATBK HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=KXUistz6s0Emu06kATBM HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=fuOE0YD01avGKg_gATBN HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=8T994wWah-1azLgoATBO HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:35 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=dReJzIK5lgkHblh4ATBP HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
10.3.0.225 - - [29/Jul/2021:18:43:36 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=eO_a9FJ0HWH_LSuDATBQ HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
I did not find anything meaningful inside the error.log
$ cat error.log
2021/07/29 06:08:01 [notice] 10145#10145: using inherited sockets from "6;"