I have an Elastic Beanstalk environment with Application Load Balancer (ALB) setup on Node.js instance which server both REST APIs as well as Socket.IO connections.
When the number of EC2 instances is more than one some of Socket.IO connections are failing with HTTP 400. The same issue is resolved when there is only one instance.
I have also tried enabling sticky sessions in the ALB and have also created a Redis adapter which connects to my Redis instance in ElasticCache.
Unable to figure out what is causing those HTTP 400 errors.
Socket.IO documentation https://socket.io/docs/v2/using-multiple-nodes/
After two days of research, I found the solution.
Use transport "websocket" instead of "polling" (default).
Server:
const io = require('socket.io')(http, {
cors: {
origin: '*'
},
transports: ['websocket']
})
Client:
import io from 'socket.io-client'
socket = io('wss://your_url/', {
transports: ['websocket']
})
Eventually had to use Network Load Balancer (NLB) which resolved this issue.
Related
According to the AWS documentation, "WebSockets and Secure WebSockets support is available natively and ready for use on an Application Load Balancer."
However, when I select Application Load Balancer in EC2, I don't have any option other than HTTP and HTTPS:
I would like to use the secure websocket protocol (wss://) which I believe would be over TLS:8888.
How can I input this option?
The solution was to use HTTPS for the listener protocol, even though the browser is making requests to wss://.
For port number, configuring both the listener and environment instance to port 8888 works.
First, let me say that this is the first time I have written an ASP.NET Core 3.1 web app and first time learning AWS with Elastic Beanstalk. So if it seems like I'm confused... it's because I am. ;-)
I have two AWS environments - one is Staging and one is Production. The Staging environment has no SSL certificate and no load balancer. It only listens on port 80.
Production has a load balancer set up with my SSL certificate, and is set up to redirect all port 80 traffic to port 443.
Port 80 = Redirect to https://#{host}:443/#{path}?#{query}
Status code:HTTP_301
Port 443 = Forward to my-target-group: 1 (100%)
Group-level stickiness: Off
When I generated the new web app in VS 2019, I opted in on HTTPS/HSTS by checking "Configure for HTTPS". So it has this in Startup.cs:
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
else
{
app.UseExceptionHandler("/Home/Error");
// The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
app.UseHsts();
}
app.UseHttpsRedirection();
I am getting this error in my Windows event log in Staging and Production: “Failed to determine the https port for redirect”
I tried the suggestion from Enforce HTTPS in ASP.NET Core
services.AddHttpsRedirection(options =>
{
options.HttpsPort = 443;
});
But that messed up the Staging environment because there's nothing listening on port 443.
Since Staging is only using HTTP, and Production is redirecting to HTTPS at the load balancer, should I just remove the UseHsts() and UseHttpsRedirection() altogether from my Startup? Will that pose any security problems - I do want traffic encrypted over the internet but I don't think it's necessary between the load balancer and the EC2 instance, correct?
Or do I need Forwarded headers, as suggested at Configure ASP.NET Core to work with proxy servers and load balancers?
I do want traffic encrypted over the internet but I don't think it's necessary between the load balancer and the EC2 instance, correct?
Correct. That's how it is usually setup. So you usually would have SSL termination on your load balancer (LB), and then from LB to your instance it would be regular http traffic:
Client----(https)---->LB----(http)---->instances
does my app still need UseHttpsRedirection() and UseHsts()?
No, as your app is just recieving http traffic only from the LB.
EDIT
I now realise that I need to install a certificate on the server and validate the client certificate separately. I'm looking at https://github.com/xavierjohn/ClientCertificateMiddleware
I believe the certificate has to be from one of the CA's listed in AWS doco - http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-supported-certificate-authorities-for-http-endpoints.html
This certificate allows API Gateway to establish a HTTPS connection to the instance and it passes along the client certificate that can be validated.
ORIGINAL POST
I am trying to configure a new microservices environment and I'm having a few issues.
Here is what I'm trying to achieve:
Angular website connects to backend API via API Gateway (URL: gateway.company.com.au)
API Gateway is configured with 4 stages - DEV, UAT, PreProd and PROD
The resources in API Gateway are a HTTP Proxy to the back-end services via a Network Load Balancer. Each service in each stage will get a different port allocated: i.e. 30,000, 30,001, etc for DEV, 31,000, 31,000, etc for UAT
The network load balancer has a DNS of services.company.com.au
AWS ECS hosts the docker containers for the back-end services. These services are .NET Core 2.0 Web API projects
The ECS task definition specifies the container image to use and has a port mapping configured - Host Port: 0 Container Port: 4430. A host port of 0 is dynamically allocated by ECS.
The network load balancer has a listener for each microservice port and forwards the request to a target group. There is a target group for each service for each environment.
The target group includes both EC2 instances in the ECS cluster and ports are dynamically assigned by ECS
This port is then mapped by ECS/Docker to the container port of 4430
In order to prevent clients from calling services.company.com.au directly, the API Gateway is configured with a Client Certificate.
In my Web API, I'm building the web host as follows:
.UseKestrel(options =>
{
options.Listen(new IPEndPoint(IPAddress.Any, 4430), listenOptions =>
{
const string certBody = "-----BEGIN CERTIFICATE----- Copied from API Gateway Client certificate -----END CERTIFICATE-----";
var cert = new X509Certificate2(Encoding.UTF8.GetBytes(certBody));
var httpsConnectionAdapterOptions = new HttpsConnectionAdapterOptions
{
ClientCertificateMode = ClientCertificateMode.AllowCertificate,
SslProtocols = System.Security.Authentication.SslProtocols.Tls,
ServerCertificate = cert
};
listenOptions.UseHttps(httpsConnectionAdapterOptions);
});
})
My DockerFile is:
FROM microsoft/aspnetcore:2.0
ARG source
WORKDIR /app
EXPOSE 80 443
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT ["dotnet", "microservice.company.com.au.dll"]
When I use Postman to try and access the service, I get a 504 Gateway timeout. The CloudWatch log shows:
(e4d594b7-c8f3-11e7-8458-ef6f94e65b64) Sending request to http://microservice.company.com.au:30000/service
(e4d594b7-c8f3-11e7-8458-ef6f94e65b64) Execution failed due to an internal error
(e4d594b7-c8f3-11e7-8458-ef6f94e65b64) Method completed with status: 504
I've been able to get the following architecture working:
API Gateway
Application Load Balancer - path-based routing to direct to the right container
ECS managing ports on the load balancer
The container listening on HTTP port 80
Unfortunately, this leaves the services open on the DNS of the Application Load Balancer due to API Gateway being able to only access public load balancers.
I'm not sure where it's failing but I suspect I've not configured .NET Core/Kestrel correctly to terminate the SSL using the Client Certificate.
In relation to this overall architecture, it would make things easier if:
The public Application Load Balancer could be used with a HTTPS listener using the Client Certificate of API Gateway to terminate the SSL connection
API Gateway could connect to internal load balancers without using Lambda as a proxy
Any tips or suggestions will be considered but at the moment, the main goal is to get the first architecture working.
I more information is required let me know and I will update the question.
The problem was caused by the security group attached to the EC2 instances that formed the ECS cluster not allowing the correct port range. The security group for each EC2 instance in the cluster needs to allow the ECS dynamic port range.
I have used EB to create an environment with Tomcat 8, Java 8 configured as load balanced, auto scaling.
Deployed a WebSocket server on this.
On the EC2 running instance, my websocket client (tyrus API) is able to communicate over websocket, for example ws://ip/chat
Now I need a TCP connection (count) based auto scaling strategy, for which switched to Application Load Balancer (ALB) with a target group pointing to this EC2 instance.
Stickiness has been enabled and ALB is using HTTP listener on port 80 with listener rule "/chat" pointing to this target group.
All involved SG have All TCP in and out traffic enabled for testing.
Invoking ws://ELB/chat results doesnt work resulting in a 404:
Caused by: org.glassfish.tyrus.core.HandshakeException: Response code was not 101: 404
Any inputs on how this should be configured.
Final aim is to be able to communicate with WebSocket server over ALB and then auto scale based on TCP "ActiveConnectionCount"
I am trying to setup an Elastic load balancer to route requests to a cluster of node.js servers running Primus.io with sockjs to manage real time communications.
I have set up the load balancer to listen with the following configuration:
HTTPS 8084 -> HTTPS 8084 (The port used on my node.js servers)
SSL 443 -> TCP 80
My understanding is that the only way to get websockets to work through ELB is via SSL->TCP, hence the above configuration.
I have correctly enabled the new proxy protocol for ELB as described here:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html
When trying to connect to the server from a client an HTTPS request is initially sent and then from what I can gather it should be upgraded to websockets. But the request is simply failing when I send it to the loadbalancer address.
If I send the initial Primus connection request to the ip of a single nodejs server like so:
var primus = new Primus('https://ip.address.of.single.server:8084');
The request is correctly returned and is upgraded to websockets correctly.
When I switch the ip address to that of the balancer, it fails and the initial https request to the node.js server returns nothing. I assume this means that the websocket transfer could not be established, but to be honest I have little experience in this area so could be completely wrong.
Does anyone have any idea what I am doing wrong?
Thanks in advance
Do you have clustered your NodeJS-instances? For example, if you use SocketIO you should use a clustered session store. Actually, I'm also currently investigating the same with SockJS running on top of Vertx.
The problem behind is Amazon ELB won't respect any forwards in the past (in opposite to Sticky Session on top of HTTP) which means that a connections via TCP level can be forwarded at any cluster's node. Yes, one tcp channel would be okay. But frameworks like SocketIO do a little bit more to support sessions (does not exist in WebSockets) and multiple transport layers (http, polling, sockets, and so on).