I am currently configuring some web servers running Apache2 and a PHP based web app. The servers are running the same PHP codebase on the same system configuration and should be placed behind a load balancer on AWS. The LB accepts and terminates HTTPS connections, and forwards them as HTTP traffic to the web servers, so in theory the Event MPM should work and make sense.
Now, since the servers are sitting behind an LB, my question is: Are the connections between the LB and the web servers being kept alive ("keepalive") in this scenario? Also, do the TLS-connections result in the event-mpm to behave like a worker-mpm or not, even if the HTTPS-connections are terminated by the LB and forwarded as unencrypted HTTP traffic?
Ref: https://serverfault.com/questions/383526/how-do-i-select-which-apache-mpm-to-use?answertab=votes#tab-top
With help of the AWS support, I was able to find an answer to the question:
The AWS LB opens an unlimited number of connections to the servers behind it, so the Apache settings have to be configured in a way such that the number of worker threads optimally uses the underlying system's resources. If you see that neither your servers' memory nor CPU load comes anywhere near its capacity (even during a stress test), then you might want to increase the number of worker threads/processes in the Apache config.
Also: If the LB terminates HTTPS connections and forwards them as HTTP traffic, the Event MPM will work as intended, which is apparently also the most optimal MPM for Apache when using an AWS LB, unless you use HTTPS between the LB and the servers. In that case, the worker MPM will do just fine.
Related
In my application:
ASP.NET Core 3.1 with Kestrel
Running in AWS ECS + Fargate
Services run in a public subnet in the VPC
Tasks listen only in the port 80
Public Network Load Balancer with SSL termination
I want to set the Security Group to allow inbound connections from anywhere (0.0.0.0/0) to port 80, and disallow any outbound connection from inside the task (except, of course, to respond to the allowed requests).
As Security Groups are stateful, the connection tracking should allow the egress of the response to the requests.
In my case, this connection tracking only works for responses without body (just headers). When the response has a body (in my case, >1MB file), they fail. If I allow outbound TCP connections from port 80, they also fail. But if I allow outbound TCP connections for the full range of ports (0-65535), it works fine.
I guess this is because when ASP.NET Core + Kestrel writes the response body it initiates a new connection which is not recognized by the Security Group connection tracking.
Is there any way I can allow only responses to requests, and no other type of outbound connection initiated by the application?
So we're talking about something like that?
Client 11.11.11.11 ----> AWS NLB/ELB public 22.22.22.22 ----> AWS ECS network router or whatever (kubernetes) --------> ECS server instance running a server application 10.3.3.3:8080 (kubernetes pod)
Do you configure the security group on the AWS NLB or on the AWS ECS? (I guess both?)
Security groups should allow incoming traffic if you allow 0.0.0.0/0 port 80.
They are indeed stateful. They will allow the connection to proceed both ways after it is established (meaning the application can send a response).
However firewall state is not kept for more than 60 seconds typically (not sure what technology AWS is using), so the connection can be "lost" if the server takes more than 1 minute to reply. Does the HTTP server take a while to generate the response? If it's a websocket or TCP server instead, does it spend whole minutes at times without sending or receiving any traffic?
The way I see it. We've got two stateful firewalls. The first with the NLB. The second with ECS.
ECS is an equivalent to kubernetes, it must be doing a ton of iptables magic to distribute traffic and track connections. (For reference, regular kubernetes works heavily with iptables and iptables have a bunch of -very important- settings like connection durations and timeouts).
Good news is. If it breaks when you open inbound 0.0.0.0:80, but it works when you open inbound 0.0.0.0:80 + outbound 0.0.0.0:*. This is definitely an issue due to the firewall dropping the connection, most likely due to losing state. (or it's not stateful in the first place but I'm pretty sure security groups are stateful).
The drop could happen on either of the two firewalls. I've never had an issue with a single bare NLB/ELB, so my guess is the problem is in the ECS or the interaction of the two together.
Unfortunately we can't debug that and we have very little information about how this works internally. Your only option will be to work with the AWS support to investigate.
I have multiple identical Scala Akka-HTTP applications, each one is installed on a dedicated server (around 10 apps), responding to HTTP requests on port 80. in front of this setup I am using single HAproxy instance that receives all the incoming traffic and balances the workload to these 10 servers.
We would like to change the HAproxy (we suspect that it causes us latency problems) and to use a different load balancer. the requirement is to adopt a different 3rd party load balancer or to develop a simple one using scala that round robin each http request to the backend akka http apps and to proxy back the response.
Is there another recommended load balancer (open source) that I can use to load balance / proxy the http incoming requests to the multiple apps other than HAproxy (maybe APACHE httpd) ?
Does it make sense to write a simple akka http application route as the loadbalancer, register the backend apps hosts in some configuration file, and to roundrobin the requests to them?
maybe I should consider Akka cluster to that purpose ? the thing is, that the applications are already standalone akka http services with no cluster support. and maybe it would be too much to go for clustering. (would like to keep it simple)
What is the best practice to load balance requests to http apps (especially akka http scala apps) as I might be missing something here?
Note - having back pressure is something that we also would like to have, meaning that if the servers are busy, we would like to response with 204 or some status code so our clients wont have timeouts in case my back end is busy.
Although Akka HTTP performance is quite impressive, I would not use it for writing a simple reverse proxy since there are tons of others out there in the community.
I am not sure where you deploy your app, but, the best (and more secure) approach is to use a LB provided by your cloud provider. Most of them has one and usually it has a good benefit-cost.
If your cloud provider does not provide one or you are hosting yourself your app, then first you should take a look on your HAProxy. Did you run tests on HAProxy in an isolated way to see it still has the same latency issues? Are you sure the config optimised for what you want? Does your HAProxy has enough resources (cpu and memory) to operate? Is your HAProxy in the same DataCenter as your deployed app?
If you follow and check all of these questions and still are having latency issues, then I would recommend you to choose another one. There are tons out there, such as Envoy and NGINX. I really like Envoy and I've been using it at work for a few months now without any complains.
Hope I could help.
[]'s
I am developing an Websocket application and I am having doubts on how to create a scalable application.
1- Should I use Nginx? And if so, where does nginx stand? It would be like this:
ELB -> Nginx -> Ec2 instances
or
Nginx -> ELB -> Ec2 instances
2- Is it necessary to use a service like Redis to make the communication between servers? Example: I am connected to server1 and my friend is connected to server2, but we are in the same room chat. If I send a message, it needs to reach my friend.
3 - Is it possible to let my Elb receives only calls in https but the conversation with the backend is http? I ask this, because I use OpsWorks and it was very hard to normalize cookbooks to create my environment.
Thank you.
Generally the architecture looks like:
ALB --> nginx1,niginx2 --> ALB --> ec2 websocket server1, server2
This allows your web servers and app servers to be load balanced independently of each other
Not necessarily. Redis is used primarily as an in memory data store for caching.
Yes - You can terminate ssl on ALB and it is in fact recommended to do it this way in order to offload ssl processing on load balancer as opposed to doing it on instances themselves. See - https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html . Additional benefit of using this is that you can use ACM to issue certificates for free that can be deployed on ALB. ACM can handle renewals for you automatically as well.
I've been searching a solution how to configure nginx proxy server to work with websockets. I have ready found a bunch of solutions make that. But, these scripts patch nginx.conf in the instances. But my instances don't have nginx running. Nginx is run on a balancer.
So my question is how to patch nginx config on a balancer.
Your question is confusing, because you are saying you are using ELB and you want Nginx. But you can't get websockets in Nginx with normal ELB, and you probably don't need Nginx with ELB except in specific situations.
You have two choices:
1) Continue to use ELB and Elasticbeanstalk. The problem is that ELB doesn't support websockets at all. See this article. You'll need to stop using ELB as a HTTP proxy, and start using it as a TCP proxy. The downside is that your app will now be exposed to your servers going up and down. (In a HTTP proxy, each request can go to a different server. In a TCP proxy, the request stays alive for the whole session, so when the server goes down, your client must 'deal with it.')
2) Run your own load balancer. Best practice is EIP + Nginx + HAProxy. This is quite a different question.
I have two ubuntu machines running django with gunicorn as my Python HTTP WSGI server. I currently have a ELB sitting in front of these two machines.
Many sources claim I should add NGINX to my stack for proxy buffering. However I don't know where Nginx should be placed and how it can be configured to point to the ELB which in turns points to the app servers.
NGINX ELB 2 Django/Gunicorn Servers
(proxy buffering, prevents DDOS attacks) -------> (Load balances between two app servers) ------> (My two app servers)
Is this setup appropriate? If so how can I configure it?
NGINX sort of becomes a single point of failure. Unless there is a reason to do so otherwise, I would likely put the ELB in front of nginx and run nginx on both app servers (could run on separate servers if needed).
The web server can also take care of static requests, which would probably be handled more efficiently than your app stack.
Since ELB is inherently scalable and fault tolerant, it is general practice to have them at front. You can attach your Web servers to ELB . By adding Nginx on the top you will be bringing Single point of failure.