Hosting multiple websites on a single server - amazon-web-services

I have a bunch of different websites, mostly random weekend projects that I'd like to keep on the web because they are still useful to me. They don't see more than 3-5 hits per day between all of them though, so I don't want to pay for a server for each of them when I could probably fit them all on a single EC2 micro instance. Is that possible? They all run off different web servers, since I tend to experiment with a lot of new tech. I was thinking I could have each webserver serve on a different port, then have incoming requests to app1.com get routed to app1.com:3000 and requests to app2.com get routed to app2.com:3001 and so on, but I don't know how I would go about setting that up.

I would suggest that what you are looking for is a reverse web proxy, which typically includes among its features the ability to understand portions of the request at layer 7, and direct the incoming traffic to the appropriate set of one or more back-end ip/port combinations based on what's observed in the request headers (or other aspects of the request).
Apache, Varnish, and Nginx all have this capacity, as does HAProxy, which is the approach that I use because it seems to be very fast and easy on memory, and thus appropriate for use on a micro instance... but that is not at all to imply that it is somehow more "correct" to use than the others. The principle is the same with any of those choices; only the configuration details are different. One service is listening to port 80, and based on the request, relays it to the appropriate server process by opening up a TCP connection to the appropriate destination, tying the ends of the two pipes together, and otherwise for the most part staying out of the way.
Here's one way (among several alternatives) that this might look in an haproxy config file:
frontend main
bind *:80
use_backend app1svr if { hdr(host) -i app1.example.com }
use_backend app2svr if { hdr(host) -i app2.example.com }
backend app1svr
server app1 127.0.0.1:3001 check inter 5000 rise 1 fall 1
backend app2svr
server app2 127.0.0.1:3002 check inter 5000 rise 1 fall 1
This says listen on port 80 of all local IP addresses; if the "Host" header contains "app1.example.com" (-i means case-insensitive) then use the "app1" backend configuration and send the request to that server; do something similar for app2.example.com. You can also declare a default_backend to use if none of the ACLs match; otherwise, if no match, it will return "503 Service Unavailable," which is what it will also return if the requested back-end isn't currently running.
You can also configure a stats endpoint to show you the current state and traffic stats of your frontends and backends in an HTML table.
Since the browser isn't connecting "directly" to the web server any more, you have to configure and rely on the X-Forwarded-For header inserted into the request headers to identify the browser's IP address, and there are other ways in which your applications may have to take the proxy into account, but this overall concept is exactly how web applications are typically scaled, so I don't see it as a significant drawback.
Note these examples do use "Anonymous ACLs," of which the documentation says:
It is generally not recommended to use this construct because it's a lot easier
to leave errors in the configuration when written that way. However, for very
simple rules matching only one source IP address for instance, it can make more
sense to use them than to declare ACLs with random names.
— http://cbonte.github.io/haproxy-dconv/configuration-1.4.html
For simple rules like these, this construct makes more sense to me than explicitly declaring an ACL and then later using that ACL to cause the action that you want, because it puts everything together on the same line.
I use this to solve a different root problem that has the same symptoms -- multiple sites for development/test projects, but only one possible external IP address (which by definition means "port 80" can only go to one place). This allows me to "host" development and test projects on different ports and platforms, all behind the single external IP of my home DSL line. The only difference in my case is that the different sites are sometimes on the same machine as the haproxy and other times they're not, but the application seems otherwise identical.

Rerouting in way you show - depends on the OS your server is hosting on. For linux you have to use iptables, for windows you could use windows firewall. You should set all incoming connections to a port 80 to be redirected do desired port 3000
But, instead of port, you could use a different host name for each service, like
app1.apps.com
app2.apps.com
and so on. You can configure it with redirecting on your DNS hosting, for apps.com IMHO this is best solution, if i got you right.
Also, you can configure a single host to reroute to all other sites, like
app1.com:3001 -> apphost1.com
app1.com:3002 -> apphost2.com
Take in mind, in this case, all traffic will pas through app1.com.

You can easily do this. Set up a different hostname for each app you want to use, create a DNS entry that points to your micro instance, and create a name-based virtual host entry for each app.
Each virtual host entry should look something like:
<VirtualHost *>
ServerName app1.example.com
DocumentRoot /var/www/html/app1/
DirectoryIndex index.html
</VirtualHost>

Related

Traefik Best Practices/Capabilities For Dynamic Vanity Domain Certificates

I'm looking for guidance on the proper tools/tech to accomplish what I assume is a fairly common need.
If there exists a web service: https://www.ExampleSaasWebService.com/ and customers can add vanity domains/subdomains to white-label or resell the service and replace the domain name with their own, there needs to be a reverse proxy to terminate vanity domains TLS traffic and route it to the statically defined (HTTPS) back-end service on the non-vanity original domain (there is essentially one "back-end" server somewhere else on the internet, not the local network, that accepts all incoming traffic no matter the incoming domain). Essentially:
"Customer A" could setup an A/CNAME record to VanityProxy.ExampleSaasWebService.com (the host running Traefik) from example.customerA.com.
"Customer B" could setup an A/CNAME record to VanityProxy.ExampleSaasWebService.com (the host running Traefik) from customerB.com and www.customerB.com.
etc...
I (surprisingly) haven't found anything that does this out of the box, but looking at Traefik (2.x) I'm seeing some promising capabilities and it seems like the most capable tool to accomplish this. Primarily because of the Let's Encrypt integration and the ability to reconfigure without a restart of the service.
I initially considered AWS's native certificate management and load balancing, but I see there is a limit of ~25 certificates per load balancer which seems like a non-starter. Presumably there could be thousands of vanity domains in place at any time.
Some of my Traefik specific questions:
Am I correct in understanding that you can get away without explicitly provisioning a generated list of explicit vanity domains to produce TLS certificates for in the config files? They can be determined on-the-fly and provisioned from Let's Encrypt based on the headers of the incoming requests/SNI?
E.g. If a request comes to www.customerZ.com and there is not yet a certificate for that domain name, one can be generated on the fly?
I found this note on the OnDemand flag in the v1.6 docs, but I'm struggling to find the equivalent documentation in the (2.x) docs.
Using AWS services, how can I easily share "state" (config/dynamic certificates that have already been created) between multiple servers to share the load? My initial thought was EFS, but I see EFS shared file system may not work because of a dependency on file change watch notifications not working on NFS mounted file systems?
It seemed like it would make sense to provision an AWS NLB (with a static IP and an associated DNS record) that delivered requests to a fleet of 1 or more of these Traefik proxies with a universal configuration/state that was safely persisted and kept in sync.
Like I mentioned above, this seems like a common/generic need. Is there a configuration file sample or project that might be a good starting point that I overlooked? I'm brand new to Traefik.
When routing requests to the back-end service, the original Host name will be identifiable still somewhere in the headers? I assume it can't remain in the Host header as the back-end recieves requests to an HTTPS hostname as well.
I will continue to experiment and post any findings back here, but I'm sure someone has setup something like this already -- so just looking to not reinvent the wheel.
I managed to do this with Caddy. It's very important that you configure the ask,interval and burst to avoid possible DDoS attacks.
Here's a simple reverse proxy example:
# https://caddyserver.com/docs/caddyfile/options#on-demand-tls
{
# General Options
debug
on_demand_tls {
# will check for "?domain=" return 200 if domain is allowed to request TLS
ask "http://localhost:5000/ask/"
interval 300s
burst 1
}
}
# TODO: use env vars for domain name? https://caddyserver.com/docs/caddyfile-tutorial#environment-variables
qrepes.app {
reverse_proxy localhost:5000
}
:443 {
reverse_proxy localhost:5000
tls {
on_demand
}
}

How to fire up all docker containers on a same local ip address in django?

I am writing a django based application with docker where there are 3 projects apps running in different containers. All django applications run at 0.0.0.0:8000.
But when I check the ip address of containers to browser the application in browser, they all run at different ip addresses:
project1 runs at 172.18.0.10:8000 can be accessed at: 172.18.0.10:8000/app1
project2 runs at 172.18.0.9:8000 can be accessed at: 172.18.0.9:8000/app2
project3 runs at 172.18.0.7:8000 can be accessed at: 172.18.0.7:8000/app3
which makes the hyperlinks of my app unusable. How do I run all the containers at one single ip, 'localhost:8000'?
Any suggestions where I am going wrong?
You are wrong in the design, mapping multiple containers to one ip+port is simply impossible. One port on one ip is always one application that listens, no matter if it is container application or not.
Simple prove: And who would then decide to which container to send the request? To all of them? Then who would decide which response is the correct one? That's what are ip addresses and ports for, to be able to send request to specific aplications on specific machines.
I think you should reconsider whatever you are doing, and do a bit more research on networking. There are several online courses on that. (I don't want to discourage you in any way, just aim you the right direction)
Simple solution without redesign you app, is putting in front of your app reverse proxy (e. g. nginx). That's the response to my rhetorical question. Reverse proxy can be a middle man that can decide to which application send the request based on something else then ip/port. Reverse proxy listens on some specific port and then by rules you provide to it (e. g. path based), can proxy the request to specific app/ip/port and proxy the response back.
But reverse proxy in this case is more a hack than proper solution, keep that in mind.

Clone http traffic to another port on same server transparently

I am experimenting with following setup.
Clone/copy (but not redirect) all incoming HTTP requests from port 80 to another port say 8080 on same machine. I have a simple NGINX + Lua based WAF which is listening on 8080. Essentially, I am running two instances of webservers here, one which is serving real requests and other one working on cloned traffic for detection purpose. I don't care about being able to block the malicious requests so I dont care about being inline.
I want to use WAF only for detection purpose i.e. it should analyze all incoming requests, raise alert and drop the request after that. This will not hamper anything from users point of view since port 80 is serving real requests.
How can I clone traffic this way and just discard it after analysis is done ? Is this feasible ? If yes, please suggest any tools which can clone traffic with minimal performance hit.
2.
Have a look : https://github.com/buger/gor
Example instructions are straightforward. Additional logging or certain forwards you could possibly add as well
In the current Nginx version, there is an ngx_http_mirror_module, which retranslates requests to another endpoint and ignores responses. See also this answer

running multiple variants of a webserver on different ports - cookies interfere

I have two instances of an identical service running on 2 ports - think of them as dev and test. They use session cookies that expire on inactivity.
When using the same browser, whenever I switch a tab from "dev" to "test", I have to log in again. Which becomes annoying after first few minutes. And this is because the HTTP cookies are tied to a server name, oblivious of the port.
I actually have multiple teams use the same test machine. My current solution is to use different browsers. I am also thinking of taking different domain names and map them to same IP address.
My question is - what am I missing, is there a better solution? Is it possible to make cookies port-aware (I don't have control over the software that names the cookies)?

Can I have multiple servers if I only have one publicly facing IP?If so, how is this done?

I have 5 servers, some of which are serving the same network service (i.e. http), but I want them to appear on the public network as separate machines. www1.mydomain.com, www2.mydomain.com
I can move the internal port numbers to another port, but how would I map the DNS name to these other machines?
What you need is a content switch, or an layer 4 through layer 7 switch. A content switch takes incoming packets and routes them to the appropriate server based on criteria you specify. Using a content switch, you can take Web traffic that is bound for one IP and spread it out over multiple IPs.
For Web requests, you can switch based on the HTTP 'Host' header. The 'Host' header specifies from which host the client desires to get the resource from.
I know that Citrix* has a free version of their Netscaler software that includes a content switch. If you want a completely free switch there is HA Proxy: http://www.haproxy.org/
Disclaimer - I work for them.
You cannot solve this with DNS. If you have only one IP address, all HTTP traffic will go to port 80 (or 443) on that address. If you want to split those requests by which HTTP host is being requested, you need something that can look at the HTTP Host header field and forward the request appropriately.