Basically, I have a couple of services. I want to forward every requests with prefix "/secured" to server1 port 80 and all other requests to server 2 port 80. The problem is that on server1, I am running service which accept the request without "/secured" prefix. In other words, I want to forward every requests such as "http://example.com/secured/api/getUser" to server1 as "http://example.com/api/getUser" (remove /secured from request' path).
With AWS ALB, currently the request is sent as http://example.com/secured/api/getUser; which forces me to update my server1's code so that the code handles requests with /secured prefix which doesn't look good.
Is there any easy way to solve this with ALB?
Thanks.
I can confirm that this is unfortunately not possible with the ALB alone - and I agree it really should be.
AWS states:
Note that the path pattern is used to route requests but does not
alter them. For example, if a rule has a path pattern of /img/*, the
rule would forward a request for /img/picture.jpg to the specified
target group as a request for /img/picture.jpg.
I had the same issue, and as Mark pointed out, you can use reverse proxy on your server and do something like this (this is Nginx configuration):
server {
listen 80 default_server;
location /secured/ {
proxy_pass http://localhost:{service_port}/;
}
}
This will strip the /secured part and proxy everything else to your service. Just be sure to have the trailing / after the service port.
Related
I searched and I couldnt' find any way to do this, so I am wondering if it is possible.
I have a service running that accepts requests and everything works fine. However, I'd like to answer some of these requests with a different service running on the same machine. These requests are the ones going to some/path/{variable}/etc and method POST.
So I would like to know if it possible to do this directly from nginx without adding any overhead.
My first solution was creating a different API that receives all the requests and if it is not the one I want to incercept, just did a proxy request to the origianl service. But this added between 200 and 500ms to every response, which is not acceptable in our use case.
So I thought that doing this through nginx would resolve much faster, but I couldn't find a way or even find out if it is possible.
Any help would be highly appreciated. If you have any other idea or alternative that I could test or implement, it would be welcome as well.
Edit: Per request by Ivan's comment.
We have already nginx running, serving all the requests by service1.
What we would like to do is:
if request.path is in the form of /path/{variable}/etc and request.method==POST:
serve using service2
else:
serve using service1
Assuming you services hosted on the same server and differs in access ports, using a chain of the map blocks should do the trick:
map $uri $service_by_uri {
~^/path/[\w]+/etc 127.0.0.1:10002; # service 2
default 127.0.0.1:10001; # service 1
}
map $request_method $service {
POST $service_by_uri; # select service according to the request URI
default 127.0.0.1:10001; # service 1
}
server {
...
proxy_pass http://$service;
...
}
I have a bit problem with my site.
So setup is ElasticBeanstalk(NGINX) + Cloudflare
But each day around 4AM I have direct IP attack to my server.
Around 300 requests in 1-2 minutes.
Bot try to access some resources like
GET /phpMyadmi/index.php HTTP/1.1
GET /shaAdmin/index.php HTTP/1.1
POST /htfr.php HTTP/1.1
For now all of them going to 80 or 8080 ports.
And successfully handled by Nginx configuration that redirect it to example:443
server {
listen 80 default_server;
listen 8080 default_server;
server_name _;
return 301 https://example.com$request_uri;
}
server {
listen 443 ssl;
server_name example.com;
ssl on;
...
So questions are,
have many site owners/devOps face the same attack. What is your action to prevent such attacks.
For now it is handled very well and did not affect on server work, should I worry about it? Or just filter out logs with /phpmy/ pattern and forgot about it.
Before this attacks I have request with method PROPFIND, should I blocked it for security reasons? It is handled by default server for now.
I know that I can use Cloudflare Argotunel or ELB + WAF. But I am not really want to do it for now.
I have found one solution on stackoverflow. Is whitelist of all cloudflare ips. But i think it is not a good one.
Also another solution that should work I guess it is to check Host header, and compare it with 'example.com'.
To answer your specific questions:
Every public IP receives unwanted traffic like you describe, sadly its pretty normal. This isnt really an attack as such, its just a bot looking for signs of specific weaknesses, or otherwise trying to provoke a response that contains useful data. This data is no doubt later used in actual attacks, but its basically automated recognisance on a potentially massive scale.
This kind of script likely isnt trying to do any damage, so as long your server is well configured & fully patched its not a big concern. However these kinds of scans are first step towards launching an attack - by identifying services & application versions with known vulnerabilities - so its wise to keep your logs for analysis.
You should follow the principle of least privilege. PROPFIND is related to WebDAV - if you dont use it, disable it (or better white list the verbs you do support and ignore the rest).
If your site is already behind CloudFlare then you really should firewall access to your IP so only Cloudflares IPs can talk to your server. Those IPs do change, so I would suggest a script to download the latest from https://www.cloudflare.com/ips-v4 and have it periodically update your firewall. Theres a slightly vuage help article from CloudFlare on the subject here: https://support.cloudflare.com/hc/en-us/articles/200169166-How-do-I-whitelist-Cloudflare-s-IP-addresses-in-iptables-
If for whatever reason you cant firewall the IP, your next best option is something like fail2ban (www.fail2ban.org) - its a log parser that can manipulate the firewall to temporarily or permanently block an IP address based on patterns found in your log files.
A final thought - id advise against redirecting from your IP to your domain name - your telling the bot/hackers your URL - which they can then use to bypass the CDN and attack your server directly. Unless you have some reason to allow HTTP/HTTPS traffic to your IP address, return a 4XX (maybe 444 a " Connection Closed Without Response") instead of redirecting when requests hit your IP. You should then create a separate server block to handle your redirects, but only have it respond to genuine named URLs.
I have a Django webapp. It runs inside Docker on Elastic Beanstalk.
I'd like to specify a health check URL for slightly more advanced health checking than "can the ELB establish a TCP connection".
Entirely reasonably, the ELB does this by connecting to the instance over HTTP, using the instance's hostname (e.g. ec2-127-0-0-1.compute-1.amazonaws.com) as the Host header.
Django has ALLOWED_HOSTS which validates the Host header of incoming requests. I set this to my application's external domain via environment variable.
Unsurprisingly and entirely reasonably, Django thus rejects ELB URL health checks due to lack of matching Host.
We don't want to disable ALLOWED_HOSTS because we'd like to be able to trust get_host().
The solutions so far seem to be:
Somehow persuade Django to not care about ALLOWED_HOSTS for certain specific paths (i.e. the health check URL)
Do something funky like calling the EC2 info API on startup to get the host's FQDN and append it to ALLOWED_HOSTS
Neither of these seem particularly pleasant. Can anyone recommend a better / existing solution?
(For the avoidance of doubt, I believe this problem to be identical to the scenario of "Disabled ALLOWED_HOSTS, fronting HTTPD that filters on host" - I want the health check to hit Django, not a fronting HTTPD)
If the ELB health check is sending its request with a host header containing the elastic beanstalk domain (*.elasticbeanstalk.com, or an EC2 domain *.amazonaws.com) then the standard ALLOWED_HOSTS can still be used with a wildcard entry of '.amazonaws.com' or '.elasticbeanstalk.com'.
In my case I received standard ipv4 addresses as the health check hosts, so a different solution was needed. If you can't predict the host at all, and it might be safer to assume you can't, you would need to take a route such as one of the following.
You can use Apache to handle approved hosts instead of propagating ambiguous requests to Django. Since the host header is intended to be the hostname of the server receiving the request, this solution changes the header of valid requests to use the expected site hostname. With elastic beanstalk you'll need to configure Apache using .ebextensions as described here. Under the .ebextensions directory in your project root, add the following to a .config file.
files:
"/etc/httpd/conf.d/eb_healthcheck.conf":
mode: "000644"
owner: root
group: root
content: |
<If "req('User-Agent') == 'ELB-HealthChecker/1.0' && %{REQUEST_URI} == '/status/'">
RequestHeader set Host "example.com"
</If>
Replacing /status/ with your health check URL and example.com with your site's appropriate domain. This tells Apache to check all incoming requests and change the host headers on requests with the appropriate health check user agent that are requesting the appropriate health check URL.
If you would really prefer not to configure Apache, you could write a custom middleware to authenticate health checks. The middleware would have to override Django's CommonMiddleware which calls HttpRequest's get_host() method that validates the request's host. You could do something like this
from django.middleware.common import CommonMiddleware
class CommonOverrideMiddleware(CommonMiddleware):
def process_request(self, request):
if not('HTTP_USER_AGENT' in request.META and request.META['HTTP_USER_AGENT'] == 'ELB-HealthChecker/1.0' and request.get_full_path() == '/status/'):
return super().process_request(request)
Which just allows any health check requests to skip the host validation. You'd then replace django.middleware.common.CommonMiddleware with path.CommonOverrideMiddleware in your settings.py.
I would recommend using the Apache configuration approach to avoid any details in the middleware, and to completely isolate Django from host issues.
This is what I use, and it works well:
import socket
local_ip = str(socket.gethostbyname(socket.gethostname()))
ALLOWED_HOSTS=[local_ip, '.mydomain.com', 'mydomain.elasticbeanstalk.com' ]
where you replace mydomain and mydomain.elasticbeanstalk.com with your own.
Here is my setup.
Public site hosted by squarespace.com (www.example-domain.com)
Web application (AWS EC2/ELB), i would like to be available via the same domain. (my.example-domain.com)
Custom profile pages available as www.example-domain.com/username
My question is how can i setup the DNS to achieve this? If can't do it just through DNS, any suggestions? The problem i am facing is that if squarespace.com is handling the www.example-domain.com traffic how can i have it only partially handle it for certain urls. Maybe i am going about this in the wrong was all together though.
The two first are ok. As you mention, (1) is not compatible with (3) for a pure DNS config as www of example-domain.com has to be configured to a single end-point.
Some ideas of non-DNS workaround:
Having the squarespace.com domain on sqsp.example-domain.com and configure your www domain to a custom web server on which you configure the root (/) to redirect (HTTP 300) to sqsp.example-domain.com. It will be quite transparent for the user, except in his browser address.
The same but setting on / a full page HTML iframe containing sqsp.example-domain.com.
The iframe approach is a "less clean", Google the solutions to build your opinion.
EDIT:
As #mike-ryan mentioned, there is the proxy solution as well where you configure you web server to request another server to get the content to return to your user. If you are already using AWS, a smart way to do this is to use CloudFront: you can setup CloudFront to proxy one server on one URL and proxy another server on other URL. Actually, this is maybe the faster to way to implement you need. Of course, a proxy is one more "hop", so it may add more delay.
If you really want to have content served from different servers while only using a single domain name, you'll need to set up a proxy server to handle the request routing for you. I am assuming your custom profile pages must be served from your EC2 instance.
Nginx will receive all requests, and will then decide whether they should be sent to Square Space or your web app. Requests will be reverse proxied to Square Space or to your app, depending on the URL.
This is similar to #smad's answer, except it will all be invisible to the users which IMHO is better than redirecting the user to a new domain name.
Example steps:
Set up an Nginx server, create two virtual hosts - one for my.example.com, and one for www.example.com
Create two upstreams in your Nginx config - one for Square Space, and one for your app
Configure the www.example.com virtual host to reverse proxy connections to the Square Space upstream, if the URL is "/". Otherwise, traffic should be proxied to your app upstream [0]
Configure the my.example.com virtual host to proxy all traffic to your app upstream
[0] how to reverse proxy via nginx a specific url?
I am curious if anyone knows, is it possible to build a varnish config which
looks at inbound headers from clients
and does regexp match, to build a certain header - that is passed on to backed web server?
Context:
-- Varnish server deployed in front of 3-(apache)-webserver pool, acting as cache and LB
-- 3rd party service DDOS proxy was dropped in front of varnish recently
-- now all inbound requests to varnish have an X-Forwarded-For: header already present, indicating what the 'real' client IP address is.
-- the back-end PHP web app running on the apache hosts - wants to see only one IP address, not a 'proper' X-F-F header which shows a comma-delimited list IP1, IP2, IP3 (for external proxy; varnish as proxy, original client IP - for example)
-- ideally what I would like to do: is get Varnish to do regexp pattern match:rebuild on the existing inbound X-F-F headers; and strip away the external DDOS proxy IP address
I had tried something like this in the 'default.vcl' file,
set req.http.X-Forwarded-For = regsub(req.http.X-Forwarded-For, "123.123.123.123"," ");
where 123.123.123.123 is the IP address of the external DDOS / proxy
but this most certainly is not working / is not the correct syntax. I'm having trouble finding good resources online that explain regexp in varnish / or good clear examples that people use.
If anyone can provide a kick in the right direction, certainly it would be greatly appreciated.
Thanks for comment below - I have adjusted my config now as follows,
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
regsub(req.http.X-Forwarded-For, "\, 123\.123\.123\.123","");
} else {
set req.http.X-Forwarded-For = client.ip;
}
and now I find in my apache logs, for example,
client.ip.real.here, 123.123.123.123 - - [22/Dec/2013:07:45:17 -0800] "GET /PATH/TO/STUFF" "Mozilla/5.0.... like Gecko"
which suggests to me the regexp is not actually doing what I thought I wanted. Is there any way to 'test' varnish interactively, ie, feed a given VCL rule set some defined input, and then get varnish to feed out specific output; kind of a 'dry run' processing as a way to facilitate debug ?
Thanks!