I found issue in elmah log about some exceptions (apparently related attract to my web site)
what is the diff between REMOTE_ADDR and LOCAL_ADDR in Elmah log ?
what does it means if IPs are different ? is it some VPN ?
Related
Since adding the option to email me (the admin) when there are problems with my Django server, I keep getting a LOT of the following emails (20 in the last hour alone).
[Django] ERROR (EXTERNAL IP): Invalid HTTP_HOST header: 'staging.menthanhgiadalat.com'. You may need to add 'staging.menthanhgiadalat.com' to ALLOWED_HOSTS.
I've set my server up to have the following at the top of the file in my sites-enabled nginx config as I read (somewhere on SO) that this would may prevent me from getting these types of emails:
server {
server_name _;
return 444;
}
But it hasn't done anything.
In the next server block I have the IP address and domain names for my site. Could this be causing the problem?
This 'staging' site isn't the only domain I'm being asked to add to my ALLOWED_HOSTS. But it is, by far, the most frequent.
Can I stop this type of alert being sent? Can I stop it from being raised? Is there something I've configured incorrectly on my server (I'm ashamed to admit I'm pretty new at this).
Thanks for any help you might be able to give.
You can configure LOGGING in your settings.py to silence django.security.DisallowedHost as directed at https://docs.djangoproject.com/en/3.2/topics/logging/#django-security
I have an ALB with 2 targets in the target group. However my health check aren't working properly. Both are showing
"Health checks failed with these codes: [404]"
My settings for the health check path are:
/var/www/html/generic/website.com/healthcheck.php
and if I do a nano /var/www/html/generic/website.com/healthcheck.php on the ec2 instance it shows this which should be all the health check needs I think.
<?php
header("Status: 200");
?>
I double checked the AZ and the ALB is in the same one and subnets as the 2 instances.
Also when I check my apache logs this is what I see:
"GET /var/www/html/generic/website.com/healthcheck.php HTTP/1.1" 404 196 "-" "ELB-HealthChecker/2.0"
What am I doing wrong that is making the healthcheck fail?
Seems you can't use a namebased vhost for a healthcheck. So in my vhost file I added the following code. What it does is if someone goes straight to your ip it will give them a 404 but if they go to your ip/healthcheck then it will show a 200 which is what ALB needs. Then in your path just put /healthcheck.
<VirtualHost *:80>
ServerName default
RewriteRule "/healthcheck" - [R=200]
Redirect 404 /
</VirtualHost>
<VirtualHost _default_:80>
RewriteRule "/healthcheck" - [R=200]
Redirect 404 /
</VirtualHost>
The ALB healthcheck is going to be accessing your healthcheck.php via the web interface and has no concept of where the files exist on the file system. This value needs to be the URI path after the hostname.
Configuring the healthcheck to be /var/www/html/generic/website.com/healthcheck.php is equivalent to telling AWS to check http://website.com//var/www/html/generic/website.com/healthcheck.php which is probably not your intention.
Assuming the actual path you want to check is more like http://website.com/healthcheck.php, update your healthcheck path to simply be /healthcheck.php and this should hopefully work for you.
Since you have multiple named hosts in nginx. What you could try is setting the healthcheck on the / with 200 OK as the status code. it should return the default nginx page with status code 200 OK.
Another method is, you can add healthcheck url that doesn't exist such as /my-health-page and in the health check configuration set the expected health check status code to be 404. So what we are assuming here is, if the nginx is inspecting url and return 404 NOT FOUND,the Nginx is working. But this method wont be checking anything more than that.
Pros:
ALB is able to whether your instance is running
ALB can check whether nginx is running
Cons:
The health check wont identify the cases for example php not working or mysql server is not reachable from the application etc
I have a Django application running on Elastic Beanstalk. I can visit my site no problem at example.com. I've set up automatic https redirect, so that it always directs to https. I've set it up so you can't view the site example.elasticbeanstalk.com domain -- if you go there you end up getting response code 400.
My auto scaling group is load balanced. My app is failing the health checks with status code 400, even though I can navigate to my site no problem with response code 200. My logs show:
***amazon IP*** (-) - - [date] "GET / HTTP/1.1" 400 26 "-" "ELB-HealthChecker/2.0"
I'm guessing the error is either from
Not allowing connection at example.elasticbeanstalk.com
Haivng automatic HTTP -> HTTPS redirect (although that would come up with a 302 I'd guess)
When the Health Check pings a site, is it pinging your custom domain (example.com) or is pining the elasticbeanstalk.com domain? What can I do to either fix this or further diagnose the error? I'd rather not allow traffic at the elasticbeanstalk.com domain, because I don't think I can get SSL on that.
The reason this is failing is because the health check checks the EC2 instance private IP. This can change with ELB, so you need to dynamically get the private IP of the instance and add it to hosts. See How to dynamically add EC2 ip addresses to Django ALLOWED_HOSTS
import requests
EC2_PRIVATE_IP = None
try: EC2_PRIVATE_IP = requests.get('http://169.254.169.254/latest/meta-data/local-ipv4', timeout=0.01).text
except requests.exceptions.RequestException: pass
if EC2_PRIVATE_IP: ALLOWED_HOSTS.append(EC2_PRIVATE_IP)
(potentially) Bad Answer
I found this answer at another SO post. While it solves the problem, I do not think it is a good answer and may be insecure.
If you add this code to your .ebextensions/something.config file, it will redirect any requests from Health Checker with a certain status request to your domain.
files:
"/etc/httpd/conf.d/eb_healthcheck.conf":
mode: "000644"
owner: root
group: root
content: |
<If "req('User-Agent') == 'ELB-HealthChecker/2.0' && %{REQUEST_URI} == '/status/'">
RequestHeader set Host "sub.example.com"
</If>
Replacing /status/ with what the health check url specified in Config -> Loan Balancer -> Health Check Path, and sub.example.com with your domain. They've also updated the health checker so it's ELB-HealthChecker/2.0 now -- another thing to pay attention to.
HOWEVER: It may not be great for security reasons, I think this could be spoofed. If you were using the default / link, someone could spoof ELB-HealthChecker/2.0 and then easily guess your link. I'm not very familiar with what someone could do with a set Host command, it may be harmless.
If you recently migrated to Amazon Linux 2 and got hit with IMDSv2 then you have to use security token like this
import requests
EC2_PRIVATE_IP = None
try:
security_token = requests.put(
'http://169.254.169.254/latest/api/token',
headers={'X-aws-ec2-metadata-token-ttl-seconds': '60'}).text
EC2_PRIVATE_IP = requests.get(
'http://169.254.169.254/latest/meta-data/local-ipv4',
headers={'X-aws-ec2-metadata-token': security_token},
timeout=0.01).text
except requests.exceptions.RequestException:
pass
if EC2_PRIVATE_IP:
ALLOWED_HOSTS.append(EC2_PRIVATE_IP)
Just to follow up. I was running a multi-container Docker environment on AWS Linux 2 with Django on Elastic Beanstalk. My environment was in a permanent severe state even though my app was accessible! Thanks to the answers above, I learned that the health checks were occurring at addresses that were simply not the Elastic Beanstalk URL! Also, the HTTP statuses were not visible on the EB environment health page, I had to go to the EC2 page and to the "target groups" health checks tab under load balancers to find out that my app was returning 400 codes to the health checks. To quickly test the solution, I just added ALLOWED_HOSTS = ['*'] (not good for production!) and sure enough, the issues disappeared!
I originally thought it was a Nginx issue and so I configured a Nginx container that worked with my Django app container. Not sure if that's necessary anymore. A totally frustrating and undocumented issue, but that comes with the territory of Elastic Beanstalk.
I have a bit problem with my site.
So setup is ElasticBeanstalk(NGINX) + Cloudflare
But each day around 4AM I have direct IP attack to my server.
Around 300 requests in 1-2 minutes.
Bot try to access some resources like
GET /phpMyadmi/index.php HTTP/1.1
GET /shaAdmin/index.php HTTP/1.1
POST /htfr.php HTTP/1.1
For now all of them going to 80 or 8080 ports.
And successfully handled by Nginx configuration that redirect it to example:443
server {
listen 80 default_server;
listen 8080 default_server;
server_name _;
return 301 https://example.com$request_uri;
}
server {
listen 443 ssl;
server_name example.com;
ssl on;
...
So questions are,
have many site owners/devOps face the same attack. What is your action to prevent such attacks.
For now it is handled very well and did not affect on server work, should I worry about it? Or just filter out logs with /phpmy/ pattern and forgot about it.
Before this attacks I have request with method PROPFIND, should I blocked it for security reasons? It is handled by default server for now.
I know that I can use Cloudflare Argotunel or ELB + WAF. But I am not really want to do it for now.
I have found one solution on stackoverflow. Is whitelist of all cloudflare ips. But i think it is not a good one.
Also another solution that should work I guess it is to check Host header, and compare it with 'example.com'.
To answer your specific questions:
Every public IP receives unwanted traffic like you describe, sadly its pretty normal. This isnt really an attack as such, its just a bot looking for signs of specific weaknesses, or otherwise trying to provoke a response that contains useful data. This data is no doubt later used in actual attacks, but its basically automated recognisance on a potentially massive scale.
This kind of script likely isnt trying to do any damage, so as long your server is well configured & fully patched its not a big concern. However these kinds of scans are first step towards launching an attack - by identifying services & application versions with known vulnerabilities - so its wise to keep your logs for analysis.
You should follow the principle of least privilege. PROPFIND is related to WebDAV - if you dont use it, disable it (or better white list the verbs you do support and ignore the rest).
If your site is already behind CloudFlare then you really should firewall access to your IP so only Cloudflares IPs can talk to your server. Those IPs do change, so I would suggest a script to download the latest from https://www.cloudflare.com/ips-v4 and have it periodically update your firewall. Theres a slightly vuage help article from CloudFlare on the subject here: https://support.cloudflare.com/hc/en-us/articles/200169166-How-do-I-whitelist-Cloudflare-s-IP-addresses-in-iptables-
If for whatever reason you cant firewall the IP, your next best option is something like fail2ban (www.fail2ban.org) - its a log parser that can manipulate the firewall to temporarily or permanently block an IP address based on patterns found in your log files.
A final thought - id advise against redirecting from your IP to your domain name - your telling the bot/hackers your URL - which they can then use to bypass the CDN and attack your server directly. Unless you have some reason to allow HTTP/HTTPS traffic to your IP address, return a 4XX (maybe 444 a " Connection Closed Without Response") instead of redirecting when requests hit your IP. You should then create a separate server block to handle your redirects, but only have it respond to genuine named URLs.
I've been getting many of these error emails from my django site. They look like they are triggered from some automated exploit. Here is one example.
Referrer: http://example.com/fck/editor/filemanager/upload/test.html
Requested URL: /fck/editor/filemanager/upload/test.html
User agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90)
IP address: 127.0.0.1
Please help me answer 2 questions:
How can I configure Django to log the real ip origin of the exploiter, i.e., something along the lines of REMOTE_ADDR instead of the localhost ip.
Is there a way to reject requests with fake referrers to begin with? The requested and referred URLs are certainly not valid links from my own example.com site, and have never been.
Thanks
I figured out my own problem so in case anyone else need this info, here goes...
I kept getting the localhost ip in the error email because my django server lives behind a reverse proxy on the same machine. In this scenario REMOTE_ADDR is always the localhost address.
There is no template or custom error reporting mechanism to get other variables into the broken link email because the email is hardcoded in django's CommonMiddleware The custom error reports mentioned in previous comments has nothing to do with this.
So in order to get the real ip address, I wrote a middleware to replace REMOTE_ADDR with HTTP_X_FORWARDED_FOR. Supposedly there is a security issue involved since HTTP_X_FORWARDED_FOR can be easily faked but that is all that can be done without a CommonMiddleWare patch to actually include both ip variables.