I've been getting many of these error emails from my django site. They look like they are triggered from some automated exploit. Here is one example.
Referrer: http://example.com/fck/editor/filemanager/upload/test.html
Requested URL: /fck/editor/filemanager/upload/test.html
User agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90)
IP address: 127.0.0.1
Please help me answer 2 questions:
How can I configure Django to log the real ip origin of the exploiter, i.e., something along the lines of REMOTE_ADDR instead of the localhost ip.
Is there a way to reject requests with fake referrers to begin with? The requested and referred URLs are certainly not valid links from my own example.com site, and have never been.
Thanks
I figured out my own problem so in case anyone else need this info, here goes...
I kept getting the localhost ip in the error email because my django server lives behind a reverse proxy on the same machine. In this scenario REMOTE_ADDR is always the localhost address.
There is no template or custom error reporting mechanism to get other variables into the broken link email because the email is hardcoded in django's CommonMiddleware The custom error reports mentioned in previous comments has nothing to do with this.
So in order to get the real ip address, I wrote a middleware to replace REMOTE_ADDR with HTTP_X_FORWARDED_FOR. Supposedly there is a security issue involved since HTTP_X_FORWARDED_FOR can be easily faked but that is all that can be done without a CommonMiddleWare patch to actually include both ip variables.
Related
<ip> is showing an actual ip address, I just didn't include it in the title.
I believe this ip address is the internal ip of my EC2 instance. I'm using AWS Elastic beanstalk to host.
I see this question has been answered a lot on SO, and the answer is always to add the ip address to the ALLOWED_HOSTS, but in my case I've set ALLOWED_HOSTS=['*'] and I'm still getting the error.
The weird thing is, I'm only getting the error when I try to access the site from my phone. When I access from the desktop browser, it works fine...
Things I've tried:
I've double checked my elastic beanstalk deployment and the changes are definitely deployed.
Ok so this probably won't be the answer for other people but it was for me..
In my case, I was doing an http GET request from my frontend and forgot the extra "/" at the end of the url. My django urls.py defines the url with a slash at the end. My fix was to add the extra "/" when doing the http GET.
On my desktop, this was automatically handled because django would reply with an automatic redirect (302) and my desktop browser would go to the url with / at the end, but my phone was not doing the redirect!
This somehow was throwing the invalid HTTP_HOST header error.
For most people, the fix for an error message like this is to add all of your domains to the list of ALLOWED_HOSTS.
Oh and if you're using elasticbeanstalk like me, don't forget to add the AWS domain name. It should look something like this:
ALLOWED_HOSTS = ['<your-unique-id>.elasticbeanstalk.com',
'example.com', '<your-subdomain>.example.com']
DON'T do ALLOWED_HOSTS = ['*'] in prod!!
The problem I am facing is that after changing the hostname and configuring the reverse proxy as described here and here, as well as following the troubleshooting guide here to resolve the 'the registered callback does not match' I am unable to get any further.
I've followed a number of other examples of how to configure nginx and add the reverseProxy property to the settings.js configs but with no luck.
As you can see below if I go to https://example.com/publisher I continue getting the error 'The registered callback does not match'
Here is what I have the callback regex set to:
regexp=(https://example.com/publisher/services/auth/callback/login|https://example.com/publisher/services/auth/callback/logout)
If I inspect the authorize request query I can see that the redirect_url is being set to 127.0.0.1 and I suspect that is the problem as when I add that url to the service provider regex callback it works, but this is not suitable in a non local environment.
And here is the request query (where I suspect the main issue lies - note redirect_uri):
https://example.com/oauth2/authorize?response_type=code&client_id=1obvNiUMBcJwMa3euoHjrsckuGIa&scope=apim:api_create%20apim:api_delete%20apim:api_import_export%20apim:api_product_import_export%20apim:api_publish%20apim:api_view%20apim:app_import_export%20apim:client_certificates_add%20apim:client_certificates_update%20apim:client_certificates_view%20apim:document_create%20apim:document_manage%20apim:ep_certificates_add%20apim:ep_certificates_update%20apim:ep_certificates_view%20apim:external_services_discover%20apim:mediation_policy_create%20apim:mediation_policy_manage%20apim:mediation_policy_view%20apim:pub_alert_manage%20apim:publisher_settings%20apim:shared_scope_manage%20apim:subscription_block%20apim:subscription_view%20apim:threat_protection_policy_create%20apim:threat_protection_policy_manage%20openid&state=/&redirect_uri=https://127.0.0.1/publisher/services/auth/callback/login
Here is how my deployment.toml is configured (I've replaced my actual domain with example.com):
Note I had to remove the ports to work behind the proxy
And here is my settings.js:
I added the reverseProxy property as suggested in a github issue
And here is my nginx conf:
This is a known limitation. Please find the steps to resolve the issue - https://apim.docs.wso2.com/en/latest/troubleshooting/troubleshooting-invalid-callback-error/#troubleshooting-registered-callback-does-not-match-with-the-provided-url-error
The reason for this error comes down to a missing X-Forwarded-For header, I ended up changing the forwardedHeader in settings.js to Host as that was being passed from my proxy server.
Thanks for the detailed question "user3745065".
I was having the exactly same issue you described in this post, and I guess I nailed the problem down.
Like you mentioned the issue is with the forwardedHeader, that in your case you switched to Host.
But checking the product documentation, the sample they provide is the following:
customUrl: { // Dynamically set the redirect origin according to the forwardedHeader host|proxyPort combination
enabled: true,
forwardedHeader: 'X-Forwarded-Host',
},
It took me a while to noticed that the forwardedHeader is supposed to be 'X-Forwarded-Host' not 'X-Forwarded-For' as it comes as default.
Few other things I needed to tweak that wasn't clear in the documentation for changing the hostname (here), I had to remove the port variable ${mgt.transport.https.port} from devportal.
That's outlined on the installation step 5 also, here. However worth mentioning:
from:
[apim.devportal]
url = "https://{Your Domain}:${mgt.transport.https.port}/devportal"
to
url = "https://{Your Domain}/devportal"
otherwise when the it tries to redirect to the portal (for instance, from the publisher) it construct the url with the port number, and that default port 9443 isn't going to work on your proxy (tested on nginx with the provided settings that is on the documentation here) which is listening and expecting calls on port 443.
Things that I noticed you configured but perhaps it's not necessary:
Set the apim.idp settings
Set the reverseProxy settings
Set the apim.gateway.environment settings (Not related to the callback url issue, this is meant for you to configure the runtime gateway urls)
Last but not the least, Following the "Troubleshooting 'Registered callback does not match with the provided url' error", again you need to remove the port number from the url, otherwise you will have the same issue aforementioned on your proxy.
Just my 2 cents! ;)
I keep getting an Invalid HOST Header error which I am trying to find the cause of. It reads as such:
Report at /GponForm/diag_Form
Invalid HTTP_HOST header: '192.168.0.1:443'. You may need to add '192.168.0.1' to ALLOWED_HOSTS
I do not know what /GponForm/diag_Form is but from the looks of it, it may be a vulnerability attacked by malware.
I also am wondering why the IP is from a router 192.168.0.1 as well as why it is coming through SSL :443
Should I consider putting a HoneyPot and blocking this IP address? Before I do, why does the IP look like a local router?
The full Request URL in the report looks like this:
Request URL: https://192.168.0.1:443/GponForm/diag_Form?style/
I am getting this error at least ~10x/day now so I would like to stop it.
Yes, this surely represents a vulnerability - someone tried to access this url on router (which usually have ip 192.168.0.1).
It looks so because request from attacker contains HOST header with this value.
Maybe django is run locally with DEBUG=True.
You may consider running it more production wised with web-server (i.e. nginx) in front filtering unwanted requests with nginx config and further adding fail2ban to parse nginx error logs and ban ip.
Or make site available only from specific ips / ads simple authorization, i.e. Basic Auth on web-server level.
Previous irrelevant answer
ALLOWED_HOSTS option specifies domains django project can serve.
In running locally - python manage.py runserver or with DEBUG=True - it defaults to localhost, 127.0.0.1 and similar.
If you are accessing django via different url - it will complain in such a manner.
To allow access from another domains - add them to ALLOWED_HOSTS: ALLOWED_HOSTS = ['localhost', '127.0.0.1', '[::1]', '192.168.0.1'].
I developed Django based web application in my company which I was able to access at localhost:
http://127.0.0.1:8000/olx
and made the ajax request without fail. When I accessed the same on my computer's IP address:
http://10.0.100.148:8000/olx
, all things were working except ajax reuquest. I got the following error message:
403 forbidden
Later I did the same thing in my personal laptop at my home and saw that I was able to access ajax request as well in both the cases: localhost and my laptop's IP address:
http://192.168.1.8:8000/olx
, which was connected to a wifi network. I also accessed the application in my mobile,connected to the same wifi-network, by typing the laptop's IP address and it was working fine.
Now I am in confusion that why I was not able to make the ajax request at the IP address of my company's computer.
What went wrong.
Can somebody help me to understand the reason behind this?
Did you check django logs, anything more than a 403? Could it be an authentication issue?
If you happen to have DEBUG = False then make sure the ALLOWED_HOSTS variable in settings has all the IP-s you listed.
I have used ShimmerCat with sc-tool to connect to my development sites as described here, and everything has worked always like a charm with it, but I also wanted to follow the "old way" configuring my /etc/hosts. In this case I had a small problem, the server ran ok, and I could access to my development site (let's say that I used https://www.example.com:4043/), but I'm also using a reverse proxy as described on this article, and on the config file reference. It redirects to a Django app I'm using. Let's say it is my devlove.yaml config file:
---
shimmercat-devlove:
domains:
www.example.com:
root-dir: site
consultant: 8080
cache-key: xxxxxxx
api.example.com:
port: 8080
The problem is that when I try to access to a URL that requests the API, a 404 response is sent from the API. Let me try to explain it through an example. I try to access to https://www.example.com:4043/country/, and on this page I do a request to the API: /api/<country>/towns/, then the API endpoint is returning a 404 response so it is not finding this URL, which does not happen when using Google Chrome with sc-tool. I had set both domains www.example.com, and api.example.com on my /etc/hosts. I have been trying to solve it, but without any luck, is there something I'm missing? Any help will be welcome. Thanks in advance.
With a bit more of data, we may be able to find the issue. In the meantime, here is a list of troubleshooting tips:
Possible issue: DNS is cached in browser, /etc/hosts is not being used (yet)
This can happen if somehow your browser has not done a DNS lookup since before you changed your /etc/hosts file. Then the connection is going to a domain in the Internet that may not have the API endpoint that you are calling.
Troubleshooting: Check ShimmerCat's log for the requests. If this is the issue, closing and opening the browser may solve the issue.
Possible issue: the host header is incorrect
ShimmerCat uses the Host header in HTTP/1.1 requests and the :authority header in HTTP/2 requests to distinguish the domains. It always discards any port number present in them. If these headers are not set or are set to a domain other than the ones ShimmerCat is configured to listen, the server will consider the situation so despicable that it will just close the connection.
Troubleshooting: This is not a 404 error, but a connection close (if trying to connect un-proxied, directly to the SSL port where ShimmerCat is listening), or a Socks Connection Failed (if trying to connect through ShimmerCat's built-in SOCKS5 proxy). In the former case, the server will print the message "Rejected request to Just https://some-domain-or-ip/some/path" in his log, using the actual value for the domain, or "Rejected request to Nothing", if no header was present. The second case is more complicated, because the SOCKS5 proxy is before the HTTP routing algorithm.
In any case, the browser will put a red line in the network panel of the developer tools. If you are accessing the server using curl, like this:
curl -k -H host:api.incorrect-domain.com https://127.0.0.1:4043/contents/blog/data-density/
or like
curl -k -H host:api.incorrect-domain.com
(notice the --http2 parameter in the second form), you will get a response:
curl: (56) Unexpected EOF
Extra-tip: There is a field for the network address in the browser's developer tools. Check it, it may tell you something!
Possible issue: something gets messed up when passing the request to the api back-end.
API backends are also sensitive to the host header, and to additional things like authentication cookies and request parameters.
Troubleshooting: A way to diagnose things is invoking ShimmerCat using the --show-proxied-headers command-line option. It makes ShimmerCat to report the proxied headers to the log:
Issuing request with headers :authority: api.example.com
:method: GET
:path: /my/api/endpoint/path/
:scheme: https
accept: */*
user-agent: curl/7.47.0
Possible issue: there are two instances or more of ShimmerCat running
...and they are using different configurations. ShimmerCat uses port sharing among several processes to increase availability. A downside of this is that is perfectly possible to mistakenly start ShimmerCat, forget about stopping it, and start it again after changing some configuration bit. The two instances will be running at the same time, and any of them will pick connections made to the listening port.
Troubleshooting: Shutdown all instances of ShimmerCat, then double-check there are none running by using the corresponding form of the ps command, and start the server with the configuration you want.