I am working on a Django project where users will have custom info given to them depending on their location. In order to do this, I use their IP address to identify their country. In order to keep data in the database consistent, I need to make sure I have an accurate IP.
I understand that using META usually uses headers sent by the client's browser, but I don't know if that applies to the REMOTE_ADDR attribute.
TLDR: what is the difference between HttpRequest.get_host() and HttpRequest.META['REMOTE_ADDR']?
The difference between HttpRequest.get_host() and HttpRequest.META['REMOTE_ADDR'] is that the first one checks IP in the following headers in order of decreasing preference:
HTTP_X_FORWARDED_HOST
HTTP_HOST
SERVER_NAME combined with SERVER_PORT
whereas the second one check the IP in the header REMOTE_ADDR.
There is a huge difference in the type of information returned: get_host() will give you the name of the server hosting your application, not the IP of the client.
More in detail, here is the implementation of get_host():
def get_host(self):
"""Returns the HTTP host using the environment or request headers."""
# We try three options, in order of decreasing preference.
if settings.USE_X_FORWARDED_HOST and (
'HTTP_X_FORWARDED_HOST' in self.META):
host = self.META['HTTP_X_FORWARDED_HOST']
elif 'HTTP_HOST' in self.META:
host = self.META['HTTP_HOST']
else:
# Reconstruct the host using the algorithm from PEP 333.
host = self.META['SERVER_NAME']
server_port = str(self.META['SERVER_PORT'])
if server_port != ('443' if self.is_secure() else '80'):
host = '%s:%s' % (host, server_port)
allowed_hosts = ['*'] if settings.DEBUG else settings.ALLOWED_HOSTS
domain, port = split_domain_port(host)
if domain and validate_host(domain, allowed_hosts):
return host
else:
msg = "Invalid HTTP_HOST header: %r." % host
if domain:
msg += "You may need to add %r to ALLOWED_HOSTS." % domain
raise DisallowedHost(msg)
If you want to check for client IP address, here are some headers that could be worth checking (see Getting the client IP address: REMOTE_ADDR, HTTP_X_FORWARDED_FOR, what else could be useful?):
REMOTE_ADDR
HTTP_X_FORWARDED_FOR
HTTP_CLIENT_IP
HTTP_X_FORWARDED_FOR can be comma delimited list of IPs
HTTP_X_FORWARDED
HTTP_X_CLUSTER_CLIENT_IP
HTTP_FORWARDED_FOR
HTTP_FORWARDED
If you don't know which one to pick (if not all), you could log those headers and pragmatically add new checkings over time.
Related
How can I capture data of someone visiting my website via a shortened link? I want to be able to capture the OS, IP, divide ID etc. Would capturing the user agent string be the best option? What are some other ways? In my case I'm building a website with DJAGNO.
To get the IP, you can use the HTTP_X_FORWARDED_FOR or REMOTE_ADDR headers from request.META
For instance:
FORWARDED_HEADER = "HTTP_X_FORWARDED_FOR"
REMOTE_HEADER = "REMOTE_ADDR"
headers = request.META
ip_header = FORWARDED_HEADER if FORWARDED_HEADER in headers else REMOTE_HEADER
ip_address = headers.get(ip_header)
For the user agent you can use the HTTP_USER_AGENT header:
USER_AGENT_HEADER = "HTTP_USER_AGENT"
user_agent = request.META.get(USER_AGENT_HEADER)
My original question was how to enable HTTPS for a Django login page, and the only response, recommended that I - make the entire site as HTTPS-only.
Given that I'm using Django 1.3 and nginx, what's the correct way to make a site HTTPS-only?
The one response mentioned a middleware solution, but had the caveat:
Django can't perform a SSL redirect while maintaining POST data.
Please structure your views so that redirects only occur during GETs.
A question on Server Fault about nginx rewriting to https, also mentioned problems with POSTs losing data, and I'm not familiar enough with nginx to determine how well the solution works.
And EFF's recommendation to go HTTPS-only, notes that:
The application must set the Secure attribute on the cookie when
setting it. This attribute instructs the browser to send the cookie
only over secure (HTTPS) transport, never insecure (HTTP).
Do apps like Django-auth have the ability to set cookies as Secure? Or do I have to write more middleware?
So, what is the best way to configure the combination of Django/nginx to implement HTTPS-only, in terms of:
security
preservation of POST data
cookies handled properly
interaction with other Django apps (such as Django-auth), works properly
any other issues I'm not aware of :)
Edit - another issue I just discovered, while testing multiple browsers. Say I have the URL https://mysite.com/search/, which has a search form/button. I click the button, process the form in Django as usual, and do a Django HttpResponseRedirect to http://mysite.com/search?results="foo". Nginx redirects that to https://mysite.com/search?results="foo", as desired.
However - Opera has a visible flash when the redirection happens. And it happens every search, even for the same search term (I guess https really doesn't cache :) Worse, when I test it in IE, I first get the message:
You are about to be redirected to a connection that is not secure - continue?
After clicking "yes", this is immediately followed by:
You are about to view pages over a secure connection - continue?
Although the second IE warning has an option to turn it off - the first warning does not, so every time someone does a search and gets redirected to a results page, they get at least one warning message.
For the 2nd part of John C's answer, and Django 1.4+...
Instead of extending HttpResponseRedirect, you can change the request.scheme to https.
Because Django is behind Nginx's reverse proxy, it doesn't know the original request was secure.
In your Django settings, set the SECURE_PROXY_SSL_HEADER setting:
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
Then, you need Nginx to set the custom header in the reverse proxy. In the Nginx site settings:
location / {
# ...
proxy_set_header X-Forwarded-Proto $scheme;
}
This way request.scheme == 'https' and request.is_secure() returns True.
request.build_absolute_uri() returns https://... and so on...
Here is the solution I've worked out so far. There are two parts, configuring nginx, and writing code for Django. The nginx part handles external requests, redirecting http pages to https, and the Django code handles internal URL generation that has an http prefix. (At least, those resulting from a HttpResponseRedirect()). Combined, it seems to work well - as far as I can tell, the client browser never sees an http page that the users didn't type in themselves.
Part one, nginx configuration
# nginx.conf
# Redirects any requests on port 80 (http) to https:
server {
listen 80;
server_name www.mysite.com mysite.com;
rewrite ^ https://mysite.com$request_uri? permanent;
# rewrite ^ https://mysite.com$uri permanent; # also works
}
# django pass-thru via uWSGI, only from https requests:
server {
listen 443;
ssl on;
ssl_certificate /etc/ssl/certs/mysite.com.chain.crt;
ssl_certificate_key /etc/ssl/private/mysite.com.key;
server_name mysite.com;
location / {
uwsgi_pass 127.0.0.1:8088;
include uwsgi_params;
}
}
Part two A, various secure cookie settings, from settings.py
SERVER_TYPE = "DEV"
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True # currently only in Dev branch of Django.
SESSION_EXPIRE_AT_BROWSER_CLOSE = True
Part two B, Django code
# mysite.utilities.decorators.py
import settings
def HTTPS_Response(request, URL):
if settings.SERVER_TYPE == "DEV":
new_URL = URL
else:
absolute_URL = request.build_absolute_uri(URL)
new_URL = "https%s" % absolute_URL[4:]
return HttpResponseRedirect(new_URL)
# views.py
def show_items(request):
if request.method == 'POST':
newURL = handle_post(request)
return HTTPS_Response(request, newURL) # replaces HttpResponseRedirect()
else: # request.method == 'GET'
theForm = handle_get(request)
csrfContext = RequestContext(request, {'theForm': theForm,})
return render_to_response('item-search.html', csrfContext)
def handle_post(request):
URL = reverse('item-found') # name of view in urls.py
item = request.REQUEST.get('item')
full_URL = '%s?item=%s' % (URL, item)
return full_URL
Note that it is possible to re-write HTTPS_Response() as a decorator. The advantage would be - not having to go through all your code and replace HttpResponseRedirect(). The disadvantage - you'd have to put the decorator in front of HttpResponseRedirect(), which is in Django at django.http.__init__.py. I didn't want to modify Django's code, but that's up to you - it's certainly one option.
if you stick your entire site behind https, you don't need to worry about it on the django end. (assuming you don't need to protect your data between nginx and django, only between users and your server)
I have a website using aspnetcore 2.1 and Kestrel (IIS as proxy) and someone else on our domain sets a cookie that contains non-ASCII characters. I know this is wrong, but I can't make sure they don't and I still want the cookie (otherwise it is easy just to remove it in my web.config).
The result being: Cookie is set incorrectly on one site and when the user "unknowingly" goes to our website, our website gives an error 400 on all requests.
Is there a way of removing the bad characters in the cookie header before it hits Kestrel? I thought I could use a HttpModule/Middleware/etc to remove it, but it looks like Kestrel is the first one to get a HttpContext.
No workaround at the side of dotnetcore. If you are using nginx behind of your dotnetcore application, you can delete all of the non-ascii character like that:
set_by_lua_block $cookie_ascii {
local cookie = ngx.var.http_cookie
if cookie == nil or cookie == '' then return cookie end
local cookie_ascii, n, err = ngx.re.gsub(cookie, "[^\\x00-\\x7F]", "")
return cookie_ascii
}
listen 80;
server_name example.com;
location / {
proxy_pass http://localhost:5000;
...
proxy_set_header Cookie $cookie_ascii;
...
}
}
In my Django's settings.py I have
SESSION_COOKIE_HTTPONLY = True
SECURE_CONTENT_TYPE_NOSNIFF = True
SECURE_BROWSER_XSS_FILTER = True
X_FRAME_OPTIONS = 'DENY'
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SECURE_SSL_REDIRECT = True
SECURE_HSTS_SECONDS = 15768000
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_HSTS_PRELOAD = True
SESSION_COOKIE_AGE = 2 * 24 * 3600
However https://detectify.com has found that this flag isn't set for csrftoken cookie. I checked what Chrome tells about the cookie, and if I understand correctly, the empty HTTP column confirms that the two cookies are not HTTP-only:
Also, if I do document.cookie in chrome's console, the csrftoken value is shown.
I wonder why this could be the case. I have Django running on uwsgi and nginx. The nginx configuration is as follows, and the site in question is https://rodichi.net:
server {
listen 443 ssl http2 default_server;
server_name rodichi.net;
ssl_certificate /etc/letsencrypt/live/rodichi.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/rodichi.net/privkey.pem;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
charset utf-8;
... # location settings follow here
```
You have only configured it to have the CSRF token be set to Secure (i.e. only sent over https requests) and not to be HttpOnly (i.e. not available to Javascript).
Looking at the Django documentation you also need to set CSRF_COOKIE_HTTPONLY. However the documentation rightly points out:
Designating the CSRF cookie as HttpOnly doesn’t offer any practical
protection because CSRF is only to protect against cross-domain
attacks. If an attacker can read the cookie via JavaScript, they’re
already on the same domain as far as the browser knows, so they can do
anything they like anyway. (XSS is a much bigger hole than CSRF.)
Although the setting offers little practical benefit, it’s sometimes
required by security auditors.
It also depends how you have implemented CSRF. There are basically two methods for forms:
Set a hidden CSRF field for each form and make this field generate a unique value each time the form is loaded. Therefore if the form submission includes a valid code, then you know the request came from your domain. This is complicated on the server side, as it requires keeping track of valid tokens and also means each form must be dynamically generated to include a random token, but is easier on client side as uses standard form requests rather than JavaScript. For this protection the CSRF cookie is not needed and is not used even if it is present.
The other method invokes setting a CSRF cookie, and having the Javascript read this and send it in a HTTP header (usually X-CSRF-TOKEN). A CSRF request from another domain won't have access to this CSRF cookie so won't be able to set the header correctly. As the cookies will be also be sent on all requests it's easy for the server to check the cookie in the HTTP Request matches the header set in the request. Which means the request came from a somewhere that has access to the cookies, which means it came from same domain. Which means it's not a CSRF attack. This is easier to implement on the server side (as no need to keep a list of active Tokens) but requires Javascript at the front end and requires a CSRF token not to be HttpOnly - precisely because the Token is supposed to be read by Client side Javascript!
Again the Django documentation warns against this:
If you enable this and need to send the value of the CSRF token with
an AJAX request, your JavaScript must pull the value from a hidden
CSRF token form input on the page instead of from the cookie.
So, all in all, it is not recommended to set the HttpOnly attribute for this cookie. It limits you, adds no real protection, and makes the cookie itself meaningless.
You will get it highlighted in and Pen Test reports on your site (including https://detectify.com by the looks of things) but should accept that as you are comfortable that this is correct. Not sure if it's possible to whitelist this cookie in https://detectify.com so it doesn't alert each time?
I have a django project running on my localhost and it is working very well, however when I uploaded it to real server, some problem started happening with the url. it happens every time HttpResponseRedirect or any redirect gets called
a page on my local host
http://127.0.0.1:8000/signin
while on the server it becomes
http://xyz.com,%20xyz.com/signin
in firebug i see
GET signin 301 MOVED PERMANENTLY
GET signin http://xyz.com,%20xyz.com/signin
I belive this happens because the urls.py has ^signin/$^ and APPEND_SLASH = True in settings.py because when I visit /signin/ it works!
404 page on my local host
Request URL: http://127.0.0.1:8000/test
on the server
Request URL: http://xyz.com,%20xyz.com/test
for some reason it is adding [comma][space] to url and redirects it.
home page is working without issues
The issue is tracked in the following ticket:
https://code.djangoproject.com/ticket/11877
It has to do with how Django handles proxy redirection. The following middleware will help you out.
class MultipleProxyMiddleware(object):
FORWARDED_FOR_FIELDS = [
'HTTP_X_FORWARDED_FOR',
'HTTP_X_FORWARDED_HOST',
'HTTP_X_FORWARDED_SERVER',
]
def process_request(self, request):
"""
Rewrites the proxy headers so that only the most
recent proxy is used.
"""
for field in self.FORWARDED_FOR_FIELDS:
if field in request.META:
if ',' in request.META[field]:
parts = request.META[field].split(',')
request.META[field] = parts[-1].strip()
If, for example, your Django site is sitting behind a proxy which includes proxy information in the X-Forwarded-For header, and then your web server also does proxying, the header will contain a list (comma separated) of the proxied addresses. By using this middleware, it will strip all but one of the proxied addresses in the headers.
It might not be an answer since I'm working with you on the same application, I fixed it.
It has something to do with nginx to apache redirection, we had proxy_set_header Host $host; and when I disabled it the redirection worked without errors.