I’m having a tough time deploying my django app (v1.9) to heroku (psql 9.5),cedar stack-14.
Here’s how I arrived here: I had tremendous migration issues that resulted in “””django.db.utils.ProgrammingError: relation already exists”””, and “””Django column “name” of relation “django_content_type” does not exist””” errors. Figuring that there were old, mishandled migrations imported int django_migrations table, I decided to push a fresh, local db up to an empty heroku with:
PGUSER=dbnameHERE PGPASSWORD=dbpassHERE heroku pg:push localDBnameHERE DATABASE --app appnameHERE
This worked flawlessly. After that, here’s what happens when I run these commands:
When I run heroku local, my full app shows locally on 0.0.0.0:5000. (/admin works, but with css issues, presumably bc the whitenoise module I imported does not deal well under production .env settings)
When I run heroku local -e .env.DEV (development .env settings) on 0.0.0.0:5000, everything, including /admin works wonderfully.
The issue begins when gunicorn comes into the picture. When I run gunicorn config.wsgi:application, it runs, but I get “This site can’t be reached, localhost took too long to respond” blank page.
Here is the request header from the blank webpage:
HTTP/1.1 301 Moved Permanently
Connection: keep-alive
Server: gunicorn/19.4.5
Date: Tue, 07 Jun 2016 22:39:00 GMT
Transfer-Encoding: chunked
Location: https://sitename.herokuapp.com/
Content-Type: text/html; charset=utf-8
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Via: 1.1 vegur
When I run heroku - heroku run python manage.py check --deploy, I get this:
WARNINGS:
?: (security.W001) You do not have 'django.middleware.security.SecurityMiddleware' in your MIDDLEWARE_CLASSES so the SECURE_HSTS_SECONDS, SECURE_CONTENT_TYPE_NOSNIFF, SECURE_BROWSER_XSS_FILTER, and SECURE_SSL_REDIRECT settings will have no effect.
?: (security.W009) Your SECRET_KEY has less than 50 characters or less than 5 unique characters. Please generate a long and random SECRET_KEY, otherwise many of Django's security-critical features will be vulnerable to attack.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W016) You have 'django.middleware.csrf.CsrfViewMiddleware' in your MIDDLEWARE_CLASSES, but you have not set CSRF_COOKIE_SECURE to True. Using a secure-only CSRF cookie makes it more difficult for network traffic sniffers to steal the CSRF token.
?: (security.W017) You have 'django.middleware.csrf.CsrfViewMiddleware' in your MIDDLEWARE_CLASSES, but you have not set CSRF_COOKIE_HTTPONLY to True. Using an HttpOnly CSRF cookie makes it more difficult for cross-site scripting attacks to steal the CSRF token.
?: (security.W018) You should not have DEBUG set to True in deployment.
?: (security.W019) You have 'django.middleware.clickjacking.XFrameOptionsMiddleware' in your MIDDLEWARE_CLASSES, but X_FRAME_OPTIONS is not set to 'DENY'. The default is 'SAMEORIGIN', but unless there is a good reason for your site to serve other parts of itself in a frame, you should change it to 'DENY'.
?: (security.W020) ALLOWED_HOSTS must not be empty in deployment.
System check identified 8 issues (0 silenced).
Same blank page and 301 redirect with no error code when I navigate to sitename.herokuapp.com that I got when running gunicorn. Any guess as to why my app throws redirects when gunicorn gets involved?
I eventually solved this issue by re-reading the Heroku docs, where it states that production environments must have their config vars loaded separately, either through their dashboard interface, or by using their heroku config:set command from the heroku cli.
I had placed my config vars $VIRTUAL_ENV/bin/postactivate, loading the different virtualenv configs following activation of the virtualenv; however, that file was not being used in production.
if ENVIRONMENT == 'production':
SECURE_BROWSER_XSS_FILTER = True
X_FRAME_OPTIONS = 'DENY'
SECURE_SSL_REDIRECT = True
SECURE_HSTS_SECONDS = 3600
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_HSTS_PRELOAD = True
SECURE_CONTENT_TYPE_NOSNIFF = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
There is some security issues can be happen
Related
My Angular4 app (running on http://127.0.0.1:4200 development server) is supposed to access a django REST backend on the web. The backend is under my control and is available only via HTTPS (running Apache that tunnels the request to a gunicorn server running on an internal port). Let's say that this is https://example.com/. For historical reasons, logging the user in is done using sessions, because I want the users to be able to also use Django's admin interface after they logged in. The workflow is as follows:
Users opens http://127.0.0.1:4200, I perform a GET request to https://example.com/REST/is_logged_in which returns a 403 when the user isn't logged in via sessions yet, 200 otherwise. In the former case, the user is redirected to https://example.com/login/, rendered by Django's template engine, allowing the user to log in. Once logged in, the user is redirected to http://127.0.0.1:4200
When clicking on some button in my Angular UI, a POST request is performed. This post request fails with 403, even though the preflight OPTIONS request explicitly lists POST as allowed actions.
Here is my CORS configuration in Django:
NG_APP_ABSOLUTE_URL = 'http://127.0.0.1:4200'
# adapt Django's to Angular's presumed XSRF cookie/header names
CSRF_COOKIE_NAME = "XSRF-TOKEN"
CSRF_HEADER_NAME = "HTTP_X_XSRF_TOKEN"
CORS_ORIGIN_WHITELIST = (
urlparse(NG_APP_ABSOLUTE_URL).netloc
)
CSRF_TRUSTED_ORIGINS = (
urlparse(NG_APP_ABSOLUTE_URL).netloc
)
CORS_ALLOW_HEADERS = default_headers + (
'x-xsrf-token',
)
CORS_ALLOW_CREDENTIALS = True
This is what Chrome reports for the (successful, 200) first REST GET request to check whether the user is logged in (after he successfully did) in the response:
Access-Control-Allow-Credentials:true
Access-Control-Allow-Origin:http://127.0.0.1:4200
Allow:GET, HEAD, OPTIONS
Connection:close
Content-Type:application/json
Date:Wed, 26 Apr 2017 15:09:26 GMT
Server:gunicorn/19.6.0
Set-Cookie:XSRF-TOKEN=...; expires=Wed, 25-Apr-2018 15:09:26 GMT; Max-Age=31449600; Path=/
Transfer-Encoding:chunked
Vary:Accept,Cookie,Origin
X-Frame-Options:SAMEORIGIN
The corresponding request had this:
Cookie:sessionid=...; XSRF-TOKEN=...
Host:example.com
Origin:http://127.0.0.1:4200
Referer:http://127.0.0.1:4200/
Now, to the actual problem:
Preflight request:
Request URL:https://example.com/REST/change_user_data/
Request Method:OPTIONS
Status Code:200 OK
Access-Control-Request-Headers:content-type
Access-Control-Request-Method:POST
Connection:keep-alive
Host:example.com
Origin:http://127.0.0.1:4200
Referer:http://127.0.0.1:4200/dashboard/account
Preflight response:
Access-Control-Allow-Credentials:true
Access-Control-Allow-Headers:accept, accept-encoding, authorization, content-type, dnt, origin, user-agent, x-csrftoken, x-requested-with, x-xsrf-token
Access-Control-Allow-Methods:DELETE, GET, OPTIONS, PATCH, POST, PUT
Access-Control-Allow-Origin:http://127.0.0.1:4200
Access-Control-Max-Age:86400
Connection:close
Content-Length:0
Content-Type:text/html; charset=utf-8
Date:Wed, 26 Apr 2017 15:36:56 GMT
Server:gunicorn/19.6.0
Vary:Origin
X-Frame-Options:SAMEORIGIN
Now my failing (403) POST request:
Accept:application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:60
Content-Type:application/json
Cookie:sessionid=...; XSRF-TOKEN=...
Host:example.com
Origin:http://127.0.0.1:4200
Referer:http://127.0.0.1:4200/dashboard/account
The response headers:
HTTP/1.1 403 Forbidden
Date: Wed, 26 Apr 2017 15:36:56 GMT
Server: gunicorn/19.6.0
Vary: Accept,Cookie,Origin
X-Frame-Options: SAMEORIGIN
Content-Type: application/json
Access-Control-Allow-Credentials: true
Allow: POST, OPTIONS
Access-Control-Allow-Origin: http://127.0.0.1:4200
Set-Cookie: XSRF-TOKEN=...; expires=Wed, 25-Apr-2018 15:36:56 GMT; Max-Age=31449600; Path=/
Connection: close
Transfer-Encoding: chunked
Why wouldn't this request work? It makes little sense to me!
Best regards!
I had the same problem, trying to send a POST request to Django (port 8000) from my Angular CLI (port 4200). I thought it was a problem of Django so I installed cors package however the "problem" is with the browser (actually is not a problem, it is a security issue, see here). Anyway, I solved the problem adding a proxy rule for my Angular CLI, as follows:
First, instead of sending my requests to http://localhost:8000/api/... is send them to /api/ (i.e. to my ng server running at port 4200).
Then I added a file in my Angular project called "proxy.conf.json" with the following content:
{
"/api": {
"target": "http://localhost:8000",
"secure": false
}
}
Finally, run your ng server with the flag "--proxy-config":
ng serve --watch --proxy-config proxy.conf.json
All API requests will be sent to the port 4200 and Angular will internally redirect them to Django, avoiding the CORS problem.
Note that this is only valid for development and won't be used when you build your app code and add it as the static code of your Django server.
Finally, with this solution I didn't need anymore the python module for cors so you could remove it.
My Django website is in HTTPS. When I am trying to POST data to the website from a script I get this error : "referer checking failed - no Referer". It seems to be a CSRF issue but I do not know how to solve it.
Example :
import requests
r = requests.post('https://mywebsite/mypage', data = {'key':'value'})
print r.text
gives me this output :
[...]
<p>Reason given for failure:</p>
<pre>
Referer checking failed - no Referer.
</pre>
<p>In general, this can occur when there is a genuine Cross Site Request Forgery, or when
<a
href="https://docs.djangoproject.com/en/1.8/ref/csrf/">Django's
CSRF mechanism</a> has not been used correctly. For POST forms, you need to
ensure:</p>
<ul>
<li>Your browser is accepting cookies.</li>
<li>The view function passes a <code>request</code> to the template's <code>render</code>
method.</li>
<li>In the template, there is a <code>{% csrf_token
%}</code> template tag inside each POST form that
targets an internal URL.</li>
<li>If you are not using <code>CsrfViewMiddleware</code>, then you must use
<code>csrf_protect</code> on any views that use the <code>csrf_token</code>
template tag, as well as those that accept the POST data.</li>
</ul>
[...]
Do I need to pass a referer to my headers before sending the POST data - which would not be convenient ? Or should I disable CSRF for this page ?
Thanks
AFAIK, This is the purpose of CSRF, to avoid posting data from unknown strange sources. You need csrf token to post this which django generates dynamically.
Upgrading Django might fix the missing Referer error.
As of Django 4.0 (release notes), the backend will first check the Origin header before falling back to the Referer header (source):
CsrfViewMiddleware verifies the Origin header, if provided by the browser, against the current host and the CSRF_TRUSTED_ORIGINS setting. This provides protection against cross-subdomain attacks.
In addition, for HTTPS requests, if the Origin header isn’t provided, CsrfViewMiddleware performs strict referer checking. This means that even if a subdomain can set or modify cookies on your domain, it can’t force a user to post to your application since that request won’t come from your own exact domain.
It's possible you have a reverse proxy running, for example an nginx proxy_pass to 127.0.0.1:8000?
In this case, Django expects the Cross-Site Forgery Protection tokens to match hostname 127.0.0.1, but they will be coming from a normal domain (for example example.com).
Expected Source
Actual Source
http://127.0.0.1
https://example.com
HTTP reverse proxy (example.com:80 -> localhost:3000) is a common way to use nginx with NodeJS applications, but it doesn't work well with Django
Client-Facing URL
Server Proxy URL
https://example.com
http://127.0.0.1:3000
It is better to run Django through a Unix socket rather than a port (example.com:80 -> <socket>). You can do this with Gunicorn:
Client-Facing URL
Server Proxy URL
https://example.com
unix:/run/example.com.sock
Here's how to do this with Django, Gunicorn, and nginx:
Let's say you've got a Django project root, which contains a system folder (the one where settings.py and wsgi.py are):
export DJANGO_PROJECT_PATH=/path/to/django/root
export DJANGO_SETTING_FOLDER=system
First, make sure you have Gunicorn installed and that you are using a virtual environment:
cd $DJANGO_PROJECT_PATH
source .venv/bin/activate # <- Use a virtual environment
pip3 install gunicorn # <- install Gunicorn in the venv
Run Gunicorn. This will start the Django project similar to running python3 manage.py runserver, except that you can listen for requests on a Unix socket:
$DJANGO_PROJECT_PATH/.venv/bin/gunicorn \
--workers=3 \
--access-logfile - \
--bind unix:/run/example.com.sock \ # <- Socket
--chdir=$DJANGO_PROJECT_PATH/ \
$DJANGO_SETTING_FOLDER.wsgi:application
Then create an HTTP proxy using nginx that passes HTTP requests from clients through the gunicon-created socket:
/etc/nginx/sites-enabled/example.com:
server {
listen 80;
listen [::]:80;
server_name example.com;
# serve static files directly through nginx
location /static/ {
autoindex off;
root /path/to/django/root;
}
# serve user-uploaded files directly through nginx
location /media/ {
autoindex off;
root /path/to/django/root;
}
# You can do fun stuff like aliasing files from other folders
location /robots.txt {
alias /path/to/django/root/static/robots.txt;
}
# here is the proxy magic
location / {
include proxy_params;
proxy_pass http://unix:/run/example.com.sock; # <- the socket!
}
}
Make sure to restart nginx:
sudo service restart nginx
After all this, your csrf tokens should match the domain name of your site and you'll be able to log in and submit forms.
Preface: I'm running nginx + gunicorn + django on an amazon ec2 instance using s3boto as a default storage backend. I am free tier. The ec2 security group allows: http, ssh, & https.
I'm attempting to send a multipart/form-data request containing a single element: a photo. When attempting to upload the photo, the iPhone (where the request is coming from) hangs. The photo is around 9.5 MB in size.
When I check the nginx-access.logs:
"POST /myUrl/ HTTP/1.1" 400 5 "-""....
When I check the nginx-error.logs:
[error] 5562#0: *1 readv() failed (104: Connection reset by peer) while reading upstream, client: my.ip.addr.iphone, server: default, request: "POST /myUrl/ HTTP/1.1", upstream: "http://127.0.0.1:8000/myUrl/", host: "ec2-my-server-ip-addr.the-location-2.compute.amazonaws.com"
[info] 5562#0: *1 client my.ip.addr.iphone closed keepalive connection
I really cannot figure out why this is happening... I have tried changing the /etc/nginx/sites-available/default timeout settings...
server { ...
client_max_body_size 20M;
client_body_buffer_size 20M;
location / {
keepalive_timeout 300;
proxy_read_timeout 300;
}
}
Any thoughts?
EDIT: After talking on IRC a little more, his problem is the 403 itself, not the nginx error. Leaving my comments on the nginx error below, in case anyone else stumbles into it someday.
I ran into this very problem last week and spent quite a while trying to figure out what was going on. See here: https://github.com/benoitc/gunicorn/issues/872
Basically, as soon as django sees the headers, it knows that the request isn't authenticated. It doesn't wait for the large request body to finish uploading; it responds immediately, and gunicorn closes the connection right after. nginx keeps sending data, and the end result is that gunicorn sends a RST packet to nginx. Once this happens, nginx cannot recover and instead of sending the actual response from gunicorn/django, it sends a 502 Bad Gateway.
I ended up putting in a piece of middleware that acecsses a couple fields in the django request, which ensures that the entire request body is downloaded before Django sends a response:
checker = re.compile(feed_url_regexp)
class AccessPostBodyMiddleware:
def process_request(self, request):
if checker.match(request.path.lstrip('/')) is not None:
# just need to access the request info here
# not sure which one of these actually does the trick.
# This will download the entire request,
# fixing this random issue between gunicorn and nginx
_ = request.POST
_ = request.REQUEST
_ = request.body
return None
However, I do not have control of the client. Since you do (in the form of your iphone app), maybe you can find a way to handle the 502 Bad Gateway. That will keep your app from having to send the entire request twice.
I have decorated a Django view with cache_control as follows:
#cache_control(
private=True,
max_age=5 * 60, # 5 minutes
)
def my_view(req):
…
When I try it with the local test server, it works as expected: subsequent page views in Chrome use the cached resource and don't make a request. When deployed in production, though, Chrome seems to ignore the Cache-control header and makes a new request every time I hit that page.
Here's the full list of headers that the production server responds with:
Cache-Control:private, max-age=300
Connection:close
Content-Encoding:gzip
Content-Length:13135
Content-Type:text/html; charset=utf-8
Date:Wed, 22 Jan 2014 20:39:29 GMT
P3P:CP="IDC CURa ADMa OUR IND PHY ONL COM STA"
Server:nginx/1.4.1
Set-Cookie:csrftoken=87y26bT5uPmyA9wt51N7m4blyqBH5nSo; expires=Wed, 21-Jan-2015 20:39:29 GMT; Max-Age=31449600; Path=/
Vary:Cookie,Accept-Encoding
What could be going wrong? Any ideas? Thanks in advance!
Got it: it was a combination of Google Analytics' cookie and the Vary:Cookie header (set by Django's SessionMiddleware). Analytics' cookie changes with each request, but since ga.js doesn't load when working on localhost, the problem only showed up in production.
I have Nginx serving my static Django files which is being run on Gunicorn. I am trying to serve MP3 files and get them to have the head 206 so that they will be accepted by Apple for podcasting. At the moment the audio files are in my static directory and are served straight through Nginx. This is the response i get:
HTTP/1.1 200 OK
Server: nginx/1.2.1
Date: Wed, 30 Jan 2013 07:12:36 GMT
Content-Type: audio/mpeg
Content-Length: 22094968
Connection: keep-alive
Last-Modified: Wed, 30 Jan 2013 05:43:57 GMT
Can someone help with the correct way to serve mp3 files so that byte-ranges will be accepted.
Update: This is the code in my view that serves the file through Django
response = HttpResponse(file.read(), mimetype=mimetype)
response["Content-Disposition"]= "filename=%s" % os.path.split(s)[1]
response["Accept-Ranges"]="bytes"
response.status_code = 206
return response
If you want to do this only in nginx then in your location directive which is responsible for serving static .mp3 files add those directives:
# here you add response header "Content-Disposition"
# with value of "filename=" + name of file (in variable $request_uri),
# so for url example.com/static/audio/blahblah.mp3
# it will be /static/audio/blahblah.mp3
# ----
set $sent_http_content_disposition filename=$request_uri;
# or
add_header content_disposition filename=$request_uri;
# here you add header "Accept-Ranges"
set $sent_http_accept_ranges bytes;
# or
add_header accept_ranges bytes;
# tell nginx that final HTTP Status Code should be 206 not 200
return 206;
There is something in your config that prevent nginx from supporting range requests for these static files. When using standard nginx modules, this may be one of the following filters (these filters modify responses, and byte-range handling is disabled if modification may happen during request body handling):
gzip
gunzip
addition filter
ssi
All these modules have directives to control MIME types they work with (gzip_types, gunzip_types, addition_types, ssi_types). By default, they are set to restrictive sets of MIME types, and range requests works fine for most static files even if these modules are enabled. But placing something like
ssi on;
ssi_types *;
into a configuration will disable byte-range support for all static files affected.
Check your nginx configuration and remove offending lines, and/or make sure to switch off modules in question for a location you serve your mp3 files from.
You can define your own status code:
response = HttpResponse('this is my response data')
response.status_code = 206
return response
If you are using Django 1.5 you may want to have a look at the new StreamingHttpResponse:
https://docs.djangoproject.com/en/dev/ref/request-response/#streaminghttpresponse-objects
This can be very helpful for big files.