[Update]
expires 30d : Static file cache expires after 30 days on client's browser
etag on : This attribute is only available after version 1.3.3. Each static file has 'etag hash value'. Client will make a request for server if the static file is changed (Even though not expired yet).
===================================================================
Here's a sample of nginx.conf file for django project
server {
listen 80;
server_name hostname.com;
...
location /static/ { # STATIC_URL
alias /path/to/static/; # STATIC_ROOT
expires 30d;
}
location /media/ { # MEDIA_URL
alias /path/to/media/; # MEDIA_ROOT
expires 30d;
}
...
}
In this code, what is the meaning of "expires 30d" ?
(1) static, media file would be deleted after 30 days, and manage.py will regenerate them automatically.
(2) static, media file would be deleted after 30 days, and I should type manage.py collectstatic manually.
Similarly, I wonder the meaning of "expires max"
This adds two HTTP headers to the responses (Expires and Cache-Control). Those headers are used by the browsers to cache content, avoid doing the same requests for static content each time a page is loaded.
expires 30d means that all content in static and media folders will be cached by browsers during 30 days, but nothing will be deleted and you won't need to regenerate anything in the server.
expires max set the Expires header to the value "Thu, 31 Dec 2037 23:55:55 GMT", and the Cache-Control to 10 years.
See the nginx documentation for more details: http://nginx.org/en/docs/http/ngx_http_headers_module.html
For more info about HTTP caching see http://www.mobify.com/blog/beginners-guide-to-http-cache-headers/
Related
I am trying to test-run a django project on my VPS for the first time. I followed a step-by-step tutorial on a blog (thanks to a nice guy on #django channel at liberachat). The setup involves uWSGI and nginx. The django project files are at /srv/www/site/*. uwsgi configuration is at /etc/uwsgi.d/mysite.com.ini, and nginx configuration is at /etc/nginx/conf.d/mysite.com.conf.
Here is what happens when I start uwsgi (systemctl start uwsgi):
Jul 29 14:35:09 mysite.com uwsgi[6998]: [uWSGI] getting INI configuration from mysite.com.ini
Jul 29 14:35:09 mysite.com uwsgi[6318]: Fri Jul 29 14:35:09 2022 - [emperor] curse the uwsgi instance mysite.com.ini (pid: 6998)
Jul 29 14:35:09 mysite.com uwsgi[6318]: Fri Jul 29 14:35:09 2022 - [emperor] removed uwsgi instance mysite.com.ini
How do I interpret and fix that?
contents of /etc/uwsgi.d/mysite.com.ini:
procname-master = demosite
# Now paths can be specified relative to here.
chdir = /srv/www/
# allow nginx
uid = 981
gid = 981
chmod-socket = 644
socket = server.sock
# Task management
; Max 4 processes
processes = 2
; Each running 4 threads
threads = 2
; Reduce to 1 process when quiet
cheaper = 1
; Save some memory per thread
thread-stack-size = 512
# Logging
plugin = logfile
; Log request details here
req-logger = file:logs/request.log
; Log other details here
logger = file:logs/error.log
log-x-forwarded-for = true
# Python app
plugin = python3
; Activate this virtualenv
virtualenv = venv/
; Add this dir to PYTHONPATH so Python can find our code
pythonpath = site/
; The WSGI module to load
module = mysite.wsgi
# Don't load the app in the Master - saves memory in quiet times
lazy-apps = true
contents of /etc/nginx/conf.d/mysite.conf:
# Allow gzip compression
gzip_types text/css application/json application/x-javascript;
gzip_comp_level 6;
gzip_proxied any;
# Look for files with .gz to serve pre-compressed data
gzip_static on;
server {
listen 80;
# The hostname(s) of my site
server_name mysite.com;
# Where to look for content (static and media)
root /srv/www/html/;
# Defines the connection for talking to our Django app service
location #proxy {
# Pass other requests to uWSGI
uwsgi_pass unix://srv/www/server.sock;
include uwsgi_params;
}
# nginx docs recommend try_files over "if"
location / {
# Try to serve existing files first
#try_files $uri #proxy =404;
root /srv/www/html/;
}
}
What did I miss?
I want to have a private media folder on my django website, accessible only to logged in users, so I got to know that I should handle authentication part on the django side, and file serving on the nginx side. However following internal location config examples I find it impossible to make it work. Nginx ignores django completely (only for the internal location case). Even if I don't have the url allowed in my urls.py and I have it listed as internal location in nginx, it will still be freely accessible to everybody.
I am posting my nginx configuration in hope that someone can find a mistake in it.
My expectation is that everything in /internal/ folder will not be accessible to anonymous users and it will only be accessible by the django application through X-Accel-Redirect header. Right now if I go to /internal/test.png in an incognito window it will show me the picture.
I am not posting my django code for now, since it is ignored anyway by nginx, so it must be the nginx config problem.
server {
server_name XXX.XX.XX.XXX example.com www.example.com;
location = /favicon.ico {
access_log off;
log_not_found off;
alias /home/user/myproject/static/favicon4.ico;
}
location /static/ {
root /home/user/myproject;
}
location /media/ {
root /home/user/myproject;
}
location / {
include proxy_params;
proxy_pass http://unix:/run/gunicorn.sock;
}
location /internal/ {
internal;
root /home/user/myproject;
}
root /home/user/myproject;
location ~* \.(jpg|jpeg|png|webp|ico|gif)$ {
expires 30d;
}
location ~* \.(css|js|pdf)$ {
expires 1d;
}
client_max_body_size 10M;
# below in this server block is only my Certbot stuff
}
P.S. I swapped identifiable data to X characters and basic names.
I had 2 more problems in this config and I will show everything I did to make it work. The original problem why nginx was ignoring django was in how nginx chooses which location block to use, as suggested by Richard Smith.
From nginx.org we can read:
To find location matching a given request, nginx first checks locations defined using the prefix strings (prefix locations). Among them, the location with the longest matching prefix is selected and remembered. Then regular expressions are checked, in the order of their appearance in the configuration file. The search of regular expressions terminates on the first match, and the corresponding configuration is used. If no match with a regular expression is found then the configuration of the prefix location remembered earlier is used.
And also:
If the longest matching prefix location has the “^~” modifier then regular expressions are not checked.
So regular expressions, if available, will be chosen first. ^~ modifier before prefix makes it chosen instead of regular expressions.
I changed location /internal/ { line to location ^~ /internal/ { and then I got 404 errors every time and no matter how I tried to access the files, but at least I knew nginx was going to this location.
The 2nd mistake was thinking that I can get away with using the same url as the folder name, or in other words, that I can put in my urls.py
path('internal/<path>', views.internal_media, name='internal_media')
together with
location ^~ /internal/ {
internal;
root /home/user/myproject;
}
in my nginx config.
I can't. The url must be different, because otherwise the url doesn't lead to django urls.py - it still leads to /internal/ location through nginx (again, due to how nginx chooses locations).
I changed my urls.py line to point to private url instead:
path('private/<path>', views.internal_media, name='internal_media')
and in the views.py file I redirect to /internal/:
def internal_media(request, path):
if request.user.groups.filter(name='team-special').exists():
response = HttpResponse()
response['X-Accel-Redirect'] = '/internal/' + path
del response['Content-Type'] # without this your images will open as text
return response
else:
raise PermissionDenied()
Aaaand this still didn't work. 404 errors every time. The 3rd mistake was forgetting about the combo of those two:
location / {
include proxy_params;
proxy_pass http://unix:/run/gunicorn.sock;
}
location ~* \.(jpg|jpeg|png|webp|ico|gif)$ {
expires 30d;
}
Now if I went to the url /private/test.jpg nginx didn't let me go to django, because location / is lower in priority than regular expressions, so location ~* took precedence and I never got to django. I noticed it by accident after a lot of time being frustrated, when I put the url incorrectly in incognito mode. When I went to /private/test.jp now I got a 403 forbidden error instead of 404.
It started working immediately when I commented out this.
location ~* \.(jpg|jpeg|png|webp|ico|gif)$ {
expires 30d;
}
location ~* \.(css|js|pdf)$ {
expires 1d;
}
So now internal files worked nicely, but I didn't have caching...
To fix that, I modified my /static/ and /media/ locations, but maybe I won't go into that here, since it is a different topic. I'll just post my full nginx config that works :)
Well, what you might want to also know is that:
~* tells nginx that we are writing a regular expression that is case insensitive
~ would tell nginx that we were writing a regular expression that is case sensitive
server {
server_name XXX.XX.XX.XXX example.com www.example.com;
location = /favicon.ico {
access_log off;
log_not_found off;
alias /home/user/myproject/static/favicon4.ico;
expires 30d;
}
location /static/ {
root /home/user/myproject;
expires 30d;
}
location /media/ {
root /home/user/myproject;
expires 30d;
}
location ~* \/(static|media)\/\S*\.(css|js|pdf) {
root /home/user/myproject;
expires 1d;
}
location ^~ /internal/ {
root /home/user/myproject;
internal;
expires 1d;
}
location / {
include proxy_params;
proxy_pass http://unix:/run/gunicorn.sock;
}
client_max_body_size 10M;
# certbot stuff
}
I have a Django app with Gunicorn, going throught Varnish and served with Nginx.
MyDjangoApp --> Gunicorn --> Varnish --> Nginx --> Client
Which one of the gzip params I have to keep ?
In Django ?
MIDDLEWARE_CLASSES = (
# Remove Django Gzip middleware as we already have it in nginx ?
'django.middleware.gzip.GZipMiddleware',
....
In Nginx ?
http {
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
....
In Varnish ?
sub vcl_backend_response {
if (bereq.url ~ "html$") {
set beresp.do_gzip = true;
}
....
Do I have to activate on all confs or Just Nginx ?
If I activate the GZipMiddleware in Django for ex, I should not need to activate it on Varnish & Nginx or I'm missing something ?
My approach to where gzip compression should be done is this:
Enable gzip on a web server that is behind Varnish.
In your case, you can keep it in Django.
Do not alter default Varnish gzip parameters. (let it handle gzip using default behaviour)
Why?
Varnish can store gzipped object in its cache (good, saves storage).
No CPU time will be spent for compression while serving a cached object (most of the clients will ask for compressed object and Varnish can just serve it directly)
So you would save both CPU and RAM by doing compression on appropriate level.
My Django website is in HTTPS. When I am trying to POST data to the website from a script I get this error : "referer checking failed - no Referer". It seems to be a CSRF issue but I do not know how to solve it.
Example :
import requests
r = requests.post('https://mywebsite/mypage', data = {'key':'value'})
print r.text
gives me this output :
[...]
<p>Reason given for failure:</p>
<pre>
Referer checking failed - no Referer.
</pre>
<p>In general, this can occur when there is a genuine Cross Site Request Forgery, or when
<a
href="https://docs.djangoproject.com/en/1.8/ref/csrf/">Django's
CSRF mechanism</a> has not been used correctly. For POST forms, you need to
ensure:</p>
<ul>
<li>Your browser is accepting cookies.</li>
<li>The view function passes a <code>request</code> to the template's <code>render</code>
method.</li>
<li>In the template, there is a <code>{% csrf_token
%}</code> template tag inside each POST form that
targets an internal URL.</li>
<li>If you are not using <code>CsrfViewMiddleware</code>, then you must use
<code>csrf_protect</code> on any views that use the <code>csrf_token</code>
template tag, as well as those that accept the POST data.</li>
</ul>
[...]
Do I need to pass a referer to my headers before sending the POST data - which would not be convenient ? Or should I disable CSRF for this page ?
Thanks
AFAIK, This is the purpose of CSRF, to avoid posting data from unknown strange sources. You need csrf token to post this which django generates dynamically.
Upgrading Django might fix the missing Referer error.
As of Django 4.0 (release notes), the backend will first check the Origin header before falling back to the Referer header (source):
CsrfViewMiddleware verifies the Origin header, if provided by the browser, against the current host and the CSRF_TRUSTED_ORIGINS setting. This provides protection against cross-subdomain attacks.
In addition, for HTTPS requests, if the Origin header isn’t provided, CsrfViewMiddleware performs strict referer checking. This means that even if a subdomain can set or modify cookies on your domain, it can’t force a user to post to your application since that request won’t come from your own exact domain.
It's possible you have a reverse proxy running, for example an nginx proxy_pass to 127.0.0.1:8000?
In this case, Django expects the Cross-Site Forgery Protection tokens to match hostname 127.0.0.1, but they will be coming from a normal domain (for example example.com).
Expected Source
Actual Source
http://127.0.0.1
https://example.com
HTTP reverse proxy (example.com:80 -> localhost:3000) is a common way to use nginx with NodeJS applications, but it doesn't work well with Django
Client-Facing URL
Server Proxy URL
https://example.com
http://127.0.0.1:3000
It is better to run Django through a Unix socket rather than a port (example.com:80 -> <socket>). You can do this with Gunicorn:
Client-Facing URL
Server Proxy URL
https://example.com
unix:/run/example.com.sock
Here's how to do this with Django, Gunicorn, and nginx:
Let's say you've got a Django project root, which contains a system folder (the one where settings.py and wsgi.py are):
export DJANGO_PROJECT_PATH=/path/to/django/root
export DJANGO_SETTING_FOLDER=system
First, make sure you have Gunicorn installed and that you are using a virtual environment:
cd $DJANGO_PROJECT_PATH
source .venv/bin/activate # <- Use a virtual environment
pip3 install gunicorn # <- install Gunicorn in the venv
Run Gunicorn. This will start the Django project similar to running python3 manage.py runserver, except that you can listen for requests on a Unix socket:
$DJANGO_PROJECT_PATH/.venv/bin/gunicorn \
--workers=3 \
--access-logfile - \
--bind unix:/run/example.com.sock \ # <- Socket
--chdir=$DJANGO_PROJECT_PATH/ \
$DJANGO_SETTING_FOLDER.wsgi:application
Then create an HTTP proxy using nginx that passes HTTP requests from clients through the gunicon-created socket:
/etc/nginx/sites-enabled/example.com:
server {
listen 80;
listen [::]:80;
server_name example.com;
# serve static files directly through nginx
location /static/ {
autoindex off;
root /path/to/django/root;
}
# serve user-uploaded files directly through nginx
location /media/ {
autoindex off;
root /path/to/django/root;
}
# You can do fun stuff like aliasing files from other folders
location /robots.txt {
alias /path/to/django/root/static/robots.txt;
}
# here is the proxy magic
location / {
include proxy_params;
proxy_pass http://unix:/run/example.com.sock; # <- the socket!
}
}
Make sure to restart nginx:
sudo service restart nginx
After all this, your csrf tokens should match the domain name of your site and you'll be able to log in and submit forms.
I have Nginx serving my static Django files which is being run on Gunicorn. I am trying to serve MP3 files and get them to have the head 206 so that they will be accepted by Apple for podcasting. At the moment the audio files are in my static directory and are served straight through Nginx. This is the response i get:
HTTP/1.1 200 OK
Server: nginx/1.2.1
Date: Wed, 30 Jan 2013 07:12:36 GMT
Content-Type: audio/mpeg
Content-Length: 22094968
Connection: keep-alive
Last-Modified: Wed, 30 Jan 2013 05:43:57 GMT
Can someone help with the correct way to serve mp3 files so that byte-ranges will be accepted.
Update: This is the code in my view that serves the file through Django
response = HttpResponse(file.read(), mimetype=mimetype)
response["Content-Disposition"]= "filename=%s" % os.path.split(s)[1]
response["Accept-Ranges"]="bytes"
response.status_code = 206
return response
If you want to do this only in nginx then in your location directive which is responsible for serving static .mp3 files add those directives:
# here you add response header "Content-Disposition"
# with value of "filename=" + name of file (in variable $request_uri),
# so for url example.com/static/audio/blahblah.mp3
# it will be /static/audio/blahblah.mp3
# ----
set $sent_http_content_disposition filename=$request_uri;
# or
add_header content_disposition filename=$request_uri;
# here you add header "Accept-Ranges"
set $sent_http_accept_ranges bytes;
# or
add_header accept_ranges bytes;
# tell nginx that final HTTP Status Code should be 206 not 200
return 206;
There is something in your config that prevent nginx from supporting range requests for these static files. When using standard nginx modules, this may be one of the following filters (these filters modify responses, and byte-range handling is disabled if modification may happen during request body handling):
gzip
gunzip
addition filter
ssi
All these modules have directives to control MIME types they work with (gzip_types, gunzip_types, addition_types, ssi_types). By default, they are set to restrictive sets of MIME types, and range requests works fine for most static files even if these modules are enabled. But placing something like
ssi on;
ssi_types *;
into a configuration will disable byte-range support for all static files affected.
Check your nginx configuration and remove offending lines, and/or make sure to switch off modules in question for a location you serve your mp3 files from.
You can define your own status code:
response = HttpResponse('this is my response data')
response.status_code = 206
return response
If you are using Django 1.5 you may want to have a look at the new StreamingHttpResponse:
https://docs.djangoproject.com/en/dev/ref/request-response/#streaminghttpresponse-objects
This can be very helpful for big files.