Passing Static Content from Django/Gunicorn to Nginx - django

I've been playing around with Nginx & Gunicorn/Django for the past two days. Thought I'd share my troubles on here in hopes someone might have a potential solution
I have two separate machines. On one, I have a public facing box with nginx installed which acts as a reverse proxy for my other internal box with Django and Gunicorn installed.
When web requests come from the outside, Nginx simply acts as the middle man and forwards the request to Django/Gunicorn. Django/Gunicorn returns a response to Nginx. Nginx then takes that response and forwards back to where the web request came from. It's a pretty neat set up because Nginx can handle slow clients and take the hit in the case of a DDOS attack. Your actual application server is safe from any sort of damage :)
Although my application doesn't serve actual static content (pictures, videos etc ...). I would still like to pass static content from Gunicorn/Django to Nginx. Currently when I access the Django admin, it's just plain text with no graphics or images. It really sucks. I did however notice that when I accessed the Gunicorn/Django server directly, all the images for the Django admin showed up. It took me a while to understand what was happening here, and I realized that the static content wasn't being passed.
Here is my current Django urls.py configuration:
from django.conf.urls import patterns, include, url
from django.contrib import admin
from django.conf.urls.static import static
from django.conf import settings
from django.views.decorators.csrf import csrf_exempt
admin.autodiscover()
urlpatterns = patterns('',
# Examples:
# url(r'^$', 'django.views.home', name='home'),
# url(r'^blog/', include('blog.urls')),
url(r'^admin/', include(admin.site.urls)),
) + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
This automatically finds the Django admin files and serves them to me through Gunicorn. I don't know how but it works. My debug settings is set to True so I don't know if that has anything to do with it.
My nginx conf looks like this:
worker_processes 1;
events {
worker_connections 1024;
}
http {
sendfile on;
gzip on;
gzip_http_version 1.0;
gzip_proxied any;
gzip_min_length 1000;
gzip_disable "MSIE [1-6]\.";
gzip_types text/plain text/xml text/css
text/comma-separated-values
text/javascript
application/x-javascript
application/atom+xml;
# Configuration containing list of application servers
upstream app_servers {
server 10.101.010.111:8000;
# server 127.0.0.1:8081;
# ..
# .
}
# I just added this in so that it will redirect all requests to HTTPs
server {
listen 80;
server_name *.mydomain.com;
rewrite ^/(.*) https://*.mydomain.com/$1 permanent;
}
# Configuration for Nginx
server {
# Running port
listen 443;
ssl on;
server_name *.mydomain.com;
### SSL log files ###
#access_log logs/ssl-access.log;
#error_log logs/ssl-error.log;
### SSL cert files ###
ssl_certificate ssl/server.crt;
ssl_certificate_key ssl/server.key;
### Add SSL specific settings here ###
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers RC4:HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
keepalive_timeout 15;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# Settings to serve static files
location ^~ /static/ {
# Example:
# root /full/path/to/application/static/file/dir;
root /app/static/;
}
# Serve a static file (ex. favico)
# outside /static directory
location = /favico.ico {
root /app/favico.ico;
}
# Proxy connections to the application servers
# app_servers
location / {
proxy_pass http://app_servers;
proxy_redirect off;
proxy_set_header Accept-Encoding "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Forwarded-Proto $scheme;
add_header Front-End-Https on;
}
}
}
How can I pass static content from Django/Gunicorn to Nginx?
Looking forward to hearing your suggestions! I'm a self taught noob so any help would be extremely appreciated!

If you tell nginx to recognize the URI /static/, it will serve all files under that directory. In your mysite.conf:
location /static/ {
alias /path/to/your/static/;
}
Edit: I'm going to try to explain why this works. Originally, Django is managing all your URLs. When it receives a request for a specific URL, it generates the appropriate response. However, when accessing something like an image, the initial response contains something like an <img> tag, which includes a source. The client then sends back a request to this source URL and asks for the image. The important part here is that the image is not served with the page initially. The other key is that nginx sees all requests first, and only passes to Django when it has to. If you tell nginx to recognize requests to /static/, you are telling it to intercept and answer all requests to this URI. Therefore, it will answer requests for static files without even passing to Gunicorn or Django. I'm new to this too, so this may not be completely accurate, but it's my best shot at an explanation!
Hope this helps.

Related

Nginx responds 404 not found on Django media URL in preprod, dev ok

I have a quite standard Django application with a Vuejs frontend.
I have different environments (preprod/dev) in which I have file upload/download features.
For files, everything works fine because they are returned through standard API views in attachment (Content-Disposition: attachment). When it comes to images though, like profile pictures, there is a problem.
In development (DEBUG=True), I have this :
from django.conf import settings
from django.conf.urls.static import static
from django.urls import include, path
from backend.applications.base.views.authentication_views import LoginAPIView, LogoutAPIView
urlpatterns = [
path("api/login", LoginAPIView.as_view()),
path("api/logout", LogoutAPIView.as_view()),
path("api/base/", include("backend.applications.base.urls")),
path("api/contact/", include("backend.applications.contact.urls")),
path("api/helpdesk/", include("backend.applications.helpdesk.urls")),
path("api/inventory/", include("backend.applications.inventory.urls")),
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT) # For serving media files when DEBUG=True
and images are correctly served (no nginx in dev mode, just frontend and backend dev servers django's runserver).
My preprod however, is made of a nginx container which serves my built vuejs frontend, and a backend container which contains my Django (DEBUG=False) application (which runs with gunicorn this time, like this : gunicorn backend.wsgi:application --bind 0.0.0.0:8000 --access-logfile="-").
Before trying to serve images, I had this nginx configuration :
http {
client_max_body_size 5M;
upstream backend_api {
server backend:8000;
# 'backend' is the name of the backend service in my docker-compose config
}
server {
listen 80;
include /etc/nginx/mime.types;
root /usr/share/nginx/html;
index index.html;
location = /favicon.ico {
access_log off;
log_not_found off;
}
location /api {
proxy_pass http://backend_api;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
}
location / {
try_files $uri $uri/ /index.html;
}
}
}
Then I thought that /media requests should also be passed to the backend and I changed
location /api
into
location ~ ^/(api|media)/ {
My /api URLs are still handled correctly but /media URLs are answered by a 404 :
(trying to load profile pictures of my user(s) in a kanban view).
Also trying directly http://localhost/media/base/users/8/picture.jpg directly in my browser doesn't work :
From here I don't know what to do to solve the issue. If something is missing, mention it and I'll update the post.
Django does not serve static- and media files with runserver, you will need WhiteNoise for that. See http://whitenoise.evans.io/en/stable/
Whitenoise however is not suitable for serving user-uploaded media files. See http://whitenoise.evans.io/en/stable/django.html#serving-media-files
(Optionally, skip whitenoise, and host static/media files through NGINX.)
You really shouldn't be hosting your server with py manage.py runserver. This is not secure. See Why not use "runserver" for production at Django? and https://docs.djangoproject.com/en/dev/ref/django-admin/#runserver
Use something like Gunicorn instead.
See https://docs.djangoproject.com/en/4.1/howto/deployment/wsgi/gunicorn/
(Or waitress, the windows alternative)
https://pypi.org/project/django-waitress/
To host static/media files with nginx, paste this into your nginx conf:
location /media {
alias /PATH/TO/DIRECTORY; #Absolute path.
}
And in your settings.py, set the media root to that same directory.

nginx 404 on root/homepage with Django + Gunicorn

I'm migrating my website from one host to another. The migration has been mostly 99% successful except for the fact that whenever I navigate to the home page of the site, I get nginx's 404 page. Every other page, static & media file is rendered properly. I've wracked my brain trying to find a solution but I haven't found any, let alone someone having a similar issue (instead, others have a working home page but 404 on all others).
I have two different domains (one that my company owns, the other is owned by the city we're in). I've updated the DNS on the domain I own to ensure it's working with the new host and it is. When I navigate using the host (example.com) or the server's IP address, the website loads all pages correctly - except the homepage, which - again - returns nginx's 404. All other 404 errors display the 404 page I've set up via Django.
Whenever this issue crops up, the gunicorn error log adds a line stating that the service is 'Booting worker with pid: ...'
nginx.conf --> save for the paths and port, this conf is identical to the conf running on my current host
http {
client_max_body_size 30M;
open_file_cache max=1000 inactive=300s;
open_file_cache_valid 360s;
open_file_cache_min_uses 2;
open_file_cache_errors off;
upstream cgac_server {
server unix:/root/cgac/cinema_backend/cinema_backend.sock fail_timeout=0;
}
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
gzip on;
server {
listen 80 default_server;
location /static/ {
expires 365d;
root /root/cgac/cinema_backend/static/;
}
location /media/ {
expires 365d;
root /root/cgac/cinema_backend/media/;
}
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
if (!-f $request_filename) {
proxy_pass http://cgac_server;
break;
}
}
}
gunicorn.service --> gunicorn is setup differently from the active server but the parameters in this file are identical, and it's working.
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=root
Group=www-data
WorkingDirectory=/root/cgac/cinema_backend
ExecStart=/root/cgac-venv/bin/gunicorn --access-logfile /root/logs/gunicorn-access.log --error-logfile /root/logs/gunicorn-error.log --workers 3 --bind unix:/root/cgac/cinema_backend/cinema_backend.sock cinema_backend.wsgi:application
[Install]
WantedBy=multi-user.target
Django URL patterns --> also 100% identical to the working site
urlpatterns = patterns('',
url(r'^admin/', include(admin.site.urls)), # NOQA
url(r'^fobi/', include('fobi.urls.view')),
url(r'^fobi/', include('fobi.urls.edit')),
url(r'^captcha/', include('captcha.urls')),
url(r'^sitemap\.xml$', 'django.contrib.sitemaps.views.sitemap',
{'sitemaps': {'cmspages': CMSSitemap}}),
url(r'^select2/', include('django_select2.urls')),
url(r'^', include('cms.urls')),
)
many option are there to fix it
put your upstream over http {}
reload your daemon
restart your gunicorn.service
your cinema_backend must be /run/cinema_backend.sock(what is normal)
try to use different config like site-availabe
The issue had absolutely nothing to do with the nginx or gunicorn configuration. It turns out that an image in a carousel on the home page somehow got corrupted or something and it was driving Django crazy. I deleted the entry in the CMS which fixed the issue, then proceeded to reupload the file and it's all good.

How to deploy django on VPS with external subdomain.?

Good day.
I have a web app that I have developed using django. I tested fine on my local, and I'm happy with how it works.
However I'm facing an issue bringing it online I used those two guides to reach my deployment:
https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04
and
http://michal.karzynski.pl/blog/2013/06/09/django-nginx-gunicorn-virtualenv-supervisor/
However my page is giving me a forbidden page.
I suspect my issue is with the way I'm handling the subdomain. So the site . has been developed using php, and I have worked on my part with django and been provided with a subdomain which is member.domain.com, So I'm deploying it on the VPS and have to make it use the subdomain.
This is how my allowed hosts looks in the settings.py
ALLOWED_HOSTS = ['member.domain.com']
and
in my nginx:
upstream app_server {
# fail_timeout=0 means we always retry an upstream even if it failed
# to return a good HTTP response (in case the Unicorn master nukes a
# single worker for timing out).
server unix:/home/path/project/run/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name member.domain.com;
client_max_body_size 4G;
access_log /home/path/project/logs/nginx-access.log;
error_log /home/path/project/logs/nginx-error.log;
location /static/ {
alias /home/path/project/src/static/;
}
location /media/ {
alias /home/path/project/src/media/;
}
location / {
# an HTTP header important enough to have its own Wikipedia entry:
# http://en.wikipedia.org/wiki/X-Forwarded-For
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# enable this if and only if you use HTTPS, this helps Rack
# set the proper protocol for doing redirects:
# proxy_set_header X-Forwarded-Proto https;
# pass the Host: header from the client right along so redirects
# can be set properly within the Rack application
proxy_set_header Host $http_host;
# we don't want nginx trying to do something clever with
# redirects, we set the Host: header above already.
proxy_redirect off;
# set "proxy_buffering off" *only* for Rainbows! when doing
# Comet/long-poll stuff. It's also safe to set if you're
# using only serving fast clients with Unicorn + nginx.
# Otherwise you _want_ nginx to buffer responses to slow
# clients, really.
# proxy_buffering off;
# Try to serve static files from nginx, no point in making an
# *application* server like Unicorn/Rainbows! serve static files.
if (!-f $request_filename) {
proxy_pass http://app_server;
break;
}
}
# Error pages
error_page 502 503 504 /500.html;
location = /500.html {
root /home/path/project/src/static/;
}
}
I'm not sure what I am doing wrong.
I will appreciate any help
To respond to 'example.com' and any subdomains, start the domain with a dot
ALLOWED_HOSTS = ['.example.com', '203.0.113.5']
I didn't even try how to run django on subdomains, but from article link you shared, you missed some configuration in your settings.py
ALLOWED_HOSTS = ['member.domain.com']
Changed
ALLOWED_HOSTS = ['.domain.com']
Hope this will solve your problem

Django, Nginx, HTTPs and HttpResponseRedirect

I'm receiving a 404 error from Nginx when attempting to return an HttpResponseRedirect from Django. This is all happening under HTTPs The flow goes something like this:
User goes to a page
Enters some information in a form
The view process the form after POST and then attempts to redirect the user to a different page.
Except, instead of redirecting to the page, Nginx just eventually servers its 404 page.
I can get this to work in development while not under Nginx and HTTPs, so I suspect this has something to do with my Nginx setup. I have this working successfully on other servers so I'm unsure why I cannot get it working here
Sample Django view:
#login_required()
def index(request):
if request.method == 'POST':
form = ShortenerForm(request.POST)
if form.is_valid():
# Do stuff
return HttpResponseRedirect(reverse('shortener_thankyou'))
else:
form = ShortenerForm()
return render(request, 'shortener/index.html', {'form': form})
Nginx
upstream apollo2_app_server {
# fail_timeout=0 means we always retry an upstream even if it failed
# to return a good HTTP response (in case the Unicorn master nukes a
# single worker for timing out).
server unix:/webapps/apollo2/run/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name apollo.mydomain.com;
rewrite ^ https://$server_name$request_uri? permanent;
}
server {
listen 443;
ssl on;
ssl_certificate /etc/nginx/ssl/bundle.crt;
ssl_certificate_key /etc/nginx/ssl/mydomain.com.key;
server_name apollo.mydomain.com;
client_max_body_size 4G;
keepalive_timeout 70;
access_log /webapps/apollo2/logs/nginx-access.log;
error_log /webapps/apollo2/logs/nginx-error.log;
location /static/ {
alias /webapps/apollo2/static/;
}
location /media/ {
alias /webapps/apollo2/media/;
}
location / {
# an HTTP header important enough to have its own Wikipedia entry:
# http://en.wikipedia.org/wiki/X-Forwarded-For
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# enable this if and only if you use HTTPS, this helps Rack
# set the proper protocol for doing redirects:
# proxy_set_header X-Forwarded-Proto https;
# pass the Host: header from the client right along so redirects
# can be set properly within the Rack application
proxy_set_header Host $http_host;
# we don't want nginx trying to do something clever with
# redirects, we set the Host: header above already.
proxy_redirect off;
# set "proxy_buffering off" *only* for Rainbows! when doing
# Comet/long-poll stuff. It's also safe to set if you're
# using only serving fast clients with Unicorn + nginx.
# Otherwise you _want_ nginx to buffer responses to slow
# clients, really.
# proxy_buffering off;
# Try to serve static files from nginx, no point in making an
# *application* server like Unicorn/Rainbows! serve static files.
if (!-f $request_filename) {
proxy_pass http://apollo2_app_server;
break;
}
}
# Error pages
error_page 500 502 503 504 /500.html;
location = /500.html {
root /webapps/apollo2/static/;
}
}
Nginx error
2015/04/24 11:04:10 [error] 18139#0: *3395 upstream prematurely closed connection while reading response header from upstream, client: 192.168.0.119, server: apollo.mydomain.com, request: "POST /shortener/ HTTP/1.1", upstream: "http://unix:/webapps/apollo2/run/gunicorn.sock:/shortener/",
I've tried a number of different solutions involving proxy_set_header X-Forwarded-Protocol $scheme; in Nginx and Djangos SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTOCOL', 'https') but without luck.
It turned out that this had nothing to do with Nginx, SSL or Django. IT changed one of our DNS servers without notifying me. The response was timing out because it could not resolve DNS. Updating to the new DNS server in /etc/resolv.conf solved the issue.

Using Django, switched from heroku + gunicorn to digitalocean + nginx + gunicorn, site now broken

I had my django app on heroku for a while with no problems. I now want to move it to a digital ocean droplet, partly as a learning exercise, partly for scalability (and cost) reasons.
After following this excellent tutorial almost to the letter, the app is working but with a huge gotcha: I now get an infinite redirect loop when I try to log in to the admin site. The first request is a POST ?next=/admin/ with the username and password, this gets a 302 response to redirect to GET /admin/, which gets a 302 response redirect to ?next=/admin/, and so on.
I have spent 2 or 3 hours with google and various nginx tutorials and this is the first time my "google the error message, copy and paste random code snippets, repeat" algorithm has ever failed me, I'm hoping the reason is that the error is trivial to solve and I just can't see it?
If it's not trivial to solve, let me know and I'll post more info.
Thanks in advance
edit 1: my nginx config file for the app is basically a verbatim copy of the tutorial. It looks like this:
upstream hello_app_server {
# fail_timeout=0 means we always retry an upstream even if it failed
# to return a good HTTP response (in case the Unicorn master nukes a
# single worker for timing out).
server unix:/webapps/hello_django/run/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name example.com;
client_max_body_size 4G;
access_log /webapps/hello_django/logs/nginx-access.log;
error_log /webapps/hello_django/logs/nginx-error.log;
location /static/ {
alias /webapps/hello_django/static/;
}
location /media/ {
alias /webapps/hello_django/media/;
}
location / {
# an HTTP header important enough to have its own Wikipedia entry:
# http://en.wikipedia.org/wiki/X-Forwarded-For
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# enable this if and only if you use HTTPS, this helps Rack
# set the proper protocol for doing redirects:
# proxy_set_header X-Forwarded-Proto https;
# pass the Host: header from the client right along so redirects
# can be set properly within the Rack application
proxy_set_header Host $http_host;
# we don't want nginx trying to do something clever with
# redirects, we set the Host: header above already.
proxy_redirect off;
# set "proxy_buffering off" *only* for Rainbows! when doing
# Comet/long-poll stuff. It's also safe to set if you're
# using only serving fast clients with Unicorn + nginx.
# Otherwise you _want_ nginx to buffer responses to slow
# clients, really.
# proxy_buffering off;
# Try to serve static files from nginx, no point in making an
# *application* server like Unicorn/Rainbows! serve static files.
if (!-f $request_filename) {
proxy_pass http://hello_app_server;
break;
}
}
# Error pages
error_page 500 502 503 504 /500.html;
location = /500.html {
root /webapps/hello_django/static/;
}
}