Django + nginx large files upload - django

I am trying to upload a 2GB file in django using nginx. The file is uploaded upto 100% and then suddenly it terminates and 502 Bad Gateway error is displayed in nginx.
Below is my application configurations
upstream app_server {
server unix:/home/training/run/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
# add here the ip address of your server
# or a domain pointing to that ip (like example.com or www.example.com)
server_name training.academy;
keepalive_timeout 30;
client_max_body_size 4G;
access_log /home/training/logs/nginx-access.log;
error_log /home/training/logs/nginx-error.log;
location /static/ {
alias /home/training/trainingapp/static/;
}
location /media/ {
alias /home/training/trainingapp/media/;
}
# checks for static file, if not found proxy to app
location / {
try_files $uri #proxy_to_app;
}
location #proxy_to_app {
proxy_read_timeout 1200;
proxy_connect_timeout 1200;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
proxy_pass http://app_server;
proxy_intercept_errors on;
}
}
And below is the gunicorn start configs
#!/bin/bash
NAME="training"
DIR=/home/training/trainingapp
USER=training
GROUP=training
WORKERS=300
TIMEOUT=1200
BIND=unix:/home/training/run/gunicorn.sock
DJANGO_SETTINGS_MODULE=training.settings
DJANGO_WSGI_MODULE=training.wsgi
LOG_LEVEL=error
cd $DIR
source ../bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DIR:$PYTHONPATH
exec ../bin/gunicorn ${DJANGO_WSGI_MODULE}:application \
--name $NAME \
--workers $WORKERS \
--user=$USER \
--group=$GROUP \
--bind=$BIND \
--log-level=$LOG_LEVEL \
--log-file=-
How can I correct this error so that i can get a full and complete upload of the file and not the 502 error?

Related

502 Bad Gateway in django app on vps why?

I am trying to deploy a Django project but I get 502 Bad Gateway
I used this tutorial
I used supervisor, Gunciorn, and Nginx
./virtualenvs/legaland_env/bin/gunicorn
#!/bin/bash
NAME="django_project"
DIR=/home/django/django_project
USER=django
GROUP=django
WORKERS=3
BIND=unix:/home/django/run/gunicorn.sock
DJANGO_SETTINGS_MODULE=django_project.settings
DJANGO_WSGI_MODULE=django_project.wsgi
LOG_LEVEL=error
cd $DIR
source ../bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DIR:$PYTHONPATH
exec ../bin/gunicorn ${DJANGO_WSGI_MODULE}:application \
--name $NAME \
--workers $WORKERS \
--user=$USER \
--group=$GROUP \
--bind=$BIND \
--log-level=$LOG_LEVEL \
--log-file=-
/etc/supervisor/conf.d/sqh.conf
[program:sqh]
startsecs=0
command=/home/admin/legaland/virtualenvs/legaland_env/bin/gunicorn
user=admin
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/home/admin/legaland/gunicorn-error.log
/etc/nginx/sites-available/sqh
upstream app_server {
server unix:/home/admin/legaland/run/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
# add here the ip address of your server
# or a domain pointing to that ip (like example.com or www.example.com)
server_name ;
keepalive_timeout 5;
client_max_body_size 4G;
access_log /home/admin/legaland/logs/nginx-access.log;
error_log /home/admin/legaland/logs/nginx-error.log;
location /static/ {
alias /home/admin/legaland/Legaland/src/static_root/;
}
# checks for static file, if not found proxy to app
location / {
try_files $uri #proxy_to_app;
}
location #proxy_to_app {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
}
Had a similar issue using the hosted platform heroku. I also got 502 bad gateway and the fix was to run the web application locally. You have to specify in the nginx config file these settings such as:
Using TCP/IP Connection:
upstream localhost {
# server unix:/tmp/nginx.socket fail_timeout=0;
server 127.0.0.1:8000
}
location / {
# Uncomment this if statement to force SSL/redirect http -> https
# if ($http_x_forwarded_proto != "https") {
# return 301 https://$host$request_uri;
# }
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host 127.0.0.1:8000;
proxy_redirect off
proxy_pass http://localhost;
}
Then in your gunicorn config file:
command='/myapp/django_env/bin/gunicorn'
pythonpath='/myapp'
bind='localhost'
workers=1
The Procfile:
web: gunicorn -c conf/gunicorn_config.py myproject.wsgi
worker: service nginx start
Last, type the following command in bash to run the heroku app:
heroku local

How can I test if gunicorn is working and communicating with nginx?

I'm deploying my first attempt at using django+gunicorn+nginx.
I have django working (curl -XGET http://127.0.0.0.1:8000 works fine if I run the development server).
I have nginx working for static content (for example I can retrieve http://example.com/static/my_pic.png in my browser).
I'm not getting any wsgi content from my website, and I haven't been able to find a good troubleshooting guide (does it just work for everyone else?!). I start gunicorn using supervisor, which reports that it is indeed running:
(in shell:)
supervisorctl status my_app
my_app RUNNING pid 1002, uptime 0:29:51
Here's the boilerplate script I used to start it:
#!/bin/bash
#script variables
NAME="gunicorn_myapp" # Name of process
DJANGODIR=/webapps/www/my_project # Django project directory
SOCKFILE=/webapps/www/run/gunicorn.sock # communicte using this socket
USER=app_user # the user to run as
GROUP=webapps # the group to run as
NUM_WORKERS=3
DJANGO_SETTINGS_MODULE=my_project.settings # settings file
DJANGO_WSGI_MODULE=my_project.wsgi # WSGI module name
# Activate the virtual environment
cd $DJANGODIR
source ../bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH
# Create the run directory if it doesn't exist
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR
exec ../bin/gunicorn ${DJANGO_WSGI_MODULE}:application \
--name $NAME \
--workers $NUM_WORKERS \
--user=$USER --group=$GROUP \
--bind=unix:$SOCKFILE
Here's the (condensed) nginx config file:
upstream my_server {
server unix:/webapps/www/run/gunicorn.sock fail_timeout=10s;
}
server {
listen 80;
server_name www.example.com;
return 301 $scheme://example.com$request_uri;
}
server {
listen 80;
server_name example.com;
client_max_body_size 4G;
access_log /webapps/www/logs/nginx-access.log;
error_log /webapps/www/logs/nginx-error.log;
location /favicon.ico { access_log off; log_not_found off; }
location /static/ {
autoindex on;
alias /webapps/www/my_project/my_app/static/;
}
location /media/ {
autoindex on;
alias /webapps/www/my_project/my_app/media/;
}
location / {
proxy_pass http://my_server;
proxy_redirect off;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header Host $http_host;
proxy_redirect off;
if (!-f $request_filename) {
proxy_pass http://example.com;
break;
}
}
location /robots.txt {
alias /webapps/www/my_project/my_app/static/robots.txt ;
}
# Error pages
error_page 500 502 503 504 /500.html;
location = /500.html {
root /webapps/www/my_project/my_app/static/;
}
}
So: gunicorn is running, nginx is running ... what tests (and how?) should I perform to determine if gunicorn is doing the wsgi stuff properly (and if nginx is proxying the said stuff through correctly)?
Edit: I've narrowed the problem down to the communication between gunicorn and nginx via the unix socket. If I change the $SOCKFILE to be bound to 0.0.0.0:80 and stop nginx, then the app's pages are served from my website. The bad news is that the socket file strings are exactly the same between the two conf files, so I don't know why they aren't communicating. I suppose this means nginx isn't correctly fetching and passing the data through then?
Go to project directory :
cd projectname
gunicorn --log-file=- projectname.wsgi:application
and
sudo systemctl status gunicorn

django+gunicorn+nginx 404 serving static files

I have django+gunicorn+nginx running on 192.168.1.81:3000. The web app will not serve any static files; it returns a 404 error. This suggests that there is a problem with the nginx virtual server config file. I have tried several solutions offered on stack overflow with no success. What is wrong with the nginx virtual server file?
upstream app_server {
server unix:/home/pi/door_site/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name 192.168.1.81;
client_max_body_size 4G;
access_log /home/pi/door_site/logs/nginx-access.log;
error_log /home/pi/door_site/logs/nginx-error.log;
location /static/ {
alias /home/pi/door_site/static/;
}
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
if (!-f $request_filename) {
proxy_pass http://app_server;
break;
}
}
}
In Nginx virtual server conf file inside server section you need to add to location sections for Nginx can serve files in /static/ and /media/ folders:
location /media {
alias /path/to/your/folder/media;
}
location /static {
alias /path/to/your/folder/static;
}
After that test Nginx configuration:
sudo nginx -t
and reload Nginx:
sudo nginx -s reload
(or restart - sudo /etc/init.d/nginx restart )
try this config using server root and try_files
upstream app_server {
server unix:/home/pi/door_site/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name 192.168.1.81;
client_max_body_size 4G;
access_log /home/pi/door_site/logs/nginx-access.log;
error_log /home/pi/door_site/logs/nginx-error.log;
root /path/to/root # place your static directories in root i.e /path/to/root/static
location / {
try_files $uri #proxy_to_app;
}
location #proxy_to_app {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
}
This will try to find your static file and then move onto your app server.
Make sure nginx is running as the right user to access your files and that your static files permissions are correctly set perhaps with:
chmod -R u=rwX,g=rwX,o=rX static_dir
I my case it was a permission issue on static directory and it worked after assigning proper permissions.

Setup two websites using Django apps and Nginx, but after enter site1.com it keeps showing content of site2.com

After I type in my browser www.site2.com, it open page of www.site1.com.
I think my configuration for both nginx and gunicorn are fine.
etc/nginx/site-available/site1 - my default website www.site1.com
upstream app_server {
server 127.0.0.1:9000 fail_timeout=0;
}
server {
listen 80 default_server;
listen [::]:80 default_server ipv6only=on;
root /usr/share/nginx/html;
index index.html index.htm;
client_max_body_size 4G;
server_name _;
keepalive_timeout 5;
# Your Django project's media files - amend as required
location /media {
alias /home/django/site1/media;
}
# your Django project's static files - amend as required
location /static {
alias /home/django/site1/static_root;
}
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
}
etc/nginx/site-available/site2 - my new website www.site2.com
upstream app_server2 {
server 127.0.0.1:9500 fail_timeout=0;
}
server {
listen 80;
listen [::]:80;
root /usr/share/nginx/html;
index index.html index.htm;
client_max_body_size 4G;
server_name www.site2.com site2.com;
keepalive_timeout 5;
# Your Django project's media files - amend as required
location /media {
alias /home/django/test-django/site2/media;
}
# your Django project's static files - amend as required
location /static {
alias /home/django/test-django/site2/static;
}
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server2;
}
}
etc/init/site1.conf
description "Gunicorn daemon for Django project"
start on (local-filesystems and net-device-up IFACE=eth0)
stop on runlevel [!12345]
# If the process quits unexpectedly trigger a respawn
respawn
setuid django
setgid django
chdir /home/django
exec gunicorn \
--name=site1\
--pythonpath=site1 \
--bind=0.0.0.0:9000 \
--config /etc/gunicorn.d/gunicorn.py \
site1.wsgi:application
etc/init/site2.conf
description "Gunicorn daemon for Django project"
start on (local-filesystems and net-device-up IFACE=eth0)
stop on runlevel [!12345]
# If the process quits unexpectedly trigger a respawn
respawn
setuid django
setgid django
chdir /home/django/test-django/
exec gunicorn \
--name=site2\
--pythonpath=site2 \
--bind=127.0.0.1:9500 \
--config /etc/gunicorn.d/gunicorn.py \
site2.wsgi:application
I didn't forget about service nginx restart and service site2 restart, also after nginx -t I don't have any erros.
In your wsgi.py files of your projects, are you using the same variable for the django.conf.ENVIRONMENT_VARIABLE? Usually the line is: os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project_site.settings")
Try to change the line for those:
django.conf.ENVIRONMENT_VARIABLE = "DJANGO_SITEA_SETTINGS_MODULE"
os.environ.setdefault("DJANGO_SITEA_SETTINGS_MODULE", "project_site.settings")
Change SITEA for whatever you want for each project and also project_site.settings for the name of the folder where the settings file is. Restart the server after this changes.
I don't know if this is the answer but it may help because it fixed for me the same problem you describe.

how to setup subdomain in following environment: nginx, supervisor, django, gunicorn?

I have a django app setup with nginx+gunicorn+supervisor and its working fine. But i need to create a subdomain for staging or development like "dev.domain.com". I have added another server block in nginx.conf for my subdomain. But my subdomain url was always pointing main domain site. so i changed the port no in proxy_pass as suggested on other posts. but due to gunicorn and supervisord i needed to add another conf file for this subdomain in "/etc/supervisord/conf.d/subdomain.conf" but when i reload supervisord its not able to start my subdomain program. below is my nginx.conf, subdomain.conf, script.sh:
nginx.conf
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
gzip on;
gzip_static on;
gzip_types application/x-javascript text/css text/html application/json text/css text/json;
server {
listen 80;
server_name domain_name
# no security problem here, since / is alway passed to upstream
root /home/path/to/project/base
# serve directly - analogous for static/staticfiles
location /static/ {
# if asset versioning is used
if ($query_string) {
expires max;
}
autoindex off;
root /home/path/to/static/;
}
location / {
proxy_pass_header Server;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_connect_timeout 10;
proxy_read_timeout 10;
proxy_pass http://localhost:8000/;
proxy_set_header X-Forwarded-For $remote_addr;
}
}
server {
listen 80;
server_name subdomain_name
# no security problem here, since / is alway passed to upstream
root /home/path/to/subdomain_directory(which is different, you can say it is fully differnt project which i want to run as development project);
# serve directly - analogous for static/staticfiles
location / {
proxy_pass_header Server;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_connect_timeout 10;
proxy_read_timeout 10;
proxy_pass http://localhost:9000/;
proxy_set_header X-Forwarded-For $remote_addr;
}
}
}
script.sh
set -e
NUM_WORKERS=4
# user/group to run as
USER=user_name
#GROUP=your_unix_group
cd /home/path/to/subdomain_base
source subdomain_virtualenv_activation
LOGFILE=log_file_path
LOGDIR=$(dirname $LOGFILE)
test -d $LOGDIR || mkdir -p $LOGDIR
exec virtualenvironment/bin/gunicorn_django -w $NUM_WORKERS \
--user=$USER --log-level=debug \
--log-file=$LOGFILE 2>>$LOGFILE
subdomain.conf
[program:programname]
directory = /home/path/to/subdomainbase/
user = user_name
command = /home/path/to/script.sh
stdout_logfile = /home/path/to/log
stderr_logfile = /home/path/to/log
I have a procfile too as suggested in gunicorn which is in base directory
Procfile
./manage.py runserver_plus 0.0.0.0:$PORT
Ok so these are my configurations. Please check where i am doing the wrong thing. I just want to run my development server as a different project but under same domain as subdomain. after all this whatever changes i am doing, main domain is working fine wioth the same process. Please let me know if you need more info on this error.
EDIT
I am reading again your post, and... Should not you must set ADDRESS in your gunicorn script? gunicorn by default uses port 8000, maybe your subdomain is trying to use the same port?
END EDIT
I have two Django applications running with nginx, gunicorn and supervisor as you want to do (well, not the same, but very similar, i have two domains and a subdomain). I don't see where is your mistake, I think must be in nginx configuration. Maybe the "root" line?
Have you seen if supervisord returns you an error when you try to start it using "supervisorctl" command?
I can show you my configuration and you can compare it:
I have two .conf files for nginx:
domain1.conf:
server {
listen 80;
server_name domain1.net;
return 301 $scheme://www.domain1.net$request_uri;
}
server {
listen 80;
server_name www.domain1.net;
access_log /var/log/nginx/domain1.log;
location /static {
alias /var/www/domain1/media/;
autoindex on;
access_log off;
}
location / {
proxy_pass_header Server;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_connect_timeout 10;
proxy_read_timeout 10;
proxy_pass http://127.0.0.1:8000/;
}
}
and domain2.conf:
server {
listen 80;
server_name subdomain.domain2.es;
access_log /var/log/nginx/domain2.log;
location /static {
alias /var/www/dev/domain2/domain2/static/;
autoindex on;
access_log off;
}
location / {
proxy_pass_header Server;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_connect_timeout 10;
proxy_read_timeout 10;
proxy_pass http://127.0.0.1:8005/;
}
}
My two gunicor scripts are the same, just changing paths and ADDRESS in one of them:
#!/bin/bash
set -e
LOGFILE=/var/log/gunicorn/domain1.log
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=1
# user/group to run as
USER=user
GROUP=user
ADDRESS=127.0.0.1:8005
cd /var/www/dev/domain1
source /path/to/venv/domain1/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec gunicorn_django -w $NUM_WORKERS --bind=$ADDRESS \
--user=$USER --group=$GROUP --log-level=debug \
--log-file=$LOGFILE 2>>$LOGFILE
My two supervisor scripts are the same too:
[program:domain1]
directory = /var/www/dev/domain1/
user = user
command = /path/to/bin/gunicorn_domain1.sh
stdout_logfile = /var/log/nginx/domain1.log
stderr_logfile = /var/log/nginx/domain1.log
I hope you found this helpful.