Deploy Django with Gunicorn and APACHE - django

I have a Django project and I wanna delivery it using gunicorn (and apache proxing).
I can't use Nginx, so that's no possible.
I've set the Apache proxy and setup a runner script to gunicorn, but i am get this weird error
2012-08-27 14:03:12 [34355] [DEBUG] GET /
2012-08-27 14:03:12 [34355] [ERROR] Error handling request
Traceback (most recent call last):
File "/home/tileone/venv/lib/python2.6/site-packages/gunicorn/workers/sync.py", line 93, in handle_request
self.address, self.cfg)
File "/home/tileone/venv/lib/python2.6/site-packages/gunicorn/http/wsgi.py", line 146, in create
path_info = path_info.split(script_name, 1)[1]
IndexError: list index out of range
I am running this script
#!/bin/bash
LOGFILE=/var/log/gunicorn/one-project.log
VENV_DIR=/path/to/venv/
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=5
# user/group to run as
USER=USER
GROUP=GROUP
BIND=127.0.0.1:9999
cd /path_to_project
echo 'Setup Enviroment'
#some libraries
echo 'Setup Venv'
source $VENV_DIR/bin/activate
export PYTHONPATH=$VENV_DIR/lib/python2.6/site-packages:$PYTHONPATH
#Setup Django Deploy
export DJANGO_DEPLOY_ENV=stage
echo 'Run Server'
test -d $LOGDIR || mkdir -p $LOGDIR
export SCRIPT_NAME='/home/tileone/one-project'
exec $VENV_DIR/bin/gunicorn_django -w $NUM_WORKERS --bind=$BIND\
--user=$USER --group=$GROUP --log-level=debug \
--log-file=$LOGFILE 2>>$LOGFILE
and my apache configuration is like this:
Alias /static/ /hpath_to_static/static/
Alias /media/ /path_to_static/media/
Alias /favicon.ico /path_to/favicon.ico
ProxyPreserveHost On
<Location />
SSLRequireSSL
ProxyPass http://127.0.0.1:9999/
ProxyPassReverse http://127.0.0.1:9999/
RequestHeader set SCRIPT_NAME /home/tileone/one-project/
RequestHeader set X-FORWARDED-PROTOCOL ssl
RequestHeader set X-FORWARDED-SSL on
</Location>
What am i doing wrong?

In case anyone has similar issues, I managed to fix this by removing the equivalent of:
RequestHeader set SCRIPT_NAME /home/tileone/one-project/
And instead adding to settings.py the equivalent of:
FORCE_SCRIPT_NAME = '/one-project'
Of course for this, the apache configuration should be more like:
ProxyPreserveHost On
<Location /one-project/>
SSLRequireSSL
ProxyPass http://127.0.0.1:9999/
ProxyPassReverse http://127.0.0.1:9999/
RequestHeader set X-FORWARDED-PROTOCOL ssl
RequestHeader set X-FORWARDED-SSL on
</Location>

The reason for the fix proposed in the accepted answer is that you need to decide between one of the following two approaches:
Let the HTTP server strip the location sub path BEFORE forwarding the request to the WSGI server (as explained above ... this is done when both the Location and the ProxyPass directive end with a forward slash. Nginx behaves the same way). In this case, you may not use the SCRIPT_NAME HTTP header/gunicorn-env-variable, because gunicorn would try to strip the value from the incoming URL (but that fails because the web server, Apache, already did that). In this case, you're forced to use Django's FORCE_SCRIPT_NAME setting.
Let the request URL passed to gunicorn unmodified (an example of an URL might be /one-project/admin), and use the SCRIPT_NAME HTTP header (or gunicorn-env-variable). Because then gunicorn will modify the request and strip the value of SCRIPT_NAME from the URL before Django handles building the response.
I would prefer option 2, because you only need to change one file, the web server configuration, and all changes are neatly together.

Related

Nginx not serving django 3 through wsgi

I am trying to serve a Django 3 build from nginx through wsgi and I cannot seem to get the last stage. I can browse to the Django site by a runserver and launching the uwsgi.ini, but, when I try to browse to the site through nginx I get a 502 error (bad gateway)(firefox). If I try from a remote site it renders the nginx home page.
I have a build of Apache on the same server. I had to spoof an IP and use a unique port to get them running side by side.
the nginx/error.log does not register any problem.
Below is the uwsgi.ini.
[uwsgi]
# variables
projectname = website
base = /opt/website/
# configuration
master = true
http = :8000
uid = nginx
virtualenv = /opt/website/djangoprojectenv
pythonpath = %(base)
chdir = %(base)
env = DJANGO_SETTINGS_MODULE=%(projectname).settings.pro
#module = %(projectname).wsgi:application
module = website.wsgi:application
socket = /tmp/%(projectname).new.sock
chown-socket = %(uid):nginx
chmod-socket = 666
And below is the conf file from nginx/conf.d
server {
listen 192.168.1.220:81;
server_name willdoit.com;
access_log off;
error_log /var/log/nginx_error.log;
location / {
uwsgi_pass unix:/tmp/website.sock;
include /etc/nginx/uwsgi_params;
uwsgi_read_timeout 300s;
client_max_body_size 32m;
}
}
The /tmp/website.sock file is owned by nginx:nginx.
If there are additional details I need to post, please advise. Any help you can provide would be welcome.
The socket was defined incorrectly. Closing.

Django 502 bad gateway with Nginx and Uwsgi

I can normally open up my web in the front several hours after deployment,but later , it occurred 502 bad gateway ,it is so wired, my web uses Django and Nginx and Uwsgi, i do research a lot on google,but failed with nothing
Here is my configuration:
1.Nginx configuration
# mysite_nginx.conf
upstream django {
server 127.0.0.1:8004; # for a web port socket (we'll use this first)
}
server {
listen 80;
server_name www.example.com # substitute your machine's IP address or FQDN
charset utf-8;
client_max_body_size 75M; # adjust to taste
location /media {
alias /home/blender_font_project/django_file/Blender_website/media;
}
location /static {
alias /home/blender_font_project/django_file/Blender_website/static;
}
location / {
uwsgi_pass 127.0.0.1:8003;
include /etc/nginx/uwsgi_params;
}
}
2.uwsgi configuration
# mysite_uwsgi.ini file
[uwsgi]
chdir = /home/blender_font_project/django_file/Blender_website
module = djangoTest5.wsgi
master = true
processes = 10
socket = :8003
vacuum = true
harakiri=60
daemonize=/home/blender_font_project/uwsgi_file/real3dfont_logfile
3.this is my Nginx error log
231 connect() failed (111: Connection refused) while connecting to upstream
BTW , i have set Django to DEBUG Ture and i can access my resource by www.example.com/static/example.jpg,but the web page shows 502
I really dont know why , thanks if you offer any help!
(...After million years struggle and strive,with inspiration from a super hero in comment named #Atul Mishra , i finally figure it out...)
It is the matter Django itself,i forget to download mysql module in View , i would have expect a Django error html if it is the django problem, but no , so i mistakenly attribute it to Nginx or Uwsgi
But the wired thing is that Django should have report the error , but no ! what an irresponsible dude!!
so , 1.remember to add Django error log function ,it saves your life , and
2.test Django with runserver before Nginx enter the stage even when comet is striking the earth!!

Django CSRF validation fails on POST requests : referer checking failed - no Referer

My Django website is in HTTPS. When I am trying to POST data to the website from a script I get this error : "referer checking failed - no Referer". It seems to be a CSRF issue but I do not know how to solve it.
Example :
import requests
r = requests.post('https://mywebsite/mypage', data = {'key':'value'})
print r.text
gives me this output :
[...]
<p>Reason given for failure:</p>
<pre>
Referer checking failed - no Referer.
</pre>
<p>In general, this can occur when there is a genuine Cross Site Request Forgery, or when
<a
href="https://docs.djangoproject.com/en/1.8/ref/csrf/">Django's
CSRF mechanism</a> has not been used correctly. For POST forms, you need to
ensure:</p>
<ul>
<li>Your browser is accepting cookies.</li>
<li>The view function passes a <code>request</code> to the template's <code>render</code>
method.</li>
<li>In the template, there is a <code>{% csrf_token
%}</code> template tag inside each POST form that
targets an internal URL.</li>
<li>If you are not using <code>CsrfViewMiddleware</code>, then you must use
<code>csrf_protect</code> on any views that use the <code>csrf_token</code>
template tag, as well as those that accept the POST data.</li>
</ul>
[...]
Do I need to pass a referer to my headers before sending the POST data - which would not be convenient ? Or should I disable CSRF for this page ?
Thanks
AFAIK, This is the purpose of CSRF, to avoid posting data from unknown strange sources. You need csrf token to post this which django generates dynamically.
Upgrading Django might fix the missing Referer error.
As of Django 4.0 (release notes), the backend will first check the Origin header before falling back to the Referer header (source):
CsrfViewMiddleware verifies the Origin header, if provided by the browser, against the current host and the CSRF_TRUSTED_ORIGINS setting. This provides protection against cross-subdomain attacks.
In addition, for HTTPS requests, if the Origin header isn’t provided, CsrfViewMiddleware performs strict referer checking. This means that even if a subdomain can set or modify cookies on your domain, it can’t force a user to post to your application since that request won’t come from your own exact domain.
It's possible you have a reverse proxy running, for example an nginx proxy_pass to 127.0.0.1:8000?
In this case, Django expects the Cross-Site Forgery Protection tokens to match hostname 127.0.0.1, but they will be coming from a normal domain (for example example.com).
Expected Source
Actual Source
http://127.0.0.1
https://example.com
HTTP reverse proxy (example.com:80 -> localhost:3000) is a common way to use nginx with NodeJS applications, but it doesn't work well with Django
Client-Facing URL
Server Proxy URL
https://example.com
http://127.0.0.1:3000
It is better to run Django through a Unix socket rather than a port (example.com:80 -> <socket>). You can do this with Gunicorn:
Client-Facing URL
Server Proxy URL
https://example.com
unix:/run/example.com.sock
Here's how to do this with Django, Gunicorn, and nginx:
Let's say you've got a Django project root, which contains a system folder (the one where settings.py and wsgi.py are):
export DJANGO_PROJECT_PATH=/path/to/django/root
export DJANGO_SETTING_FOLDER=system
First, make sure you have Gunicorn installed and that you are using a virtual environment:
cd $DJANGO_PROJECT_PATH
source .venv/bin/activate # <- Use a virtual environment
pip3 install gunicorn # <- install Gunicorn in the venv
Run Gunicorn. This will start the Django project similar to running python3 manage.py runserver, except that you can listen for requests on a Unix socket:
$DJANGO_PROJECT_PATH/.venv/bin/gunicorn \
--workers=3 \
--access-logfile - \
--bind unix:/run/example.com.sock \ # <- Socket
--chdir=$DJANGO_PROJECT_PATH/ \
$DJANGO_SETTING_FOLDER.wsgi:application
Then create an HTTP proxy using nginx that passes HTTP requests from clients through the gunicon-created socket:
/etc/nginx/sites-enabled/example.com:
server {
listen 80;
listen [::]:80;
server_name example.com;
# serve static files directly through nginx
location /static/ {
autoindex off;
root /path/to/django/root;
}
# serve user-uploaded files directly through nginx
location /media/ {
autoindex off;
root /path/to/django/root;
}
# You can do fun stuff like aliasing files from other folders
location /robots.txt {
alias /path/to/django/root/static/robots.txt;
}
# here is the proxy magic
location / {
include proxy_params;
proxy_pass http://unix:/run/example.com.sock; # <- the socket!
}
}
Make sure to restart nginx:
sudo service restart nginx
After all this, your csrf tokens should match the domain name of your site and you'll be able to log in and submit forms.

Django + uWSGI + nginx requests hang

I'm running a Django web application using Nginx and uWSGI. I'm having problems with the requests hanging for no apparent reason.
I have added a bunch of logging in the application, and this snippet is where it seems to hang. There are two log lines at the start of the try block, and the first one gets printed, but not he second one, so it would seem that it hangs in the middle of the code. This code is from a middleware class that I added in the Django configuration.
def process_request(self, request):
if 'auth' not in request.session:
try:
log.info("Auth not found") # this line is logged
log.info("another log line") # this line is never logged
if request.is_ajax():
return HttpResponse(status=401)
...
I managed to get a backtrace from the uWSGI thread and this is where it's stuck:
*** backtrace of 76 ***
/usr/bin/uwsgi(uwsgi_backtrace+0x2e) [0x45121e]
/usr/bin/uwsgi(what_i_am_doing+0x30) [0x451350]
/lib/x86_64-linux-gnu/libc.so.6(+0x36c30) [0x7f8a4b2b8c30]
/lib/x86_64-linux-gnu/libc.so.6(epoll_wait+0x33) [0x7f8a4b37d653]
/home/vdr/vdr-ui/env/local/lib/python2.7/site-packages/gevent/core.so(+0x27625) [0x7f8a44092625]
/home/vdr/vdr-ui/env/local/lib/python2.7/site-packages/gevent/core.so(ev_run+0x29b) [0x7f8a4409d11b]
/home/vdr/vdr-ui/env/local/lib/python2.7/site-packages/gevent/core.so(+0x32bc0) [0x7f8a4409dbc0]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4bd4) [0x7f8a4a0c30d4]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7f8a4a0c517d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x162310) [0x7f8a4a0c5310]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f8a4a08ce23]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x7d30d) [0x7f8a49fe030d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f8a4a08ce23]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7f8a4a04b837]
/home/vdr/vdr-ui/env/local/lib/python2.7/site-packages/greenlet.so(+0x375c) [0x7f8a49b1c75c]
/home/vdr/vdr-ui/env/local/lib/python2.7/site-packages/greenlet.so(+0x30a6) [0x7f8a49b1c0a6]
[0x7f8a42f26f38]
*** end of backtrace ***
SIGUSR2: --- uWSGI worker 3 (pid: 76) is managing request /login?next=/&token=45092ca6-c1a0-4c23-9d44-4d171fc561b8 since Wed Dec 2 09:52:44 2015 ---
The Nginx error log prints out [error] 619#0: *55 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.17.0.1, server: vdr
There are no errors in the printouts from uWSGI, so I'm a bit at a loss. Has anyone seen anything similar? All this is running within a Docker container if that makes any difference.
Nginx conf:
upstream uwsgi {
server unix:///tmp/vdr.sock;
}
server {
listen 80;
charset utf-8;
client_max_body_size 500M;
server_name localhost 172.17.0.2;
location /static {
alias /home/vdr/vdr-ui/static;
}
location / {
include uwsgi_params;
uwsgi_pass uwsgi;
uwsgi_read_timeout 200s;
}
}
uWSGI conf:
[uwsgi]
chdir = %d
module = alft_ui.wsgi:application
uid=1000
master=true
pidfile=/tmp/vdr.pid
vacuum=true
max-requests=5000
processes=4
env=DJANGO_SETTINGS_MODULE=alft_ui.settings.prod-live
home=/home/vdr/vdr-ui/env
socket=/tmp/vdr.sock
chmod-socket=666
So I finally found the cause for this. It turns out that my setup script added some logstash settings to the Django configuration. These settings pointed to the IP 10.8.0.1 which wasn't reachable from this environment. This would explain why the app got stuck on a logging line. Removing these settings made everything work again.
Always good to know that it was your own fault all along :)

Apache 2.4 setenvif dontlog pingdom.com

Many of webmasters use pingdom.com as a monitoring ping service.
But the problem is that /httpd/access_log is full of
208.64.28.194 - - [06/Aug/2015:12:20:22 -0500] "GET / HTTP/1.1" 200 2917 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
I set
CustomLog "logs/access_log" combined env=!dontlog
and tried to get rid of it using variations like
SetEnvIf Remote_Host "^pingdom\.com$" dontlog
SetEnvIFNoCase Remote_Host "pingdom.com$" dontlog
SetEnvIfNoCase Referer "www\.pingdom\.com" dontlog
SetEnvIFNoCase Host "^pingdom.com$" dontlog
but still no a success with any of them - so thanks for any else hint to try.
I'll put my comment here as answer so anyone will find this easier.
Since one can see from the log file, the host name is not Pingdom.com but a part of the user agent string.
Solutions to try:
First be sure you have enabled the setenvif-module. Write the command
sudo apache2ctl -M | grep setenv
It should return something like "setenvif_module (shared)"
Then you can try setting by remote address
SetEnvIf Remote_Addr "208\.64\.28\.194$" dontlog
The final working solution is this, dont log if the user agent string contains Pingdom string:
SetEnvIfNoCase User-Agent "^Pingdom" dontlog
Edit: enhanced some parts of the answer.