Empty docstrings when running django-piston docs under nginx - django

I'm using django-piston for my REST json api, and I have it all set up for documentation through pistons built in generate_doc function. Under the django runserver, it works great. The template that loops over the doc objects successfully lists the docstrings for both the class and for each method.
When I serve the site via nginx and uwsgi, the docstrings are empty. At first I thought this was a problem with the django markup filter and using restructuredtext formatting, but when I turned that off and just simply tried to see the raw docstring values in the template, they are None.
I don't see any issues in the logs, and I can't understand why nginx/uwsgi is the factor here, but honestly it does work great over the dev runserver. I'm kind of stuck on how to start debugging this through nginx/uwsgi. Has anyone run into this situation or have a suggestion of where I can start to look?
My doc view is pretty simple:
views.py
def ApiDoc(request):
docs = [
generate_doc(handlers.UsersHandler),
generate_doc(handlers.CategoryHandler),
]
c = {
'docs': docs,
'model': 'Users'
}
return render_to_response("api/docs.html", c, RequestContext(request))
And my template is almost identical to the stock piston template:
api/docs.html
{% load markup %}
...
{% for doc in docs %}
<h5>top</h5>
<h3><a id="{{doc.name}}">{{ doc.name|cut:"Handler" }}:</a></h3>
<p>
{{ doc.doc|default:""|restructuredtext }}
</p>
...
{% for method in doc.get_all_methods %}
{% if method.http_name in doc.allowed_methods %}
<dt><a id="{{doc.name}}_{{method.http_name}}">request</a> <i>{{ method.http_name }}</i></dt>
{% if method.doc %}
<dd>
{{ method.doc|default:""|restructuredtext }}
<dd>
{% endif %}
The rendered result of this template under nginx would be that doc.doc and method.doc are None. I have tried removing the filter and just checking the raw value to confirm this.
I'm guessing the problem would have to be somewhere in the uwsgi layer, and its environment. Im running uwsgi with a config like this:
/etc/init/uwsgi.conf
description "uWSGI starter"
start on (local-filesystems
and runlevel [2345])
stop on runlevel [016]
respawn
exec /usr/sbin/uwsgi \
--uid www-data \
--socket /opt/run/uwsgi.sock \
--master \
--logto /opt/log/uwsgi_access.log \
--logdate \
--optimize 2 \
--processes 4 \
--harakiri 120 \
--post-buffering 8192 \
--buffer-size 8192 \
--vhost \
--no-site
And my nginx server entry location snippet looks like this:
sites-enabled/mysite.com
server {
listen 80;
server_name www.mysite.com mysite.com;
set $home /var/www/mysite.com/projects/mysite;
set $pyhome /var/www/mysite.com/env/mysite;
root $home;
...
location ~ ^/(admin|api)/ {
include uwsgi_params;
uwsgi_pass uwsgi_main;
uwsgi_param UWSGI_CHDIR $home;
uwsgi_param UWSGI_SCRIPT wsgi_app;
uwsgi_param UWSGI_PYHOME $pyhome;
expires epoch;
}
...
}
Edit: Configuration Info
Server: Ubuntu 11.04
uWSGI version 1.0
nginx version: nginx/1.0.11
django non-rel 1.3.1
django-piston latest pypi 0.2.3
python 2.7

uWSGI is starting up the interpreter in way equivalent to -OO option to Python command line. This second level of optimisation removes doc strings.
-OO : remove doc-strings in addition to the -O optimizations
Change:
--optimize 2
to:
--optimize 1

Related

Docker + Django if_debug template tag is not working

I am running a docker-compose with nginx routing requests to my Django server, with this config:
upstream django {
server backend:8000;
}
server {
listen 80;
location / {
proxy_pass http://django;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
}
To use the {% if debug %} template tag I know I need both the correct internal IP and the DEBUG setting turned on. I get the correct internal IPs by running the following code snippet in my Django settings.py:
import socket
hostname, _, ips = socket.gethostbyname_ex(socket.gethostname())
INTERNAL_IPS = [ip for ip in ips] + ['127.0.0.1']
When I docker exec -it backend sh, and import the DEBUG and INTERNAL_IPS settings, I get that the values are ['172.19.0.4', '127.0.0.1'] and True, respectively. Then, when I run docker inspect -f '{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq) to check the containers' IP address, I see that the backend container that runs Django has an IP of 172.19.0.4, same as the internal IP.
Despite this all, in the Django template, I get a Django debug mode error (further proving that debug is turned on) that says that an error got triggered on line 61!!!
57 {% if debug %}
58 <!-- This url will be different for each type of app. Point it to your main js file. -->
59 <script type="module" src="http://frontend:3000/src/main.jsx"></script>
60 {% else %}
61 {% render_vite_bundle %}
62 {% endif %}
Can someone help me understand what I'm doing wrong and why this template tag won't work? Is there an easier way to do an if statement based on the Django debug setting? Thanks in advance!
I figured it out! I was getting the IP of the backend container when I typed hostname, _, ips = socket.gethostbyname_ex(socket.gethostname()), but what I really wanted was to mark Nginx's forwarding as "internal". So, I changed the line to hostname, _, ips = socket.gethostbyname_ex("nginx") and the request got forwarded correctly. Might be worth it to consider whitelisting all of the DOcker containers' internal IPs so we can receive messages freely.

Django CSRF validation fails on POST requests : referer checking failed - no Referer

My Django website is in HTTPS. When I am trying to POST data to the website from a script I get this error : "referer checking failed - no Referer". It seems to be a CSRF issue but I do not know how to solve it.
Example :
import requests
r = requests.post('https://mywebsite/mypage', data = {'key':'value'})
print r.text
gives me this output :
[...]
<p>Reason given for failure:</p>
<pre>
Referer checking failed - no Referer.
</pre>
<p>In general, this can occur when there is a genuine Cross Site Request Forgery, or when
<a
href="https://docs.djangoproject.com/en/1.8/ref/csrf/">Django's
CSRF mechanism</a> has not been used correctly. For POST forms, you need to
ensure:</p>
<ul>
<li>Your browser is accepting cookies.</li>
<li>The view function passes a <code>request</code> to the template's <code>render</code>
method.</li>
<li>In the template, there is a <code>{% csrf_token
%}</code> template tag inside each POST form that
targets an internal URL.</li>
<li>If you are not using <code>CsrfViewMiddleware</code>, then you must use
<code>csrf_protect</code> on any views that use the <code>csrf_token</code>
template tag, as well as those that accept the POST data.</li>
</ul>
[...]
Do I need to pass a referer to my headers before sending the POST data - which would not be convenient ? Or should I disable CSRF for this page ?
Thanks
AFAIK, This is the purpose of CSRF, to avoid posting data from unknown strange sources. You need csrf token to post this which django generates dynamically.
Upgrading Django might fix the missing Referer error.
As of Django 4.0 (release notes), the backend will first check the Origin header before falling back to the Referer header (source):
CsrfViewMiddleware verifies the Origin header, if provided by the browser, against the current host and the CSRF_TRUSTED_ORIGINS setting. This provides protection against cross-subdomain attacks.
In addition, for HTTPS requests, if the Origin header isn’t provided, CsrfViewMiddleware performs strict referer checking. This means that even if a subdomain can set or modify cookies on your domain, it can’t force a user to post to your application since that request won’t come from your own exact domain.
It's possible you have a reverse proxy running, for example an nginx proxy_pass to 127.0.0.1:8000?
In this case, Django expects the Cross-Site Forgery Protection tokens to match hostname 127.0.0.1, but they will be coming from a normal domain (for example example.com).
Expected Source
Actual Source
http://127.0.0.1
https://example.com
HTTP reverse proxy (example.com:80 -> localhost:3000) is a common way to use nginx with NodeJS applications, but it doesn't work well with Django
Client-Facing URL
Server Proxy URL
https://example.com
http://127.0.0.1:3000
It is better to run Django through a Unix socket rather than a port (example.com:80 -> <socket>). You can do this with Gunicorn:
Client-Facing URL
Server Proxy URL
https://example.com
unix:/run/example.com.sock
Here's how to do this with Django, Gunicorn, and nginx:
Let's say you've got a Django project root, which contains a system folder (the one where settings.py and wsgi.py are):
export DJANGO_PROJECT_PATH=/path/to/django/root
export DJANGO_SETTING_FOLDER=system
First, make sure you have Gunicorn installed and that you are using a virtual environment:
cd $DJANGO_PROJECT_PATH
source .venv/bin/activate # <- Use a virtual environment
pip3 install gunicorn # <- install Gunicorn in the venv
Run Gunicorn. This will start the Django project similar to running python3 manage.py runserver, except that you can listen for requests on a Unix socket:
$DJANGO_PROJECT_PATH/.venv/bin/gunicorn \
--workers=3 \
--access-logfile - \
--bind unix:/run/example.com.sock \ # <- Socket
--chdir=$DJANGO_PROJECT_PATH/ \
$DJANGO_SETTING_FOLDER.wsgi:application
Then create an HTTP proxy using nginx that passes HTTP requests from clients through the gunicon-created socket:
/etc/nginx/sites-enabled/example.com:
server {
listen 80;
listen [::]:80;
server_name example.com;
# serve static files directly through nginx
location /static/ {
autoindex off;
root /path/to/django/root;
}
# serve user-uploaded files directly through nginx
location /media/ {
autoindex off;
root /path/to/django/root;
}
# You can do fun stuff like aliasing files from other folders
location /robots.txt {
alias /path/to/django/root/static/robots.txt;
}
# here is the proxy magic
location / {
include proxy_params;
proxy_pass http://unix:/run/example.com.sock; # <- the socket!
}
}
Make sure to restart nginx:
sudo service restart nginx
After all this, your csrf tokens should match the domain name of your site and you'll be able to log in and submit forms.

Media folder permission while deploying django app with Ansible

I am deploying with Ansible, in task I do check whether a folder exists or not for e.g. log, assets etc. and give them 755 permission
- name: Ensure source, static, media and logs directories exist
file: dest={{ item }} mode=755 owner={{ app_user }} group={{ app_group }} state=directory
with_items:
- "{{ static_dir }}"
- "{{ media_dir }}"
- "{{ logs_dir }}"
I am running the app with app_user and who is in the group of apache so my all files and directories have app_user:apache permission.
With the above permission I'am not able to upload files to media directory, but when I give chown -R g+w media permission to media directory uploads happens, but then ansible stops working as media gets apache:apache permission.
How do I resolve this issue, what permission do I give to media folder?
My django project resides in /var/www/www.example.com/ and media is in /var/www/www.example.com/src/media/
www.example.com folder has app_user:apache chown.
The Ansible file module needs the full octal number supplied to the mode parameter, rather than the shorthand 3 digit version we are used to using with the chmod command.
As mentioned on http://docs.ansible.com/ansible/file_module.html, "Leaving off the leading zero will likely have unexpected results.".
Try:
file: dest={{ item }} mode=0755 owner={{ app_user }} group={{ app_group }} state=directory
Hope that helps.

Django+gunicorn+nginx upload large file 502 error

Problem
Uploading 1-2mb files works fine.
When I attempt to upload 16mb file, i get 502 error after several seconds
More detalied:
I click "Upload"
Google Chrome uploads file (upload status is changing from 0% to 100% in left bottom corner)
Status changes to "Waiting for HOST", where HOST is my site hostname
After a half of minute server returns "502 Bad Gateway"
My view:
def upload(request):
if request.method == 'POST':
f = File(data=request.FILES['file'])
f.save()
return redirect(reverse(display), f.id)
else:
return render('filehosting_upload.html', request)
render(template, request [,data]) is my own shorthand that deals with some ajax stuff;
The filehosting_upload.html:
{% extends "base.html" %}
{% block content %}
<h2>File upload</h2>
<form action="{% url nexus.filehosting.views.upload %}" method="post" enctype="multipart/form-data">
{% csrf_token %}
<input type="file" name="file">
<button type="submit" class="btn">Upload</button>
</form>
{% endblock %}
Logs & specs
There are nothing informative in logs i can find.
Versions:
Django==1.4.2
Nginx==1.2.1
gunicorn==0.17.2
Command line parameters
command=/var/www/ernado/data/envs/PROJECT_NAME/bin/gunicorn -b localhost:8801 -w 4 PROJECT_NAME:application
Nginx configuration for related location:
location /files/upload {
client_max_body_size 100m;
proxy_pass http://HOST;
proxy_connect_timeout 300s;
proxy_read_timeout 300s;
}
Nginx log entry (changed MY_IP and HOST)
2013/03/23 19:31:06 [error] 12701#0: *88 upstream prematurely closed connection while reading response header from upstream, client: MY_IP, server: HOST, request: "POST /files/upload HTTP/1.1", upstream: "http://127.0.0.1:8801/files/upload", host: "HOST", referrer: "http://HOST/files/upload"
Django log
2013-03-23 19:31:06 [12634] [CRITICAL] WORKER TIMEOUT (pid:12829)
2013-03-23 19:31:06 [12634] [CRITICAL] WORKER TIMEOUT (pid:12829)
2013-03-23 19:31:06 [13854] [INFO] Booting worker with pid: 13854
Question(s)
how to fix that?
is it possible to fix that without nginx upload module?
Update 1
Tried suggested config
gunicorn --workers=3 --worker-class=tornado --timeout=90 --graceful-timeout=10 --log-level=DEBUG --bind localhost:8801 --debug
Works fine for me now.
I run my gunicorn with that parameters, try :
python manage.py run_gunicorn --workers=3 --worker-class=tornado --timeout=90 --graceful-timeout=10 --log-level=DEBUG --bind 127.0.0.1:8151 --debug
or if you run differently, you may run with that options
For large files handling you should use a worker-class. Also I had some trouble using gevent in python 3.7, better to use 3.6.
Django, Python 3.6 example:
Install:
pip install gevent
Run
gunicorn --chdir myApp myApp.wsgi --workers 4 --worker-class=gevent --bind 0.0.0.0:80 --timeout=90 --graceful-timeout=10
You need to used an other worker type class an async one like gevent or tornado see this for more explanation :
First explantion :
You may also want to install Eventlet or Gevent if you expect that your application code may need to pause for extended periods of time during request processing
Second one :
The default synchronous workers assume that your application is resource bound in terms of CPU and network bandwidth. Generally this means that your application shouldn’t do anything that takes an undefined amount of time. For instance, a request to the internet meets this criteria. At some point the external network will fail in such a way that clients will pile up on your servers.

Deploy Django with Gunicorn and APACHE

I have a Django project and I wanna delivery it using gunicorn (and apache proxing).
I can't use Nginx, so that's no possible.
I've set the Apache proxy and setup a runner script to gunicorn, but i am get this weird error
2012-08-27 14:03:12 [34355] [DEBUG] GET /
2012-08-27 14:03:12 [34355] [ERROR] Error handling request
Traceback (most recent call last):
File "/home/tileone/venv/lib/python2.6/site-packages/gunicorn/workers/sync.py", line 93, in handle_request
self.address, self.cfg)
File "/home/tileone/venv/lib/python2.6/site-packages/gunicorn/http/wsgi.py", line 146, in create
path_info = path_info.split(script_name, 1)[1]
IndexError: list index out of range
I am running this script
#!/bin/bash
LOGFILE=/var/log/gunicorn/one-project.log
VENV_DIR=/path/to/venv/
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=5
# user/group to run as
USER=USER
GROUP=GROUP
BIND=127.0.0.1:9999
cd /path_to_project
echo 'Setup Enviroment'
#some libraries
echo 'Setup Venv'
source $VENV_DIR/bin/activate
export PYTHONPATH=$VENV_DIR/lib/python2.6/site-packages:$PYTHONPATH
#Setup Django Deploy
export DJANGO_DEPLOY_ENV=stage
echo 'Run Server'
test -d $LOGDIR || mkdir -p $LOGDIR
export SCRIPT_NAME='/home/tileone/one-project'
exec $VENV_DIR/bin/gunicorn_django -w $NUM_WORKERS --bind=$BIND\
--user=$USER --group=$GROUP --log-level=debug \
--log-file=$LOGFILE 2>>$LOGFILE
and my apache configuration is like this:
Alias /static/ /hpath_to_static/static/
Alias /media/ /path_to_static/media/
Alias /favicon.ico /path_to/favicon.ico
ProxyPreserveHost On
<Location />
SSLRequireSSL
ProxyPass http://127.0.0.1:9999/
ProxyPassReverse http://127.0.0.1:9999/
RequestHeader set SCRIPT_NAME /home/tileone/one-project/
RequestHeader set X-FORWARDED-PROTOCOL ssl
RequestHeader set X-FORWARDED-SSL on
</Location>
What am i doing wrong?
In case anyone has similar issues, I managed to fix this by removing the equivalent of:
RequestHeader set SCRIPT_NAME /home/tileone/one-project/
And instead adding to settings.py the equivalent of:
FORCE_SCRIPT_NAME = '/one-project'
Of course for this, the apache configuration should be more like:
ProxyPreserveHost On
<Location /one-project/>
SSLRequireSSL
ProxyPass http://127.0.0.1:9999/
ProxyPassReverse http://127.0.0.1:9999/
RequestHeader set X-FORWARDED-PROTOCOL ssl
RequestHeader set X-FORWARDED-SSL on
</Location>
The reason for the fix proposed in the accepted answer is that you need to decide between one of the following two approaches:
Let the HTTP server strip the location sub path BEFORE forwarding the request to the WSGI server (as explained above ... this is done when both the Location and the ProxyPass directive end with a forward slash. Nginx behaves the same way). In this case, you may not use the SCRIPT_NAME HTTP header/gunicorn-env-variable, because gunicorn would try to strip the value from the incoming URL (but that fails because the web server, Apache, already did that). In this case, you're forced to use Django's FORCE_SCRIPT_NAME setting.
Let the request URL passed to gunicorn unmodified (an example of an URL might be /one-project/admin), and use the SCRIPT_NAME HTTP header (or gunicorn-env-variable). Because then gunicorn will modify the request and strip the value of SCRIPT_NAME from the URL before Django handles building the response.
I would prefer option 2, because you only need to change one file, the web server configuration, and all changes are neatly together.