Django + uWSGI hold a long time to response - django

I'm running a Django web application using Nginx and uWSGI. Now I meet a problem that the finish_process view in django
I have added logging at the begin and the end of Django finish_process view.
I make a request at 17:20:18, and the view finished at 17:20:48. But uWSGI does not return response at this time, and after 577 seconds, it throws IOError when it try to write response to client, because nginx close the connection (uwsgi_read_timeout is 300 seconds).
My question is why uWSGI holds the response so long after Django handled the view? I'm a bit at a loss.
Django log:
[INFO]246 views.py/finish_process 2016-03-06 17:20:18: [VIEW][START] finish_process: id=4
[INFO]282 views.py/finish_process 2016-03-06 17:20:48: [VIEW][END] finish_process: id=4
uWSGI log:
Sun Mar 6 17:29:55 2016 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 296] during POST /api/finish_process/ (10.11.16.251)
IOError: write error
[pid: 3275|app: 0|req: 48689/48688] 10.11.16.251 () {34 vars in 553 bytes} [Sun Mar 6 17:20:18 2016] POST /api/finish_process/ => generated 0 bytes in 577024 msecs (HTTP/1.1 200) 3 headers in 0 bytes (0 switches on core 4)
Nginx error.log:
2016/03/06 17:25:18 [error] 3052#0: *44561 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.11.16.251, server: skyline, request: "POST /api/finish_process/ HTTP/1.1", upstream: "uwsgi://unix:/var/run/skyline.sock:", host: "10.11.16.253"
uwsgi.ini:
[uwsgi]
socket = /var/run/skyline.sock
chdir = /opt/skyline
processes = 1
threads = 10
master = true
env = DJANGO_SETTINGS_MODULE=skyline.prod_settings
module = skyline.wsgi:application
chmod-socket = 666
vacuum = true
die-on-term = true
Nginx conf:
server {
listen 80;
server_name skyline;
charset utf-8;
client_max_body_size 50M;
uwsgi_read_timeout 300;
location / {
include uwsgi_params;
uwsgi_pass unix:/var/run/skyline.sock;
}
}
Updated:
Solved. I made a mistake.

Related

".sock" does not exist error; connect() error

I'm trying to set up my Mezzanine/Django project on a virtual Ubuntu 14.04 box on Linode, but get a "502 Bad Gateway error" when trying to navigate to my site in my browser.
I'm following the instructions here: https://linode.com/docs/web-servers/nginx/deploy-django-applications-using-uwsgi-and-nginx-on-ubuntu-14-04.
I ran git clone in /home/django/, so everything's in a folder named FOLDER.
/home/django/ has these directories: Env/ and FOLDER/.
This should give some idea of what FOLDER/ tree looks like:
FOLDER
- product_blog
-- product_blog
--- settings.py
--- wsgi.py
-- manage.py
In my settings.py, I have:
ALLOWED_HOSTS = ["PUBLIC IP OF MY LINODE UBUNTU INSTANCE HERE"]
wsgi.py has this:
"""
WSGI config for product_blog project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/1.10/howto/deployment/wsgi/
"""
import os
from django.core.wsgi import get_wsgi_application
from mezzanine.utils.conf import real_project_name
os.environ.setdefault("DJANGO_SETTINGS_MODULE",
"%s.settings" % real_project_name("product_blog"))
application = get_wsgi_application()
/etc/uwsgi/sites/product_blog.ini has this:
[uwsgi]
project = product_blog
base = /home/django
chdir = %(base)/%(project)
home = %(base)/Env/%(project)
module = %(project).wsgi:application
master = true
processes = 2
socket = %(base)/%(project)/%(project).sock
chmod-socket = 664
vacuum = true
/etc/init/uwsgi.conf has this:
description "uWSGI"
start on runlevel [2345]
stop on runlevel [06]
respawn
env UWSGI=/usr/local/bin/uwsgi
env LOGTO=/var/log/uwsgi.log
exec $UWSGI --master --emperor /etc/uwsgi/sites --die-on-term --uid django --gid www-data --logto $LOGTO
/etc/nginx/sites-available/product_blog has this:
server {
listen 80;
server_name mydomain.com;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /home/django/FOLDER;
}
location / {
include uwsgi_params;
uwsgi_pass unix:/home/django/FOLDER/product_blog/product_blog.sock;
}
}
When I go to the IP address for the Linode box, I see a 502 Bad Gateway error.
/var/log/uwsgi.log has this...
*** has_emperor mode detected (fd: 6) ***
[uWSGI] getting INI configuration from product_blog.ini
*** Starting uWSGI 2.0.17 (64bit) on [Sun Apr 22 17:17:56 2018] ***
compiled with version: 4.8.4 on 22 April 2018 19:53:45
os: Linux-4.15.12-x86_64-linode105 #1 SMP Thu Mar 22 02:13:40 UTC 2018
nodename: ubuntu-linode
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /etc/uwsgi/sites
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
chdir() to /home/django/product_blog
chdir(): No such file or directory [core/uwsgi.c line 2629]
chdir(): No such file or directory [core/uwsgi.c line 1644]
Sun Apr 22 17:17:56 2018 - [emperor] curse the uwsgi instance product_blog.ini (pid: 4238)
Sun Apr 22 17:17:59 2018 - [emperor] removed uwsgi instance product_blog.ini
And /var/log/nginx/error.log has this (identifying info removed):
2018/04/22 17:18:14 [crit] 3953#0: *9 connect() to unix:/home/django/FOLDER/product_blog/product_blog.sock failed (2: No such file or directory) while connecting to upstream, client: 73.49.35.42, server: mydomain.com, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/home/django/FOLDER/product_blog/product_blog.sock:", host: "ADDRESS TO LINODE BOX"
It seems that product_blog.sock is supposed to be somewhere in my directory, but it is not there. How do I fix the 502 error so that I can navigate my browser to the Linode box's address and see a working website?
The socket file is supposed to be created by uWSGI. But the uWSGI log is telling you that it can't start up because it can't cd to the directory you have specified in product_blog.ini, /home/django/product_blog.
I can't tell if FOLDER is a placeholder or not, but in any case you don't seem to have included it in that path. I suppose it should be:
chdir = %(base)/FOLDER/%(project)
...
socket = %(base)/FOLDER/%(project)/%(project).sock

nginx & uwsgi deploy django app get SIGPIPE 502 error

I use nginx and uwsgi deploy my django webapp, the start few minutes all things well but after some time the ngnix will report 502 error and I check the uwsgi stderr log. there are some error marked => generated & SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request such as blow error message:
>[pid: 27427|app: 0|req: 1/1] 10.209.78.30 () {38 vars in 1030 bytes} [Mon Sep 29 15:46:00 2014] GET blabla...(just request url address) => generated 16018 bytes in 1428 msecs (HTTP/1.1 200) 6 headers in 157 bytes (1 switches on core 0)
Mon Sep 29 15:46:03 2014 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request blabla...(just request url address) (ip 10.209.78.30) !!!
Mon Sep 29 15:46:03 2014 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 287] during GET blabla...(just request url address) (10.209.78.30)
IOError: write error
[pid: 27425|app: 0|req: 1/2] 10.209.78.30 () {38 vars in 2050 bytes} [Mon Sep 29 15:46:00 2014] GET blabla...(just request url address) => generated 0 bytes in 3578 msecs (HTTP/1.1 200) 6 headers in 0 bytes (0 switches on core 0)
Mon Sep 29 15:46:04 2014 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request blabla...(just request url address) (ip 10.217.95.130) !!!
Mon Sep 29 15:46:04 2014 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 287] during GET blabla...(just request url address) (10.217.95.130)
IOError: write error
[pid: 27426|app: 0|req: 1/3] 10.217.95.130 () {38 vars in 1871 bytes} [Mon Sep 29 15:46:00 2014] GET blabla...(just request url address) => generated 0 bytes in 4415 msecs (HTTP/1.1 200) 6 headers in 0 bytes (0 switches on core 0)
Mon Sep 29 15:46:05 2014 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request blabla...(just request url address) (ip 10.217.95.130) !!!
Mon Sep 29 15:46:05 2014 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 287] during GET blabla...(just request url address) (10.217.95.130)
IOError: write error
I replace the request url by blabla...
My nginx.conf is bleow:
location / {
add_header "Access-Control-Allow-Origin" "*";
add_header "Access-Control-Allow-Methods" "GET, OPTIONS";
add_header "Access-Control-Allow-Headers" "origin, authorization, accept";
uwsgi_pass unix:///tmp/djangoapp.sock;
include uwsgi_params;
}
uWSGI launch command:
uwsgi --socket /tmp/djangoapp.sock -p 10 --wsgi-file /opt/graphite/conf/graphite_wsgi.py --chmod-socket=666

uwsgi ImportError: No module named os

I'm teaching myself how to setup an Ubuntu Server to run my Django application. I want to use Nginx + uwsgi. I know that this question can be very easy for experts but I've spent 6 days looking for it over the internet without getting it (in any case, forgive me if there is any link with the answer). I've followed a lot of tutorials and posts but I didn't found a solution.
I describe my file structure below:
My django project is located in /usr/local/projects/myproject
My virtualenv is in /root/.virtualenvs/myproject
My uwsgi config file myproject.ini is in /etc/uwsgi/apps-available/ and correctly symbolic linked in /etc/uwsgi/apps-enabled/
[uwsgi]
plugins = python
socket = /tmp/myproject.sock
chmod-socket = 644
uid = www-data
gid = www-data
master = true
enable-threads = true
processes = 2
no-site=true
virtualenv = /root/.virtualenvs/myproject
chdir = /usr/local/projects/myproject
module = myproject.wsgi:application
pidfile = /usr/local/projects/myproject/myproject.pid
logto = /var/log/uwsgi/myproject_uwsgi.log
vacuum = true
My nginx config file myproject.conf is in /etc/nginx/sites-available/ and correctly symbolic linked in /etc/nginx/sites-enabled/
# the upstream component nginx needs to connect to
upstream django {
server unix:///tmp/myproject.sock; # for a file socket
}
server {
listen 80;
server_name dev.myproject.com www.dev.myproject.com;
access_log /var/log/nginx/myproject_access.log;
error_log /var/log/nginx/myproject_error.log;
location / {
uwsgi_pass unix:///tmp/myproject.sock;
include /etc/nginx/uwsgi_params;
uwsgi_param UWSGI_SCRIPT myproject.wsgi;
}
location /media/ {
alias /usr/local/projects/myproject/media/;
}
location /static/ {
alias /usr/local/projects/myproject/static/;
}
}
When I try to access to dev.myproject.com I get an Internal Server Error. Then I take a look to my uwsgi log:
Traceback (most recent call last):
File "./myproject/wsgi.py", line 9, in <module>
import os
ImportError: No module named os
Sat Jul 26 17:39:16 2014 - unable to load app 0 (mountpoint='') (callable not found or import error)
Sat Jul 26 17:39:16 2014 - --- no python application found, check your startup logs for errors ---
[pid: 8559|app: -1|req: -1/8] 79.148.138.10 () {40 vars in 685 bytes} [Sat Jul 26 17:39:16 2014] GET / => generated 21 bytes in 0 msecs (HTTP/1.1 500) 1 headers in 57 bytes (0 switches on core 0)
I need your help because I'm not able to find a solution despite the posibility of being very simple.
If you need to know something else let me know and I will update my question as soon as possible.
I finally found a solution.
I followed kchan's suggestion about not putting any of the contents in the /root/ directory. Basically, I did some small changes in my myproject.conf file and my myproject.ini file. I created a user and structured everything like below:
uwsgi config file myproject.ini in /etc/uwsgi/apps-available/ and correctly symbolic linked in /etc/uwsgi/apps-enabled/
[uwsgi]
plugins = python
socket = /tmp/myproject.sock
chmod-socket = 644
uid = www-data
gid = www-data
master = true
enable-threads = true
processes = 2
virtualenv = /home/user/.virtualenvs/myproject
chdir = /home/user/projects/myproject
module = myproject.wsgi:application
pidfile = /home/user/projects/myproject/myproject.pid
daemonize = /var/log/uwsgi/myproject_uwsgi.log
vacuum = true
nginx config file myproject.conf in /etc/nginx/sites-available/ and correctly symbolic linked in /etc/nginx/sites-enabled/
# the upstream component nginx needs to connect to
upstream django {
server unix:///tmp/myproject.sock; # for a file socket
}
server {
listen 80;
server_name dev.myproject.com www.dev.myproject.com;
access_log /var/log/nginx/myproject_access.log;
error_log /var/log/nginx/myproject_error.log;
location / {
uwsgi_pass django;
include /etc/nginx/uwsgi_params;
}
location /media/ {
alias /home/user/projects/myproject/media/;
}
location /static/ {
alias /home/user/projects/myproject/static/;
}
}
I must say that I think the real problem was to try to setup my DB configuration in the postactivate file of my virtualenv. Hope to help someone else.

Request is too slow with nginx, uwsgi, django, pgpool, postgresql

I have deployed a service.
It is served by nginx, uwsgi, django, pgpool, postgresql stack.
The page that has no db access is no problem.
However, the page that has some data from postgres is tpo slow.
Actually, db query time is quite fast. under 10 ms.
But result to client is over the 120000 ms
uwsgi log
[pid: 2056|app: 0|req: 4/10] 211.207.245.120 () {44 vars in 1116 bytes} [Thu Jul 19 00:53:31 2012] GET /account/admin/cb_main/invitationuser/ => generated 38606 bytes in 122126 msecs (HTTP/1.1 200) 8 headers in 373 bytes (1 switches on core 0)
My settings are below
Please check my settings and solve my problems
nginx settings
upstream cuying {
ip_hash;
server 127.0.0.1:9001;
}
server {
listen 8080;
root /home/cuying_mgr/;
client_max_body_size 20M;
server_name cuying.com;
location / {
uwsgi_pass cuying;
include uwsgi_params;
uwsgi_read_timeout 120;
uwsgi_send_timeout 120;
}
}
uwsgi settings
chdir=/home/cuying_mgr/develop/virenv/cuying/cuying
processes=2
workers=8
enable-threads=true
socket=127.0.0.1:9001
module=cuyingProject.wsgi:application
master=True
pidfile=/tmp/cuying-master.pid
vacuum=True
close-on-exec=True
max-requests=3000
post-buffering=8192
socket-timeout=120
limit-post=20480000
virtualenv=/home/cuying_mgr/develop/virenv/cuying
daemonize=/var/log/uwsgi/cuying.log
Please help me out!
It's too slow.....
Use profiler to find out the slowest point while processing request. It looks like the problem was not caused by postgresql or uwsgi things.
See ProfilingDjango wiki page.

Nginx connection reset, response from uWsgi lost

I have a django app hosted via Nginx and uWsgi. In a certain very simple request, I get different behaviour for GET and POST, which should not be the case.
The uWsgi daemon log:
[pid: 32454|app: 0|req: 5/17] 127.0.0.1 () {36 vars in 636 bytes} [Tue Oct 19 11:18:36 2010] POST /buy/76d4f520ae82e1dfd35564aed64a885b/a_2/10/ => generated 80 bytes in 3 msecs (HTTP/1.0 440) 1 headers in 76 bytes (0 async switches on async core 0)
[pid: 32455|app: 0|req: 5/18] 127.0.0.1 () {32 vars in 521 bytes} [Tue Oct 19 11:18:50 2010] GET /buy/76d4f520ae82e1dfd35564aed64a885b/a_2/10/ => generated 80 bytes in 3 msecs (HTTP/1.0 440) 1 headers in 76 bytes (0 async switches on async core 0)
The Nginx accesslog:
127.0.0.1 - - [19/Oct/2010:18:18:36 +0200] "POST /buy/76d4f520ae82e1dfd35564aed64a885b/a_2/10/ HTTP/1.0" 440 0 "-" "curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15"
127.0.0.1 - - [19/Oct/2010:18:18:50 +0200] "GET /buy/76d4f520ae82e1dfd35564aed64a885b/a_2/10/ HTTP/1.0" 440 80 "-" "curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15"
The Nginx errorlog:
2010/10/19 18:18:36 [error] 4615#0: *5 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST /buy/76d4f520ae82e1dfd35564aed64a885b/a_2/10/ HTTP/1.0", upstream: "uwsgi://unix:sock/uwsgi.sock:", host: "localhost:9201"
In essence, Nginx somewhere loses the response if I use POST, not so if I use GET.
Anybody knows something about that?
Pass --post-buffering 1 to uwsgi
This will automatically buffer all the http body > 1 byte
The problem is raised by the way nginx manages upstream disconnections
I hit the same issue, but on my case I can't disable "uwsgi_pass_request_body" as most times (but not always) my app do need the POST data.
This is the workaround I found, while this issue is not fixed in uwsgi:
http://permalink.gmane.org/gmane.comp.python.wsgi.uwsgi.general/813
import django.core.handlers.wsgi
class ForcePostHandler(django.core.handlers.wsgi.WSGIHandler):
"""Workaround for: http://lists.unbit.it/pipermail/uwsgi/2011-February/001395.html
"""
def get_response(self, request):
request.POST # force reading of POST data
return super(ForcePostHandler, self).get_response(request)
application = ForcePostHandler()
I am facing the same issues. I tried all solutions above, but they were not working. Ignoring the response body in my case is simply not an option.
Apparently it is a bug with nginx and uwsgi when dealing with POST requests whose response is smaller than 4052 bytes
What solved it for me was adding "--pep3333-input" to the parameter list of uwsgi. After that all POSTs are returned correctly.
Versions of nginx/uwsgi I'm using:
$ nginx -V
nginx: nginx version: nginx/0.9.6
$ uwsgi --version
uWSGI 0.9.7
After a lucky find in further research (http://answerpot.com/showthread.php?577619-Several%20Bugs/Page2) I found something that helped...
Supplying the uwsgi_pass_request_body off; parameter in the Nginx conf resolves this problem...