I hit a strange issue with my site(nginx1.7+uwsgi2.0+django1.6).
Today I see there are some strange log entries in my uwsgi logs.
Snippet here:
Mon Aug 31 10:43:17 2015 - WSGI app 1 (mountpoint='zc.qq.com|') ready in 0 seconds on interpreter 0xf627c0 pid: 18360
zc.qq.com {address space usage: 421933056 bytes/402MB} {rss usage: 102522880 bytes/97MB} [pid: 18360|app: 1|req: 1/7] 61.132.52.107 () {42 vars in 684 bytes} [Mon Aug 31 10:43:17 2015] GET /cgi-bin/common/attr?id=260714&r=0.6131902049963026 => generated 0 bytes in 6113 msecs (HTTP/1.1 301) 4 headers in 210 bytes (2 switches on core 0)
zc.qq.com {address space usage: 421933056 bytes/402MB} {rss usage: 102522880 bytes/97MB} [pid: 18360|app: 1|req: 2/8] 61.132.52.105 () {44 vars in 986 bytes} [Mon Aug 31 10:43:29 2015] GET /cgi-bin/common/attr?id=260714&r=0.1676001222494321 => generated 0 bytes in 3 msecs (HTTP/1.1 301) 4 headers in 210 bytes (2 switches on core 0)
Actually, zc.qq.com is nothing to do with my site.
So, how this guy comes into my server?
It sits here as an wsgi app, so, sometimes keep restarting together with my own django app, thus, sometimes it takes django app more than 5 secs to respond to http request.
I see the pid for strange app is 18360. So, here:
[root#localhost uwsgi]# ps -ef|grep uwsgi
root 18352 1 0 10:40 ? 00:00:00 uwsgi -x /home/uwsgi/uwsgi2.xml
root 18353 18352 0 10:40 ? 00:00:00 uwsgi -x /home/uwsgi/uwsgi2.xml
root 18354 18352 0 10:40 ? 00:00:00 uwsgi -x /home/uwsgi/uwsgi2.xml
root 18355 18352 0 10:40 ? 00:00:00 uwsgi -x /home/uwsgi/uwsgi2.xml
root 18356 18352 0 10:40 ? 00:00:00 uwsgi -x /home/uwsgi/uwsgi2.xml
root 18357 18352 0 10:40 ? 00:00:12 uwsgi -x /home/uwsgi/uwsgi2.xml
root 18358 18352 0 10:40 ? 00:00:00 uwsgi -x /home/uwsgi/uwsgi2.xml
root 18359 18352 0 10:40 ? 00:00:13 uwsgi -x /home/uwsgi/uwsgi2.xml
root 18360 18352 1 10:40 ? 00:00:18 uwsgi -x /home/uwsgi/uwsgi2.xml
root 18871 18818 0 11:07 pts/2 00:00:00 grep uwsgi
It comes from uwsgi...But here is uwsgi config file:
<uwsgi>
<socket>/var/run/uwsgi.socket</socket>
<listen>100</listen>
<master>true</master>
<vhost>true</vhost>
<no-site>true</no-site>
<pidfile>/usr/local/nginx/uwsgi.pid</pidfile>
<processes>8</processes>
<profiler>true</profiler>
<memory-report>true</memory-report>
<enable-threads>true</enable-threads>
<logdate>true</logdate>
<limit-as>6048</limit-as>
<daemonize>/home/django.log</daemonize>
</uwsgi>
And, here is snippet in nginx.conf(domainname is just sample here)
server {
listen 80;
server_name www.mysite.com;
location / {
uwsgi_pass unix:///var/run/uwsgi.socket;
include uwsgi_params;
uwsgi_param UWSGI_CHDIR /home/mysite;
uwsgi_param UWSGI_SCRIPT wsgi;
access_log off;
}
location /static/ {
root /home/mysite/;
access_log off;
log_not_found off;
autoindex on;
}
}
So, exactly nothings in any config files related to zc.qq.com or other strage domains(I also see proxyjudge.info).
Anyone hit this before?
Thanks.
Wesley
It is because you have enabled virtualhosting and dynamic apps but you do not make any check from the nginx side. The first request for a non-configured (in uWSGI) domain, will result in a new app to load
Related
I tried to deploy my Django project on server with uwsgi.
I can initial uwsgi with uwsgi --ini conf/uwsgi.ini
But when I tried to visit the website, I got this error
--- no python application found, check your startup logs for errors ---
[pid: 11228|app: -1|req: -1/1] 127.0.0.1 () {46 vars in 946 bytes} [Mon Nov 28 17:06:37 2022] GET / => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 83 bytes (0 switches on core 0)
I put my repo at https://github.com/dolphinfafa/MyLife
Anyone knows how to resolve this?
My deploy enviroment is "Django 1.11.13 + Python3.6.5(with virtualenv) + uWSGI 2.0 + Nginx 1.12". Here is my project:
(ncms) [ncms#localhost ncms]$ pwd
/home/ncms/ncms
(ncms) [ncms#localhost ncms]$ ll
总用量 36
drwxrwxr-x 12 ncms ncms 157 5月 17 13:48 apps
-rwxrwxr-x 1 ncms ncms 16384 5月 14 18:49 celerybeat-schedule
drwxrwxr-x 2 ncms ncms 66 5月 17 10:40 db_tools
drwxrwxr-x 4 ncms ncms 41 5月 17 10:40 extra_apps
drwxrwxr-x 5 ncms ncms 233 5月 17 10:40 libs
drwxrwxr-x 2 ncms ncms 152 5月 17 10:40 logfiles
-rwxrwxr-x 1 ncms ncms 855 5月 6 22:23 manage.py
drwxrwxr-x 3 ncms ncms 201 5月 21 14:19 ncms
-rwxrwxr-x 1 ncms ncms 351 5月 15 18:25 ncms.conf
-rwxrwxr-x 1 ncms ncms 2766 5月 17 13:43 notes.md
-rwxrwxr-x 1 ncms ncms 518 5月 14 15:52 requirements.txt
drwxrwxr-x 3 ncms ncms 23 5月 18 16:09 static
drwxrwxr-x 10 ncms ncms 120 5月 18 16:05 static_files
drwxrwxr-x 11 ncms ncms 4096 5月 17 15:38 templates
my virtualenv path and name:
(ncms) [ncms#localhost ncms]$ pwd
/home/ncms/.virtualenvs/ncms
(ncms) [ncms#localhost ncms]$ ll
总用量 8
drwxrwxr-x 3 ncms ncms 4096 5月 21 13:46 bin
drwxrwxr-x 2 ncms ncms 24 5月 15 11:49 include
drwxrwxr-x 3 ncms ncms 23 5月 15 11:49 lib
-rw-rw-r-- 1 ncms ncms 61 5月 15 11:50 pip-selfcheck.json
Three important files you must know:
1./etc/uwsgi/ncms.ini
[uwsgi]
# Django diretory that contains manage.py
chdir = /home/ncms/ncms
module = ncms.wsgi:application
env = DJANGO_SETTINGS_MODULE=ncms.settings
# enable master process manager
master = true
# bind to UNIX socket
socket = /run/uwsgi/ncms.sock
# number of worker processes
processes = 4
# user identifier of uWSGI processes
uid = ncms
# group identifier of uWSGI processes
gid = ncms
#respawn processes after serving 5000 requests
max-requests = 5000
# clear environment on exit
vacuum = true
# the virtualenv you are using (full path)
home = /home/ncms/.virtualenvs/ncms
# set mode and own of created UNIX socket
chown-socket = ncms:nginx
chmod-socket = 660
# place timestamps into log
log-date = true
logto = /var/log/uwsgi.log
no-site = true
2./etc/systemd/system/uwsgi.service
[Unit]
Description=ncms uWSGI service
[Service]
ExecStartPre=/usr/bin/bash -c 'mkdir -p /run/uwsgi; chown ncms:nginx /run/uwsgi'
ExecStart=/usr/bin/uwsgi --emperor /etc/uwsgi
Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all
[Install]
WantedBy=graphical.target
3./etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/mime.types;
default_type application/octet-stream;
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
location = favicon.ico { access_log off; log_not_found off; }
location /static {
root /home/ncms/ncms;
}
location / {
include uwsgi_params;
uwsgi_pass unix:/run/uwsgi/ncms.sock;
}
}
}
After they were configured well:
sudo nginx –t
sudo usermod -a -G ncms nginx
chmod 710 /home/ncms
sudo systemctl daemon-reload
sudo systemctl restart nginx
sudo systemctl restart uwsgi
Then I always got this error at the Operational MODE: preforking step when I looked /var/log/uwsgi.log:
Mon May 21 16:38:35 2018 - SIGINT/SIGQUIT received...killing workers...
Mon May 21 16:38:35 2018 - received message 0 from emperor
Mon May 21 16:38:36 2018 - worker 1 buried after 1 seconds
Mon May 21 16:38:36 2018 - worker 2 buried after 1 seconds
Mon May 21 16:38:36 2018 - worker 3 buried after 1 seconds
Mon May 21 16:38:36 2018 - worker 4 buried after 1 seconds
Mon May 21 16:38:36 2018 - goodbye to uWSGI.
Mon May 21 16:38:36 2018 - VACUUM: unix socket /run/uwsgi/ncms.sock removed.
Mon May 21 16:38:38 2018 - *** Starting uWSGI 2.0.17 (64bit) on [Mon May 21 16:38:38 2018] ***
Mon May 21 16:38:38 2018 - compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-16) on 26 April 2018 05:37:29
Mon May 21 16:38:38 2018 - os: Linux-3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018
Mon May 21 16:38:38 2018 - nodename: localhost.localdomain
Mon May 21 16:38:38 2018 - machine: x86_64
Mon May 21 16:38:38 2018 - clock source: unix
Mon May 21 16:38:38 2018 - pcre jit disabled
Mon May 21 16:38:38 2018 - detected number of CPU cores: 4
Mon May 21 16:38:38 2018 - current working directory: /etc/uwsgi
Mon May 21 16:38:38 2018 - detected binary path: /usr/bin/uwsgi
Mon May 21 16:38:38 2018 - chdir() to /home/ncms/ncms
Mon May 21 16:38:38 2018 - your processes number limit is 7164
Mon May 21 16:38:38 2018 - your memory page size is 4096 bytes
Mon May 21 16:38:38 2018 - detected max file descriptor number: 1024
Mon May 21 16:38:38 2018 - lock engine: pthread robust mutexes
Mon May 21 16:38:38 2018 - thunder lock: disabled (you can enable it with --thunder-lock)
Mon May 21 16:38:38 2018 - uwsgi socket 0 bound to UNIX address /run/uwsgi/ncms.sock fd 3
Mon May 21 16:38:38 2018 - setgid() to 2014
Mon May 21 16:38:38 2018 - setuid() to 2030
Mon May 21 16:38:38 2018 - Python version: 2.7.5 (default, Aug 4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
Mon May 21 16:38:38 2018 - Set PythonHome to /home/ncms/.virtualenvs/ncms
Mon May 21 16:38:38 2018 - *** Python threads support is disabled. You can enable it with --enable-threads ***
Mon May 21 16:38:38 2018 - Python main interpreter initialized at 0x1cea860
Mon May 21 16:38:38 2018 - your server socket listen backlog is limited to 100 connections
Mon May 21 16:38:38 2018 - your mercy for graceful operations on workers is 60 seconds
Mon May 21 16:38:38 2018 - mapped 364600 bytes (356 KB) for 4 cores
Mon May 21 16:38:38 2018 - *** Operational MODE: preforking ***
Traceback (most recent call last):
File "./ncms/__init__.py", line 1, in <module>
from __future__ import absolute_import, unicode_literals
ImportError: No module named __future__
Mon May 21 16:38:38 2018 - unable to load app 0 (mountpoint='') (callable not found or import error)
Mon May 21 16:38:38 2018 - *** no app loaded. going in full dynamic mode ***
Mon May 21 16:38:38 2018 - *** uWSGI is running in multiple interpreter mode ***
Mon May 21 16:38:38 2018 - spawned uWSGI master process (pid: 3456)
Mon May 21 16:38:38 2018 - spawned uWSGI worker 1 (pid: 3458, cores: 1)
Mon May 21 16:38:38 2018 - spawned uWSGI worker 2 (pid: 3459, cores: 1)
Mon May 21 16:38:38 2018 - spawned uWSGI worker 3 (pid: 3460, cores: 1)
Mon May 21 16:38:38 2018 - spawned uWSGI worker 4 (pid: 3462, cores: 1)
When I removed the line "from future import absolute_import, unicode_literals" in my code, it rasied the same error, like:
Mon May 21 16:43:30 2018 - *** Operational MODE: preforking ***
Traceback (most recent call last):
File "./ncms/__init__.py", line 4, in <module>
from .celery import app as celery_app
File "./ncms/celery.py", line 6, in <module>
import os
ImportError: No module named os
Mon May 21 16:43:30 2018 - unable to load app 0 (mountpoint='') (callable not found or import error)
Mon May 21 16:43:30 2018 - *** no app loaded. going in full dynamic mode ***
looks like can't import anything...
When I access my website, the /var/log/uwsgi.log displayed:
Mon May 21 16:55:28 2018 - --- no python application found, check your startup logs for errors ---
[pid: 3812|app: -1|req: -1/3] 192.168.10.1 () {46 vars in 854 bytes} [Mon May 21 16:55:28 2018] GET / => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 83 bytes (0 switches on core 0)
Mon May 21 16:55:28 2018 - --- no python application found, check your startup logs for errors ---
[pid: 3812|app: -1|req: -1/4] 192.168.10.1 () {48 vars in 855 bytes} [Mon May 21 16:55:28 2018] GET /favicon.ico => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 83 bytes (1 switches on core 0)
I've google a lot and tried many time changing here and there, just don't get the right answer.
Anyone could help me? Please!
You can first using uwsgi command line to start app as root
uwsgi --socket 127.0.0.1:8080 --chdir /home/ncms/ncms/ --wsgi-file ncms/wsgi.py
then if ok, debug the config file mode
I'm running a Django web application using Nginx and uWSGI. Now I meet a problem that the finish_process view in django
I have added logging at the begin and the end of Django finish_process view.
I make a request at 17:20:18, and the view finished at 17:20:48. But uWSGI does not return response at this time, and after 577 seconds, it throws IOError when it try to write response to client, because nginx close the connection (uwsgi_read_timeout is 300 seconds).
My question is why uWSGI holds the response so long after Django handled the view? I'm a bit at a loss.
Django log:
[INFO]246 views.py/finish_process 2016-03-06 17:20:18: [VIEW][START] finish_process: id=4
[INFO]282 views.py/finish_process 2016-03-06 17:20:48: [VIEW][END] finish_process: id=4
uWSGI log:
Sun Mar 6 17:29:55 2016 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 296] during POST /api/finish_process/ (10.11.16.251)
IOError: write error
[pid: 3275|app: 0|req: 48689/48688] 10.11.16.251 () {34 vars in 553 bytes} [Sun Mar 6 17:20:18 2016] POST /api/finish_process/ => generated 0 bytes in 577024 msecs (HTTP/1.1 200) 3 headers in 0 bytes (0 switches on core 4)
Nginx error.log:
2016/03/06 17:25:18 [error] 3052#0: *44561 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.11.16.251, server: skyline, request: "POST /api/finish_process/ HTTP/1.1", upstream: "uwsgi://unix:/var/run/skyline.sock:", host: "10.11.16.253"
uwsgi.ini:
[uwsgi]
socket = /var/run/skyline.sock
chdir = /opt/skyline
processes = 1
threads = 10
master = true
env = DJANGO_SETTINGS_MODULE=skyline.prod_settings
module = skyline.wsgi:application
chmod-socket = 666
vacuum = true
die-on-term = true
Nginx conf:
server {
listen 80;
server_name skyline;
charset utf-8;
client_max_body_size 50M;
uwsgi_read_timeout 300;
location / {
include uwsgi_params;
uwsgi_pass unix:/var/run/skyline.sock;
}
}
Updated:
Solved. I made a mistake.
I use nginx and uwsgi deploy my django webapp, the start few minutes all things well but after some time the ngnix will report 502 error and I check the uwsgi stderr log. there are some error marked => generated & SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request such as blow error message:
>[pid: 27427|app: 0|req: 1/1] 10.209.78.30 () {38 vars in 1030 bytes} [Mon Sep 29 15:46:00 2014] GET blabla...(just request url address) => generated 16018 bytes in 1428 msecs (HTTP/1.1 200) 6 headers in 157 bytes (1 switches on core 0)
Mon Sep 29 15:46:03 2014 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request blabla...(just request url address) (ip 10.209.78.30) !!!
Mon Sep 29 15:46:03 2014 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 287] during GET blabla...(just request url address) (10.209.78.30)
IOError: write error
[pid: 27425|app: 0|req: 1/2] 10.209.78.30 () {38 vars in 2050 bytes} [Mon Sep 29 15:46:00 2014] GET blabla...(just request url address) => generated 0 bytes in 3578 msecs (HTTP/1.1 200) 6 headers in 0 bytes (0 switches on core 0)
Mon Sep 29 15:46:04 2014 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request blabla...(just request url address) (ip 10.217.95.130) !!!
Mon Sep 29 15:46:04 2014 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 287] during GET blabla...(just request url address) (10.217.95.130)
IOError: write error
[pid: 27426|app: 0|req: 1/3] 10.217.95.130 () {38 vars in 1871 bytes} [Mon Sep 29 15:46:00 2014] GET blabla...(just request url address) => generated 0 bytes in 4415 msecs (HTTP/1.1 200) 6 headers in 0 bytes (0 switches on core 0)
Mon Sep 29 15:46:05 2014 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request blabla...(just request url address) (ip 10.217.95.130) !!!
Mon Sep 29 15:46:05 2014 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 287] during GET blabla...(just request url address) (10.217.95.130)
IOError: write error
I replace the request url by blabla...
My nginx.conf is bleow:
location / {
add_header "Access-Control-Allow-Origin" "*";
add_header "Access-Control-Allow-Methods" "GET, OPTIONS";
add_header "Access-Control-Allow-Headers" "origin, authorization, accept";
uwsgi_pass unix:///tmp/djangoapp.sock;
include uwsgi_params;
}
uWSGI launch command:
uwsgi --socket /tmp/djangoapp.sock -p 10 --wsgi-file /opt/graphite/conf/graphite_wsgi.py --chmod-socket=666
I have deployed a service.
It is served by nginx, uwsgi, django, pgpool, postgresql stack.
The page that has no db access is no problem.
However, the page that has some data from postgres is tpo slow.
Actually, db query time is quite fast. under 10 ms.
But result to client is over the 120000 ms
uwsgi log
[pid: 2056|app: 0|req: 4/10] 211.207.245.120 () {44 vars in 1116 bytes} [Thu Jul 19 00:53:31 2012] GET /account/admin/cb_main/invitationuser/ => generated 38606 bytes in 122126 msecs (HTTP/1.1 200) 8 headers in 373 bytes (1 switches on core 0)
My settings are below
Please check my settings and solve my problems
nginx settings
upstream cuying {
ip_hash;
server 127.0.0.1:9001;
}
server {
listen 8080;
root /home/cuying_mgr/;
client_max_body_size 20M;
server_name cuying.com;
location / {
uwsgi_pass cuying;
include uwsgi_params;
uwsgi_read_timeout 120;
uwsgi_send_timeout 120;
}
}
uwsgi settings
chdir=/home/cuying_mgr/develop/virenv/cuying/cuying
processes=2
workers=8
enable-threads=true
socket=127.0.0.1:9001
module=cuyingProject.wsgi:application
master=True
pidfile=/tmp/cuying-master.pid
vacuum=True
close-on-exec=True
max-requests=3000
post-buffering=8192
socket-timeout=120
limit-post=20480000
virtualenv=/home/cuying_mgr/develop/virenv/cuying
daemonize=/var/log/uwsgi/cuying.log
Please help me out!
It's too slow.....
Use profiler to find out the slowest point while processing request. It looks like the problem was not caused by postgresql or uwsgi things.
See ProfilingDjango wiki page.