I'm running a django application inside a container using supervisord.
But sometimes i need to view the log to fix some errors and i could'nt find a way to do it.
I tried to add an stdout_logfile and stderr_logfile but always the err logfile is empty
this is my supervisor.conf
[supervisord]
loglevel=info
logfile=/tmp/supervisord.log
[program:myapp]
command = python3 -u /usr/src/app/manage.py runserver 0.0.0.0:8000
stdout_logfile=/usr/src/app/out.log
stderr_logfile=/usr/src/app/err.log
And always the same result, the out.log file will contain the lines before the exception happen and the err.log won't be created
This is the output that i get when i run docker compose
2020-05-13 17:33:44,140 INFO supervisord started with pid 1
2020-05-13 17:33:45,144 INFO spawned: 'myapp' with pid 9
2020-05-13 17:33:46,201 INFO success: myapp entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
After a big struggling i found the log is being buffered, so the solution is by adding environment = PYTHONUNBUFFERED=1 to the supervisor.conf file
my conf file after modification
[supervisord]
loglevel=info
logfile=/tmp/supervisord.log
[program:myapp]
command = python3 -u /usr/src/app/manage.py runserver 0.0.0.0:8000
environment = PYTHONUNBUFFERED=1
stdout_logfile=/usr/src/app/out.log
stderr_logfile=/usr/src/app/err.log
Related
all. Got a weird problem in running gunicorn with supervisor to serve a flask application. Before I start stating my problem, just want to say that I'm aware of this SO post. Nevertheless, my problem was that I could run gunicorn directly in the foreground to server the app. The problem only occurred when I used supervisor.
Here's the structure of my flask application:
myproject/
|---webapp/
| |---__init__.py # create_app function is here
| |--- ... # other files in webapp
|---wsgi.py
|---manage.py
|---venv # gunicorn is in venv/bin
The wsgi.py file looks like the following
from webapp import create_app
app = create_app()
if __name__ == "__main__":
app.run()
Sitting in myproject directory, I can launch the app by running
$ venv/bin/gunicorn -w 4 wsgi:app
Then I created a webapp.conf in /etc/supervisor/conf.d/ directory, as follows.
[program:webapp]
directory=/home/<my username>/projects/myproject
command=/home/<my username>/projects/myproject/venv/bin/gunicorn -w 4 wsgi:app
user=<my username>
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=<some path>
stderr_logfile=<some path>
When I reloaded my supervisor, the stdout log file said
usage: gunicorn [OPTIONS] [APP_MODULE] gunicorn: error: No application module specified.
My question is, why do I get an error here if I don't get an error when running gunicorn directly? What's the difference?
By the way, I then tried to use systemd to run the gunicorn following this post, and systemd can launch the gunicorn service normally.
Any idea what I did wrong with supervisor? Thanks very much.
I have a Django app which runs on Gunicorn, and is managed by SupervisorD which is managed by Ansible.
I want Django to read the DJANGO_SECRET_KEY variable from the environment, since I don't want to store my secret key in a config file or VCS. For that I read the key from the environment in my settings.py:
SECRET_KEY = os.environ['DJANGO_SECRET_KEY']
Looking at Supervisor docs it says:
Note that the subprocess will inherit the environment variables of the shell used to start “supervisord” except for the ones overridden here. See Subprocess Environment.
Here's my supervisor.conf:
[program:gunicorn]
command=/.../.virtualenvs/homepage/bin/gunicorn homepage.wsgi -w 1 --bind localhost:8001 --pid /tmp/gunicorn.pid
directory=/.../http/homepage
When I set the variable and run Gunicorn command from the shell, it starts up just fine:
$ DJANGO_SECRET_KEY=XXX /.../.virtualenvs/homepage/bin/gunicorn homepage.wsgi -w 1 --bind localhost:8001 --pid /tmp/gunicorn.pid
However when I set the variable in the shell and restart the Supervisor service my app fails to start with error about not found variable:
$ DJANGO_SECRET_KEY=XXX supervisorctl restart gunicorn
gunicorn: ERROR (not running)
gunicorn: ERROR (spawn error)
Looking at Supervisor error log:
File "/.../http/homepage/homepage/settings.py", line 21, in <module>
SECRET_KEY = os.environ['DJANGO_SECRET_KEY']
File "/.../.virtualenvs/homepage/lib/python2.7/UserDict.py", line 40, in __getitem__
raise KeyError(key)
KeyError: 'DJANGO_SECRET_KEY'
[2017-08-27 08:22:09 +0000] [19353] [INFO] Worker exiting (pid: 19353)
[2017-08-27 08:22:09 +0000] [19349] [INFO] Shutting down: Master
[2017-08-27 08:22:09 +0000] [19349] [INFO] Reason: Worker failed to boot.
I have also tried restarting the supervisor service, but same error occurs:
$ DJANGO_SECRET_KEY=XXX systemctl restart supervisor
...
INFO exited: gunicorn (exit status 3; not expected)
My question is how do I make Supervisor to "pass" environment variables to it's child processes?
Create executable file similar to this and try to start it manually.
i.e create file and copy script below /home/user/start_django.sh
You need to fill in DJANGODIR and make other adjustments according to your case. also, you may need to adjust permissions accordingly.
#!/bin/bash
DJANGODIR=/.../.../..
ENVBIN=/.../.virtualenvs/homepage/bin/bin
# Activate the virtual environment
cd $DJANGODIR
source $ENVBIN/activate
DJANGO_SECRET_KEY=XXX
#define other env variables if you need
# Start your Django
exec gunicorn homepage.wsgi -w 1 --bind localhost:8001 --pid /tmp/gunicorn.pid
If it starts manually then just use this file in your conf.
[program:django_project]
command = /home/user/start_django.sh
user = {your user}
stdout_logfile = /var/log/django.log
redirect_stderr = true
# you can also try to define enviroment variables in this conf
environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8,DJANGO_SECRET_KEY=XXX
references that might be helpful,
Ref 1 - environment-variables
Ref 2
Ref 3
OK figured it out myself. Turns out ansible has feature called Vault, which is used for exactly this kind of jobs - encrypting keys.
Now I added the vaulted secret key to ansible's host_vars, see:
Vault: single encrypted variable and Inventory: Splitting out host and group specific data.
I added a task to my ansible playbook to copy the key file from ansible vault to the server:
- name: copy django secret key to server
copy: content="{{ django_secret_key }}" dest=/.../http/homepage/deploy/django_secret_key.txt mode=0600
And made Django read the secret from that file:
with open(os.path.join(BASE_DIR, 'deploy', 'django_secret_key.txt')) as secret_key_file:
SECRET_KEY = secret_key_file.read().strip()
If anyone has a simpler/better solution, please post it and I will accept it as the answer.
I have the following config file for Upstart, and it starts the Flask server fine, but whenever there is an exception in the app the log file doesn't have the exception information.
start on [2345]
stop on [06]
respawn
script
cd /var/www/binary-fission/server
export BF_CONFIG=config/staging.py
exec uwsgi --http 0.0.0.0:5000 --wsgi-file server.py --callable app --master --threads 2 --processes 4 --logto /var/log/binary-fission/server.log
end script
However, if I run the same uwsgi command manually without Upstart, the exception is logged.
How do I make upstart+uwgi log the exception from a Flask application?
It turned out that turning on the "PROPAGATE_EXCEPTIONS" option in the flask configuration file (config/staging.py) fixed the issue. This is because in that configuration file, "DEBUG" is turned off which turns "PROPAGATE_EXCEPTIONS" off at the same time.
When I ran uwsgi command manually, I didn't specify the configuration file and my Flask app fell back to the default configuration with "DEBUG" on.
I have a django app in which it has a celery functionality, so i can able to run the celery sucessfully like below
celery -A tasks worker --loglevel=info
but as a known fact that we need to run it as a daemon and so i have written the below celery.conf file inside /etc/supervisor/conf.d/ folder
; ==================================
; celery worker supervisor example
; ==================================
[program:celery]
; Set full path to celery program if using virtualenv
command=/root/Envs/proj/bin/celery -A app.tasks worker --loglevel=info
user=root
environment=C_FORCE_ROOT="yes"
environment=HOME="/root",USER="root"
directory=/root/apps/proj/structure
numprocs=1
stdout_logfile=/var/log/celery/worker.log
stderr_logfile=/var/log/celery/worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
but when i tried to update the supervisor like supervisorctl reread and supervisorctl update i was getting the message from supervisorctl status
celery FATAL Exited too quickly (process log may have details)
So i went to worker.log file and seen the error message as below
Running a worker with superuser privileges when the
worker accepts messages serialized with pickle is a very bad idea!
If you really want to continue then you have to set the C_FORCE_ROOT
environment variable (but please think about this before you do).
User information: uid=0 euid=0 gid=0 egid=0
So why it was complaining about C_FORCE_ROOT even though we had set it as environment variable inside supervisor conf file ? what am i doing wrong in the above conf file ?
I had the same problem,so I added
environment=C_FORCE_ROOT="yes"
in my program config,but It didn't work
so I used
environment=C_FORCE_ROOT="true"
it's working
You'll need to run celery with a non superuser account, Please remove following lines from your config:
user=root
environment=C_FORCE_ROOT="yes"
environment=HOME="/root",USER="root"
And the add these lines to your config, I assume that you use django as a non superuser and developers as the user group:
user=django
group=developers
Note that subprocesses will inherit the environment variables of the
shell used to start supervisord except for the ones overridden here
and within the program’s environment option. See supervisord documents.
So Please note that when you change environment variables via supervisor config files, Changes won't apply by running supervisorctl reread and supervisorctl reload . You should run supervisor from the very start by following command:
supervisord -c /path/to/config/file.conf
From this other thread on stackoverflow. I managed to add the following settings and it worked for me.
app.conf.update(
CELERY_ACCEPT_CONTENT = ['json'],
CELERY_TASK_SERIALIZER = 'json',
CELERY_RESULT_SERIALIZER = 'json',
)
For my current flask deployment, I had to set up a uwsgi server.
This is how I have created the uwsgi daemon:
sudo vim /etc/init/uwsgi.conf
# file: /etc/init/uwsgi.conf
description "uWSGI server"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
exec /myproject/myproject-env/bin/uwsgi --uid www-data --gid www-data --home /myproject/myproject-env/site/F11/Engineering/ --socket /tmp/uwsgi.sock --chmod-socket --module F11 --callable app --pythonpath /myproject/myproject-env/site/F11/Engineering/ -H /myproject/myproject-env
However after running this successfully: sudo start uwsgi
uwsgi start/running, process 1286
And trying to access the application via browser:
I get a 502 Bad Gateway
and an error entry in nginx error.log:
2013/06/13 23:47:28 [error] 743#0: *296 upstream prematurely closed
connection while reading response header from upstream, client:
xx.161.xx.228, server: myproject.com, request: "GET /show_records/2013/6 HTTP/1.1", upstream:
"uwsgi://unix:///tmp/uwsgi.sock:", host: "myproject.com"
But the sock file has the permission it needs:
srw-rw-rw- 1 www-data www-data 0 Jun 13 23:46 /tmp/uwsgi.sock
If I run the exec command from above in the command line as a process, it works perfectly fine. Why is the daemon not working correctly please?
btw Nginx is running as
vim /etc/nginx/nginx.conf
user www-data;
and vim /etc/nginx/sites-available/default
location / {
uwsgi_pass unix:///tmp/uwsgi.sock;
include uwsgi_params;
}
and it is started as sudo service nginx start
I am running this on Ubuntu 12.04 LTS.
I hope I have provided all the necessary data, hope someone can guide me into the right direction. Thanks.
Finally I have solved this problem after working nearly 2 days on it. I hope this solution will help other flask/uwsgi users that are experiencing a similar problem.
I had two major issues that caused this.
1) The best way to find the problems with a daemon is obviously a log file and a cleaner structure.
sudo vim /etc/init/uwsgi.conf
Change the daemon script to the following:
# file: /etc/init/uwsgi.conf
description "uWSGI server"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
exec /home/ubuntu/uwsgi-1.9.12/uwsgi -c /myproject/uwsgi.ini
vim /myproject/uwsgi.ini
[uwsgi]
socket = /tmp/uwsgi.sock
master = true
enable-threads = true
processes = 5
chdir= /myproject/F11/Engineering
module=F11:app
virtualenv = /myproject/myproject-env/
uid = www-data
gid = www-data
logto = /myproject/error.log
This is much cleaner way of setting up the daemon. Also notice the last line how to setup the log file. Initially I had set the log file to /var/log/uwsgi/error.log. After a lot of sweat and tears I realized the daemon is running as www-data and hence can not access the /var/log/uwsgi/error.log since the error.log was owned by root:root. This made the uwsgi fail silently.
I found it much more efficient to just point the log file to my own /myproject, where the daemon has guaranteed access as www-data. And also don't forget to make the whole project accessible to www-data or the daemon will fail with an Internal Server error message. -->
sudo chown www-data:www-data -R /myproject/
Restart uwsgi daemon:
sudo service uwsgi restart
2) Now you have three log files to lookout for:
tail -f /var/log/upstart/uwsgi.log --> Shows problems with your daemon upon start
tail -f /var/log/nginx/error.log --> Shows permission problems when wsgi access is refused, often because /tmp/uwsgi.sock file is owned by root instead of www-data. In that case simply delete the sock file sudo rm /tmp/uwsgi.sock
tail -f /myproject/error.log --> Shows errors thrown by uwsgi in your application
This combination of log files helped me to figure out that I also had a bad import with Flask-Babel in my Flask application. Bad in that sense, that the way I utilized the library was falling back to the system's locale to determine the datetime format.
File "/myproject/F11/Engineering/f11_app/templates/show_records.html", line 25, in block "body"
<td>{{ record.record_date|format_date }}</td>
File "./f11_app/filters.py", line 7, in format_date
day = babel_dates.format_date(value, "EE")
File "/myproject/myproject-env/local/lib/python2.7/site-packages/babel/dates.py", line 459, in format_date
return pattern.apply(date, locale)
File "/myproject/myproject-env/local/lib/python2.7/site-packages/babel/dates.py", line 702, in apply
return self % DateTimeFormat(datetime, locale)
File "/myproject/myproject-env/local/lib/python2.7/site-packages/babel/dates.py", line 699, in __mod__
return self.format % other
File "/myproject/myproject-env/local/lib/python2.7/site-packages/babel/dates.py", line 734, in __getitem__
return self.format_weekday(char, num)
File "/myproject/myproject-env/local/lib/python2.7/site-packages/babel/dates.py", line 821, in format_weekday
return get_day_names(width, context, self.locale)[weekday]
File "/myproject/myproject-env/local/lib/python2.7/site-packages/babel/dates.py", line 69, in get_day_names
return Locale.parse(locale).days[context][width]
AttributeError: 'NoneType' object has no attribute 'days'
This is the way I was using the Flask filter:
import babel.dates as babel_dates
#app.template_filter('format_date')
def format_date(value):
day = babel_dates.format_date(value, "EE")
return '{0} {1}'.format(day.upper(), affix(value.day))
The strangest part is that this code is working perfectly fine within the dev environment (!). It works even fine when running the uwsgi as a root process from the command line. But it fails when ran by the www-data daemon. This must have something to do with how the locale is set, which Flask-Babel is trying to fall back to.
When I changed the import like this, it all worked finally with the daemon:
from flask.ext.babel import format_date
#app.template_filter('format_date1')
def format_date1(value):
day = format_date(value, "EE")
return '{0} {1}'.format(day.upper(), affix(value.day))
Hence be careful when using Eclipse/Aptana Studio that is trying to pick the right namespace for your classes in code. It can really turn ugly.
It is now working perfectly fine as a uwsgi daemon on an Amazon Ec2 (Ubuntu 12.04) since 2 days. I hope this experience helps fellow python developers.
I gave up, there was no uwsgi.log generated, and nginx just kept complaining about :
2014/03/06 01:06:28 [error] 23175#0: *22 upstream prematurely closed connection while reading response header from upstream, client: client.IP, server: my.server.IP, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/var/web/the_gelatospot/uwsgi.sock:", host: "host.ip"
for every single request. This only happened if running uwsgi as a service, as a process it started fine under any user. So this would work from command line (page responds):
$exec /var/web/the_gelatospot/mez_server.sh
This didn't (/etc/init/site_service.conf):
description "mez sites virtualenv and uwsgi_django" start on runlevel
[2345] stop on runlevel [06] respawn respawn limit 10 5 exec
/var/web/the_gelatospot/mez_server.sh
The process would start but on each request nginx would complain about the closed connection. Strangely I have this same config working just fine for 2 other apps using the same nginx version and same uwsgi version as well as both apps are mezzanine CMS apps. I tried everything I could think of and what was suggested. In the end I switched to gunicorn which works fine:
#!/bin/bash
NAME="the_gelatospot" # Name of the application
DJANGODIR=/var/web/the_gelatospot # Django project directory
SOCKFILE=/var/web/the_gelatospot/gunicorn.sock # we will communicte using this unix socket
USER=ec2-user
GROUP=ec2-user # the user to run as, the group to run as
NUM_WORKERS=3 # how many worker processes should Gunicorn spawn
DJANGO_SETTINGS_MODULE=settings
#DJANGO_SETTINGS_MODULE=the_gelatospot.settings # which settings file should Django use
#DJANGO_WSGI_MODULE=the_gelatospot.wsgi # WSGI module name
DJANGO_WSGI_MODULE=wsgi
echo "Starting $NAME as `the_gelatospot`"
# Activate the virtual environment
cd $DJANGODIR
source ../mez_env/bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH
# Create the run directory if it doesn't exist
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR
cd ..
# Start your Django GUnicorn
# Programs meant to be run under supervisor should not daemonize themselves (do not use --daemon)
exec gunicorn -k eventlet ${DJANGO_WSGI_MODULE}:application \
--name $NAME \
--workers $NUM_WORKERS \
--log-level=debug \
--bind=unix:$SOCKFILE
And here is the one that wouldn't work as a service (nginx complaining of prematurely closed connection and no app log data coming through).
#!/bin/bash
DJANGODIR=/var/web/the_gelatospot/ # Django project directory
cd $DJANGODIR
source ../mez_env/bin/activate
uwsgi --ini uwsgi.ini
And the uwsgi.ini:
[uwsgi]
uid = 222
gid = 500
socket = /var/web/the_gelatospot/uwsgi.sock
virtualenv = /var/web/mez_env
chdir = /var/web/the_gelatospot/
wsgi-file = /var/web/the_gelatospot/wsgi.py
pythonpath = ..
env = DJANGO_SETTINGS_MODULE=the_gelatospot.settings
die-on-term = true
master = true
chmod-socket = 666
;experiment using uwsgitop
worker = 1
;gevent = 100
processes = 1
daemonize = /var/log/nginx/uwsgi.log
logto = /var/log/nginx/uwsgi.logi
log-maxsize = 10000000
enable-threads = true
I went from gunicorn to uWSGI last year and I had no issues with it till now, it also seemed a bit faster than gunicorn. Right now I'm thinking of sticking to gunicorn. Its getting better, puts up much nicer numbers with eventlet installed, and it is easier to configure.
Hope this workaround helps. I would still like to know the issue with uWSGI and nginx but I'm stumped.
Update:
So using gunicorn allowed me to run the server as a service and while toying in mezzanine I ran into this error: FileSystemEncodingChanged
To fix this I found the solution here:
https://groups.google.com/forum/#!msg/mezzanine-users/bdln_Y99zQw/9HrhNSKFyZsJ
And I had to modify it a bit since I don't use supervisord, I only use upstart and a shell script. I added this to right before I execute gunicorn in my mez_server.sh file:
export LANG=en_US.UTF-8, LC_ALL=en_US.UTF-8, LC_LANG=en_US.UTF-8
exec gunicorn -k eventlet ${DJANGO_WSGI_MODULE}:application \
--name $NAME \
--workers $NUM_WORKERS \
--log-level=debug \
--bind=unix:$SOCKFILE
I also tried this fix using uWSGI like so ( a lot shorter since using uwsgi.ini ):
export LANG=en_US.UTF-8, LC_ALL=en_US.UTF-8, LC_LANG=en_US.UTF-8
exec uwsgi --ini uwsgi.ini
I am still sticking to gunicorn since it still worked with the problem and lead me in the right direction to resolve it. I was very disappointed that uWSGI provided no output in the log file even with these params, I only seen the server start process and that's it:
daemonize = /var/log/nginx/uwsgi.log
logto = /var/log/nginx/uwsgi.logi
While nginx kept throwing the disconnect error, uWSGI sat there like nothing is happening.
As a single line running with daemon true command is
gunicorn app.wsgi:application -b 127.0.0.1:8000 --daemon
bind your app with 127.0.0.1:8000 and --deamon force it to run as daemon
but define a gunicorn_config.cfg file and run with -c flag is good practice
for more:
https://gunicorn-docs.readthedocs.org/en/develop/configure.html