I'm running a Django app with Gunicorn in a Kubernetes environment. Since a few weeks the memory usage of the pods keeps growing. After looking into the process list I noticed that there are many gunicorn processes which seem dead but are still using memory.
The command I'm starting gunicorn is:
gunicorn app.wsgi -w 3 -b 0.0.0.0:8000 --env DJANGO_SETTINGS_MODULE=app.settings.prod --reload
I don't know what is causing the growth of the gunicorn processes and how to avoid this behaviour.
Here is a screenshot of the process list:
Related
I use a Daphne server for my ASGI Django application. When I run htop on my Ubuntu server it shows all the load on only one processor and the app gets slow, so I'd like to know what is the easiest way to speed up my server. App is inside a Docker container.
I run Daphne with command: daphne WBT.asgi:application --bind "0.0.0.0" --port "8000".
I use Nginx as a proxy.
I tried using uvicorn server with multiple workers but then I get problems with layout. Ie. let's say that I have 4 workers when I visit my web app, it loads it on only one worker correctly, so in 25% cases I get no layout.
I'm using django-channels therefore I need to use daphne but for the static files and other things I want to use gunicorn. I can start daphne alongside gunicorn but I can not start both of them at the same time.
My question is should I start both of them at the same time or is there any better option?
If should I how can I do that?
Here is my runing server command:
gunicorn app.wsgi:application --bind 0.0.0.0:8000 --reload && daphne -b 0.0.0.0 -p 8089 app.asgi:application
PS:
I splited location of / and /ws/ for gunicorn and daphne in nginx.conf.
Your problem is you're calling both processes in the same context/line and one never gets called because the first one never "ends".
this process: gunicorn app.wsgi:application --bind 0.0.0.0:8000 --reload
won't terminate at any point so the && command never gets ran unless you kill that command manually, and at that point I'm not sure that wouldn't kill the whole process chain entirely.
if you want to run both you can background both processes with & e.g.
(can't test but this should work)
gunicorn app.wsgi:application --bind 0.0.0.0:8000 --reload & daphne -b 0.0.0.0 -p 8089 app.asgi:application &
The teach a man to fish info on this type of issue is here
I'm fairly certain you will lose the normal console logging you would normally have by running these in the background, as such i'd suggest looking into nohup instead of & or sending the logs somewhere with a logging utility so you aren't flying blind.
As for other options if you plan to scale up to a large number of users, probably 100+ I would just run two servers, one for wsgi django http requests and one for asgi daphne ws requests. Have nginx proxy between the two for whatever you need and you're done. That's also what channels recommends for larger applications.
It is good practice to use a common path prefix like /ws/ to distinguish WebSocket connections from ordinary HTTP connections because it will make deploying Channels to a production environment in certain configurations easier.
In particular for large sites it will be possible to configure a production-grade HTTP server like nginx to route requests based on path to either (1) a production-grade WSGI server like Gunicorn+Django for ordinary HTTP requests or (2) a production-grade ASGI server like Daphne+Channels for WebSocket requests.
Note that for smaller sites you can use a simpler deployment strategy where Daphne serves all requests - HTTP and WebSocket - rather than having a separate WSGI server. In this deployment configuration no common path prefix like /ws/ is necessary.
it is not needed to run both.Daphne is a HTTP, HTTP2 and WebSocket protocol server.
take a look at README at this link:
https://github.com/django/daphne
I have flask app and starting that with gunicorn with 5 threads. I have two options so far to stop that running. Either grep for gunicorn and kill all 5 pids at once will kill command or pkill command.
But both are not what i am looking, especially with pkill, there are other applications running with same user id.
Anyone has a script I can use? Or an idea how I can implement?
Gunicorn can write his pid to a file.
gunicorn ... -p /path/to/your/file/gunicorn.pid ...
Then you can run something like that to kill the application:
kill `cat /path/to/your/file/gunicorn.pid`
We're using Django + Gunicorn + Nginx in our server. The problem is that after a while we see lot's of gunicorn worker processes that have became orphan, and a lot other ones that have became zombie. Also we can see that some of Gunicorn worker processes spawn some other Gunicorn workers. Our best guess is that these workers become orphans after their parent workers have died.
Why Gunicorn workers spawn child workers? Why do they die?! And how can we prevent this?
I should also mention that we've set Gunicorn log level to debug and still we don't see any thing significant, other than periodical log of workers number, which reports count of workers we wanted from it.
UPDATE
This is the line we used to run gunicorn:
gunicorn --env DJANGO_SETTINGS_MODULE=proj.settings proj.wsgi --name proj --workers 10 --user proj --group proj --bind 127.0.0.1:7003 --log-level=debug --pid gunicorn.pid --timeout 600 --access-logfile /home/proj/access.log --error-logfile /home/proj/error.log
In my case I deploy in Ubuntu servers (LTS releases, now almost are 14.04 LTS servers) and I never did have problems with gunicorn daemons, I create a gunicorn.conf.py and launch gunicorn with this config from upstart with an script like this in /etc/init/djangoapp.conf
description "djangoapp website"
start on startup
stop on shutdown
respawn
respawn limit 10 5
script
cd /home/web/djangoapp
exec /home/web/djangoapp/bin/gunicorn -c gunicorn.conf.py -u web -g web djangoapp.wsgi
end script
I configure gunicorn with a .py file config and i setup some options (details below) and deploy my app (with virtualenv) in /home/web/djangoapp and no problems with zombie and orphans gunicorn processes.
i verified your options, timeout can be a problem but another one is that you don't setup max-requests in your config, by default is 0, so, no automatic worker restart in your daemon and can generate memory leaks (http://gunicorn-docs.readthedocs.org/en/latest/settings.html#max-requests)
We will use a .sh file to start the gunicorn process. Later you will use a supervisord configuration file. what is supervisord? some external know how information link about how to install supervisord with Django,Nginx,Gunicorn Here
gunicorn_start.sh remember to give chmod +x to the file.
#!/bin/sh
NAME="myDjango"
DJANGODIR="/var/www/html/myDjango"
NUM_WORKERS=3
echo "Starting myDjango -- Django Application"
cd $DJANGODIR
exec gunicorn -w $NUM_WORKERS $NAME.wsgi:application --bind 127.0.0.1:8001
mydjango_django.conf : Remember to install supervisord on your OS. and
Copy this on the configuration folder.
[program:myDjango]
command=/var/www/html/myDjango/gunicorn_start.sh
user=root
autorestart=true
redirect_sderr=true
Later on use the command:
Reload the daemon’s configuration files, without add/remove (no restarts)
supervisordctl reread
Restart all processes Note: restart does not reread config files. For that, see reread and update.
supervisordctl start all
Get all process status info.
supervisordctl status
This sounds like a timeout issue.
You have multiple timeouts going on and they all need to be in a descending order. It seems they may not be.
For example:
Nginx has a default timeout of 60 seconds
Gunicorn has a default timeout of 30 seconds
Django has a default timeout of 300 seconds
Postgres default timeout is complicated but let's pose 60 seconds for this example.
In this example, when 30 seconds has passed and Django is still waiting for Postgres to respond. Gunicorn tells Django to stop, which in turn should tell Postgres to stop. Gunicorn will wait a certain amount of time for this to happen before it kills django, leaving the postgres process as an orphan query. The user will re-initiate their query and this time the query will take longer because the old one is still running.
I see that you have set your Gunicorn tiemeout to 300 seconds.
This would probably mean that Nginx tells Gunicorn to stop after 60 seconds, Gunicorn may wait for Django who waits for Postgres or any other underlying processes, and when Nginx gets tired of waiting, it kills Gunicorn, leaving Django hanging.
This is still just a theory, but it is a very common problem and hopefully leads you and any others experiencing similar problems, to the right place.
I'm trying to resize workers but the following message appears:
MacBook-Pro-de-Ricardo:subastas ricardo$ heroku ps:resize worker=2X
Resizing and restarting the specified dynos... failed
! No such process type worker defined in Procfile.
this is my procfile:
web: gunicorn -k flask_sockets.worker chat:app
Unicorn workers and Heroku workers are not the same.
Unicorn workers are in your web process, responding to requests.
https://devcenter.heroku.com/articles/python-gunicorn
Heroku workers are background workers.
https://devcenter.heroku.com/articles/background-jobs-queueing
If you need more ram et. al. for your workers, you just need to re-size the web type:
https://devcenter.heroku.com/articles/dyno-size
ex:
$ heroku ps:scale web=n:2X
That reads: "scale to n web worker(s) at 2X dyno size", where n is the number of web dynos you want running.