I am running django application in apache using mod wsgi. I have set server limit to 1 as of the following configuration.
00-mpm.conf
<IfModule worker.c>
StartServers 1
ServerLimit 1
#MaxClients 300
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
At beginning apache runs with one server. But after some time when i am checking it runs with two servers. And i got the following error in the error.log
[mpm_worker:notice] [pid 46754:tid 139924109117568] AH00297: SIGUSR1 received. Doing graceful restart.
How can i make sure that apache to run with only one server. And not doing restart.
Apache runs a parent process and then one child worker process with that configuration. The parent process would be quite small in size as all it is doing is managing the child worker process and restarts it if it dies etc. Requests are only handled by the child worker process. So what you are seeing is normal.
Why are you trying to restrict it to one process? If it is to satisfy some requirement of your WSGI application that can only have one process, you should not be using embedded mode anyway. Use daemon mode of mod_wsgi, in which case the WSGI application will use one process by default.
So put back the Apache MPM settings and then use:
WSGIRestrictEmbedded On
WSGIDaemonProcess myapp
WSGIScriptAlias / /some/path/myapp.wsgi process-group=myapp application-group=%{GLOBAL}
This will result in the Apache parent process, however many child worker process Apache needs to accept and proxy requests, and the single mod_wsgi daemon process running the WSGI application, to which the child worker processes are proxying requests.
Related
I have an AWS beanstalk env and have old setting of wsgi (given below), I do not have idea how does this work internally, can anybody guide me?
NumProcesses:7 -- number of process
NumThreads:5 -- number of thread in each process
How memory and cpu are being used with this configuration because there is no memory and cpu settings in AWS beanstalk level.
These parameters are part of configuration option for Python environment:
aws:elasticbeanstalk:application:environment.
They mean (from docs):
NumProcesses: The number of daemon processes that should be started for the process group when running WSGI applications (default value 1).
NumThreads: The number of threads to be created to handle requests in each daemon process within the process group when running WSGI applications (default value 15).
Internally, these values map to uwsgi or gunicorn configuration options in your EB environment. For example:
uwsgi --http :8000 --wsgi-file application.py --master --processes 4 --threads 2
Their impact on memory and cpu usage of your instance(s) is based on your application and how resource intensive it is. If you are not sure how to set them up, maybe keeping them at default values would be a good start.
The settings are also available in the EB console, under Software category:
To add on to #Marcin
Amazon linux 2 uses gunicorn
workers are processes in gunicorn
Gunicorn should only need 4-12 worker processes to handle hundreds or thousands of requests per second.
Gunicorn relies on the operating system to provide all of the load balancing when handling requests. Generally, we (gunicorn creators) recommend (2 x $num_cores) + 1 as the number of workers to start off with. While not overly scientific, the formula is based on the assumption that for a given core, one worker will be reading or writing from the socket while the other worker is processing a request.
To see how the settings in the option settings map to gunicorn you can ssh into your eb instance, go
$ eb ssh
$ cd cd /var/app/current/
$ cat Procfile
web: gunicorn --bind 127.0.0.1:8000 --workers=3 --threads=20 api.wsgi:application
--threads
A positive integer generally in the 2-4 x $(NUM_CORES) range. You’ll want to vary this a bit to find the best for your particular application’s work load.
The threads option only applies to gthread worker type. gunicons default worker class is sync, If you try to use the sync worker type and set the threads setting to more than 1, the gthread worker type will be used instead automatically
based on all the above I would personally choose
workers = (2 x $NUM_CORES ) + 1
threads = 4 x $NUM_CORES
for a t3.medum instance that has 2 cores that translates to
workers = 5
threads = 8
obviously, you need to tweak this for your use case, and treat these as defaults that could very well not be right for your particular application use case, read the refs below to see how to choose the right setup for you use case
References:
REF: Gunicorn Workers and Threads
REF: https://medium.com/building-the-system/gunicorn-3-means-of-concurrency-efbb547674b7
REF: https://docs.gunicorn.org/en/stable/settings.html#worker-class
I have a problem with uwsgi everytime I restart the server when I have a code updates.
When I restart the uwsgi using "sudo restart accounting", there's a small gap between stop and start instance that results to downtime and stops all the current request.
When I try "sudo reload accounting", it works but my memory goes up (double). When I run the command "ps aux | grep accounting", it shows that I have 10 running processes (accounting.ini) instead of 5 and it freezes up my server when the memory hits the limit.
accounting.ini
I am running
Ubuntu 14.04
Django 1.9
nginx 1.4.6
uwsgi 2.0.12
This is how uwsgi does graceful reload. Keeps old processes until requests are served and creates new ones that will take over incoming requests.
Read Things that could go wrong
Do not forget, your workers/threads that are still running requests
could block the reload (for various reasons) for more seconds than
your proxy server could tolerate.
And this
Another important step of graceful reload is to avoid destroying
workers/threads that are still managing requests. Obviously requests
could be stuck, so you should have a timeout for running workers (in
uWSGI it is called the “worker’s mercy” and it has a default value of
60 seconds).
So i would recommend trying worker-reload-mercy
Default value is to wait 60 seconds, just lower it to something that your server can handle.
Tell me if it worked.
Uwsgi chain reload
This is another try to fix your issue. As you mentioned your uwsgi workers are restarting in a manner described below:
send SIGHUP signal to the master
Wait for running workers.
Close all of the file descriptors except the ones mapped to sockets.
Call exec() on itself.
One of the cons of this kind of reload might be stuck workers.
Additionaly you report that your server crashes when uwsgi maintains 10 proceses (5 old and 5 new ones).
I propose trying chain reload. DIrect quote from documentation explains this kind of reload best:
When triggered, it will restart one worker at time, and the following worker is not reloaded until the previous one is ready to accept new requests.
It means that you will not have 10 processes on your server but only 5.
Config that should work:
# your .ini file
lazy-apps = true
touch-chain-reload = /path/to/reloadFile
Some resources on chain reload and other kinds are in links below:
Chain reloading uwsgi docs
uWSGI graceful Python code deploy
I am using Easy apache 4, mod_wsgi, and Python 3.5. When I called a Django project in the server I got the following error:
(13)Permission denied: mod_wsgi (pid=24223): Unable to connect to WSGI daemon
process 'user123' on '/var/run/wsgi.8442.6.7.sock' as user with uid=3708.
Your Apache installation is likely set up to run with SECURE privileges mode. This means that the Apache child worker process is forked and privileges dropped before handling the request, which in this case is simply trying to proxy the request through to the mod_wsgi daemon process. The consequence of this is that it cannot connect to the socket for the daemon process, as it was setup with ownership to match the original Apache child worker process before privileges were dropped.
This is evident because the error message has uid in the range of a normal user and not the special apache or nobody user.
To fix it, you need to modify the WSGIDaemonProcess directive configuration and add the option:
socket-user=#3708
or:
socket-user=username
where username is replaced with the actual name of the user with uid of 3708.
The addition of this option seems to be required due to recent changes in CPanel configurations for Apache.
Sometime socket-user setting does not work.
This may occur because of WSGISocketPrefix path not found
If it is VPS:
WSGISocketPrefix /var/run/wsgi
If you are in a shared hosting:
WSGISocketPrefix ../../var/run/wsgi
Thank You
I'm running a Django app on Apache 2.4.7 with mod_wsgi 3.4. The whole setup is on an EC2 ubuntu instance. Ever since I deployed the app, the server has gone down with 504/503 errors every few days with this message in the logs:
Script timed out before returning headers: wsgi.py
I've searched extensively but all I can conclude is I have a memory leak somewhere? I can't seem to figure out what's actually going wrong, since my Django install is pretty vanilla. This is the relevant part of my conf file:
WSGIScriptAlias / /home/ubuntu/projects/appname/app/app/app/wsgi.py
WSGIDaemonProcess app python-path=/home/ubuntu/projects/appname/app user=ubuntu
WSGIProcessGroup app
WSGIApplicationGroup %{GLOBAL}
Could this be from some third party library? The only extras I have installed are ImageMagick and exiftool, the latter of which isn't being used. Is there anything else I can do to debug?
Does your application call out to any backend services?
If you are getting 503/504 and that message, then it would generally indicate that your code is either hanging on backend services or that your code is blocking indefinitely on thread locks.
So basically all request threads become busy and get used up.
If they didn't provide such an ancient version of mod_wsgi then newer versions do at least have better options to combat this sort of problem in your application and recover automatically, plus log information to help you work out why.
For such an old version, you can set 'inactivity-timeout' option to WSGIDaemonProcess to '60' as a way of recovering, but this will also have the effect of restarting your application after 60 seconds if it wasn't receiving any requests as well, which may in itself not be ideal for some apps. In newer versions the inactivity timeout is separated from the concept of a request timeout.
My setup is django 1.3 and the default mod_wsgi and apache packages for ubuntu 10.04. I tested one view of my app on my development VM (DEBUG and debugging toolbar off):
ab -n 200 -c 5 http://127.0.0.1/
and got 4 requests per second. This seemed slow so I simplified the queries, used indexes, etc. to the point where debugging toolbar tells me I have 4 queries taking 8ms. Running the same test, I only get 8 requests per second. The CPU seems to be at 100% the whole time. This seems quite slow for what is now a quite simple view, but it is just a low powered VM.
I decided to start up a large ec2 instance (4 cpu) to see what kind of performance I would get on that class of machine and was surprised to only get 13 requests per second. How can I change the configuration of apache/mod_wsgi to get more performance out of this class of machine?
I think I am using worker rather than prefork:
$ /usr/sbin/apache2 -l
Compiled in modules:
core.c
mod_log_config.c
mod_logio.c
worker.c
http_core.c
mod_so.c
My worker configuration is:
<IfModule mpm_worker_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 0
</IfModule>
and my WSGI settings look like this:
WSGIScriptAlias / /home/blah/site/proj/wsgi.py
WSGIDaemonProcess blah user=blah group=blah processes=1 threads=10
WSGIProcessGroup blah
Thanks very much for your help!
NOTE: I tried the ab test from another instance and got the same result
Make sure keep-alive is off.
More processes and single-threaded I have seen better performance where CPU is the limiting factor; try processes=4 threads=1.
The best way to tweak mod_wsgi is to not use it :)
First: I don't think your problem is the web server: with mod_wsgi you can get hundreds requests/s. You can get better results with caching and with DB connection pooling. If you're using postgres, take a look at pgpool II: http://pgpool.projects.postgresql.org/ .
However, if you want to go deeper into wsgi web servers, read carefully this nice benchmark: http://nichol.as/benchmark-of-python-web-servers .
If you don't need asyncronous workers, gunicorn is a good choice. It's very easy to setup (you can run it with manage.py run_gunicorn) and it's pretty fast. If you want to be sure that mod_wsgi is not the cuprit, try it. If you want better performance go with gevent or uWSGI.
But the Web Server won't change your benchmark: you can go from 4 req/s to 4.01 req/s.