I have a django+uwsgi based website. some of the tables have almost 1 million rows.
After a few website usages, the VIRT memory used the uwsgi process reaches almost 20GB...almost kill my server...
Could you someone tell what may caused this problem? is it my table rows too big? (unlikely. Pinterest has much more data). now, i had to use reload-on-as= 10024
reload-on-rss= 4800 to kill the workers every few minutes....it is painful...
any help?
Here is my uwsgi.ini file
[uwsgi]
chdir = xxx
module = xxx.wsgi
master = true
processes = 2
socket =127.0.0.1:8004
chmod-socket = 664
no-orphans = true
#limit-as=256
reload-on-as= 10024
reload-on-rss= 4800
max-requests=250
uid = www-data
gid = www-data
#chmod-socket = 777
chown-socket = www-data
# clear environment on exit
vacuum = true
After some digging on stackflow and google search, here is the solution.
read this how django memory works and why it keeps going up
read this django app profiling
then I figured out the major parameter to set in uwsgi.ini is max_request. originally, I set it as 2000. now set it as 50. so it will respawn workers when memory goes up too much.
Then i try to figure out which request causes huge data query results from database. I ended up finding this little line:
amount=sum(x.amount for x in Project.objects.all())
While Project table has over 1 million complex entries.Occupying huge memory.... since I commented this out... everything runs smooth now.
So it is good to understand how the [django query works with database]
(Sorry I don't have enough reputation to comment - so apologies if this answer doesn't help in your case)
I had the same issue running Django on uwsgi/gninx and uwsgi controlled via supervisor. uwsgi-supervisor process started using lots of memory and consuming 100% CPU so only option was to repeatedly restart uwsgi.
Turned out the solution was to set up logging in the uwsgi.ini file:
logto = /var/log/uwsgi.log
There is some discussion on this here: https://github.com/unbit/uwsgi/issues/296
Related
I am following the tutorial on
Center for High Throughput Computing and Introduction to Configuration in the HTCondor website to set up a Partitionable slot. Before any configuration I run
condor_status
and get the following output.
I update the file 00-minicondor in /etc/condor/config.d by adding the following lines at the end of the file.
NUM_SLOTS = 1
NUM_SLOTS_TYPE_1 = 1
SLOT_TYPE_1 = cpus=4
SLOT_TYPE_1_PARTITIONABLE = TRUE
and reconfigure
sudo condor_reconfig
Now with
condor_status
I get this output as expected. Now, I run the following command to check everything is fine
condor_status -af Name Slotype Cpus
and find slot1#ip-172-31-54-214.ec2.internal undefined 1 instead of slot1#ip-172-31-54-214.ec2.internal Partitionable 4 61295 that is what I would expect. Moreover, when I try to summit a job that asks for more than 1 cpu it does not allocate space for it (It stays waiting forever) as it should.
I don't know if I made some mistake during the installation process or what could be happening. I would really appreciate any help!
EXTRA INFO: If it can be of any help have have installed HTCondor with the command
curl -fsSL https://get.htcondor.org | sudo /bin/bash -s – –no-dry-run
on Ubuntu 18.04 running on an old p2.xlarge instance (it has 4 cores).
UPDATE: After rebooting the whole thing it seems to be working. I can now send jobs with different CPUs requests and it will start them properly.
The only issue I would say persists is that Memory allocation is not showing properly, for example:
But in reality it is allocating enough memory for the job (in this case around 12 GB).
If I run again
condor_status -af Name Slotype Cpus
I still get something I am not supposed to
But at least it is showing the correct number of CPUs (even if it just says undefined).
What is the output of condor_q -better when the job is idle?
when I run unit tests for my application, first tests are successful, then around 100, tests start to fail, due to PDOException (Too many connections). I have already searched about this problem, but was not able to solve it.
My config is as follows:
<phpunit
backupGlobals = "false"
backupStaticAttributes = "false"
colors = "true"
convertErrorsToExceptions = "true"
convertNoticesToExceptions = "true"
convertWarningsToExceptions = "true"
processIsolation = "false"
stopOnFailure = "false"
syntaxCheck = "false"
bootstrap = "bootstrap.php.cache" >
If I change processIsolation to "true", all tests generate an error (E):
Caused by ErrorException: unserialize(): Error at offset 0 of 79 bytes
For that I tried setting "detect_unicode = Off" inside php.ini file.
If I run tests in smaller batches, like with "--group something", all tests are successful.
Can someone help me solve the issue when running all the tests at once? I really want to get rid of the PDOException.
Thanks in advance!
You should increase the maximum number of concurrent connections in your DB server.
If you're using MySQL, edit /etc/mysql/my.cnf and set the max_connections parameter to the number of concurrent connections you need. Then restart the MySQL server.
Keep in mind: In theory, the physical limits are very high. But if your queries cause a high CPU load or memory consumption, your DB server could eat up the resources required for other processes. This means, you could run out of memory, or your system can become overloaded.
For people who are having the same issue, here is more specific steps to config the my.cnf file.
If you are sure you are on the right my.cnf file, put max_connections = 500 (default is 151) to [mysqld] section in the my.cnf. Don't put it in the [client] section.
To make sure you are on the right my.cnf, if you have multiple mysqld installed from Homebrew or XAMMPP, find the right mysqld, for XAMMPP using /Applications/XAMPP/xamppfiles/sbin/mysqld --verbose --help | grep -A 1 "Default options" and you will get something like this:
Default options are read from the following files in the given order:
/Applications/XAMPP/xamppfiles/etc/xampp/my.cnf /Applications/XAMPP/xamppfiles/etc/my.cnf ~/.my.cnf
Normally it's at /Applications/XAMPP/xamppfiles/etc/my.cnf.
In my django project I have the following dependencies:
django==1.5.4
django-celery==3.1.9
amqp==1.4.3
kombu==3.0.14
librabbitmq==1.0.3 (as suggested by https://stackoverflow.com/a/17541942/1452356)
In dev_settings.py:
DEBUG = False
BROKER_URL = "django://"
import djcelery
djcelery.setup_loader()
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
CELERYD_CONCURRENCY = 2
# CELERYD_TASK_TIME_LIMIT = 10
CELERYD_TASK_TIME_LIMIT is commented as suggested here https://stackoverflow.com/a/17561747/1452356 along with debug_toolbar as suggested by https://stackoverflow.com/a/19931261/1452356
I start my worker in a shell with:
./manage.py celeryd --settings=dev_settings
Then I send a task:
class ExempleTask(Task):
def run(self, piProjectId):
table = []
for i in range(50000000):
table.append(1)
return None
Using a django command:
class Command(BaseCommand):
def handle(self, *plArgs, **pdKwargs):
loResult = ExempleTask.delay(1)
loResult.get()
return None
With:
./manage.py purge_and_delete_test --settings=dev_settings
I monitor the memory usage with:
watch -n 1 'ps ax -o rss,user,command | sort -nr | grep celery |head -n 5'
Every time I call the task, it increase the memory consumption of the celeryd/worker process, proportionally to the amount of data allocated in it...
It seems like a common issue (c.f. others stackoverflow link), however I couldn't fix it, even with the latest dependencies.
Thanks.
This is a Python and OS issue, not really a django or celery issue. Without getting too deep:
1) A process will never free memory addressing space once it has requested it from the OS. It never says "hey, I'm done here, you can have it back". In the example you've given, I'd expect the process size to grow for a while, and then stabilize, possibly at a high base line. After your example allocation, you might call the gc interface to force a garbage collect to see how
2) This isn't usually a problem, because unused pages are paged out by the OS because your process stops accessing that address space that it has deallocated.
3) It is a problem if your process is leaking object references, preventing python from garbage collecting to re-appropriate the space for later reuse by that process, and requiring your process to ask for more address space from the OS. At some point, the OS cries uncle and will (probably) kill your process with its oomkiller or similar mechanism.
4) If you are leaking, either fix the leak or set CELERYD_MAX_TASKS_PER_CHILD, and your child processes will (probably) commit suicide before upsetting the OS.
This is a good general discussion on Python's memory management:
CPython memory allocation
And a few minor things:
Use xrange not range - range will generate all values then iterate over that list. xrange is just a generator. Have set Django DEBUG=False?
kinda a Celery noob here but, I think I have a configuration issue where Celery is putting too much stuff in Redis
my goal is to attempt to reduce or optimize the amount of memory Redis is using, if I can
i have a pretty large Django production thing, where Celery jobs are run "a lot". In my settings.py I have
BROKER_BACKEND = "redis"
From a top -p13907 Redis is using a ton of memory (on the box it's only used by Celery):
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13907 redis 20 0 10.5g 3.3g 532 S 0 42.8 109:38.94 redis-server
I found this CELERY_TASK_RESULT_EXPIRES setting which looks like something I should add to my settings file.
By default, from the documentation it looks like it's set to 1 day (86400 seconds)
Is this what I wanna change? Or is there more settings I should look into? Another thing I'm unsure about is if I add it, how should I go about deciding whats a "safe" number of seconds to set it to?
i guess maybe your celery caller forget to clean up result and these result will be stored in message queue server until expiration. In celery, you have to call
r.get()
to get result and clean it in message queue. if you only access the result without calling this function:
r.result
the result would be still holding by message queue server and consume your memory a lot!
Common situation: I have a client on my server who may update some of the code in his python project. He can ssh into his shell and pull from his repository and all is fine -- but the code is stored in memory (as far as I know) so I need to actually kill the fastcgi process and restart it to have the code change.
I know I can gracefully restart fcgi but I don't want to have to manually do this. I want my client to update the code, and within 5 minutes or whatever, to have the new code running under the fcgi process.
Thanks
First off, if uptime is important to you, I'd suggest making the client do it. It can be as simple as giving him a command called deploy-code. Using your method, if there is an error in their code, your method requires a 10 minute turnaround (read: downtime) for fixing it, assuming he gets it correct.
That said, if you actually want to do this, you should create a daemon which will look for files modified within the last 5 minutes. If it detects one, it will execute the reboot command.
Code might look something like:
import os, time
CODE_DIR = '/tmp/foo'
while True:
if restarted = True:
restarted = False
time.sleep(5*60)
for root, dirs, files in os.walk(CODE_DIR):
if restarted=True:
break
for filename in files:
if restared=True:
break
updated_on = os.path.getmtime(os.path.join(root, filename))
current_time = time.time()
if current_time - updated_on <= 6 * 60: # 6 min
# 6 min could offer false negatives, but that's better
# than false positives
restarted = True
print "We should execute the restart command here."