Django celery task execution - django

I have 3 tasks in tasks.py file when demon is working only one task execute . Rest of two display in debug mode but not execute.
celery worker --concurrency=1
Is there any changes need ?
Edit :
#celery.decorators.periodic_task(run_every=datetime.timedelta(minutes=1))
def task1():
try:
sc = SampleCount.objects.get(pk=1)
except:
sc = SampleCount()
sc.num = sc.num + 1
sc.save()
return(sc)
same task2 for add data into SampleCount2(minute=2) and task3 for add data into SampleCount3(minute=3).
Now i run
python manage.py celeryd -E -B --loglevel=DEBUG
so it will display 3 task. but when I check status its display only 1 task1 active . No other two in scheduled.if i run
python manage.py celeryd -E -B --loglevel=DEBUG
command again then task3 is active and rest of two not working. How to run three task using one worker .
I have wait till 10 minute but only one task working(may be its task1, or task2 or task3 ).

It depends on what resources are available on server.
There is not such requirement, that number of tasks must be equal to number of workers. The only thing is needed to understand:
the more workers are running, the faster all tasks will be processed, but more server resources will be needed.
This command:
celery worker --concurrency=1
Will spawn two processes
1. Control process, that will manage workers
2. worker process, the one, that will actually do the job
And this command:
celery worker --concurrency=2
Will spawn three processes
1. Control process, that will manage workers
2. worker process #1, the one, that will actually do the job
2. worker process #2, the one, that will actually do the job
And so on.
One worker can process all of your tasks, but two worker will do it faster.

Related

Celery worker stop without any error after task completion

Celery worker running in ubuntu stops without any error after the process completion But The celery beat keeps running without any issues.
Recently this problem occurs when deployed to the new server the DB schema and some code also were changed previously it was running fine both worker and beat.
Comand use to run celery worker
celery -A base worker -l info --detach --logfile=logs/celery.log -n celery_worker
No error after the process completes which takes around 80, 90 min to complete.
After completion, there is no celery worker process running and the next task doesn't execute.
What might be the issue here how can I debug it?

Celery is repeating my tasks three times

I call some tasks in celery one time but celery executes all of them three times.
Is it an expected behavior of celery or is it a misconfiguration?
I'm using Django 1.5.11, Celery 3.1.23 and Redis 3.0.6.
You may have some stray workers executing the tasks or an celery flower instance may try to "help" recover unacked messages.
Make sure that only one instance of celery is running with ps -Af | grep celerybeat and check if you have any flower instance running by accessing http://localhost:5555 (it usually runs on that port).

Correct Way for Celery Process Architecture and demonizing

I have a Python/Django Project running on uwsgi/nginx. For asynchronous task we are using rabbitmq/celeryd and supervisord to manage all the daemons
Versions:
python: 2.7
django: 1.9.7
celery: 3.1.23
django-celery: 3.1.17
Celery has 10 queue of type Direct (say queue1, queue2, ...)
Each queue is handled by a separate celeryd process which is manage via supervisord. each supervisord process looks as following
[program:app_queue_worker]
command=/var/www/myproj/venv/bin/celery worker -A myproj -c 2 --queue=queue1 --loglevel=INFO
directory=/var/www/myproj/
user=ubuntu
numprocs=1
autostart=true
autorestart=true
startsecs=10
exitcodes=1
stopwaitsecs = 600
killasgroup=true
priority=1000
Hence Supervisord is running 10 Mainprocess and 20 Worker process
Other Thing I have noticed is uwsgi also spawns some celery workers(Dont understand how and why, YET ) with concurrency=2. So if I have 4 uwsgi process running i will have an addition 10 celery workers running
All these workers are each taking 200-300M memory? Something is wrong here I feel it but I am not able to put my finger on it. Celery shouldn't be running such memory heavy process?
Note: Debug=False, there is no memory leakage due to debug
Can someone please comment on the architecture if it is correct or wrong?
Would it be better to run 2-3 celery MainProcesses which listen all queues at once and increase its concurrency?
Update : celery.py Config
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'MyProject.settings')
from django.conf import settings # noqa
from chatterbox import celery_settings
app = Celery('MyProject')
# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.conf.update(
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
CELERYD_CONCURRENCY=1,
)
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
There is no simple answer to this.
To me, the fact that uwsgi spawns celery workers is wrong.
Creating only worker processes that consume all queues might lead to the situation where long running tasks make some queues overflow whereas separate workers that consume specific queues with short running tasks could make the situation better. Everything depends on your use case.
The 300mb residual memory is quite a lot. If the tasks are i/o bound go multi-thread/gevent. However, if the tasks are CPU bound, you have no other option than to scale process wise.
If you start a celery worker with concurrency of n, it will spawn n + 1 process by default. Since you are spawning 10 workers with a concurrency of 2, celery will start 30 processes.
Each worker consumes ~60MB(~30MB for main process & 2*~15MB for subprocesses) of memory when not consuming queues. It might vary depending what your worker is doing. If you start 10 workers, it will consume ~600MB of memory.
I am not sure how you came to know that uwsgi also spawns some celery workers. Only supervisor should spawn the process.
You can run just 1 celery worker which listens to all queues with a concurrency of 20. This will reduce your memory usage at the cost of flexibility. With this setup, you can't start/stop consuming from selected queues. Also, there is no guarantee that all queues will be consumed equally.

Sidekiq: How to assign process to a worker?

I'm having bit struggle in Sidekiq with multiple workers and multiple Sidekiq processes.
I'ld like to run three sidekiq processes for a environemt.
I'm having three workers class (Lets say "worker1", "worker2" and "worker3") and three sidekiq processes (Lets say "process1", "process2" and "process3").
In my current structure the workers are running on any process which is available. But what I want is the worker1 should run on process1 and worker2 should run on process2 and so on.
I'm bit confused about achieving it and I'll be glad if I know how to set a worker to a particular sidekiq process.
Sidekiq processes are:
process1: bundle exec sidekiq -q default
process2: bundle exec sidekiq -C config/myapp_sidekiq.yml
process3: bundle exec sidekiq -q process3
Thanks in advance...
You assign processes to pull jobs from queues. You assign workers to go into queues.
process1: bundle exec sidekiq -q queue1
class Worker1
sidekiq_options queue: 'queue1'
end
# will only be processed by process1
Worker1.perform_async

celery control add_consumer giving Error: No nodes replied within time constraint

I want to configure celery worker to consume only from a particular queue,
I saw in celery docs that control add_consumer does exactly that.
Problem is when I try :
celery control -A [App_name] add_consumer [queue_name] worker1.h%
it gives me error :
Error: No nodes replied within time constraint
Any help is really appreciated.
Is there any other way I can make my worker consume from a specific queue?
Note : celery -A [App_name] worker1.h%
starts the celery worker, and everything works fine just that is works on all my queues. I want to dedicate a worker to consume from specific queue.
Broker used: rabbitmq
I would just run a separate worker
celery -A app_name -Q queue_name --concurrency=1