Celery with Supervisord workers are not working Isolated - django

Question

.conf file
[program:task1]
directory=/home/ubuntu/proj1
command=/usr/bin/python3 /usr/local/bin/celery -A proj1 worker -l info --concurrency=10 -n proj1_worker#%%h
user=ubuntu
numprocs=1
stdout_logfile=/var/log/proj1_celeryd.log
stderr_logfile=/var/log/proj1_celeryd.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
priority=998
[program:task2]
directory=/home/ubuntu/proj2/
command=/usr/bin/python3 /usr/local/bin/celery -A proj2 worker -l info --concurrency=10 -n proj2_worker#%%h
user=ubuntu
numprocs=1
stdout_logfile=/var/log/proj2_celeryd.log
stderr_logfile=/var/log/proj2_celeryd.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
priority=998
[group:celeryworkers]
programs=task1,task2
proj1_worker and proj2_worker are not getting isolated.
At first, always proj1_worker is called even I called proj2_worker
I don't know where I am going wrong. Kindly assist.
Thank you in advance

Answer 1

First of all I really recommend you to use virtualenv for each project. Create 2 separate virtualenvs (you can specify your own location), see https://docs.python.org/3/library/venv.html.
python3 -m venv /home/ubuntu/virtualenvs/proj1
python3 -m venv /home/ubuntu/virtualenvs/proj2
Activate virtualenv and install celery:
source /home/ubuntu/virtualenvs/proj1/bin/activate
pip install --upgrade celery
source /home/ubuntu/virtualenvs/proj2/bin/activate
pip install --upgrade celery
Your supervisor configuration should then look like this:
[program:task1]
directory=/home/ubuntu/proj1
command=/home/ubuntu/virtualenvs/proj1/bin/celery worker -A proj1 -l info --concurrency=10 -n proj1_worker#%%h
# ...
[program:task2]
directory=/home/ubuntu/proj2
command=/home/ubuntu/virtualenvs/proj1/bin/celery worker -A proj2 -l info --concurrency=10 -n proj2_worker#%%h
# ...
Next, create 2 separate virtual hosts for your projects:
rabbitmqctl add_user proj_1 <PASSWORD>
rabbitmqctl add_vhost proj_1_vhost
rabbitmqctl set_permissions -p proj_1_vhost proj_1 ".*" ".*" ".*"
rabbitmqctl add_user proj_2 <PASSWORD>
rabbitmqctl add_vhost proj_2_vhost
rabbitmqctl set_permissions -p proj_2_vhost proj_2 ".*" ".*" ".*"
Finally modify celery configuration to use newly created virtual hosts:
app = Celery('proj1_celery_app')
app.conf.update(
# ...
broker_url='amqp://proj1:<PASSWORD>#localhost:5672/proj_1_vhost'
# ...
)
app = Celery('proj2_celery_app')
app.conf.update(
# ...
broker_url='amqp://proj2:<PASSWORD>#localhost:5672/proj_2_vhost'
# ...
)
For more info about rabbit vhosts see this SO post: Running multiple instances of celery on the same server.