django-celery-beat doesn't start within supervisor just manually - django

Question

I've installed Celery[sqs] and django-celery-beat in my Django 1.10 project.
I've trying to run them both (worker and beat) using Supervisor on and Elastic Beanstalk instance.
The Supervisor config is being created dynamically with the following script:
#!/usr/bin/env bash
# get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}
# create celery beat config script
celerybeatconf="[program:celery-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A phsite --loglevel=DEBUG --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=false
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 10
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# create celery worker config script
celeryworkerconf="[program:celery-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A phsite --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999
environment=$celeryenv"
# create files for the scripts
echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf
echo "$celeryworkerconf" | tee /opt/python/etc/celeryworker.conf
# add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: celerybeat.conf celeryworker.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# reread the supervisord config
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf reread
# update supervisord in cache without restarting all services
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf update
After which the following ebextension is running:
container_commands:
01_create_celery_beat_configuration_file:
command: "cat .ebextensions/files/celery_configuration.sh > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && sed -i 's/\r$//' /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
02_chmod_supervisor_sock:
command: "chmod 777 /opt/python/run/supervisor.sock"
03_create_logs:
command: "touch /var/log/celery-beat.log /var/log/celery-worker.log"
04_chmod_logs:
command: "chmod 777 /var/log/celery-beat.log /var/log/celery-worker.log"
05_start_celery_worker:
command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart celery-worker"
06_start_celery_beat:
command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf start celery-beat"
When logging-in to the Instance, and running
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf status
celery-beat is already "not started" (with an empty log file) while the celery-worker is running.
The weirdest part is that if I run it manually (e.g.
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf start celery-beat
It's running without errors.
Anyone has any idea how to debug it?
Why would it not load within the eb_extension while it does load later?
Maybe that has to do with the fact the Django is not up yet and am using django_celery_beat.schedulers:DatabaseScheduler configuration?

Answer 1

So the simple reason is, the shell script created in the eb_extension:
container_commands:
01_create_celery_beat_configuration_file:
command: "cat .ebextensions/files/celery_configuration.sh > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && sed -i 's/\r$//' /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
Is created in the appdeploy/post directory and therefore runs (post deployment and basically) after the following commands are executed.
The start/restart command don't do a thing because the shell script hasn't registered those services yet. 🤦‍♂️