Running celeryd-worker on elastic-beanstalk - django

Question

I am trying to get celery set up on elastic-beanstalk to run periodic-tasks. I have gotten everything running except for the celeryd-worker on elastic-beanstalk. It deploys fine and it is registering tasks but when I set up a periodic task either in celery.py or through django-celery-beat it doesn't execute. I pulled the logs from EB and got
/opt/python/log/supervisord.log
-------------------------------------
2019-09-20 16:32:50,831 INFO spawned: 'celeryd-worker' with pid 712
2019-09-20 16:32:53,929 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 16:32:54,930 INFO gave up: celeryd-worker entered FATAL state, too many start retries too quickly
2019-09-20 16:41:43,327 INFO stopped: celeryd-beat (exit status 0)
2019-09-20 16:41:44,333 INFO spawned: 'celeryd-beat' with pid 1664
2019-09-20 16:41:54,442 INFO success: celeryd-beat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2019-09-20 16:41:54,689 INFO spawned: 'celeryd-worker' with pid 1670
2019-09-20 16:41:57,844 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 16:41:58,847 INFO spawned: 'celeryd-worker' with pid 1676
2019-09-20 16:42:02,037 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 16:42:04,045 INFO spawned: 'celeryd-worker' with pid 1711
2019-09-20 16:42:05,271 INFO stopped: httpd (exit status 0)
2019-09-20 16:42:05,275 INFO spawned: 'httpd' with pid 1717
2019-09-20 16:42:07,241 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-20 16:42:07,898 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 16:42:11,041 INFO spawned: 'celeryd-worker' with pid 1841
2019-09-20 16:42:11,344 INFO stopped: celeryd-beat (exit status 0)
2019-09-20 16:42:12,349 INFO spawned: 'celeryd-beat' with pid 1845
2019-09-20 16:42:14,527 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 16:42:15,529 INFO gave up: celeryd-worker entered FATAL state, too many start retries too quickly
2019-09-20 16:42:22,426 INFO success: celeryd-beat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2019-09-20 16:42:22,673 INFO spawned: 'celeryd-worker' with pid 1852
2019-09-20 16:42:25,760 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 16:42:26,763 INFO spawned: 'celeryd-worker' with pid 1857
2019-09-20 16:42:29,890 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 16:42:31,897 INFO spawned: 'celeryd-worker' with pid 1872
2019-09-20 16:42:35,010 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 16:42:38,360 INFO spawned: 'celeryd-worker' with pid 1879
2019-09-20 16:42:41,484 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 16:42:42,485 INFO gave up: celeryd-worker entered FATAL state, too many start retries too quickly
2019-09-20 17:01:45,639 INFO spawned: 'celeryd-worker' with pid 3075
2019-09-20 17:01:46,392 INFO stopped: celeryd-beat (exit status 0)
2019-09-20 17:01:47,397 INFO spawned: 'celeryd-beat' with pid 3080
2019-09-20 17:01:47,924 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:01:49,695 INFO spawned: 'celeryd-worker' with pid 3085
2019-09-20 17:01:51,339 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:01:53,343 INFO spawned: 'celeryd-worker' with pid 3089
2019-09-20 17:01:54,890 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:01:57,894 INFO spawned: 'celeryd-worker' with pid 3094
2019-09-20 17:01:57,895 INFO success: celeryd-beat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2019-09-20 17:01:58,386 INFO stopped: celeryd-worker (terminated by SIGTERM)
2019-09-20 17:01:59,391 INFO spawned: 'celeryd-worker' with pid 3099
2019-09-20 17:02:00,957 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:02:01,958 INFO gave up: celeryd-worker entered FATAL state, too many start retries too quickly
2019-09-20 17:02:03,508 INFO stopped: httpd (exit status 0)
2019-09-20 17:02:03,513 INFO spawned: 'httpd' with pid 3144
2019-09-20 17:02:04,678 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-20 17:02:08,612 INFO stopped: celeryd-beat (exit status 0)
2019-09-20 17:02:09,617 INFO spawned: 'celeryd-beat' with pid 3268
2019-09-20 17:02:20,058 INFO success: celeryd-beat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2019-09-20 17:02:20,306 INFO spawned: 'celeryd-worker' with pid 3274
2019-09-20 17:02:21,860 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:02:22,863 INFO spawned: 'celeryd-worker' with pid 3278
2019-09-20 17:02:24,429 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:02:26,437 INFO spawned: 'celeryd-worker' with pid 3291
2019-09-20 17:02:27,995 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:02:31,970 INFO spawned: 'celeryd-worker' with pid 3296
2019-09-20 17:02:33,535 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:02:34,537 INFO gave up: celeryd-worker entered FATAL state, too many start retries too quickly
2019-09-20 17:17:05,637 INFO spawned: 'celeryd-worker' with pid 4245
2019-09-20 17:17:07,195 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:17:08,197 INFO spawned: 'celeryd-worker' with pid 4250
2019-09-20 17:17:09,750 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:17:11,757 INFO spawned: 'celeryd-worker' with pid 4268
2019-09-20 17:17:12,996 INFO stopped: httpd (exit status 0)
2019-09-20 17:17:13,667 INFO spawned: 'httpd' with pid 4274
2019-09-20 17:17:14,000 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:17:15,001 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-20 17:17:17,749 INFO spawned: 'celeryd-worker' with pid 4356
2019-09-20 17:17:19,939 INFO stopped: celeryd-worker (terminated by SIGTERM)
2019-09-20 17:17:20,944 INFO spawned: 'celeryd-worker' with pid 4400
2019-09-20 17:17:22,486 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:17:23,005 INFO gave up: celeryd-worker entered FATAL state, too many start retries too quickly
2019-09-20 17:28:31,137 INFO stopped: celeryd-beat (exit status 0)
2019-09-20 17:28:32,142 INFO spawned: 'celeryd-beat' with pid 5121
2019-09-20 17:28:42,799 INFO success: celeryd-beat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2019-09-20 17:28:43,049 INFO spawned: 'celeryd-worker' with pid 5127
2019-09-20 17:28:44,655 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:28:45,745 INFO spawned: 'celeryd-worker' with pid 5145
2019-09-20 17:28:46,959 INFO stopped: httpd (exit status 0)
2019-09-20 17:28:46,963 INFO spawned: 'httpd' with pid 5151
2019-09-20 17:28:47,955 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:28:48,957 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-20 17:28:49,961 INFO spawned: 'celeryd-worker' with pid 5221
2019-09-20 17:28:52,555 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:28:53,967 INFO stopped: celeryd-beat (exit status 0)
2019-09-20 17:28:54,972 INFO spawned: 'celeryd-beat' with pid 5278
2019-09-20 17:28:55,975 INFO spawned: 'celeryd-worker' with pid 5282
2019-09-20 17:28:57,801 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:28:58,803 INFO gave up: celeryd-worker entered FATAL state, too many start retries too quickly
2019-09-20 17:29:05,811 INFO success: celeryd-beat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2019-09-20 17:29:06,058 INFO spawned: 'celeryd-worker' with pid 5304
2019-09-20 17:29:07,616 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:29:08,618 INFO spawned: 'celeryd-worker' with pid 5309
2019-09-20 17:29:10,142 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:29:12,149 INFO spawned: 'celeryd-worker' with pid 5322
2019-09-20 17:29:13,708 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:29:17,450 INFO spawned: 'celeryd-worker' with pid 5326
2019-09-20 17:29:18,998 INFO exited: celeryd-worker (exit status 1; not expected)
2019-09-20 17:29:20,000 INFO gave up: celeryd-worker entered FATAL state, too many start retries too quickly
It appears that the celery beat worker is starting/restarting just fine so the problem must be with the celery worker? I am very novice at this so im not sure where the problem.
.ebextensions/files/celery_configuration.txt
#!/usr/bin/env bash
# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}
# Create celery configuraiton script
celeryconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A backend -P solo --loglevel=INFO -n worker.%%h
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv
[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A backend --loglevel=INFO --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf
# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# Reread the supervisord config
supervisorctl -c /opt/python/etc/supervisord.conf reread
# Update supervisord in cache without restarting all services
supervisorctl -c /opt/python/etc/supervisord.conf update
# Start/Restart celeryd through supervisord
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker
02-python.config
packages:
yum:
libcurl-devel: []
container_commands:
01_upgrade_pip_for_venv:
command: "/opt/python/run/venv/bin/pip install --upgrade pip"
04_celery_tasks:
command: "cat .ebextensions/files/celery_configuration.txt > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
leader_only: true
05_celery_tasks_run:
command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
leader_only: true
celery.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from celery.schedules import crontab
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'backend.settings')
app = Celery('backend')
# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
app.conf.beat_schedule = {
'task_every_5_seconds': { #name of the scheduler
'task': 'add_new_task', # task name which we have created in tasks.py
'schedule': 10.0, # set the period of running
},
}

Answer 1

Here are a couple of things to check:
the start command for celery beat looks wrong if you are using celery v4. According to the docs, you should be using something like:
/opt/python/run/venv/bin/celery beat -A backend --loglevel=INFO --workdir=/tmp --scheduler django_celery_beat.schedulers:DatabaseScheduler --pidfile /tmp/celerybeat.pid
make sure that you only have one beat process running across all of your beanstalk instances (i.e., one beat instance per broker URL).