I'm running celery with django and works great in development. But now I want to make it live
on my production server and I am running into some issues.
My setup is as follows:
Ubuntu
Nginx
Vitualenv
Upstart
Gunicorn
Django
I'm not sure how to now start celery with django when starting it with upstart and where does it log to?
Im starting django here:
~$ cd /var/www/webapps/minamobime_app
~$ source ../bin/activate
exec /var/www/webapps/bin/gunicorn_django -w $NUM_WORKERS \
--user=$USER --group=$GROUP --bind=$IP:$PORT --log-level=debug \
--log-file=$LOGFILE 2>>$LOGFILE
how do I start celery?
exec python manage.py celeryd -E -l info -c 2
Consider configuring celery as a daemon. For logging speciy:
CELERYD_LOG_FILE="/var/log/celery/%n.log"
where %s will be replaced by the node name
You can install supervisor using apt-get and then you can add the following to a file named celeryd.conf (or any name you wish) to etc/supervisor/conf.d folder (create the conf.d folder if it is not present)
; ==================================
; celery worker supervisor example
; ==================================
[program:celery]
; Set full path to celery program if using virtualenv
command=/home/<path to env>/env/bin/celery -A <appname> worker -l info
;enter the directory in which you want to run the app
directory=/home/<path to the app>
user=nobody
numprocs=1
stdout_logfile=/home/<path to the log file>/worker.log
stderr_logfile=/home/<path to the log file>/worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 1000
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
Also add the following lines to etc/supervisor/supervisord.conf
[include]
files = /etc/supervisor/conf.d/*.conf
Now start the supervisor by typing supervisord in terminal and celery will start automatically according to the settings you made above.
You can run:
python manage.py celery worker
this will work if you have djcelery in your INSTALLED_APPS
Related
This tutorial covers channels 1.x's deployment. However, this does not work with channels 2.x. The failing part is the daemon script, which is as follows:
files:"/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_daemon.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
# Get django environment variables
djangoenv=`cat /opt/python/current/env
| tr '\n' ',' | sed 's/%/%%/g' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g'
| sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
djangoenv=${djangoenv%?}
# Create daemon configuraiton script
daemonconf="[program:daphne]
; Set full path to channels program if using virtualenv
command=/opt/python/run/venv/bin/daphne -b 0.0.0.0 -p 5000 <your_project>.asgi:channel_layer
directory=/opt/python/current/app
user=ec2-user
numprocs=1
stdout_logfile=/var/log/stdout_daphne.log
stderr_logfile=/var/log/stderr_daphne.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$djangoenv
[program:worker]
; Set full path to program if using virtualenv
command=/opt/python/run/venv/bin/python manage.py runworker
directory=/opt/python/current/app
user=ec2-user
numprocs=1
stdout_logfile=/var/log/stdout_worker.log
stderr_logfile=/var/log/stderr_worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$djangoenv"
# Create the supervisord conf script
echo "$daemonconf" | sudo tee /opt/python/etc/daemon.conf
# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
then
echo "[include]" | sudo tee -a /opt/python/etc/supervisord.conf
echo "files: daemon.conf" | sudo tee -a /opt/python/etc/supervisord.conf
fi
# Reread the supervisord config
sudo /usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf reread
# Update supervisord in cache without restarting all services
sudo /usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf update
# Start/Restart processes through supervisord
sudo /usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart daphne
sudo /usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart worker
After deployment, AWS has 2 errors in logs: daphne: No such process and worker: No such process.
How should this script be changed, so that it runs on channels 2.x as well?
Thanks
I had the same error and the reason for mine was the supervisor process that runs these additional scripts wasn't picking up the Daphne process because of this line of code:
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
This checks if the supervisord.conf file has [include] present and adds the daemon process ONLY if there is no [include].
In my case I had a
[include]
celery.conf
in my supervisord file that prevented this Daphne script from adding the daemon.conf.
There are a few things you can do:
If you have another script creating a .conf file, combine these into one using the same include logic
Rewrite the include logic to check for daemon.conf specifically
Manually add daemon.conf to the supervisord.conf by SSH into your EC2 instance
I am trying to use django and celery to automate the task and want
to run celery as daemon. I copied the code for celeryd and
celerybeat from its official documentation and put it inside the
/etc/init.d/ folder.
Below is the code of my celeryd file and I put it inside the 'etc/default' folder. The code in the bold letters is commented code.
CELERY_BIN="/home/user/.local/bin/celery"
# Name of nodes to start, here we have a single node
CELERYD_NODES="worker"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"
# Where to chdir at start.
CELERYD_CHDIR="/home/user/django_project"
CELERY_APP="django_project"
#How to call "manage.py celeryd_multi"
#CELERYD_MULTI="$CELERYD_CHDIR/django_project/manage.py celeryd_multi"
# Extra arguments to celeryd
#CELERYD_OPTS="--time-limit 300 --concurrency=8"
# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"
# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.
CELERYD_USER="user"
CELERYD_GROUP="user"
# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="settings"
# beat settings
CELERYBEAT_OPTS="--scheduler django_celery_beat.schedulers:DatabaseScheduler"
CELERYBEAT_LOG_FILE="/var/log/celery/celeryBeat.log"
CELERYBEAT_PID_FILE="/var/run/celery/celeryBeat.pid"
# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1
I started the celeryd using "sudo /etc/init.d/celeryd/ start" command.
I started the celerybeat using "sudo /etc/init.d/celerybeat/ start" command.
When I checked the status of celery and celerybeat it is showing
celery init v10.1.
Using config script: /etc/default/celeryd
celeryd down: no pidfiles found
Please do let me know how to run celery as daemon.
I am trying to start a celery worker on EB but get an error which doesn't explain much.
Command in config file in .ebextensions dir:
03_celery_worker:
command: "celery worker --app=config --loglevel=info -E --workdir=/opt/python/current/app/my_project/"
The listed command works fine on my local machine (just change workdir parameter).
Errors from the EB:
Activity execution failed, because: /opt/python/run/venv/local/lib/python3.6/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
and
Starting new HTTPS connection (1): eu-west-1.queue.amazonaws.com
(ElasticBeanstalk::ExternalInvocationError)
I have updated celery worker command with parameter --uid=2 and privileges error disappeared but command execution is still failed due to
ExternalInvocationError
Any suggestions what I do wrong?
ExternalInvocationError
As I understand it means that listed command cannot be run from EB container commands. It is needed to create a script on the server and run celery from the script. This post describes how to do it.
Update:
It is needed to create a config file in .ebextensions directory. I called it celery.config. Link to the post above provides a script which works almost fine. It is needed to make some minor additions to work 100% correct. I had issues with schedule periodic tasks (celery beat). Below are steps on how to fix is:
Install (add to requirements) django-celery beat pip install django-celery-beat, add it to installed apps and use --scheduler parameter when starting celery beat. Instructions are here.
In the script you specify user which run the script. For celery worker it is celery user which was added earlier in the script (if doesn't exist). When I tried to start celery beat I got error PermissionDenied. It means that celery user doesn't have all necessary rights. using ssh I logged in to EB, looked a list of all users (cat /etc/passwd) and decided to use daemon user.
Listed steps resolved celery beat errors. Updated config file with the script is below (celery.config):
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
# Create required directories
sudo mkdir -p /var/log/celery/
sudo mkdir -p /var/run/celery/
# Create group called 'celery'
sudo groupadd -f celery
# add the user 'celery' if it doesn't exist and add it to the group with same name
id -u celery &>/dev/null || sudo useradd -g celery celery
# add permissions to the celery user for r+w to the folders just created
sudo chown -R celery:celery /var/log/celery/
sudo chown -R celery:celery /var/run/celery/
# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/%/%%/g' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
celeryenv=${celeryenv%?}
# Create CELERY configuration script
celeryconf="[program:celeryd]
directory=/opt/python/current/app
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A config.celery:app --loglevel=INFO --logfile=\"/var/log/celery/%%n%%I.log\" --pidfile=\"/var/run/celery/%%n.pid\"
user=celery
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 60
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# Create CELERY BEAT configuraiton script
celerybeatconf="[program:celerybeat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A config.celery:app --loglevel=INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler --logfile=\"/var/log/celery/celery-beat.log\" --pidfile=\"/var/run/celery/celery-beat.pid\"
directory=/opt/python/current/app
user=daemon
numprocs=1
stdout_logfile=/var/log/celerybeat.log
stderr_logfile=/var/log/celerybeat.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 60
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999
environment=$celeryenv"
# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf
echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf
# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "celery.conf" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: uwsgi.conf celery.conf celerybeat.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# Enable supervisor to listen for HTTP/XML-RPC requests.
# supervisorctl will use XML-RPC to communicate with supervisord over port 9001.
# Source: https://askubuntu.com/questions/911994/supervisorctl-3-3-1-http-localhost9001-refused-connection
if ! grep -Fxq "[inet_http_server]" /opt/python/etc/supervisord.conf
then
echo "[inet_http_server]" | tee -a /opt/python/etc/supervisord.conf
echo "port = 127.0.0.1:9001" | tee -a /opt/python/etc/supervisord.conf
fi
# Reread the supervisord config
supervisorctl -c /opt/python/etc/supervisord.conf reread
# Update supervisord in cache without restarting all services
supervisorctl -c /opt/python/etc/supervisord.conf update
# Start/Restart celeryd through supervisord
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd
supervisorctl -c /opt/python/etc/supervisord.conf restart celerybeat
commands:
01_killotherbeats:
command: "ps auxww | grep 'celery beat' | awk '{print $2}' | sudo xargs kill -9 || true"
ignoreErrors: true
02_restartbeat:
command: "supervisorctl -c /opt/python/etc/supervisord.conf restart celerybeat"
leader_only: true
One thing to focus attention on: in my project celery.py file is in the config directory, that is why I write -A config.celery:app when start celery worker and celery beat
I have spent the past few days implementing Channels into my Django app in order to use Channel's websocket support. My Django project is written in Python 3.4 and I am using Daphne and Channel's redis backend.
I have been able to get everything functioning locally by wrapping Supervisord in a Python2 virtualenv and using it to run scripts that start Daphne/Redis/workers inside a Python3 virtualenv, but have had no success deploying to our Elastic Beanstalk (Python 3.4) environment.
Is there any way to set up my EB deployment configs to run Supervisor in a Python2 virtualenv, like I can locally? If not, how can I go about getting Daphne, redis, and my workers up and running on EB deploy? I am willing to switch process managers if necessary, but have found Supervisor's syntax to be easier to understand/implement than Circus, and am not aware of any other viable alternatives.
With this configuration, I am able to successfully deploy to my EB environment and ssh into it, but Supervisor fails to start every process, and if I try to start Supervisor manually to check on the processes supervisorctl status gives me FATAL "Exited too quickly (process log may have details) for everything I try to initialize. The logs are empty.
Channels backend config:
CHANNEL_LAYERS = {
"default": {
"BACKEND": "asgi_redis.RedisChannelLayer",
"ROUTING": "<app>.routing.channel_routing",
"CONFIG": {
"hosts": [
os.environ.get('REDIS_URL', 'redis://localhost:6379')
],
},
},
}
asgi.py:
import os
from channels.asgi import get_channel_layer
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<app>.settings")
channel_layer = get_channel_layer()
supervisor conf (rest of the conf file was left default):
[program:Redis]
environment=PATH="/opt/python/run/venv/bin"
command=sh /opt/python/current/app/<app>/start_redis.sh
directory=/opt/python/current/app
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/redis.out.log
[program:Daphne]
environment=PATH="/opt/python/run/venv/bin"
command=sh /opt/python/current/app/<app>/start_daphne.sh
directory=/opt/python/current/app
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/daphne.out.log
[program:Worker]
environment=PATH="/opt/python/run/venv/bin"
command=sh /opt/python/current/app/<app>/start_worker.sh
directory=/opt/python/current/app
process_name=%(program_name)s_%(process_num)02d
numprocs=4
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/workers.out.log
.ebextensions/channels.config:
container_commands:
01_start_supervisord:
command: "sh /supervisord/start_supervisor.sh"
start_supervisor.sh:
#!/usr/bin/env bash
virtualenv -p /usr/bin/python2.7 /tmp/senv
source /tmp/senv/bin/activate
sudo pip install supervisor
sudo /usr/local/bin/supervisord -c
/opt/python/current/app/<app>/supervisord.conf
supervisorctl -c /opt/python/current/app/<app>/supervisord.conf status
start_redis:
#!/usr/bin/env bash
sudo wget http://download.redis.io/releases/redis-3.2.8.tar.gz
sudo tar xzf redis-3.2.8.tar.gz
cd redis-3.2.8
sudo make
source /opt/python/run/venv/bin/activate
sudo src/redis-server
start_daphne:
#!/usr/bin/env bash
source /opt/python/run/venv/bin/activate
/opt/python/run/venv/bin/daphne -b 0.0.0.0 -p 5000 <app>.asgi:channel_layer
start_worker:
#!/usr/bin/env bash
source /opt/python/run/venv/bin/activate
python manage.py runworker
I was loosely following this guide but since it was written for a python2 EB environment it is really only good for the ALB setup and base supervisor configuration.
Thank you guys for reading this, and please let me know if I can provide anything else by way of code/output etc.
So thanks to the logging advice from Berlin's answer, and an virtual environment tweak suggested by the AWS support team (I forwarded this question along to them), I was finally able to get this to work.
First off, I ended up completely removing Redis from Supervisor, and instead chose to run an ElastiCache Redis instance which I then connected to my EB instance. I don't think this is the only way to handle this, but it was the best route for my implementation.
I then changed from using the pre-existing start_supervisor.sh script and instead added a command to channels.config ebextension to create the script and add it to EB's postdeployment operations. This was necessary because .ebextension config files are run during deployment, but do not live past environment creation (this may not be entirely correct, but for the sake of this solution that is how I think of them), so even though my script was mostly correct the Supervisor process it started up would just die as soon as deployment was done.
so my .ebextensions/channels.config is now:
container_commands:
01_create_post_dir:
command: "mkdir /opt/elasticbeanstalk/hooks/appdeploy/post"
ignoreErrors: true
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/start_supervisor.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
virtualenv -p /usr/bin/python2.7 /tmp/senv
source /tmp/senv/bin/activate && source /opt/python/current/env
python --version > /tmp/version_check.txt
sudo pip install supervisor
/usr/local/bin/supervisord -c /opt/python/current/app/<app>/supervisord.conf
supervisorctl -c /opt/python/current/app/<app>/supervisord.conf status
This alone was enough to get Supervisor running on EB deployment, but I had to make some more changes to get Daphne and my Django workers to stay alive:
start_daphne.sh:
#!/usr/bin/env bash
source /opt/python/run/venv/bin/activate && source /opt/python/current/env
/opt/python/run/venv/bin/daphne -b 0.0.0.0 -p 5000 <app>.asgi:channel_layer
start_worker.sh:
#!/usr/bin/env bash
source /opt/python/run/venv/bin/activate && source /opt/python/current/env
python manage.py runworker
Adding && source /opt/python/current/env to the virtualenv activation command was suggested to me by AWS support, as env variables are not pulled into virtualenvs automatically, which was causing Daphne and workers to die on creation due to Import Errors.
I also made some changes to my supervisord.conf file:
[unix_http_server]
file=/tmp/supervisor.sock ; (the path to the socket file)
[supervisord]
logfile=/tmp/supervisord.log ; supervisord log file
loglevel=error ; info, debug, warn, trace
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
[program:Daphne]
environment=PATH="/opt/python/run/venv/bin"
command=sh /opt/python/current/app/<app>/start_daphne.sh --log-file /tmp/start_daphne.log
directory=/opt/python/current/app
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/daphne.out.log
stderr_logfile=/tmp/daphne.err.log
[program:Worker]
environment=PATH="/opt/python/run/venv/bin"
command=sh /opt/python/current/app/<app>/start_worker.sh --log-file /tmp/start_worker.log
directory=/opt/python/current/app
process_name=%(program_name)s_%(process_num)02d
numprocs=4
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/workers.out.log
stderr_logfile=/tmp/workers.err.log
You said logs are empty, so hard to debug, make sure to have the log line on the main supervisor config file /etc/supervisord.conf, see what the errors and share them.
[supervisord]
logfile=/var/log/supervisord/supervisord.log ; supervisord log file
loglevel=error ; info, debug, warn, trace
And to each of your program on your supervisor conf file add a log error and see what is the errors and share them.
command=sh /opt/python/current/app/<app>/start_redis.sh --log-file /path/to/your/logs/start_redis.log
stdout_logfile=/tmp/redis.out.log
stderr_logfile=/tmp/redis.err.log
I wrote a long and detailed google doc on how to do this linking to it here
https://docs.google.com/document/d/1naZsjO05Pa6uB9D-26eCeUVROZhz_MKkL8GI2knGLeo/edit?usp=sharing
And for any moderators, the reason I'm not typing it all in here is that the guide is long and with a lot of images. Figured it's worth sharing as it will help someone
I really enjoy using upstart. I currently have upstart jobs to run different gunicorn instances in a number of virtualenvs. However, the 2-3 examples I found for Celery upstart scripts on the interwebs don't work for me.
So, with the following variables, how would I write an Upstart job to run django-celery in a virtualenv.
Path to Django Project:
/srv/projects/django_project
Path to this project's virtualenv:
/srv/environments/django_project
Path to celery settings is the Django project settings file (django-celery):
/srv/projects/django_project/settings.py
Path to the log file for this Celery instance:
/srv/logs/celery.log
For this virtual env, the user:
iamtheuser
and the group:
www-data
I want to run the Celery Daemon with celerybeat, so, the command I want to pass to the django-admin.py (or manage.py) is:
python manage.py celeryd -B
It'll be even better if the script starts after the gunicorn job starts, and stops when the gunicorn job stops. Let's say the file for that is:
/etc/init/gunicorn.conf
You may need to add some more configuration, but this is an upstart script I wrote for starting django-celery as a particular user in a virtualenv:
start on started mysql
stop on stopping mysql
exec su -s /bin/sh -c 'exec "$0" "$#"' user -- /home/user/project/venv/bin/python /home/user/project/django_project/manage.py celeryd
respawn
It works great for me.
I know that it looks ugly, but it appears to be the current 'proper' technique for running upstart jobs as unprivileged users, based on this superuser answer.
I thought that I would have had to do more to get it to work inside of the virtualenv, but calling the python binary inside the virtualenv is all it takes.
Here is my working config using the newer systemd running on Ubuntu 16.04 LTS. Celery is in a virtualenv. App is a Python/Flask.
Systemd file: /etc/systemd/system/celery.service
You'll want to change the user and paths.
[Unit]
Description=Celery Service
After=network.target
[Service]
Type=forking
User=nick
Group=nick
EnvironmentFile=-/home/nick/myapp/server_configs/celery_env.conf
WorkingDirectory=/home/nick/myapp
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
--pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
[Install]
WantedBy=multi-user.target
Environment file (referenced above):/home/nick/myapp/server_configs/celery_env.conf
# Name of nodes to start
# here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"
# Absolute or relative path to the 'celery' command:
CELERY_BIN="/home/nick/myapp/venv/bin/celery"
# App instance to use
CELERY_APP="myapp.tasks"
# How to call manage.py
CELERYD_MULTI="multi"
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
# and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"
To automatically create the log and run folder with the correct permissions for your user, create a file in /usr/lib/tmpfiles.d. I was having trouble with the /var/run/celery folder being deleted on rebooting and then celery could not start properly.
My /usr/lib/tmpfiles.d/celery.conf file:
d /var/log/celery 2775 nick nick -
d /var/run/celery 2775 nick nick -
To enable: sudo systemctl enable celery.service
Now you'll need to reboot your system for the /var/log/celery and /var/run/celery folders to be created. You can check to see if celery started after rebooting by checking the logs in /var/log/celery.
To restart celery: sudo systemctl restart celery.service
Debugging: tail -f /var/log/syslog and try restarting celery to see what the error is. It could be related to the backend or other things.
Hope this helps!