I am trying to use django and celery to automate the task and want
to run celery as daemon. I copied the code for celeryd and
celerybeat from its official documentation and put it inside the
/etc/init.d/ folder.
Below is the code of my celeryd file and I put it inside the 'etc/default' folder. The code in the bold letters is commented code.
CELERY_BIN="/home/user/.local/bin/celery"
# Name of nodes to start, here we have a single node
CELERYD_NODES="worker"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"
# Where to chdir at start.
CELERYD_CHDIR="/home/user/django_project"
CELERY_APP="django_project"
#How to call "manage.py celeryd_multi"
#CELERYD_MULTI="$CELERYD_CHDIR/django_project/manage.py celeryd_multi"
# Extra arguments to celeryd
#CELERYD_OPTS="--time-limit 300 --concurrency=8"
# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"
# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.
CELERYD_USER="user"
CELERYD_GROUP="user"
# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="settings"
# beat settings
CELERYBEAT_OPTS="--scheduler django_celery_beat.schedulers:DatabaseScheduler"
CELERYBEAT_LOG_FILE="/var/log/celery/celeryBeat.log"
CELERYBEAT_PID_FILE="/var/run/celery/celeryBeat.pid"
# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1
I started the celeryd using "sudo /etc/init.d/celeryd/ start" command.
I started the celerybeat using "sudo /etc/init.d/celerybeat/ start" command.
When I checked the status of celery and celerybeat it is showing
celery init v10.1.
Using config script: /etc/default/celeryd
celeryd down: no pidfiles found
Please do let me know how to run celery as daemon.
Related
I cannot seem to get celery working in production using supervisor. I am normally using it in local development running the command. However, when I make a supervisor setup, it is not working. Reading the logs I am getting the error:
Unable to load celery application.
The module inp_proj was not found.
My .conf file for supervisor is:
[program:inp_proj]
directory=/www/[project_directory]/inp_proj
command=/www/[project_directory]/venv/bin/celery -A inp_proj worker --loglevel=INFO
user=jan
numprocs=1
stdout_logfile=/var/log/celery/inp_proj.log
stderr_logfile=/var/log/celery/inp_proj.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600
killasgroup=true
priority=998
this is my celery.py file:
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'inp_proj.settings')
app = Celery('inp_proj')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
It is located inside project directory and inside inp_proj.
I tried changing the celery command in the created config file, adding environment path to supervisor config file but nothing seems to be working. If I manually active the venv with
source venv/bin/activate
and then start the worker manually, it works normally. But when using supervisor, it doesn't work.
I'm using ECS for the first time. I have dockerized my Django 2.2 application and using ECS and uwsgi to run the Django application in production.
While in the development environment, I had to run three commands to run Django server, celery and celery beat
python manage.py runserver
celery -A qcg worker -l info
celery beat -A qcg worker -l info
Where qcg is my application.
My Dockerfile has following uwsgi configuration
EXPOSE 8000
ENV UWSGI_WSGI_FILE=qcg/wsgi.py
ENV UWSGI_HTTP=:8000 UWSGI_MASTER=1 UWSGI_HTTP_AUTO_CHUNKED=1 UWSGI_HTTP_KEEPALIVE=1 UWSGI_LAZY_APPS=1 UWSGI_WSGI_ENV_BEHAVIOR=holy
ENV UWSGI_WORKERS=2 UWSGI_THREADS=4
ENV UWSGI_STATIC_MAP="/static/=/static_cdn/static_root/" UWSGI_STATIC_EXPIRES_URI="/static/.*\.[a-f0-9]{12,}\.(css|js|png|jpg|jpeg|gif|ico|woff|ttf|otf|svg|scss|map|txt) 315360000"
USER ${APP_USER}:${APP_USER}
ENTRYPOINT ["/app/scripts/docker/entrypoint.sh"]
The entrypoint.sh file has
exec "$#"
I have created the ECS task definition and in the container's command input, I have
uwsgi --show-config
This starts the uwsgi server.
Now I'm running 1 EC2 instance in the cluster and running one service with two instances of the task definition.
I couldn't figure out how to run the celery task and celery beat in my application.
Do I need to create separate tasks for running celery and celery-beat?
Yes, you need to run separate ECS tasks (or separate ECS services) for celery beat and celery worker. Celery Beat will send the Celery tasks to the Celery worker.
I use separate Dockerfiles for Celery, Celery beat, and Django.
Worker Dockerfile something like this:
FROM python:3.8
ENV PYTHONUNBUFFERED 1
ADD requirements.txt /requirements.txt
RUN pip install -r /requirements.txt
ADD . /src
WORKDIR /src
CMD ["celery", "-A", "<appname>", "worker"]
and Beat Dockerfile:
FROM python:3.8
ENV PYTHONUNBUFFERED 1
ADD requirements.txt /requirements.txt
RUN pip install -r /requirements.txt
ADD . /src
WORKDIR /src
CMD ["celery", "-A", "<appname>", "beat"]
What is the best practice to run celery as a daemon in a production virtualenv? I use the following in the local environment which works perfect and receiving tasks works as expected. But in production always stuck at
WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) error
I use the following configuration in local and in production:
/etc/default/celeryd:
CELERY_BIN="path/to/celery/bin"
CELERY_APP="myproj"
CELERYD_CHDIR="home/myuser/project/myproj"
CELERYD_OPTS="--time-limit=300 --concurrency=4"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_USER="myuser"
CELERYD_GROUP="myuser"
CELERY_CREATE_DIRS=1
/etc/init.d/celeryd:[celeryd]
Package & OS version info:
Ubuntu == 16.04.2
Celery == 4.1.0
rabbitmq == 3.5.7
django == 2.0.1
I also use these commands while making celery to run as daemon:
sudo chown -R root:root /var/log/celery/
sudo chown -R root:root /var/run/celery/
sudo update-rc.d celeryd defaults
sudo update-rc.d celeryd enable
sudo /etc/init.d/celeryd start
Here is my django settings.py configuration for celery:
CELERY_BROKER_URL = 'amqp://localhost'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_RESULT_BACKEND = 'db+sqlite:///results.sqlite'
CELERY_TASK_SERIALIZER = 'json'
Need expert advise to make the celery daemon to work correctly in production virtualenv. Thanks in advance!
Unless you've created a seperate vhost & user for rabbitmq, set the CELERY_BROKER_URL to amqp://guest#localhost//
Also, rather than root, you should set the owner of /var/log/celery/ and /var/run/celery/ to "myuser" as you have set in your celeryd config
I guess this can be a symptom of OOM. I was deploying a Celery backend in a Docker container - "it worked on my machine" but not in the cluster. I allocated more ram to the task and no longer have the problem.
I got this error due to out of memory error caught in
/var/log/kern.log
I have Tensorflow running in one of my tasks, which needs additional computational power, but my physical memory(RAM) is not sufficient to handle that much load. I weird that there's no log in celery except SigKill 9 error. But the kernel log helped me to fix it.
I'm running celery with django and works great in development. But now I want to make it live
on my production server and I am running into some issues.
My setup is as follows:
Ubuntu
Nginx
Vitualenv
Upstart
Gunicorn
Django
I'm not sure how to now start celery with django when starting it with upstart and where does it log to?
Im starting django here:
~$ cd /var/www/webapps/minamobime_app
~$ source ../bin/activate
exec /var/www/webapps/bin/gunicorn_django -w $NUM_WORKERS \
--user=$USER --group=$GROUP --bind=$IP:$PORT --log-level=debug \
--log-file=$LOGFILE 2>>$LOGFILE
how do I start celery?
exec python manage.py celeryd -E -l info -c 2
Consider configuring celery as a daemon. For logging speciy:
CELERYD_LOG_FILE="/var/log/celery/%n.log"
where %s will be replaced by the node name
You can install supervisor using apt-get and then you can add the following to a file named celeryd.conf (or any name you wish) to etc/supervisor/conf.d folder (create the conf.d folder if it is not present)
; ==================================
; celery worker supervisor example
; ==================================
[program:celery]
; Set full path to celery program if using virtualenv
command=/home/<path to env>/env/bin/celery -A <appname> worker -l info
;enter the directory in which you want to run the app
directory=/home/<path to the app>
user=nobody
numprocs=1
stdout_logfile=/home/<path to the log file>/worker.log
stderr_logfile=/home/<path to the log file>/worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 1000
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
Also add the following lines to etc/supervisor/supervisord.conf
[include]
files = /etc/supervisor/conf.d/*.conf
Now start the supervisor by typing supervisord in terminal and celery will start automatically according to the settings you made above.
You can run:
python manage.py celery worker
this will work if you have djcelery in your INSTALLED_APPS
I really enjoy using upstart. I currently have upstart jobs to run different gunicorn instances in a number of virtualenvs. However, the 2-3 examples I found for Celery upstart scripts on the interwebs don't work for me.
So, with the following variables, how would I write an Upstart job to run django-celery in a virtualenv.
Path to Django Project:
/srv/projects/django_project
Path to this project's virtualenv:
/srv/environments/django_project
Path to celery settings is the Django project settings file (django-celery):
/srv/projects/django_project/settings.py
Path to the log file for this Celery instance:
/srv/logs/celery.log
For this virtual env, the user:
iamtheuser
and the group:
www-data
I want to run the Celery Daemon with celerybeat, so, the command I want to pass to the django-admin.py (or manage.py) is:
python manage.py celeryd -B
It'll be even better if the script starts after the gunicorn job starts, and stops when the gunicorn job stops. Let's say the file for that is:
/etc/init/gunicorn.conf
You may need to add some more configuration, but this is an upstart script I wrote for starting django-celery as a particular user in a virtualenv:
start on started mysql
stop on stopping mysql
exec su -s /bin/sh -c 'exec "$0" "$#"' user -- /home/user/project/venv/bin/python /home/user/project/django_project/manage.py celeryd
respawn
It works great for me.
I know that it looks ugly, but it appears to be the current 'proper' technique for running upstart jobs as unprivileged users, based on this superuser answer.
I thought that I would have had to do more to get it to work inside of the virtualenv, but calling the python binary inside the virtualenv is all it takes.
Here is my working config using the newer systemd running on Ubuntu 16.04 LTS. Celery is in a virtualenv. App is a Python/Flask.
Systemd file: /etc/systemd/system/celery.service
You'll want to change the user and paths.
[Unit]
Description=Celery Service
After=network.target
[Service]
Type=forking
User=nick
Group=nick
EnvironmentFile=-/home/nick/myapp/server_configs/celery_env.conf
WorkingDirectory=/home/nick/myapp
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
--pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
[Install]
WantedBy=multi-user.target
Environment file (referenced above):/home/nick/myapp/server_configs/celery_env.conf
# Name of nodes to start
# here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"
# Absolute or relative path to the 'celery' command:
CELERY_BIN="/home/nick/myapp/venv/bin/celery"
# App instance to use
CELERY_APP="myapp.tasks"
# How to call manage.py
CELERYD_MULTI="multi"
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
# and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"
To automatically create the log and run folder with the correct permissions for your user, create a file in /usr/lib/tmpfiles.d. I was having trouble with the /var/run/celery folder being deleted on rebooting and then celery could not start properly.
My /usr/lib/tmpfiles.d/celery.conf file:
d /var/log/celery 2775 nick nick -
d /var/run/celery 2775 nick nick -
To enable: sudo systemctl enable celery.service
Now you'll need to reboot your system for the /var/log/celery and /var/run/celery folders to be created. You can check to see if celery started after rebooting by checking the logs in /var/log/celery.
To restart celery: sudo systemctl restart celery.service
Debugging: tail -f /var/log/syslog and try restarting celery to see what the error is. It could be related to the backend or other things.
Hope this helps!