How run a daemon of djcelery with celerycam - django

I'm working with django celery,I have a deamon of celery with supervisor but I have a problem in the django admin I can't see the state of the task,
I can only see the state of my tasks in the django admin, when I typed in console python manage.py celerycam,
How I run a daemon of celerycam.

You can start your celerycam daemon with your app and celery all together with supervisorctl.
Example config file (/etc/supervisor/conf.d/app_name.conf):
# app config
[program:app_name]
user = www-data
directory = /var/www/app_name
command = /var/www/app_name/bin/python /var/www/app_name/bin/gunicorn agora.wsgi_server:application --bind 127.0.0.1:8022 -t 90 --workers 4 --settings='app_name.settings.production'
redirect_stderr = true
autorestart=true
stdout_logfile = /var/log/supervisor/app_name.log
stderr_logfile = /var/log/supervisor/app_name_err.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn
autostart = true
stopsignal=KILL
environment=LANG="en_US.UTF-8",LC_ALL="en_US.UTF-8",LC_LANG="en_US.UTF-8"
stopasgroup=true
killasgroup=true
# celerycam config
[program:app_name_celerycam]
user = www-data
directory = /var/www/app_name
command = /var/www/app_name/bin/python manage.py celerycam --settings='app_name.settings.production'
redirect_stderr = true
autorestart=true
stdout_logfile = /var/log/supervisor/app_name_celerycam.log
stderr_logfile = /var/log/supervisor/app_name_celerycam_err.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn
autostart = true
stopwaitsecs=5
# celery config
[program:app_name_celery]
user = www-data
directory = /var/www/app_name
command = /var/www/app_name/bin/python manage.py celeryd -l INFO -E -B --settings='app_name.settings.production' --concurrency=1 --pidfile=/var/run/celery/app_name_celery.pid
redirect_stderr = true
autorestart=true
stdout_logfile = /var/log/supervisor/app_name_celery.log
stderr_logfile = /var/log/supervisor/app_name_celery_err.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn
autostart=true
stopwaitsecs=5
environment=C_FORCE_ROOT=1
stopasgroup=true
killasgroup=true
# group of our daemons
[group:app_name]
programs=app_name,app_name_celerycam,app_name_celery
priority=999
Reload our configuration:
supervisorctl reread
Now we can manage all daemons of our application with simple commands:
supervisorctl start app_name:*
supervisorctl stop app_name:*
supervisorctl restart app_name:*
supervisorctl status app_name:*

Related

uWSGI error every time I use the requests module in Django app on Docker

I'm running a Django app with uWSGI in Docker with docker-compose. I get the same error every time I:
Send a POST request with AJAX
In handling said request in my view, I use python's requests module, i.e. r = requests.get(some_url)
uWSGI says the following:
!!! uWSGI process 13 got Segmentation Fault !!!
DAMN ! worker 1 (pid: 13) died :( trying respawn ...
Respawned uWSGI worker 1 (new pid: 24)
spawned 4 offload threads for uWSGI worker 1
The console in the browser says net::ERR_EMPTY_RESPONSE
I've tried using the requests module in different places, and wherever I put it I get the same Segmentation Fault error. I'm also able to run everything fine outside of docker with no errors, so I've narrowed it down to: docker + requests module = errror.
Is there something that could be blocking the requests sent with the requests module from within the docker container? Thanks in advance for your help.
Here's my uwsgi.ini file:
[uwsgi]
chdir = %d
module = my_project.wsgi:application
master = true
processes = 2
http = 0.0.0.0:8000
vacuum = true
pidfile = /tmp/my_project.pid
daemonize = %d/my_project.log
check-static = %d
static-expires = /* 7776000
offload-threads = %k
uid = 1000
gid = 1000
# there is no /etc/mime.types on the docker Arch Linux image
mime-file = %d/mime.types
Dockerfile:
FROM alpine:3.8
ENV PYTHONUNBUFFERED 1
RUN mkdir /my_project
WORKDIR /my_project
RUN apk add build-base python3-dev py3-pip python3
# deps for python cryptography
RUN apk add libffi-dev musl-dev openssl-dev
# dep for uwsgi
RUN apk add linux-headers
ADD requirements.txt /my_project/
RUN pip3 install -r requirements.txt
ADD . /my_project/
ENTRYPOINT ./start.sh
docker-compose.yml:
version: '3'
services:
web:
build: .
entrypoint: ./start.sh
volumes:
- .:/my_project
ports:
- "8000:8000"
environment:
- DEBUG_LEVEL=INFO
network_mode: "host"
start.sh:
#!/bin/sh
echo '' > logfile.log
uwsgi --ini uwsgi.ini
tail -f logfile.log
Solution: Change base image to Ubuntu 16.04 and everything works fine now.

Celery with Supervisord workers are not working Isolated

.conf file
[program:task1]
directory=/home/ubuntu/proj1
command=/usr/bin/python3 /usr/local/bin/celery -A proj1 worker -l info --concurrency=10 -n proj1_worker#%%h
user=ubuntu
numprocs=1
stdout_logfile=/var/log/proj1_celeryd.log
stderr_logfile=/var/log/proj1_celeryd.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
priority=998
[program:task2]
directory=/home/ubuntu/proj2/
command=/usr/bin/python3 /usr/local/bin/celery -A proj2 worker -l info --concurrency=10 -n proj2_worker#%%h
user=ubuntu
numprocs=1
stdout_logfile=/var/log/proj2_celeryd.log
stderr_logfile=/var/log/proj2_celeryd.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
priority=998
[group:celeryworkers]
programs=task1,task2
proj1_worker and proj2_worker are not getting isolated.
At first, always proj1_worker is called even I called proj2_worker
I don't know where I am going wrong. Kindly assist.
Thank you in advance
First of all I really recommend you to use virtualenv for each project. Create 2 separate virtualenvs (you can specify your own location), see https://docs.python.org/3/library/venv.html.
python3 -m venv /home/ubuntu/virtualenvs/proj1
python3 -m venv /home/ubuntu/virtualenvs/proj2
Activate virtualenv and install celery:
source /home/ubuntu/virtualenvs/proj1/bin/activate
pip install --upgrade celery
source /home/ubuntu/virtualenvs/proj2/bin/activate
pip install --upgrade celery
Your supervisor configuration should then look like this:
[program:task1]
directory=/home/ubuntu/proj1
command=/home/ubuntu/virtualenvs/proj1/bin/celery worker -A proj1 -l info --concurrency=10 -n proj1_worker#%%h
# ...
[program:task2]
directory=/home/ubuntu/proj2
command=/home/ubuntu/virtualenvs/proj1/bin/celery worker -A proj2 -l info --concurrency=10 -n proj2_worker#%%h
# ...
Next, create 2 separate virtual hosts for your projects:
rabbitmqctl add_user proj_1 <PASSWORD>
rabbitmqctl add_vhost proj_1_vhost
rabbitmqctl set_permissions -p proj_1_vhost proj_1 ".*" ".*" ".*"
rabbitmqctl add_user proj_2 <PASSWORD>
rabbitmqctl add_vhost proj_2_vhost
rabbitmqctl set_permissions -p proj_2_vhost proj_2 ".*" ".*" ".*"
Finally modify celery configuration to use newly created virtual hosts:
app = Celery('proj1_celery_app')
app.conf.update(
# ...
broker_url='amqp://proj1:<PASSWORD>#localhost:5672/proj_1_vhost'
# ...
)
app = Celery('proj2_celery_app')
app.conf.update(
# ...
broker_url='amqp://proj2:<PASSWORD>#localhost:5672/proj_2_vhost'
# ...
)
For more info about rabbit vhosts see this SO post: Running multiple instances of celery on the same server.

How to run a celery worker with Django app scalable by AWS Elastic Beanstalk?

How to use Django with AWS Elastic Beanstalk that would also run tasks by celery on main node only?
This is how I set up celery with django on elastic beanstalk with scalability working fine.
Please keep in mind that 'leader_only' option for container_commands works only on environment rebuild or deployment of the App. If service works long enough, leader node may be removed by Elastic Beanstalk. To deal with that, you may have to apply instance protection for your leader node. Check: http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html#instance-protection-instance
Add bash script for celery worker and beat configuration.
Add file root_folder/.ebextensions/files/celery_configuration.txt:
#!/usr/bin/env bash
# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}
# Create celery configuraiton script
celeryconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A django_app --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv
[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A django_app --loglevel=INFO --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf
# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# Reread the supervisord config
supervisorctl -c /opt/python/etc/supervisord.conf reread
# Update supervisord in cache without restarting all services
supervisorctl -c /opt/python/etc/supervisord.conf update
# Start/Restart celeryd through supervisord
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker
Take care about script execution during deployment, but only on main node (leader_only: true).
Add file root_folder/.ebextensions/02-python.config:
container_commands:
04_celery_tasks:
command: "cat .ebextensions/files/celery_configuration.txt > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
leader_only: true
05_celery_tasks_run:
command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
leader_only: true
Beat is configurable without need of redeployment, with separate django applications: https://pypi.python.org/pypi/django_celery_beat.
Storing task results is good idea to: https://pypi.python.org/pypi/django_celery_beat
File requirements.txt
celery==4.0.0
django_celery_beat==1.0.1
django_celery_results==1.0.1
pycurl==7.43.0 --global-option="--with-nss"
Configure celery for Amazon SQS broker
(Get your desired endpoint from list: http://docs.aws.amazon.com/general/latest/gr/rande.html)
root_folder/django_app/settings.py:
...
CELERY_RESULT_BACKEND = 'django-db'
CELERY_BROKER_URL = 'sqs://%s:%s#' % (aws_access_key_id, aws_secret_access_key)
# Due to error on lib region N Virginia is used temporarily. please set it on Ireland "eu-west-1" after fix.
CELERY_BROKER_TRANSPORT_OPTIONS = {
"region": "eu-west-1",
'queue_name_prefix': 'django_app-%s-' % os.environ.get('APP_ENV', 'dev'),
'visibility_timeout': 360,
'polling_interval': 1
}
...
Celery configuration for django django_app app
Add file root_folder/django_app/celery.py:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'django_app.settings')
app = Celery('django_app')
# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
Modify file root_folder/django_app/__init__.py:
from __future__ import absolute_import, unicode_literals
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from django_app.celery import app as celery_app
__all__ = ['celery_app']
Check also:
How do you run a worker with AWS Elastic Beanstalk? (solution without scalability)
Pip Requirements.txt --global-option causing installation errors with other packages. "option not recognized" (solution for problems coming from obsolate pip on elastic beanstalk that cannto deal with global options for properly solving pycurl dependency)
This is how I extended the answer by #smentek to allow for multiple worker instances and a single beat instance - same thing applies where you have to protect your leader. (I still don't have an automated solution for that yet).
Please note that envvar updates to EB via the EB cli or the web interface are not relflected by celery beat or workers until app server restart has taken place. This caught me off guard once.
A single celery_configuration.sh file outputs two scripts for supervisord, note that celery-beat has autostart=false, otherwise you end up with many beats after an instance restart:
# get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}
# create celery beat config script
celerybeatconf="[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A lexvoco --loglevel=INFO --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=false
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 10
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# create celery worker config script
celeryworkerconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A lexvoco --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999
environment=$celeryenv"
# create files for the scripts
echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf
echo "$celeryworkerconf" | tee /opt/python/etc/celeryworker.conf
# add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: celerybeat.conf celeryworker.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# reread the supervisord config
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf reread
# update supervisord in cache without restarting all services
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf update
Then in container_commands we only restart beat on leader:
container_commands:
# create the celery configuration file
01_create_celery_beat_configuration_file:
command: "cat .ebextensions/files/celery_configuration.sh > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && sed -i 's/\r$//' /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
# restart celery beat if leader
02_start_celery_beat:
command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat"
leader_only: true
# restart celery worker
03_start_celery_worker:
command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker"
If someone is following smentek's answer and getting the error:
05_celery_tasks_run: /usr/bin/env bash does not exist.
know that, if you are using Windows, your problem might be that the "celery_configuration.txt" file has WINDOWS EOL when it should have UNIX EOL. If using Notepad++, open the file and click on "Edit > EOL Conversion > Unix (LF)". Save, redeploy, and error is no longer there.
Also, a couple of warnings for really-amateur people like me:
Be sure to include "django_celery_beat" and "django_celery_results" in your "INSTALLED_APPS" in settings.py file.
To check celery errors, connect to your instance with "eb ssh" and then "tail -n 40 /var/log/celery-worker.log" and "tail -n 40 /var/log/celery-beat.log" (where "40" refers to the number of lines you want to read from the file, starting from the end).
Hope this helps someone, it would've saved me some hours!

start: Job failed to start UWSGI,

Trying to configure ubuntu+nginx+uwsgi+django
upstart script /etc/init/uwsgi.conf
description "uWSGI application server in Emperor mode"
start on runlevel [2345]
stop on runlevel [!2345]
setuid voxa
setgid www-data
exec /usr/local/bin/uwsgi --emperor /etc/uwsgi/sites
uwsgi configuration
[uwsgi]
project = project
base = /home/user
chdir = %(base)/%(project)
home = home/user/Env/project_env
module = %(project).wsgi:application
master = true
processes = 5
socket = %(base)/%(project)/%(project).sock
chmod-socket = 664
vacuum = true
socket = %(base)/%(project)/%(project).sock
chmod-socket = 664
vacuum = true
But after running the command sudo service uwsgi start I get an error
start: Job failed to start
What should I check to handle it?
UPD:
with virtualenv enabled app successfully runs with uwsgi command
uwsgi --http :8000 --module project.wsgi
uWSGI doesn't have permissions to create socket file in specified directory. To solve that, you can run emperor as root and drop privileges in vassal, after creating socket. Simply add to your vassal config:
username = voxa
gropuname = www-data
And remove setuid and setgid from your upstart config file.
If you're worried that someone will abuse that and use other user/group, you can use emperor tyrant mode, by adding --emperor-tyrant to uwsgi start line in upstart config. That will disallow change of username and groupname to other than owner of vassal config file.

Supervisord with django writing separate logs for each program

I'm using supervisord (through django-supervisor a thin wrapper around supervisor) to run multiple processes with my Django installation.
My problem is all the logs are written to the supervisord log file (in this example out.log) instead of the different log files.
the conf file (cleaned up):
[supervisord]
logfile=/var/log/server/ourserver/out.log
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
[program:webserver]
command=uwsgi uwsgi.ini
stout_logfile = /var/log/server/ourserver/django.log
redirect_stderr = true
;autostart = true
;autorestart = true
[program:celery]
command=celery worker -B -A server.celery --loglevel=info --concurrency=4
;autostart = true
;autorestart = true
stout_logfile = /var/logs/server/ourserver/celery.log
redirect_stderr = true
[program:updater]
command=python -u updater.py
;directory=/home/ubuntu/server/ourserver
;autostart = true
;autorestart = true
stout_logfile = /var/logs/server/ourserver/updater.log
redirect_stderr = true
replace stout_logfile with stdout_logfile