Update!
Got it to work by adding export LC_ALL=en_US.UTF-8 at the top of my bash script.
I am attempting to daemonize celery beat and worker. I have no troubles running celery or beat when I ssh into my Elastic Beanstalk instance and do the following steps:
cd /opt/python/current/app
/opt/python/run/venv/bin/celery -A myDjangoApp beat --loglevel=INFO
/opt/python/run/venv/bin/celery -A myDjangoApp worker --loglevel=INFO
Tasks are scheduled and able to execute with ease. However when I run the exact same commands with supervisor I am getting an uniformative error. From looking at supervisorctl status I see the process runs for a few seconds then fails. Upon further examination the the log files show me the following error:
File "/opt/python/run/venv/bin/celery", line 11, in <module>
sys.exit(main())
File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/__main__.py", line 15, in main
sys.exit(_main())
File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/celery.py", line 213, in main
return celery(auto_envvar_prefix="CELERY")
File "/opt/python/run/venv/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/click/core.py", line 760, in main
_verify_python3_env()
File "/opt/python/run/venv/local/lib/python3.6/site-packages/click/_unicodefun.py", line 130, in _verify_python3_env
" mitigation steps.{}".format(extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/python3/ for mitigation steps.
Listed below are my supervisor.conf and celery.sh(runs celery) files.
Following is my supervisor.conf file.
[unix_http_server]
file=/opt/python/run/supervisor.sock ; (the path to the socket file)
chmod=0777 ; socket file mode (default 0700)
;chown=nobody:nogroup ; socket file uid:gid owner
[supervisord]
logfile=/opt/python/log/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=10MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/opt/python/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
directory=/opt/python/current/app ; (default is not to cd during start)
;nocleanup=true ; (don't clean up tempfiles at start;default false)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///opt/python/run/supervisor.sock
[program:httpd]
command=/opt/python/bin/httpdlaunch
numprocs=1
directory=/opt/python/current/app
autostart=true
autorestart=unexpected
startsecs=1 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
killasgroup=false ; SIGKILL the UNIX process group (def false)
redirect_stderr=false
;[program:celeryWorker]
;user=root
;command=/opt/python/run/venv/bin/celery -A Daash worker --loglevel=INFO --daemon
;directory=/opt/python/current/app
;numprocs=1
;autostart=true
;autorestart=true
;startsecs=0
;stopwaitsecs=60
;killasgroup=true
[program:celeryBeat]
user=root
command=/opt/python/etc/celery.sh
directory=/opt/python/current/app
numprocs=1
autostart=true
autorestart=true
startsecs=0
stopwaitsecs=60
killasgroup=true
stdout_logfile = /opt/python/log/cel_stdout.log
stderr_logfile = /opt/python/log/cel.log
Following is my celery.sh file.I have mirrored the exact steps I run when manually running celery in the .sh file.
#!/usr/bin/env bash
cd /opt/python/current/app
export PATH=/opt/python/run/venv/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin
sudo /opt/python/run/venv/bin/celery -A myDjangoApp beat --loglevel=INFO
Judging by the stack trace you provided, the error happens inside the Click dependency, associated with an encoding limitation.
Click's documentation explains this error a bit further: https://click.palletsprojects.com/en/8.0.x/unicode-support/?highlight=ascii
The fact that you were able to run it interactively is because your user session has the locale and other environment variables all correctly set, which the root session created by supervisord probably does not.
In any case, setting these variables within your celery.sh script should do the trick:
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
I've installed Celery[sqs] and django-celery-beat in my Django 1.10 project.
I've trying to run them both (worker and beat) using Supervisor on and Elastic Beanstalk instance.
The Supervisor config is being created dynamically with the following script:
#!/usr/bin/env bash
# get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}
# create celery beat config script
celerybeatconf="[program:celery-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A phsite --loglevel=DEBUG --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=false
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 10
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# create celery worker config script
celeryworkerconf="[program:celery-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A phsite --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999
environment=$celeryenv"
# create files for the scripts
echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf
echo "$celeryworkerconf" | tee /opt/python/etc/celeryworker.conf
# add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: celerybeat.conf celeryworker.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# reread the supervisord config
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf reread
# update supervisord in cache without restarting all services
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf update
After which the following ebextension is running:
container_commands:
01_create_celery_beat_configuration_file:
command: "cat .ebextensions/files/celery_configuration.sh > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && sed -i 's/\r$//' /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
02_chmod_supervisor_sock:
command: "chmod 777 /opt/python/run/supervisor.sock"
03_create_logs:
command: "touch /var/log/celery-beat.log /var/log/celery-worker.log"
04_chmod_logs:
command: "chmod 777 /var/log/celery-beat.log /var/log/celery-worker.log"
05_start_celery_worker:
command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart celery-worker"
06_start_celery_beat:
command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf start celery-beat"
When logging-in to the Instance, and running
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf status
celery-beat is already "not started" (with an empty log file) while the celery-worker is running.
The weirdest part is that if I run it manually (e.g.
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf start celery-beat
It's running without errors.
Anyone has any idea how to debug it?
Why would it not load within the eb_extension while it does load later?
Maybe that has to do with the fact the Django is not up yet and am using django_celery_beat.schedulers:DatabaseScheduler configuration?
So the simple reason is, the shell script created in the eb_extension:
container_commands:
01_create_celery_beat_configuration_file:
command: "cat .ebextensions/files/celery_configuration.sh > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && sed -i 's/\r$//' /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
Is created in the appdeploy/post directory and therefore runs (post deployment and basically) after the following commands are executed.
The start/restart command don't do a thing because the shell script hasn't registered those services yet. 🤦♂️
I've a Django 2 application deployed on AWS Elastic Beanstalk.
I configured Celery in order to exec async tasks on the same machine.
Since I added Celery, every time I redeploy my application eb deploy myapp-env I get the following error:
ERROR: [Instance: i-0bfa590abfb9c4878] Command failed on instance. Return code: 2 Output: (TRUNCATED)...
ERROR: already shutting down
error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 800
error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 800.
Hook /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
ERROR: Unsuccessful command execution on instance id(s) 'i-0bfa590abfb9c4878'. Aborting the operation.
ERROR: Failed to deploy application.
To be able to update my application code, I have to re-create a new environment by doing the following:
$ eb terminate myapp-env # terminate the environment
$ eb create myapp-env # re-create it
The Celery config file that I'm using is the following:
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
# Create required directories
sudo mkdir -p /var/log/celery/
sudo mkdir -p /var/run/celery/
# Create group called 'celery'
sudo groupadd -f celery
# add the user 'celery' if it doesn't exist and add it to the group with same name
id -u celery &>/dev/null || sudo useradd -g celery celery
# add permissions to the celery user for r+w to the folders just created
sudo chown -R celery:celery /var/log/celery/
sudo chown -R celery:celery /var/run/celery/
# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/%/%%/g' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
celeryenv=${celeryenv%?}
# Create CELERY configuration script
celeryconf="[program:celeryd]
directory=/opt/python/current/app
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A celery_conf.celery_app:app --loglevel=INFO --logfile=\"/var/log/celery/%%n%%I.log\" --pidfile=\"/var/run/celery/%%n.pid\"
user=celery
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 60
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# Create CELERY BEAT configuraiton script
celerybeatconf="[program:celerybeat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A celery_conf.celery_app:app --loglevel=INFO --logfile=\"/var/log/celery/celery-beat.log\" --pidfile=\"/var/run/celery/celery-beat.pid\"
directory=/opt/python/current/app
user=celery
numprocs=1
stdout_logfile=/var/log/celerybeat.log
stderr_logfile=/var/log/celerybeat.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 60
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999
environment=$celeryenv"
# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf
echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf
# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "celery.conf" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: uwsgi.conf celery.conf celerybeat.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# Enable supervisor to listen for HTTP/XML-RPC requests.
# supervisorctl will use XML-RPC to communicate with supervisord over port 9001.
# Source: https://askubuntu.com/questions/911994/supervisorctl-3-3-1-http-localhost9001-refused-connection
if ! grep -Fxq "[inet_http_server]" /opt/python/etc/supervisord.conf
then
echo "[inet_http_server]" | tee -a /opt/python/etc/supervisord.conf
echo "port = 127.0.0.1:9001" | tee -a /opt/python/etc/supervisord.conf
fi
# Reread the supervisord config
supervisorctl -c /opt/python/etc/supervisord.conf reread
# Update supervisord in cache without restarting all services
supervisorctl -c /opt/python/etc/supervisord.conf update
# Start/Restart celeryd through supervisord
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd
supervisorctl -c /opt/python/etc/supervisord.conf restart celerybeat
commands:
01_killotherbeats:
command: "ps auxww | grep 'celery beat' | awk '{print $2}' | sudo xargs kill -9 || true"
ignoreErrors: true
02_restartbeat:
command: "supervisorctl -c /opt/python/etc/supervisord.conf restart celerybeat"
leader_only: true
/var/log/eb-activity.log
[2018-05-02T13:30:06.402Z] INFO [5456] - [Application update app-v2_0-226-g83fa-180502_152745#2/AppDeployStage1/AppDeployEnactHook] : Completed activity. Result:
Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/enact.
[2018-05-02T13:30:06.402Z] INFO [5456] - [Application update app-v2_0-226-g83fa-180502_152745#2/AppDeployStage1/AppDeployPostHook] : Starting activity...
[2018-05-02T13:30:06.402Z] INFO [5456] - [Application update app-v2_0-226-g83fa-180502_152745#2/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Starting activity...
[2018-05-02T13:30:07.544Z] INFO [5456] - [Application update app-v2_0-226-g83fa-180502_152745#2/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Activity execution failed, because: [program:celeryd]
directory=/opt/python/current/app
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A celery_conf.celery_app:app --loglevel=INFO --logfile="/var/log/celery/%%n%%I.log" --pidfile="/var/run/celery/%%n.pid"
user=celery
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 60
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",APP_ENV="test",RDS_PASSWORD="XXXXXXXXXX",DJANGO_SETTINGS_MODULE="olem.settings_prod",PYCURL_SSL_LIBRARY="nss",RDS_USERNAME="testuser",RDS_DB_NAME="olemapptestingDB",RDS_HOSTNAME="olemapptesting.cw2kqid7nxi9.us-east-1.rds.amazonaws.com"
[program:celerybeat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A celery_conf.celery_app:app --loglevel=INFO --logfile="/var/log/celery/celery-beat.log" --pidfile="/var/run/celery/celery-beat.pid"
directory=/opt/python/current/app
user=celery
numprocs=1
stdout_logfile=/var/log/celerybeat.log
stderr_logfile=/var/log/celerybeat.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 60
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999
environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",APP_ENV="test",RDS_PASSWORD="XXXXXXXXXX",DJANGO_SETTINGS_MODULE="olem.settings_prod",PYCURL_SSL_LIBRARY="nss",RDS_USERNAME="testuser",RDS_DB_NAME="olemapptestingDB",RDS_HOSTNAME="olemapptesting.cw2kqid7nxi9.us-east-1.rds.amazonaws.com"
[include]
files: uwsgi.conf celery.conf celerybeat.conf
ERROR: supervisor shutting down
ERROR: already shutting down
error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 800
error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 800 (ElasticBeanstalk::ExternalInvocationError)
caused by: [program:celeryd]
directory=/opt/python/current/app
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A celery_conf.celery_app:app --loglevel=INFO --logfile="/var/log/celery/%%n%%I.log" --pidfile="/var/run/celery/%%n.pid"
user=celery
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 60
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",APP_ENV="test",RDS_PASSWORD="XXXXXXXXXX",DJANGO_SETTINGS_MODULE="olem.settings_prod",PYCURL_SSL_LIBRARY="nss",RDS_USERNAME="testuser",RDS_DB_NAME="olemapptestingDB",RDS_HOSTNAME="olemapptesting.cw2kqid7nxi9.us-east-1.rds.amazonaws.com"
[program:celerybeat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A celery_conf.celery_app:app --loglevel=INFO --logfile="/var/log/celery/celery-beat.log" --pidfile="/var/run/celery/celery-beat.pid"
directory=/opt/python/current/app
user=celery
numprocs=1
stdout_logfile=/var/log/celerybeat.log
stderr_logfile=/var/log/celerybeat.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 60
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999
environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",APP_ENV="test",RDS_PASSWORD="XXXXXXXXXX",DJANGO_SETTINGS_MODULE="olem.settings_prod",PYCURL_SSL_LIBRARY="nss",RDS_USERNAME="testuser",RDS_DB_NAME="olemapptestingDB",RDS_HOSTNAME="olemapptesting.cw2kqid7nxi9.us-east-1.rds.amazonaws.com"
[include]
files: uwsgi.conf celery.conf celerybeat.conf
ERROR: supervisor shutting down
ERROR: already shutting down
error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 800
error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 800 (Executor::NonZeroExitStatus)
[2018-05-02T13:30:07.544Z] INFO [5456] - [Application update app-v2_0-226-g83fa-180502_152745#2/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Activity failed.
[2018-05-02T13:30:07.545Z] INFO [5456] - [Application update app-v2_0-226-g83fa-180502_152745#2/AppDeployStage1/AppDeployPostHook] : Activity failed.
[2018-05-02T13:30:07.545Z] INFO [5456] - [Application update app-v2_0-226-g83fa-180502_152745#2/AppDeployStage1] : Activity failed.
[2018-05-02T13:30:07.545Z] INFO [5456] - [Application update app-v2_0-226-g83fa-180502_152745#2] : Completed activity. Result:
Application update - Command CMD-AppDeploy failed
Supervisor logs (/opt/python/log/supervisord.log)
2018-05-02 16:13:07,913 CRIT Supervisor running as root (no user in config file)
2018-05-02 16:13:07,928 INFO RPC interface 'supervisor' initialized
2018-05-02 16:13:07,928 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-05-02 16:13:07,929 INFO supervisord started with pid 2991
2018-05-02 16:13:08,931 INFO spawned: 'httpd' with pid 3076
2018-05-02 16:13:10,017 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:15:03,412 INFO stopped: httpd (exit status 0)
2018-05-02 16:15:04,418 INFO spawned: 'httpd' with pid 4568
2018-05-02 16:15:05,554 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:15:16,700 INFO waiting for httpd to die
2018-05-02 16:15:17,481 INFO stopped: httpd (exit status 0)
2018-05-02 16:15:17,485 CRIT Supervisor running as root (no user in config file)
2018-05-02 16:15:17,485 WARN Included extra file "/opt/python/etc/uwsgi.conf" during parsing
2018-05-02 16:15:17,485 INFO RPC interface 'supervisor' initialized
2018-05-02 16:15:17,485 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-05-02 16:15:17,485 INFO supervisord started with pid 2991
2018-05-02 16:15:17,873 INFO spawned: 'httpd' with pid 4826
2018-05-02 16:15:17,875 INFO spawned: 'uwsgi' with pid 4827
2018-05-02 16:15:18,509 INFO spawned: 'celeryd' with pid 4953
2018-05-02 16:15:18,511 INFO spawned: 'celerybeat' with pid 4954
2018-05-02 16:15:19,481 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:15:19,481 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:15:19,496 INFO stopped: celeryd (terminated by SIGTERM)
2018-05-02 16:15:19,835 INFO spawned: 'celeryd' with pid 4962
2018-05-02 16:15:28,859 INFO success: celerybeat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2018-05-02 16:15:29,860 INFO success: celeryd entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2018-05-02 16:15:30,307 INFO stopped: celerybeat (exit status 0)
2018-05-02 16:15:31,312 INFO spawned: 'celerybeat' with pid 4975
2018-05-02 16:15:41,326 INFO success: celerybeat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
#################### NEW LOGS AFTER eb deploy ####################
2018-05-02 16:20:51,734 INFO exited: celerybeat (terminated by SIGKILL; not expected)
2018-05-02 16:20:51,958 INFO spawned: 'celerybeat' with pid 5275
2018-05-02 16:21:52,037 WARN killing 'celerybeat' (5275) with SIGKILL
2018-05-02 16:21:52,040 INFO stopped: celerybeat (terminated by SIGKILL)
2018-05-02 16:21:53,046 INFO spawned: 'celerybeat' with pid 5301
2018-05-02 16:22:03,060 INFO success: celerybeat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2018-05-02 16:22:20,654 INFO stopped: httpd (exit status 0)
2018-05-02 16:22:21,659 INFO spawned: 'httpd' with pid 5687
2018-05-02 16:22:22,716 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:22:29,436 INFO waiting for httpd, celerybeat, uwsgi, celeryd to die
2018-05-02 16:22:33,272 INFO waiting for httpd, celerybeat, uwsgi, celeryd to die
2018-05-02 16:22:36,276 INFO waiting for httpd, celerybeat, uwsgi, celeryd to die
2018-05-02 16:22:39,279 INFO waiting for httpd, celerybeat, uwsgi, celeryd to die
2018-05-02 16:22:40,281 WARN killing 'uwsgi' (4827) with SIGKILL
2018-05-02 16:22:40,292 INFO stopped: uwsgi (terminated by SIGKILL)
2018-05-02 16:22:40,477 INFO stopped: celerybeat (exit status 0)
2018-05-02 16:22:40,554 INFO stopped: httpd (exit status 0)
2018-05-02 16:22:42,177 INFO stopped: celeryd (exit status 0)
2018-05-02 16:22:42,184 CRIT Supervisor running as root (no user in config file)
2018-05-02 16:22:42,184 WARN Included extra file "/opt/python/etc/uwsgi.conf" during parsing
2018-05-02 16:22:42,184 WARN Included extra file "/opt/python/etc/celery.conf" during parsing
2018-05-02 16:22:42,184 WARN Included extra file "/opt/python/etc/celerybeat.conf" during parsing
2018-05-02 16:22:42,185 INFO RPC interface 'supervisor' initialized
2018-05-02 16:22:42,185 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2018-05-02 16:22:42,185 INFO RPC interface 'supervisor' initialized
2018-05-02 16:22:42,185 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-05-02 16:22:42,185 INFO supervisord started with pid 2991
2018-05-02 16:22:43,187 INFO spawned: 'celeryd' with pid 5924
2018-05-02 16:22:43,189 INFO spawned: 'celerybeat' with pid 5925
2018-05-02 16:22:43,190 INFO spawned: 'uwsgi' with pid 5926
2018-05-02 16:22:43,191 INFO spawned: 'httpd' with pid 5927
2018-05-02 16:22:44,326 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:22:44,326 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:22:53,364 INFO success: celeryd entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2018-05-02 16:22:53,365 INFO success: celerybeat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
How I can solve this?
UPDATE 1
Looking at the supervisor log file, I think that the error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'> is due to a SIGKILL signal where not expected.
See line 2018-05-02 16:20:51,734 INFO exited: celerybeat (terminated by SIGKILL; not expected)
UPDATE 2
Supervisor config file (/opt/python/etc/supervisord.conf)
[unix_http_server]
file=/opt/python/run/supervisor.sock ; (the path to the socket file)
;chmod=0700 ; socket file mode (default 0700)
;chown=nobody:nogroup ; socket file uid:gid owner
[supervisord]
logfile=/opt/python/log/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=10MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/opt/python/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
directory=/opt/python/current/app ; (default is not to cd during start)
;nocleanup=true ; (don't clean up tempfiles at start;default false)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///opt/python/run/supervisor.sock
[program:httpd]
command=/opt/python/bin/httpdlaunch
numprocs=1
directory=/opt/python/current/app
autostart=true
autorestart=unexpected
startsecs=1 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
killasgroup=false ; SIGKILL the UNIX process group (def false)
redirect_stderr=false
[include]
files = uwsgi.conf
[include]
files: uwsgi.conf celery.conf celerybeat.conf
[inet_http_server]
port = 127.0.0.1:9001
[include]
files: uwsgi.conf celery.conf celerybeat.conf
So there two types of fixes for this
Just add a sleep 15 seconds to give enough time for previous supervisor instance to shutdown
Create a pre hooks script which will check if supervisor is already running and if so it will just wait for it to die. So something like /opt/elasticbeanstalk/hooks/appdeploy/pre/wait_for_supervised_to_die.sh
How to use Django with AWS Elastic Beanstalk that would also run tasks by celery on main node only?
This is how I set up celery with django on elastic beanstalk with scalability working fine.
Please keep in mind that 'leader_only' option for container_commands works only on environment rebuild or deployment of the App. If service works long enough, leader node may be removed by Elastic Beanstalk. To deal with that, you may have to apply instance protection for your leader node. Check: http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html#instance-protection-instance
Add bash script for celery worker and beat configuration.
Add file root_folder/.ebextensions/files/celery_configuration.txt:
#!/usr/bin/env bash
# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}
# Create celery configuraiton script
celeryconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A django_app --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv
[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A django_app --loglevel=INFO --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf
# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# Reread the supervisord config
supervisorctl -c /opt/python/etc/supervisord.conf reread
# Update supervisord in cache without restarting all services
supervisorctl -c /opt/python/etc/supervisord.conf update
# Start/Restart celeryd through supervisord
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker
Take care about script execution during deployment, but only on main node (leader_only: true).
Add file root_folder/.ebextensions/02-python.config:
container_commands:
04_celery_tasks:
command: "cat .ebextensions/files/celery_configuration.txt > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
leader_only: true
05_celery_tasks_run:
command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
leader_only: true
Beat is configurable without need of redeployment, with separate django applications: https://pypi.python.org/pypi/django_celery_beat.
Storing task results is good idea to: https://pypi.python.org/pypi/django_celery_beat
File requirements.txt
celery==4.0.0
django_celery_beat==1.0.1
django_celery_results==1.0.1
pycurl==7.43.0 --global-option="--with-nss"
Configure celery for Amazon SQS broker
(Get your desired endpoint from list: http://docs.aws.amazon.com/general/latest/gr/rande.html)
root_folder/django_app/settings.py:
...
CELERY_RESULT_BACKEND = 'django-db'
CELERY_BROKER_URL = 'sqs://%s:%s#' % (aws_access_key_id, aws_secret_access_key)
# Due to error on lib region N Virginia is used temporarily. please set it on Ireland "eu-west-1" after fix.
CELERY_BROKER_TRANSPORT_OPTIONS = {
"region": "eu-west-1",
'queue_name_prefix': 'django_app-%s-' % os.environ.get('APP_ENV', 'dev'),
'visibility_timeout': 360,
'polling_interval': 1
}
...
Celery configuration for django django_app app
Add file root_folder/django_app/celery.py:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'django_app.settings')
app = Celery('django_app')
# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
Modify file root_folder/django_app/__init__.py:
from __future__ import absolute_import, unicode_literals
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from django_app.celery import app as celery_app
__all__ = ['celery_app']
Check also:
How do you run a worker with AWS Elastic Beanstalk? (solution without scalability)
Pip Requirements.txt --global-option causing installation errors with other packages. "option not recognized" (solution for problems coming from obsolate pip on elastic beanstalk that cannto deal with global options for properly solving pycurl dependency)
This is how I extended the answer by #smentek to allow for multiple worker instances and a single beat instance - same thing applies where you have to protect your leader. (I still don't have an automated solution for that yet).
Please note that envvar updates to EB via the EB cli or the web interface are not relflected by celery beat or workers until app server restart has taken place. This caught me off guard once.
A single celery_configuration.sh file outputs two scripts for supervisord, note that celery-beat has autostart=false, otherwise you end up with many beats after an instance restart:
# get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}
# create celery beat config script
celerybeatconf="[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A lexvoco --loglevel=INFO --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=false
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 10
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# create celery worker config script
celeryworkerconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A lexvoco --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999
environment=$celeryenv"
# create files for the scripts
echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf
echo "$celeryworkerconf" | tee /opt/python/etc/celeryworker.conf
# add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: celerybeat.conf celeryworker.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# reread the supervisord config
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf reread
# update supervisord in cache without restarting all services
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf update
Then in container_commands we only restart beat on leader:
container_commands:
# create the celery configuration file
01_create_celery_beat_configuration_file:
command: "cat .ebextensions/files/celery_configuration.sh > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && sed -i 's/\r$//' /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
# restart celery beat if leader
02_start_celery_beat:
command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat"
leader_only: true
# restart celery worker
03_start_celery_worker:
command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker"
If someone is following smentek's answer and getting the error:
05_celery_tasks_run: /usr/bin/env bash does not exist.
know that, if you are using Windows, your problem might be that the "celery_configuration.txt" file has WINDOWS EOL when it should have UNIX EOL. If using Notepad++, open the file and click on "Edit > EOL Conversion > Unix (LF)". Save, redeploy, and error is no longer there.
Also, a couple of warnings for really-amateur people like me:
Be sure to include "django_celery_beat" and "django_celery_results" in your "INSTALLED_APPS" in settings.py file.
To check celery errors, connect to your instance with "eb ssh" and then "tail -n 40 /var/log/celery-worker.log" and "tail -n 40 /var/log/celery-beat.log" (where "40" refers to the number of lines you want to read from the file, starting from the end).
Hope this helps someone, it would've saved me some hours!
I'm using this guide to setting up an intranet server. Everything goes ok, the server works and I can checked it is working in my network.
But when I logout, I get 404 error.
The sock file is in the path indicated in gunicorn_start.
(cmi2014)javier#sgc:~/workspace/cmi/cmi$ ls -l run/
total 0
srwxrwxrwx 1 javier javier 0 mar 10 17:31 cmi.sock
Actually I can se the workers when listed the process list.
(cmi2014)javier#sgc:~/workspace/cmi/cmi$ ps aux | grep cmi
javier 17354 0.0 0.2 14652 8124 ? S 17:27 0:00 gunicorn: master [cmi]
javier 17365 0.0 0.3 18112 10236 ? S 17:27 0:00 gunicorn: worker [cmi]
javier 17366 0.0 0.3 18120 10240 ? S 17:27 0:00 gunicorn: worker [cmi]
javier 17367 0.0 0.5 36592 17496 ? S 17:27 0:00 gunicorn: worker [cmi]
javier 17787 0.0 0.0 4408 828 pts/0 S+ 17:55 0:00 grep --color=auto cmi
And supervisorctl responds that the process is running:
(cmi2014)javier#sgc:~/workspace/cmi/cmi$ sudo supervisorctl status cmi
[sudo] password for javier:
cmi RUNNING pid 17354, uptime 0:29:21
There is an error in nginx logs,
(cmi2014)javier#sgc:~/workspace/cmi/cmi$ tail logs/nginx-error.log
2014/03/10 17:38:57 [error] 17299#0: *19 connect() to
unix:/home/javier/workspace/cmi/cmi/run/cmi.sock failed (111: Connection refused) while
connecting to upstream, client: 10.69.0.174, server: , request: "GET / HTTP/1.1",
upstream: "http://unix:/home/javier/workspace/cmi/cmi/run/cmi.sock:/", host:
"10.69.0.68:2014"
Again, the error appears only when I logout or close session, but everything works fine when run or reload supervisor and stay connected.
By the way, ngnix, supervisor and gunicorn run on my uid.
Thanks in advance.
Edit Supervisor conf
[program:cmi]
command = /home/javier/entornos/cmi2014/bin/cmi_start
user = javier
stdout_logfile = /home/javier/workspace/cmi/cmi/logs/cmi_supervisor.log
redirect_stderr = true
autostart=true
autorestart=true
Gnunicor start script
#!/bin/bash
NAME="cmi" # Name of the application
DJANGODIR=/home/javier/workspace/cmi/cmi # Django project directory
SOCKFILE=/home/javier/workspace/cmi/cmi/run/cmi.sock # we will communicte using this unix socket
USER=javier # the user to run as
GROUP=javier # the group to run as
NUM_WORKERS=3 # how many worker processes should Gunicorn spawn
DJANGO_SETTINGS_MODULE=cmi.settings # which settings file should Django use
DJANGO_WSGI_MODULE=cmi.wsgi # WSGI module name
echo "Starting $NAME as `whoami`"
# Activate the virtual environment
cd $DJANGODIR
source /home/javier/entornos/cmi2014/bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH
export CMI_SECRET_KEY='***'
export CMI_DATABASE_HOST='***'
export CMI_DATABASE_NAME='***'
export CMI_DATABASE_USER='***'
export CMI_DATABASE_PASS='***'
export CMI_DATABASE_PORT='3306'
# Create the run directory if it doesn't exist
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR
# Start your Django Unicorn
# Programs meant to be run under supervisor should not daemonize themselves (do not use --daemon)
exec /home/javier/entornos/cmi2014/bin/gunicorn ${DJANGO_WSGI_MODULE}:application --name $NAME --workers $NUM_WORKERS --user=$USER --group=$GROUP --log-level=debug --bind=unix:$SOCKFILE