Celery wont run when initiated by supervisor on aws elasticbeanstalk - django

Question

Update!
Got it to work by adding export LC_ALL=en_US.UTF-8 at the top of my bash script.
I am attempting to daemonize celery beat and worker. I have no troubles running celery or beat when I ssh into my Elastic Beanstalk instance and do the following steps:
cd /opt/python/current/app
/opt/python/run/venv/bin/celery -A myDjangoApp beat --loglevel=INFO
/opt/python/run/venv/bin/celery -A myDjangoApp worker --loglevel=INFO
Tasks are scheduled and able to execute with ease. However when I run the exact same commands with supervisor I am getting an uniformative error. From looking at supervisorctl status I see the process runs for a few seconds then fails. Upon further examination the the log files show me the following error:
File "/opt/python/run/venv/bin/celery", line 11, in <module>
sys.exit(main())
File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/__main__.py", line 15, in main
sys.exit(_main())
File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/celery.py", line 213, in main
return celery(auto_envvar_prefix="CELERY")
File "/opt/python/run/venv/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/click/core.py", line 760, in main
_verify_python3_env()
File "/opt/python/run/venv/local/lib/python3.6/site-packages/click/_unicodefun.py", line 130, in _verify_python3_env
" mitigation steps.{}".format(extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/python3/ for mitigation steps.
Listed below are my supervisor.conf and celery.sh(runs celery) files.
Following is my supervisor.conf file.
[unix_http_server]
file=/opt/python/run/supervisor.sock ; (the path to the socket file)
chmod=0777 ; socket file mode (default 0700)
;chown=nobody:nogroup ; socket file uid:gid owner
[supervisord]
logfile=/opt/python/log/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=10MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/opt/python/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
directory=/opt/python/current/app ; (default is not to cd during start)
;nocleanup=true ; (don't clean up tempfiles at start;default false)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///opt/python/run/supervisor.sock
[program:httpd]
command=/opt/python/bin/httpdlaunch
numprocs=1
directory=/opt/python/current/app
autostart=true
autorestart=unexpected
startsecs=1 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
killasgroup=false ; SIGKILL the UNIX process group (def false)
redirect_stderr=false
;[program:celeryWorker]
;user=root
;command=/opt/python/run/venv/bin/celery -A Daash worker --loglevel=INFO --daemon
;directory=/opt/python/current/app
;numprocs=1
;autostart=true
;autorestart=true
;startsecs=0
;stopwaitsecs=60
;killasgroup=true
[program:celeryBeat]
user=root
command=/opt/python/etc/celery.sh
directory=/opt/python/current/app
numprocs=1
autostart=true
autorestart=true
startsecs=0
stopwaitsecs=60
killasgroup=true
stdout_logfile = /opt/python/log/cel_stdout.log
stderr_logfile = /opt/python/log/cel.log
Following is my celery.sh file.I have mirrored the exact steps I run when manually running celery in the .sh file.
#!/usr/bin/env bash
cd /opt/python/current/app
export PATH=/opt/python/run/venv/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin
sudo /opt/python/run/venv/bin/celery -A myDjangoApp beat --loglevel=INFO

Answer 1

Judging by the stack trace you provided, the error happens inside the Click dependency, associated with an encoding limitation.
Click's documentation explains this error a bit further: https://click.palletsprojects.com/en/8.0.x/unicode-support/?highlight=ascii
The fact that you were able to run it interactively is because your user session has the locale and other environment variables all correctly set, which the root session created by supervisord probably does not.
In any case, setting these variables within your celery.sh script should do the trick:
export LC_ALL=C.UTF-8
export LANG=C.UTF-8