setting up django & celery: location for celery path - django

I'm setting up celery based on an example and at this point...
$ export PYTHONPATH=/webapps/hello_django/hello:$PYTHONPATH
$ /webapps/hello_django/bin/celery --app=hello.celery:app worker --loglevel=INFO
on my end set as
samuel#samuel-pc:~/Documents/code/revamp$ export PYTHONPATH=/home/samuel/Documents/code/revamp/gallery:$PYTHONPATH
samuel#samuel-pc:~/Documents/code/revamp$ /home/samuel/Documents/code/revamp/revamp/celery --app=revamp.celery:app worker --loglevel=INFO
bash: /home/samuel/Documents/code/revamp/revamp/celery: No such file or directory
not sure what it did to the path and this is what the result should be
-------------- celery#django v3.1.11 (Cipater)
---- **** -----
--- * *** * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.5
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: hello_django:0x15ae410
- ** ---------- .> transport: redis://localhost:6379/0
- ** ---------- .> results: disabled
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
[tasks]
. testapp.tasks.test
[2014-05-20 13:53:59,740: INFO/MainProcess] Connected to redis://localhost:6379/0
[2014-05-20 13:53:59,748: INFO/MainProcess] mingle: searching for neighbors
[2014-05-20 13:54:00,756: INFO/MainProcess] mingle: all alone
[2014-05-20 13:54:00,769: WARNING/MainProcess] celery#django ready.
my guess is I need to set path to the path for celery installation, if so anyone who can tell me the path.

Had lots of headeaches with celery tutorials out there. Try this:
First thing you need a virtual enviroment for your project, so you dont need to setup paths.
bash:
sudo pip3 install virtualenv
virtualenv env
source env/bin/activate
Then you need a django project and an app.
bash:
pip install django
django-admin starproject myproject
cd myproject
python manage.py startapp myapp
Then you should pip install celery
Next make a tasks.py on the same level of your view.py on myapp directory:
tasks.py
from celery import Celery
from celery.decorators import task
app = Celery('tasks', broker='pyamqp://guest#localhost//')
#task(bind=True,name="my_task")
def my_task(self):
print('hello')
return 1+1
Install your broker (rabbitmq)
bash:
sudo apt-get install rabbitmq-server
sudo service rabbitmq-server restart
Go to your app directory, the one that has tasks.py and run celery -A tasks worker --loglevel=info. This only works on the directory that tasks is defined. Then you should have your worker up and runnning. When you print or return something from your task it should appear here.
Finally you should use your task. Setup a view(setup url, make a template, etc.) and call your task from the view:
views.py
from django.shortcuts import render
from .tasks import my_task
def index(request):
my_task.delay()
return render({},'index.html')
The magic is that the delay call is assync, non-blockable.
From this minimalistic example I hope you can understand better the paths you tutorials talk about and do complicated stuff that they make you do: like placing celery settings on settings.py, calling the worker from other directories, putting things on path... This was a real pain when I was trying to learn this from the oficial docs.
Good luck!

Related

Celery and Django on production machine

/opt/fubar/fubar/celery.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'fubar.settings')
app = Celery('fubar')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
/opt/fubar/fubar/init.py
from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app
__all__ = ('celery_app', )
/etc/supervisor/conf.d/celeryworker.conf
[program:fubar-celery]
command=/opt/fubar/env/bin/celery worker -A fubar --loglevel=INFO
directory=/opt/fubar
user=www-data
numprocs=1
stdout_logfile=/var/log/celery/fubar/worker.log
stderr_logfile=/var/log/celery/fubar/worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600
killasgroup=true
priority=998
$ service rabbitmq-server status
● rabbitmq-server.service - RabbitMQ Messaging Server
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2020-07-26 06:19:47 UTC; 19h ago
Main PID: 21884 (rabbitmq-server)
Tasks: 92 (limit: 4915)
CGroup: /system.slice/rabbitmq-server.service
├─21884 /bin/sh /usr/sbin/rabbitmq-server
├─21905 /bin/sh /usr/lib/rabbitmq/bin/rabbitmq-server
├─22073 /usr/lib/erlang/erts-9.2/bin/epmd -daemon
├─22185 /usr/lib/erlang/erts-9.2/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -B i
├─22299 erl_child_setup 65536
├─22372 inet_gethost 4
└─22373 inet_gethost 4
$ celery worker -A fubar --loglevel=INFO on localhost returns
[tasks]
. fubar.celery.debug_task
. apps.raptor.tasks.launchraptor
. apps.raptor.tasks.nolaunch
while I see no tasks in the log file in production
Apache error log shows:
mod_wsgi (pid=26854): Exception occurred processing WSGI script '/var/www/fubar/fubar.wsgi'.
...
celery.exceptions.NotRegistered: 'apps.raptor.tasks.launchraptor'
I installed supervisor with pip install supervisor to get v4.2.0
What can I run to test whether things are configured properly?
Why is celery worker started with supervisor not finding the tasks that show up when run as celery worker.
I got rid of RabbitMQ and moved to redis. I had more success installing/configuring redis. Doesn't answer the question asked.
Observation was that supervisor installed with pip install supervisor doesn't work and does when done via apt install supervisor. I don't know why.
tail -f /var/log/celery/fubar/worker.log is the best way to see what's going on with the worker.
celery inspect ... also works for a snapshot of whats happening
I had to change command= in the conf.d/fubar*.conf files to point to a shell script which worked better than calling celery from the conf file itself. Also, shell scripts must be owned by the user=value and set to +x.

celery task not processed by celery

total celery and django noob here, so sorry if the problem is trivial. Basically the problem is that any function defined by #app.task is not being processed by celery, it just runs normally as if celery isn't there.
My celery_app.py file is -
from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
app = Celery(broker=settings.CELERY_BROKER_URL)
app.config_from_object('django.conf:settings')
app.autodiscover_tasks()
if __name__ == '__main__':
app.start()
While my tasks.py file is -
from project.celery_app import app
#app.task
def mytask():
...
I get the following output on running celery in the terminal -
-------------- celery#LAPTOP v4.1.0 (latentcall)
---- **** -----
--- * *** * -- Windows-10-10.0.16299-SP0 2017-12-20 19:27:24
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: __main__:0x229ce2884e0
- ** ---------- .> transport: amqp://user:**#localhost:5672/myvhost
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 8 (solo)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. account.tasks.mytask
[2017-12-20 19:27:24,085: INFO/MainProcess] Connected to amqp://user:**#127.0.0.1:5672/myvhost
[2017-12-20 19:27:24,101: INFO/MainProcess] mingle: searching for neighbors
[2017-12-20 19:27:25,126: INFO/MainProcess] mingle: all alone
[2017-12-20 19:27:25,141: WARNING/MainProcess] c:\programdata\anaconda2\envs\myenv\lib\site- packages\celery\fixups\django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2017-12-20 19:27:25,141: INFO/MainProcess] celery#LAPTOP- ready.
So my task is known to celery, but it doesn't do anything about it. The task runs on a button click, and using -loglevel=debug it is seen that celery isn't affected by it. I am using RabbitMQ as broker, celery 4.1.0, python3 and django-1.10.5. Any help would be greatly appreciated!
As I had thought, a simple mistake. Just needed to change mytask() to mytask.delay() and celery started receiving it.
.delay() is actually a shorcut method. If want to provide additional options you have to use .apply_async()
official doc can be found here:
http://docs.celeryproject.org/en/latest/userguide/calling.html

Django - Celery 4.1 with django-celery-beat/rabbitmq : Nothing?

I followed the tutorial on http://docs.celeryproject.org/en/latest/ and I am on virtualbox (Xubuntu 16.XX TLS), Django 1.11.3, Celery 4.1 . rabbitmq 3.6.14, Python 2.7 .
and when I started the daemonization with the init-script: celerybeat (with /etc/default/celeryd config file)
[2017-11-19 01:13:00,912: INFO/MainProcess] beat: Starting...
and nothing more after. Do you see what could I make wrong ?
My celery.py:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'oscar.settings')
app = Celery('oscar')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Broker settings
app.conf.broker_url = 'amqp://oscar:oscar#localhost:5672/oscarRabbit'
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
some_app/tasks.py:
from __future__ import absolute_import, unicode_literals
from oscar import celery_app
from celery.schedules import crontab
from .models import HelpRequest
from datetime import datetime, timedelta
import logging
""" CONSTANTS FOR THE TIMER """
# Can be changed (by default 1 week)
WEEKS_BEFORE_PENDING = 0
DAYS_BEFORE_PENDING = 0
HOURS_BEFORE_PENDING = 0
MINUTES_BEFORE_PENDING = 1
# http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html
# for schedule : http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#crontab-schedules
#celery_app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
sender.add_periodic_task(
crontab(minute=2),
set_open_help_request_to_pending
)
#celery_app.task(name="HR_OPEN_TO_PENDING")
def set_open_help_request_to_pending():
""" For timedelay idea : https://stackoverflow.com/a/27869101/6149867 """
logging.info("RUNNING CRON TASK FOR STUDENT COLLABORATION : set_open_help_request_to_pending")
request_list = HelpRequest.objects.filter(
state=HelpRequest.OPEN,
timestamp__gte=datetime.now() - timedelta(hours=HOURS_BEFORE_PENDING,
minutes=MINUTES_BEFORE_PENDING,
days=DAYS_BEFORE_PENDING,
weeks=WEEKS_BEFORE_PENDING)
)
if request_list:
logging.info("FOUND ", request_list.count(), " Help request(s) => PENDING")
for help_request in request_list.all():
help_request.change_state(HelpRequest.PENDING)
/etc/default/celeryd:
# Names of nodes to start
# most people will only start one node:
CELERYD_NODES="worker1"
# but you can also start multiple and configure settings
# for each in CELERYD_OPTS
#CELERYD_NODES="worker1 worker2 worker3"
# alternatively, you can specify the number of nodes to start:
#CELERYD_NODES=10
# Absolute or relative path to the 'celery' command:
CELERY_BIN="/home/jy95/Documents/oscareducation/ve/local/bin/celery"
# App instance to use
# comment out this line if you don't use an app
CELERY_APP="oscar"
# Where to chdir at start.
CELERYD_CHDIR="/home/jy95/Documents/oscareducation"
# Extra command-line arguments to the worker
# django_celery_beat for admin purpuse
CELERYD_OPTS="--scheduler django_celery_beat.schedulers:DatabaseScheduler -f /var/log/celery/celery_tasks.log"
# Set logging level to DEBUG
#CELERYD_LOG_LEVEL="DEBUG"
# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.
# You need to create this user manually (or you can choose
# a user/group combination that already exists (e.g., nobody).
CELERYD_USER="celery"
CELERYD_GROUP="celery"
# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1
My setup of rabbitmq :
$ sudo rabbitmqctl add_user oscar oscar
$ sudo rabbitmqctl add_vhost oscarRabbit
$ sudo rabbitmqctl set_user_tags oscar administrator
$ sudo rabbitmqctl set_permissions -p oscarRabbit oscar ".*" ".*" ".*"
The commands I run to start (and their messages) :
sudo rabbitmq-server -detached
sudo /etc/init.d/celerybeat start
Warning: PID file not written; -detached was passed.
/etc/init.d/celerybeat: lerybeat: not found
celery init v10.1.
Using configuration: /etc/default/celeryd
Starting celerybeat...
sudo /etc/init.d/celerybeat start
source ve/bin/activate
python manage.py runserver
Performing system checks...
System check identified no issues (0 silenced).
November 19, 2017 -01:49:22 Django version 1.11.3, using settings 'oscar.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Thanks for your answer
It looks like you've started a celerybeat process and your server, but haven't started a celery worker process.
python celery -A proj worker -B
(where proj is the name of your project).
Note that you can start a celery worker with an embedded beat process rather than needing to run celerybeat separately.

Django Celery Error

I'm been trying to schedule a task for celery to do at the background but I keep encountering issues.
Quick reference on what I've done so far
I installed celery and django-celery via easy_install, added to INSTALLED_APPS:
'djcelery',
'kombu.transport.django',
imported the following and ran syncdb:
import djcelery
djcelery.setup_loader()
BROKER_URL = "django://"
The task I'm trying to run in the background is password reset. So when a user forget his password , I want to make the sending email task run in the background so what I done is
I moved my forgot_password function from views.py into tasks.py so it can run.
My tasks.py:
from django.contrib.auth.views import password_reset
from django.shortcuts import render
from celery.decorators import task
#task()
def forgot_password(request):
if request.method == 'POST':
return password_reset(request,
from_email=request.POST.get('email'))
else:
return render(request, 'forgot_password.html')
Their nothing in views.py now.
The problem is even though, I can send email if I lose my password. I'm not sure if it's running in the background. What I did to check is was:
manage.py celery worker --loglevel=info
but I get an error: KeyError: 'processName':
C:\o\17\mysite>manage.py celery worker --loglevel=info
-------------- celery#gg-PC v3.0.19 (Chiastic Slide)
---- **** -----
--- * *** * -- Windows-Vista-6.0.6001-SP1
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> broker: django://localhost//
- ** ---------- .> app: default:0x319c930 (djcelery.loaders.DjangoLoader
)
- ** ---------- .> concurrency: 2 (processes)
- *** --- * --- .> events: OFF (enable -E to monitor this worker)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery: exchange:celery(direct) binding:celery
[Tasks]
. accounts.tasks.forgot_password
[2013-05-15 17:49:45,279: WARNING/MainProcess] C:\Python26\lib\site-packages\dja
ngo_celery-3.0.17-py2.6.egg\djcelery\loaders.py:133: UserWarning: Using settings
.DEBUG leads to a memory leak, never use this setting in production environments
!
warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2013-05-15 17:49:45,292: WARNING/MainProcess] celery#gg-PC ready.
[2013-05-15 17:49:45,292: INFO/MainProcess] consumer: Connected to django://loca
lhost//.
Traceback (most recent call last):
File "C:\Python26\lib\logging\__init__.py", line 754, in emit
msg = self.format(record)
File "C:\Python26\lib\logging\__init__.py", line 637, in format
return fmt.format(record)
File "C:\Python26\lib\logging\__init__.py", line 428, in format
s = self._fmt % record.__dict__
KeyError: 'processName'
Traceback (most recent call last):
File "C:\Python26\lib\logging\__init__.py", line 754, in emit
msg = self.format(record)
File "C:\Python26\lib\logging\__init__.py", line 637, in format
return fmt.format(record)
File "C:\Python26\lib\logging\__init__.py", line 428, in format
s = self._fmt % record.__dict__
KeyError: 'processName'
Can someone please kindly help me by tell me if forget password is configured probably to send emails in the background and why do I get this error KeyError: 'processName'?
First of all there is a simple way of sending emails using celery:
django-celery-email
Second, your task is wrong. Task is a background job, not view. It should perform the final operation. In your case this is sending email

Django and Celery: Tasks not imported

I am using Django with Celery to run two tasks in the background related to contacts/email parsing.
Structure is:
project
/api
/core
tasks.py
settings.py
settings.py file contains:
BROKER_URL = 'django://'
BROKER_BACKEND = "djkombu.transport.DatabaseTransport"
#celery
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
BROKER_VHOST = "/"
sys.path.append(os.path.dirname(os.path.basename(__file__)))
CELERY_IMPORTS = ['project.core.tasks']
import djcelery
djcelery.setup_loader()
# ....
INSTALLED_APPS = (
#...
'kombu.transport.django',
'djcelery',
)
tasks.py contains:
from celery.task import Task
from celery.registry import tasks
class ParseEmails(Task):
#...
class ImportGMailContactsFromGoogleAccount(Task):
#...
tasks.register(ParseEmails)
tasks.register(ImportGMailContactsFromGoogleAccount)
Also, added in wsgi.py
os.environ["CELERY_LOADER"] = "django"
Now, I have this app hosted on a WebFactional server. On my localhost this runs fine, but on the WebFaction server, where the Django app is deployed on a Apache server, I get:
2013-01-23 17:25:00,067: ERROR/MainProcess] Task project.core.tasks.ImportGMailContactsFromGoogleAccount[df84e03f-9d22-44ed-a305-24c20407f87c] raised exception: Task of kind 'project.core.tasks.ImportGMailContactsFromGoogleAccount' is not registered, please make sure it's imported.
But the tasks show up as registered. If I run
python2.7 manage.py celeryd -l info
I obtain:
-------------- celery#web303.webfaction.com v3.0.13 (Chiastic Slide)
---- **** -----
--- * *** * -- [Configuration]
-- * - **** --- . broker: django://localhost//
- ** ---------- . app: default:0x1e55350 (djcelery.loaders.DjangoLoader)
- ** ---------- . concurrency: 8 (processes)
- ** ---------- . events: OFF (enable -E to monitor this worker)
- ** ----------
- *** --- * --- [Queues]
-- ******* ---- . celery: exchange:celery(direct) binding:celery
--- ***** -----
[Tasks]
. project.core.tasks.ImportGMailContactsFromGoogleAccount
. project.core.tasks.ParseEmails
I thought it could be a relative import error, but I assumed the changes in settings.py and wsgi.py would prevent that.
I am thinking the multiple Python version supported by WebFactional could have to do with this, however I installed all the libraries for Python 2.7 and I am also running Django for 2.7, so there should be no problem with that.
Running in localhost using celeryd -l info the Tasks also show up in the list when I start the worker but it doesn't output the error when I call the task - it runs perfectly.
Thank you
I had the same issue in a new Ubuntu 12.04 / Apache / mod_wsgi / Django 1.5 / Celery 3.0.13 production environment. Everything works fine on my Mac Os X 10.8 laptop and my old server (which has Celery 3.0.12), but not on the new server.
It seems there is some issue in Celery:
https://github.com/celery/celery/issues/1150
My initial solution was changing my Task class based task to #task decorator based, from something like this:
class CreateInstancesTask(Task):
def run(self, pk):
management.call_command('create_instances', verbosity=0, pk=pk)
tasks.register(CreateInstancesTask)
to something like this:
#task()
def create_instances_task(pk):
management.call_command('create_instances', verbosity=0, pk=pk)
Now this task seems to work, but of course I've to do some further testing...