I'm trying to make use of periodic tasks but can't make it work.
I have this test task
# handler/tasks.py
from celery import Celery
app = Celery()
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
# Calls test('hello') every 2 seconds.
sender.add_periodic_task(2, test.s('hello'), name='add every 2')
#app.task
def test(arg):
print(arg)
Celery is configured
# project dir
# salaryx_django/celery.py
from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'salaryx_django.settings')
app = Celery('salaryx_django')
# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
# salaryx_django/settings.py
# CELERY STUFF
BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Europe/London'
The workers are intiated
[2022-04-25 14:57:55,424: INFO/MainProcess] Connected to redis://localhost:6379//
[2022-04-25 14:57:55,426: INFO/MainProcess] mingle: searching for neighbors
[2022-04-25 14:57:56,433: INFO/MainProcess] mingle: all alone
[2022-04-25 14:57:56,452: WARNING/MainProcess] /Users/jonas/Desktop/salaryx_django/venv/lib/python3.8/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
leak, never use this setting in production environments!
warnings.warn('''Using settings.DEBUG leads to a memory
[2022-04-25 14:57:56,453: INFO/MainProcess] celery#Air-von-Jonas ready.
and Redis is waiting for connections
but nothing happens at all..
Celery Beat
(venv) jonas#Air-von-Jonas salaryx_django % celery -A salaryx_django beat
celery beat v5.2.6 (dawn-chorus) is starting.
__ - ... __ - _
LocalTime -> 2022-04-26 05:38:27
Configuration ->
. broker -> redis://localhost:6379//
. loader -> celery.loaders.app.AppLoader
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule
. logfile -> [stderr]#%WARNING
. maxinterval -> 5.00 minutes (300s)
You also have to run beat.
From beat entries documentation
The add_periodic_task() function will add the entry to the beat_schedule setting behind the scenes
Simply running celery -A salaryx_django beat in another process should get you going. Read docs for more info.
Related
I am using Celery beat to perform a task that is supposed to be executed at on specific time. I was trying to excute it now by changing the time just to see if it works correctly. What I have noticed is it sends the task correctly when I run a fresh command that is celery -A jgs beat -l INFO but then suppose I change the time in the schedule section from two minutes or three minutes from now and then again run the above command, beat does not send the task. Then I noticed something strange. If I go to the admin area and delete all the other old tasks that were created in the crontab table, and then run the command again it sends the task again to the worker.
The tasks are being traced by the worker correctly and also the celery worker is working correctly. Below are the codes that I wrote to perform the task.
celery.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from django.conf import settings
from celery.schedules import crontab
from django.utils import timezone
from datetime import timezone
# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'jgs.settings')
app = Celery('jgs')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.conf.enable_utc = False
app.conf.update(timezone = 'Asia/Kolkata')
# app.conf.update(BROKER_URL=os.environ['REDIS_URL'],
# CELERY_RESULT_BACKEND=os.environ['REDIS_URL'])
app.config_from_object('django.conf:settings', namespace='CELERY')
# Celery beat settings
app.conf.beat_schedule = {
'send-expiry-email-everyday': {
'task': 'control.tasks.send_expiry_mail',
'schedule': crontab(hour=1, minute=5),
}
}
# Load task modules from all registered Django apps.
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print(f'Request: {self.request!r}')
control/tasks.py
from celery import shared_task
from django.core.mail import message, send_mail, EmailMessage
from django.conf import settings
from django.template.loader import render_to_string
from datetime import datetime, timedelta
from account.models import CustomUser
from home.models import Contract
#shared_task
def send_expiry_mail():
template = render_to_string('expiry_email.html')
email = EmailMessage(
'Registration Successfull', #subject
template, # body
settings.EMAIL_HOST_USER,
['emaiid#gmail.com'], # sender email
)
email.fail_silently = False
email.content_subtype = 'html' # WITHOUT THIS THE HTML WILL GET RENDERED AS PLAIN TEXT
email.send()
return "Done"
settings.py
############# CELERY SETTINGS #######################
CELERY_BROKER_URL = 'redis://127.0.0.1:6379'
# CELERY_BROKER_URL = os.environ['REDIS_URL']
CELERY_ACCEPT_CONTENT =['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Asia/Kolkata'
CELERY_RESULT_BACKEND = 'django-db'
# CELERY BEAT CONFIGURATIONS
CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'
commands that I am using
for worker
celery -A jgs.celery worker --pool=solo -l info
for beat
celery -A jgs beat -l INFO
Please correct me where I going wrong or what I am writing wrong, I completely in beginer phase in this async part.
I am really sorry if my sentences were confusing above.
I configured a Celery instance like this:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
app = Celery(
'project',
backend='rpc://',
broker='pyamqp://',
result_backend = 'rpc://'
)
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print(f'Request: {self.request!r}')
I have a task:
app = Celery('spotifycluster',broker='pyamqp://guest#localhost//')
#app.task
def some_task(X):
time.sleep(2)
return sum(X)
When I call the task and check its state, it's always 'PENDING'.
task = some_task.delay(features)
task_id = task.task_id
state = AsyncResult(id=task_id).state
But the terminal shows:
[2021-06-30 16:11:00,072: INFO/MainProcess] Task spotify.tasks.AffinityPropagation_task[09a1812b-1044-480a-a0fb-49be2e5cdc94] received
[2021-06-30 16:11:02,077: INFO/ForkPoolWorker-2] Task spotify.tasks.AffinityPropagation_task[09a1812b-1044-480a-a0fb-49be2e5cdc94] succeeded in 2.002569585999993s: 4
Which is confusing to me. I read other issues, but those were mostly related to a bug on Windows. I'm running on Mac. What am I missing here? Suggestions are much appreciated.
Try to change to another broker. In my case nothing helped until I totally moved to Redis from RabbitMQ. The setting below is for Django running on Heroku.
CELERY_RESULT_BACKEND = os.environ.get('REDIS_URL', 'redis://localhost:6379/0')
CELERY_BROKER_URL = os.environ.get('REDIS_URL', 'redis://localhost:6379/0')
Trying to get a celery-based scraper up and running. The celery worker seems to function on its own, but when I also run the celery beat server, the worker gives me this keyerror.
File "c:\users\myusername\.virtualenvs\django-news-scraper-dbqk-dk5\lib\site-packages\celery\worker\consumer\consumer.py", line 555, in on_task_received
strategy = strategies[type_]
KeyError: 'core.tasks.scrape_dev_to'
[2020-10-04 16:51:41,231: ERROR/MainProcess] Received unregistered task of type 'core.tasks.scrape_dev_to'.
The message has been ignored and discarded.
I've been through many similar answers on stackoverflow, but none solved my problem. I'll list things I tried at the end.
Project structure:
core -tasks
newsscraper -celery.py -settings.py
tasks:
import time
from newsscraper.celery import shared_task, task
from .scrapers import scrape
#task
def scrape_dev_to():
URL = "https://dev.to/search?q=django"
scrape(URL)
return
settings.py:
INSTALLED_APPS = [
'django.contrib.admin',
...
'django_celery_beat',
'core',
]
...
# I Added this setting while troubleshooting, got a new ModuleNotFound error for core.tasks
#CELERY_IMPORTS = (
# 'core.tasks',
#)
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_BEAT_SCHEDULE = {
"ScrapeStuff": {
'task': 'core.tasks.scrape_dev_to',
'schedule': 10 # crontab(minute="*/30")
}
}
celery.py:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'newsscraper.settings')
app = Celery('newsscraper')
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
When I run debug for the celery worker, I see that celery doesn't have the task I want (scrape_dev_to) registered. Shouldn't the app.autodiscover_tasks() call in celery.py take care of this? Here's the output:
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
I also get a ModuleNotFoundError when I try to add core.tasks to a CELERY_IMPORTS setting. This is my best guess for where the problem is, but I don't know how to solve it.
Things I tried:
Add core.tasks to a celery_imports setting. This causes a new error when I try to run the celery beat: ‘no module named ‘core.tasks’ ‘.
Hardcoding the name in the task: name='core.tasks.scrape_dev_to'
Specified the celery config explicitly when calling the worker: celery -A newsscraper worker -l INFO -settings=celeryconfig
Playing with my imports (from newsscraper.celery instead of from celery, for instance)
Adding some config code to the init.py for the module containing tasks (already had it in the init.py for module containing settings and celery.py)
Python manage.py check identified no issues
Calling the work with core.tasks explicitly: celery -A core.tasks worker -l INFO
I had the same problem and this setup solved it for me.
in your settings
CELERY_IMPORTS = [
'app_name.tasks',
]
and
# app_name/tasks.py
from celery import shared_task
#shared_task
def my_task(*args, **kwargs):
pass
Docs ref for imports.
This can occur when you configured a celery task and then removed it.
Just deconfigure the tasks and configure again
$ celery -A proj purge
or
from proj.celery import app
app.control.purge()
In settings.py, I have added the below line:
CELERY_IMPORTS = [
'app_name.tasks',]
and it worked for me.
I have found this question and tried all recommendations as well as the accepted answer. Still no luck.
Here is my mysite/taskapp/celery.py:
import os
from celery import Celery
from django.apps import AppConfig
from django.conf import settings
if not settings.configured:
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.local') # pragma: no cover
app = Celery('mysite')
class CeleryConfig(AppConfig):
name = 'mysite.taskapp'
verbose_name = 'Celery Config'
def ready(self):
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request)) # pragma: no cover
Here is my mysite's __init__.py:
from mysite.taskapp.celery import app as celery_app
__all__ = ['celery_app']
Here is an example mysite/myapp/tasks.py:
from celery import shared_task
#shared_task
def mytask():
print('Do something')
When running the celery worker with celery -A mysite worker -l info here is the output:
celery#tim-office v4.1.0 (latentcall)
Darwin-16.7.0-x86_64-i386-64bit 2018-02-27 21:38:55
[config]
.> app: mysite:0x1058e0fd0
.> transport: amqp://guest:**#localhost:5672//
.> results: disabled://
.> concurrency: 4 (prefork)
.> task events: OFF (enable -E to monitor tasks in this worker)
[queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. mysite.taskapp.celery.debug_task
Note that the mysite/myapp/tasks.py tasks are not found. I've been spinning my wheels for a full day trying all kinds of things from adding the app to CELERY_IMPORTS in the settings file:
CELERY_IMPORTS = ('myapp.tasks',)
To trying to force them into autodiscover_tasksline in the abovemysite/taskapp/celery.py`:
app.autodiscover_tasks(['myapp',])
Where am I going wrong here? Any help is greatly appreciated.
UPDATE: Possibly worth noting I am using the project structure of cookiecutter-django.
I would comment but don't have 50 reputation.
Do you have logging setup? If so, temporarily disable logging and try to start celery again.
It's possible that there's an uncaught error when Django starts. In my case celery didn't have the correct permissions to a log file.
I had the same problem, but since I solved it safely, I will respond.
My environment is as follows.
Django 4.0
Celery 2.0
First of all, I will show you the sample code. (I am sorry, the language is Japanese.
Sample Code
The main points for making Django recognize tasks are as follows.
In order for Server to recognize Task, app.autodiscover_tasks () is called only once when loading settings.py specified byDJANGO_SETTINGS_MODULE. It does not need to be called on other sub applications.
When starting worker and scheduler, specify DJANGO_SETTINGS_MODULE.
# Server
python3 manage.py runserver 0.0.0.0:8000
# Worker
DJANGO_SETTINGS_MODULE=tm.settings celery -A tm worker
# Scheduler
DJANGO_SETTINGS_MODULE=tm.settings celery -A tm beat --scheduler django_celery_beat.schedulers:DatabaseScheduler
With these two, Django side should recognize the task. Please see Sample Code for details.
References:
http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html
https://github.com/celery/django-celery-beat
I had the same/a similar issue when using Django 2 and Celery 4.1. It appears that the AppConfig messes up some of the auto-configuration with Celery. I found two solutions:
Don't pass in settings.INSTALLED_APPS to the app.autodiscover_tasks function. This may mess something else up down the road, but haven't run into it yet
Do pass it in, but include an array with just the names of the apps as pointed out here https://stackoverflow.com/a/39129453/502572 (looks you like you mentioned a similar approach as a comment)
Here's my <project_name>/<project_name>/celery.py file:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<project_name>.settings')
app = Celery('<project_name>')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
if __name__ == '__main__':
app.start()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
My __init__.py and <project_name>/<app_name>/tasks.py is setup the same as yours.
I have to use Celery 4.0.2 with RabbitMQ 3.6.10 to handle an async task. Then, I have followed this tutorial: https://www.codementor.io/uditagarwal/asynchronous-tasks-using-celery-with-django-du1087f5k
However, I have a slight problem with my task because it's impossible to have a result. My task always remains in "PENDING" state.
My question is what i have to do to get a result ?
Thank you in advance for your answer.
Here my code:
Here my __init_.py:
from __future__ import absolute_import, unicode_literals
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
__all__ = ['celery_app']
Here a part of my setting.py:
BROKER_URL = 'amqp://guest:guest#localhost//'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
Here my celery.py:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')
app = Celery('mysite',
backend='amqp',
broker='amqp://guest#localhost//')
# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
And my tasks.py
# Create your tasks here
from __future__ import absolute_import, unicode_literals
from celery import shared_task
#shared_task
def add(x, y):
test = "ok"
current_task.update_state(state='PROGRESS',
meta={'test': ok})
return x + y
And here my Django Shell:
>>> from blog.tasks import *
>>> job = add.delay(2,3)
>>> job.state
'PENDING'
>>> job.result
>>>
With a picture of my RabbitMQ interface:
You need to start a worker that will process the tasks you add to queue. From your virtualenv run:
celery worker -A blog -l info