I'm running multiple simulations as tasks through celery (version 2.3.2) from django. The simulations get set up by another task:
In views.py:
result = setup_simulations.delay(parameters)
request.session['sim'] = result.task_id # Store main task id
In tasks.py:
#task(priority=1)
def setup_simulations(parameters):
task_ids = []
for i in range(number_of_simulations):
result = run_simulation.delay(other_parameters)
task_ids.append(result.task_id)
return task_ids
After the initial task (setup_simulations) has finished, I try to revoke the simulation tasks as follows:
main_task_id = request.session['sim']
main_result = AsyncResult(main_task_id)
# Revoke sub tasks
from celery.task.control import revoke
for sub_task_id in main_result.get():
sub_result = AsyncResult(sub_task_id); sub_result.revoke() # Does not work
# revoke(sub_task_id) # Does not work neither
When I look at the output from "python manage.py celeryd -l info", the tasks get executed as if nothing had happened. Any ideas somebody what could have gone wrong?
As you mention in the comment, revoke is a remote control command so it's only currently supported by the amqp and redis transports.
You can accomplish this yourself by storing a revoked flag in your database, e.g:
from celery import states
from celery import task
from celery.exceptions import Ignore
from myapp.models import RevokedTasks
#task
def foo():
if RevokedTasks.objects.filter(task_id=foo.request.id).count():
if not foo.ignore_result:
foo.update_state(state=states.REVOKED)
raise Ignore()
If your task is working on some model you could even store a flag in that.
Related
I am setting up multiple tasks in my tasks.py, and calling the tasks from views.py. I want to invoke all the different tasks in a for loop so I can collect the status easily and build a progress bar. I am able to invoke all the tasks line by line currently (as shown below).
Here are my questions: how do I invoke the different tasks from views.py in a for loop? it always give me an error of "unicode does not have attribute delay()". Or is there a better way to collect the statuses of different tasks and build the progress bar from them?
I have tried to invoke the functions in views.py like this:
for i in range (1, 6):
functionName = "calculation" + str(i)
functionName.delay(accountNumber)
But this gives an error as stated above "unicode does not have attribute delay()"
my guess is that the tasks are imported from tasks.py to views.py
my current tasks.py:
#shared_task
def calculation1(arg):
some action here
#shared_task
def calculation2(arg):
some action here
#shared_task
def calculation3(arg):
some action here
#shared_task
def calculation4(arg):
some action here
#shared_task
def calculation5(arg):
some action here
my views.py:
result_calculation1= calculation1.delay(accountNumber)
result_calculation2 = calculation2.delay(accountNumber)
result_calculation3 = calculation3.delay(accountNumber)
result_calculation4= calculation4.delay(accountNumber)
result_calculation5 = calculation5.delay(accountNumber)
I want to collect all the tasks statuses in a for loop, so I can build a progress bar, but if there is any other better suggestion on collecting task status and building a progress bar , that's great.
Thank you very much for help in advance.
You need to use getattr() to retrieve the functions from the tasks.py module once you've built the names:
from myapp import tasks # Make sure you import the tasks module
for i in range (1, 6):
functionName = "calculation" + str(i)
task = getattr(tasks, functionName) # Get the task by name from the tasks module
After you've retrieved the task function, you can build up a list of signatures:
signatures = []
signatures.append(task.s(accountNumber)) # Add task signature
From the signatures you can create a group and execute the group as a whole:
from celery import group
task_group = group(signatures)
group_result = group() # Execute the group
And from the group_result you can access each individual task result and build the progress bar around that (perhaps iterating the results in group_result and checking each result's status):
for result in group_result:
status = result.status
# Your progress bar logic...
Putting it all together:
from celery import group
from myapp import tasks # Make sure you import the tasks module
signatures = []
for i in range (1, 6):
functionName = "calculation" + str(i)
task = getattr(tasks, functionName) # Get the task from the tasks module
signatures.append(task.s(accountNumber)) # Add each task signature
task_group = group(signatures)
group_result = group() # Execute the group
for result in group_result:
status = result.status
# Your progress bar logic...
you can put the tasks into a group and you can receive a GroupResult. you can refer to celery doc
In my views.py I am using celery to run a shared task present in tasks.py.
Here is how I call from views.py
task = task_addnums.delay()
task_id = task.id
tasks.py looks as
from celery import shared_task
from celery.result import AsyncResult
#shared_task
def task_addnums():
# print self.request.id
# do something
return True
Now, as we can see we already have task_id from task.id in views.py . But, Let's say If I want to fetch task id from the shared_task itself how can I ? The goal is to get task id from the task_addnums itself so I can use that to pass into some other function.
I tried using self.request.id considering the first param is self . But it didn't worked.
Solved.
This answer is a gem Getting task_id inside a Celery task
You can do function_name.request.id to get task id.
current_task from celery will get the current task.Code like this:
from celery import shared_task, current_task
#shared_task
def task_addnums():
print(current_task.request)
# do something
return True
I feel a bit stupid for asking, but it doesn't appear to be in the documentation for RQ. I have a 'failed' queue with thousands of items in it and I want to clear it using the Django admin interface. The admin interface lists them and allows me to delete and re-queue them individually but I can't believe that I have to dive into the django shell to do it in bulk.
What have I missed?
The Queue class has an empty() method that can be accessed like:
import django_rq
q = django_rq.get_failed_queue()
q.empty()
However, in my tests, that only cleared the failed list key in Redis, not the job keys itself. So your thousands of jobs would still occupy Redis memory. To prevent that from happening, you must remove the jobs individually:
import django_rq
q = django_rq.get_failed_queue()
while True:
job = q.dequeue()
if not job:
break
job.delete() # Will delete key from Redis
As for having a button in the admin interface, you'd have to change django-rq/templates/django-rq/jobs.html template, who extends admin/base_site.html, and doesn't seem to give any room for customizing.
The redis-cli allows FLUSHDB, great for my local environment as I generate a bizzallion jobs.
With a working Django integration I will update. Just adding $0.02.
You can empty any queue by name using following code sample:
import django_rq
queue = "default"
q = django_rq.get_queue(queue)
q.empty()
or even have Django Command for that:
import django_rq
from django.core.management.base import BaseCommand
class Command(BaseCommand):
def add_arguments(self, parser):
parser.add_argument("-q", "--queue", type=str)
def handle(self, *args, **options):
q = django_rq.get_queue(options.get("queue"))
q.empty()
As #augusto-men method seems not to work anymore, here is another solution:
You can use the the raw connection to delete the failed jobs. Just iterate over rq:job keys and check the job status.
from django_rq import get_connection
from rq.job import Job
# delete failed jobs
con = get_connection('default')
for key in con.keys('rq:job:*'):
job_id = key.decode().replace('rq:job:', '')
job = Job.fetch(job_id, connection=con)
if job.get_status() == 'failed':
con.delete(key)
con.delete('rq:failed:default') # reset failed jobs registry
The other answers are outdated with the RQ updates implementing Registries.
Now, you need to do this to loop through and delete failed jobs. This would work any particular Registry as well.
import django_rq
from rq.registry import FailedJobRegistry
failed_registry = FailedJobRegistry('default', connection=django_rq.get_connection())
for job_id in failed_registry.get_job_ids():
try:
failed_registry.remove(job_id, delete_job=True)
except:
# failed jobs expire in the queue. There's a
# chance this will raise NoSuchJobError
pass
Source
You can empty a queue from the command line with:
rq empty [queue-name]
Running rq info will list all the queues.
I'm struggling with Django, Celery, djcelery & PeriodicTasks.
I've created a task to pull a report for Adsense to generate a live stat report. Here is my task:
import datetime
import httplib2
import logging
from apiclient.discovery import build
from celery.task import PeriodicTask
from django.contrib.auth.models import User
from oauth2client.django_orm import Storage
from .models import Credential, Revenue
logger = logging.getLogger(__name__)
class GetReportTask(PeriodicTask):
run_every = datetime.timedelta(minutes=2)
def run(self, *args, **kwargs):
scraper = Scraper()
scraper.get_report()
class Scraper(object):
TODAY = datetime.date.today()
YESTERDAY = TODAY - datetime.timedelta(days=1)
def get_report(self, start_date=YESTERDAY, end_date=TODAY):
logger.info('Scraping Adsense report from {0} to {1}.'.format(
start_date, end_date))
user = User.objects.get(pk=1)
storage = Storage(Credential, 'id', user, 'credential')
credential = storage.get()
if not credential is None and credential.invalid is False:
http = httplib2.Http()
http = credential.authorize(http)
service = build('adsense', 'v1.2', http=http)
reports = service.reports()
report = reports.generate(
startDate=start_date.strftime('%Y-%m-%d'),
endDate=end_date.strftime('%Y-%m-%d'),
dimension='DATE',
metric='EARNINGS',
)
data = report.execute()
for row in data['rows']:
date = row[0]
revenue = row[1]
try:
record = Revenue.objects.get(date=date)
except Revenue.DoesNotExist:
record = Revenue()
record.date = date
record.revenue = revenue
record.save()
else:
logger.error('Invalid Adsense Credentials')
I'm using Celery & RabbitMQ. Here are my settings:
# Celery/RabbitMQ
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "myuser"
BROKER_PASSWORD = "****"
BROKER_VHOST = "myvhost"
CELERYD_CONCURRENCY = 1
CELERYD_NODES = "w1"
CELERY_RESULT_BACKEND = "amqp"
CELERY_TIMEZONE = 'America/Denver'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
import djcelery
djcelery.setup_loader()
On first glance everything seems to work, but after turning on the logger and watching it run I have found that it is running the task at least four times in a row - sometimes more. It also seems to be running every minute instead of every two minutes. I've tried changing the run_every to use a crontab but I get the same results.
I'm starting celerybeat using supervisor. Here is the command I use:
python manage.py celeryd -B -E -c 1
Any ideas as to why its not working as expected?
Oh, and one more thing, after the day changes, it continues to use the date range it first ran with. So as days progress it continues to get stats for the day the task started running - unless I run the task manually at some point then it changes to the date I last ran it manually. Can someone tell me why this happens?
Consider creating a separate queue with one worker process and fixed rate for this type of tasks and just add the tasks in this new queue instead of running them in directly from celerybeat. I hope that could help you to figure out what is wrong with your code, is it problem with celerybeat or your tasks are running longer than expected.
#task(queue='create_report', rate_limit='0.5/m')
def create_report():
scraper = Scraper()
scraper.get_report()
class GetReportTask(PeriodicTask):
run_every = datetime.timedelta(minutes=2)
def run(self, *args, **kwargs):
create_report.delay()
in settings.py
CELERY_ROUTES = {
'myapp.tasks.create_report': {'queue': 'create_report'},
}
start additional celery worker with that would handle tasks in your queue
celery worker -c 1 -Q create_report -n create_report.local
Problem 2. Your YESTERDAY and TODAY variables are set at class level, so within one thread they are set only once.
I'm using Django and Celery and I'm trying to setup routing to multiple queues. When I specify a task's routing_key and exchange (either in the task decorator or using apply_async()), the task isn't added to the broker (which is Kombu connecting to my MySQL database).
If I specify the queue name in the task decorator (which will mean the routing key is ignored), the task works fine. It appears to be a problem with the routing/exchange setup.
Any idea what the problem could be?
Here's the setup:
settings.py
INSTALLED_APPS = (
...
'kombu.transport.django',
'djcelery',
)
BROKER_BACKEND = 'django'
CELERY_DEFAULT_QUEUE = 'default'
CELERY_DEFAULT_EXCHANGE = "tasks"
CELERY_DEFAULT_EXCHANGE_TYPE = "topic"
CELERY_DEFAULT_ROUTING_KEY = "task.default"
CELERY_QUEUES = {
'default': {
'binding_key':'task.#',
},
'i_tasks': {
'binding_key':'important_task.#',
},
}
tasks.py
from celery.task import task
#task(routing_key='important_task.update')
def my_important_task():
try:
...
except Exception as exc:
my_important_task.retry(exc=exc)
Initiate task:
from tasks import my_important_task
my_important_task.delay()
You are using the Django ORM as a broker, which means declarations are only stored in memory
(see the, inarguably hard to find, transport comparison table at http://readthedocs.org/docs/kombu/en/latest/introduction.html#transport-comparison)
So when you apply this task with routing_key important_task.update it will not be able
to route it, because it hasn't declared the queue yet.
It will work if you do this:
#task(queue="i_tasks", routing_key="important_tasks.update")
def important_task():
print("IMPORTANT")
But it would be much simpler for you to use the automatic routing feature,
since there's nothing here that shows you need to use a 'topic' exchange,
to use automatic routing simply remove the settings:
CELERY_DEFAULT_QUEUE,
CELERY_DEFAULT_EXCHANGE,
CELERY_DEFAULT_EXCHANGE_TYPE
CELERY_DEFAULT_ROUTING_KEY
CELERY_QUEUES
And declare your task like this:
#task(queue="important")
def important_task():
return "IMPORTANT"
and then to start a worker consuming from that queue:
$ python manage.py celeryd -l info -Q important
or to consume from both the default (celery) queue and the important queue:
$ python manage.py celeryd -l info -Q celery,important
Another good practice is to not hardcode the queue names into the
task and use CELERY_ROUTES instead:
#task
def important_task():
return "DEFAULT"
then in your settings:
CELERY_ROUTES = {"myapp.tasks.important_task": {"queue": "important"}}
If you still insist on using topic exchanges then you could
add this router to automatically declare all queues the first time
a task is sent:
class PredeclareRouter(object):
setup = False
def route_for_task(self, *args, **kwargs):
if self.setup:
return
self.setup = True
from celery import current_app, VERSION as celery_version
# will not connect anywhere when using the Django transport
# because declarations happen in memory.
with current_app.broker_connection() as conn:
queues = current_app.amqp.queues
channel = conn.default_channel
if celery_version >= (2, 6):
for queue in queues.itervalues():
queue(channel).declare()
else:
from kombu.common import entry_to_queue
for name, opts in queues.iteritems():
entry_to_queue(name, **opts)(channel).declare()
CELERY_ROUTES = (PredeclareRouter(), )