I am building an interval based scheduler using apscheduler. Here's the code:
from flask import Flask
import time
from apscheduler.schedulers.background import BackgroundScheduler
def job1():
print('performed job1')
def job2():
print('performed job2')
sched = BackgroundScheduler(daemon=True)
sched.add_job(lambda : sched.print_jobs(),'interval',minutes=1)
sched.add_job(job1,'interval', minutes=1)
sched.add_job(job2, 'interval',minutes=2)
try:
sched.start()
except (KeyboardInterrupt, SystemExit):
pass
app = Flask(__name__)
if __name__ == "__main__":
app.run()
whenever two jobs are triggered simultaneously, apscheduler performs the second one. I just want to know, on what basis does apscheduler decide which job to perform out of the two clashed jobs. And is it possible to change that criteria as i want to perform the job which has the higher priority.I'm defining priorities explicitly.
Related
I have stepA and stepB which needs to be applied to ~100 sets of data. Since these sets are CPU intensive I want them to happen sequentially.
I am using Celery to create a chain with ~100 chains with (stepA and StepB).
It celery execution works fine, I want to show the status of each of the chain in a django page as each of the stepA and stepB may take an hour to complete.
I want to show the status of each of the sub chains in a django page. The problem is the AsyncResult sees all the children of the parent chain as tasks.
The following is a sample code snippet.
test_celery.py
from __future__ import absolute_import, division, print_function
from celery import Celery, chord, chain, group
from datetime import datetime
import time
app = Celery('tasks', backend='redis', broker='redis://')
#app.task
def ident(x):
print("Guru: inside ident: {}".format(datetime.now()))
print(x)
return x
#app.task
def tsum(numbers):
print("Guru: inside tsum: {}".format(datetime.now()))
print(numbers)
time.sleep(5)
return sum(numbers)
test.py
import test_celery
from celery import chain
from celery.result import AsyncResult
from celery.utils.graph import DependencyGraph
def method():
async_chain = chain(chain(test_celery.tsum.si([1, 2]), test_celery.ident.si(2)), chain(test_celery.tsum.si([2,3]), test_celery.ident.si(3)))
chain_task_names = [task.task for task in async_chain.tasks]
# run the chain
chain_results_tasks = async_chain.apply_async()
print("async_chain=", dir(chain_results_tasks))
print("result.get={}".format(chain_results_tasks.status))
# create a list of names and tasks in the chain
chain_tasks = zip(chain_task_names, reversed(list(get_chain_nodes(chain_results_tasks))))
xx = list(get_chain_nodes(chain_results_tasks))
#
print(dir(chain_results_tasks))
for task in async_chain.tasks:
print("dir task={}".format(dir(task)))
print("task_name={} task_id={}".format(task.task, task.parent_id))
for i in xx:
res = AsyncResult(i)
# print("res={}".format(dir(res)))
parent = get_parent_node(i)
print(parent.build_graph(intermediate=True))
print("parent_task={}".format(dir(parent)))
print(xx[-1].build_graph(intermediate=True))
method()
Any help is appreciated.
I am currently developing a Django application based on django-tenants-schema. You don't need to look into the actual code of the module, but the idea is that it has a global setting for the current database connection defining which schema to use for the application tenant, e.g.
tenant = tenants_schema.get_tenant()
And for setting
tenants_schema.set_tenant(xxx)
For some of the tasks I would like them to remember the current global tenant selected during the instantiation, e.g. in theory:
class AbstractTask(Task):
'''
Run this method before returning the task future
'''
def before_submit(self):
self.run_args['tenant'] = tenants_schema.get_tenant()
'''
This method is run before related .run() task method
'''
def before_run(self):
tenants_schema.set_tenant(self.run_args['tenant'])
Is there an elegant way of doing it in celery?
Celery (as of 3.1) has signals you can hook into to do this. You can alter the kwargs that were passed in, and on the other side, undo your alterations before they're given to the actual task:
from celery import shared_task
from celery.signals import before_task_publish, task_prerun, task_postrun
from threading import local
current_tenant = local()
#before_task_publish.connect
def add_tenant_to_task(body=None, **unused):
body['kwargs']['tenant_middleware.tenant'] = getattr(current_tenant, 'id', None)
print 'sending tenant: {t}'.format(t=current_tenant.id)
#task_prerun.connect
def extract_tenant_from_task(kwargs=None, **unused):
tenant_id = kwargs.pop('tenant_middleware.tenant', None)
current_tenant.id = tenant_id
print 'current_tenant.id set to {t}'.format(t=tenant_id)
#task_postrun.connect
def cleanup_tenant(**kwargs):
current_tenant.id = None
print 'cleaned current_tenant.id'
#shared_task
def get_current_tenant():
# Here is where you would do work that relied on current_tenant.id being set.
import time
time.sleep(1)
return current_tenant.id
And if you run the task (not showing logging from the worker):
In [1]: current_tenant.id = 1234; ct = get_current_tenant.delay(); current_tenant.id = 5678; ct.get()
sending tenant: 1234
Out[1]: 1234
In [2]: current_tenant.id
Out[2]: 5678
The signals are not called if no message is sent (when you call the task function directly, without delay() or apply_async()). If you want to filter on the task name, it is available as body['task'] in the before_task_publish signal handler, and the task object itself is available in the task_prerun and task_postrun handlers.
I am a Celery newbie, so I can't really tell if this is the "blessed" way of doing "middleware"-type stuff in Celery, but I think it will work for me.
I'm not sure what you mean here, is before_submit executed before the task is called by a client?
In that case I would rather use a with statement here:
from contextlib import contextmanager
#contextmanager
def set_tenant_db(tenant):
prev_tenant = tenants_schema.get_tenant()
try:
tenants_scheme.set_tenant(tenant)
yield
finally:
tenants_schema.set_tenant(prev_tenant)
#app.task
def tenant_task(tenant=None):
with set_tenant_db(tenant):
do_actions_here()
tenant_task.delay(tenant=tenants_scheme.get_tenant())
You can of course create a base task that does this automatically,
you can apply the context in Task.__call__ for example, but I'm not sure
if that saves you much if you can just use the with statement explicitly.
I feel a bit stupid for asking, but it doesn't appear to be in the documentation for RQ. I have a 'failed' queue with thousands of items in it and I want to clear it using the Django admin interface. The admin interface lists them and allows me to delete and re-queue them individually but I can't believe that I have to dive into the django shell to do it in bulk.
What have I missed?
The Queue class has an empty() method that can be accessed like:
import django_rq
q = django_rq.get_failed_queue()
q.empty()
However, in my tests, that only cleared the failed list key in Redis, not the job keys itself. So your thousands of jobs would still occupy Redis memory. To prevent that from happening, you must remove the jobs individually:
import django_rq
q = django_rq.get_failed_queue()
while True:
job = q.dequeue()
if not job:
break
job.delete() # Will delete key from Redis
As for having a button in the admin interface, you'd have to change django-rq/templates/django-rq/jobs.html template, who extends admin/base_site.html, and doesn't seem to give any room for customizing.
The redis-cli allows FLUSHDB, great for my local environment as I generate a bizzallion jobs.
With a working Django integration I will update. Just adding $0.02.
You can empty any queue by name using following code sample:
import django_rq
queue = "default"
q = django_rq.get_queue(queue)
q.empty()
or even have Django Command for that:
import django_rq
from django.core.management.base import BaseCommand
class Command(BaseCommand):
def add_arguments(self, parser):
parser.add_argument("-q", "--queue", type=str)
def handle(self, *args, **options):
q = django_rq.get_queue(options.get("queue"))
q.empty()
As #augusto-men method seems not to work anymore, here is another solution:
You can use the the raw connection to delete the failed jobs. Just iterate over rq:job keys and check the job status.
from django_rq import get_connection
from rq.job import Job
# delete failed jobs
con = get_connection('default')
for key in con.keys('rq:job:*'):
job_id = key.decode().replace('rq:job:', '')
job = Job.fetch(job_id, connection=con)
if job.get_status() == 'failed':
con.delete(key)
con.delete('rq:failed:default') # reset failed jobs registry
The other answers are outdated with the RQ updates implementing Registries.
Now, you need to do this to loop through and delete failed jobs. This would work any particular Registry as well.
import django_rq
from rq.registry import FailedJobRegistry
failed_registry = FailedJobRegistry('default', connection=django_rq.get_connection())
for job_id in failed_registry.get_job_ids():
try:
failed_registry.remove(job_id, delete_job=True)
except:
# failed jobs expire in the queue. There's a
# chance this will raise NoSuchJobError
pass
Source
You can empty a queue from the command line with:
rq empty [queue-name]
Running rq info will list all the queues.
I have to run tasks on approximately 150k Django objects. What is the best way to do this? I am using the Django ORM as the Broker. The database backend is MySQL and chokes and dies during the task.delay() of all the tasks. Related, I was also wanting to kick this off from the submission of a form, but the resulting request produced a very long response time that timed out.
I would also consider using something other than using the database as the "broker". It really isn't suitable for this kind of work.
Though, you can move some of this overhead out of the request/response cycle by launching a task to create the other tasks:
from celery.task import TaskSet, task
from myapp.models import MyModel
#task
def process_object(pk):
obj = MyModel.objects.get(pk)
# do something with obj
#task
def process_lots_of_items(ids_to_process):
return TaskSet(process_object.subtask((id, ))
for id in ids_to_process).apply_async()
Also, since you probably don't have 15000 processors to process all of these objects
in parallel, you could split the objects in chunks of say 100's or 1000's:
from itertools import islice
from celery.task import TaskSet, task
from myapp.models import MyModel
def chunks(it, n):
for first in it:
yield [first] + list(islice(it, n - 1))
#task
def process_chunk(pks):
objs = MyModel.objects.filter(pk__in=pks)
for obj in objs:
# do something with obj
#task
def process_lots_of_items(ids_to_process):
return TaskSet(process_chunk.subtask((chunk, ))
for chunk in chunks(iter(ids_to_process),
1000)).apply_async()
Try using RabbitMQ instead.
RabbitMQ is used in a lot of bigger companies and people really rely on it, since it's such a great broker.
Here is a great tutorial on how to get you started with it.
I use beanstalkd ( http://kr.github.com/beanstalkd/ ) as the engine. Adding a worker and a task is pretty straightforward for Django if you use django-beanstalkd : https://github.com/jonasvp/django-beanstalkd/
It’s very reliable for my usage.
Example of worker :
import os
import time
from django_beanstalkd import beanstalk_job
#beanstalk_job
def background_counting(arg):
"""
Do some incredibly useful counting to the value of arg
"""
value = int(arg)
pid = os.getpid()
print "[%s] Counting from 1 to %d." % (pid, value)
for i in range(1, value+1):
print '[%s] %d' % (pid, i)
time.sleep(1)
To launch a job/worker/task :
from django_beanstalkd import BeanstalkClient
client = BeanstalkClient()
client.call('beanstalk_example.background_counting', '5')
(source extracted from example app of django-beanstalkd)
Enjoy !
I`ve followed the guidelines in http://celeryq.org/docs/django-celery/getting-started/first-steps-with-django.html and created a view that calls my test method in tasks.py:
import time
from celery.decorators import task
#task()
def add(x, y):
time.sleep(10)
return x + y
But if my add-method takes a long time to respond, how can I store the result-object I got when calling add.delay(1,2) and use it to check the progress/success/result using get later?
You only need the task-id:
result = add.delay(2, 2)
result.task_id
With this you can poll the status of the task (using e.g. AJAX)
Django-celery comes with a view that returns results and status in JSON:
http://celeryq.org/docs/django-celery/reference/djcelery.views.html