I have a Django 1.5.1 webapp using Celery 3.0.23 with RabbitMQ 3.1.5. and sqlite3.
I can submit jobs using a simple result = status.tasks.mymethod.delay(parameter), all tasks executes correctly:
[2013-09-30 17:04:11,369: INFO/MainProcess] Got task from broker: status.tasks.prova[a22bf0b9-0d5b-4ce5-967a-750f679f40be]
[2013-09-30 17:04:11,566: INFO/MainProcess] Task status.tasks.mymethod[a22bf0b9-0d5b-4ce5-967a-750f679f40be] succeeded in 0.194540023804s: u'Done'
I want to display in a page the latest 10 jobs submitted and their status. Is there a way in Django to get such objects? I see a couple of tables in the database (celery_taskmeta and celery_taskmeta_2ff6b945) and tried some accesses to the objects but Django always displays a AttributeError page.
What is the correct way to access Celery results from Django?
Doing
cel = celery.status.tasks.get(None)
cel = status.tasks.all()
does not work, resulting in the aforementioned AttributeError. (status is the name of my app)
EDIT: I am sure tasks are saved, as this small tutorial says:
By default django-celery stores this state in the Django database. You may consider choosing an alternate result backend or disabling states alltogether (see Result Backends).
Following the links there are only references on how to setup the DB connection and not how to retrieve the results.
Try this:
from djcelery.models import TaskMeta
TaskMeta.objects.all()
Related
Normally, we can deploy celery for async. Can celery be used for asynchronous file uploading, which allows client to continue working on the website while big size of file being uploaded? I passed the forms to the task of the celery and I had an error like 'Object of type module is not JSON serializable'. Is there any way for async file uploading?
i'm pretty sure it's not possible, what you need to do is more like open a popup page and do the job inside.
'Object of type module is not JSON serializable'
One of the best practice of Celery is to stock data in database (for a lot of data), and create a celery task with ID's only. The object you pass to celery need to be json formatted.
I'm using django-celery-beat in a django app (this stores the schedule in the database instead of a local file). I've configured my schedule via celery_beat that Celery is initialized with via app.config_from_object(...)
I recently renamed/removed a few tasks and restarted the app. The new tasks showed up, but the tasks removed from the celery_beat dictionary didn't get removed from the database.
Is this expected workflow -- requiring manual removal of tasks from the database? Is there a workaround to automatically reconcile the schedule at Django startup?
I tried a PeriodicTask.objects.all().delete() in celery/__init__.py
def _clean_schedule():
from django.db import transaction
from django_celery_beat.models import PeriodicTask
from django_celery_beat.models import PeriodicTasks
with transaction.atomic():
PeriodicTask.objects.\
exclude(task__startswith='celery.').\
exclude(name__in=settings.CELERY_CONFIG.celery_beat.keys()).\
delete()
PeriodicTasks.update_changed()
_clean_schedule()
but that is not allowed because Django isn't properly started up yet:
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
You also can't use Django's AppConfig.ready() because making queries / db connections in ready() is not supported.
Looking at how django-celery-beat actually works to install the schedules, I thought I maybe I could hook into that process.
It doesn't happen when Django starts -- it happens when beat starts. It calls setup_schedule() against the class passed on the beat command line.
Therefore, we can just override the scheduler with
--scheduler=myproject.lib.scheduler:DatabaseSchedulerWithCleanup
to do cleanup:
import logging
from django_celery_beat.models import PeriodicTask
from django_celery_beat.models import PeriodicTasks
from django_celery_beat.schedulers import DatabaseScheduler
from django.db import transaction
class DatabaseSchedulerWithCleanup(DatabaseScheduler):
def setup_schedule(self):
schedule = self.app.conf.beat_schedule
with transaction.atomic():
num, info = PeriodicTask.objects.\
exclude(task__startswith='celery.').\
exclude(name__in=schedule.keys()).\
delete()
logging.info("Removed %d obsolete periodic tasks.", num)
if num > 0:
PeriodicTasks.update_changed()
super(DatabaseSchedulerWithCleanup, self).setup_schedule()
Note, you only want this if you are exclusively managing tasks with beat_schedule. If you add tasks via Django admin or programatically, they will also be deleted.
I put Celery in my Django app so that the two other python programs can process the input from my Django app via doing subprocess method.
My question is how do I access the output from the subprocess? Back then when I made just a python program, I access the log files (output from the two apps) via stdout and stderr. Is this the same when I use Celery in Django? Is the value of CELERY_RESULT_BACKEND (if I should assign my Django app's db here) affected by the log files?
So far what I've done is:
Access the two apps via subprocess in my tasks.py
I assigned my broker's db, Redis, as my db for now for CELERY_RESULT_BACKEND. My plan is to get the log files and then save them to my Django app's db so that I can just access that db.
Can you offer some help?
Typically, you only care about the task result, which is the return value of the celery task, and that is stored in your result_backend for at least result_expires time (usually 1 day). So, to the extent that you want to access any particular task's result, you can just do so using the task ID.
This morning in my dbshell, I tried to wipe my database with:
=>drop schema public cascade;
I waited for about 10 minutes and I got no response back. So I quit the terminal (In retrospect probably a bad decision). Now, when I try and access my models:
>>> from myapp.models import Tool
>>> Tool.objects.all()
I get no response back. When I try to log into the app on heroku I get a timeout error. When I run the app locally I get waiting for 127.0.0.1 which continues onto infinity with out a response.
I'm stumped and not sure what's happening. Any ideas would be greatly appreciated.
Thank you
I found the solution to wipe the database and get responses back with this unaccepted answer
I wish I would have found this earlier when I was searching stack and the heroku-postgres docs.
My User collection contains data such as
{"user1":"zera",
"my_status":"active",
"date_creation" : ISODate("2013-10-01T10:15:52.055Z")
}
{"user2":"dfgf",
"my_status":"noactive",
"date_creation": ISODate("2013-10-01T08:55:41.212Z")
}
I need to find each user with my_status :"active" and update their my_status after 24 hours from each user's date_creation.
Can anyone suggest a method to do it using django?
Well, I'd write an async task to keep polling the database to check for users with active status. If the user is active, update their status.
For the asynchronous tasks, you can use python-rq but to make things easier there's a django module for python-rq, it's django-rq. Also, Celery is another popular and good option. There's also a module for Django, you can find it here.