DJCelery not storing task results in Django SQLite DB - django

Question

DJCelery is not storing task results in my Django SQLite DB.
I have an existing Django project that I have started setting up Celery w/ RabbitMQ on. I started my RabbitMQ server. I can run Celery python manage.py celeryd --verbosity=2 --loglevel=DEBUG and Celerybeat python manage.py celerybeat --verbosity=2 --loglevel=DEBUG. Everything starts up w/ out error and my periodic example tasks also runs without error.
I used pip install django-celery to install. I have djcelery in my installed apps and ran python manage.py migrate djcelery. I added CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend' to the end of my settings.py file.
When I run python manage.py celeryd --verbosity=2 --loglevel=DEBUG, the startup text shows:
...
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results:
- *** --- * --- .> concurrency: 1 (prefork)
...
The results section being blank indicates to me that the configuration isn't right somehow but I can't figure out how. I tried using app.conf.update in my celery.py file to set the CELERY_RESULT_BACKEND but got the same results. I left out CELERY_RESULT_BACKEND, but that defaulted to no results. I also tried putting 'database' instead of 'djcelery.backends.database:DatabaseBackend' but that indicated it was attempting to use sqlalchemy instead of djcelery.
When I run python manage.py runserver I can see a DJCELERY section with tables Crontabs, Intervals, Periodic tasks, Tasks, and Workers. There isn't any data on my Tasks though.
Can anyone point out what could be wrong or missing? Thank you for your time.

Answer 1

tutuDajuju led me in the right direction - there's more to it so I'll write it all up. I abandoned using djcelery in favor of sqlalchemy with a separate back-end database outside of Django.
Inside my venv I ran pip install sqlalchemy. I then put CELERY_RESULT_BACKEND = 'db+sqlite:///celery_results.sqlite3' in settings.py. This connected Celery to the new SQLite database to use for state/results.
Running celery -A <projectapp>.celery:app worker then showed the database in the startup message:
...
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: sqlite:///celery_results.sqlite3
- *** --- * --- .> concurrency: 1 (prefork)
...
At first I was worried because the database file wasn't created in my Django project dir. This is because I hadn't ran a task yet. Once I ran my first task, the database & tables were created correctly.
I verified task results were stored in the database by running a script:
from sqlalchemy import create_engine
engine = create_engine("sqlite:///celery_results.sqlite3")
connection = engine.connect()
result = connection.execute("select * from celery_taskmeta")
for row in result:
print(row)
connection.close()
I found the table names by:
print(engine.table_name())
Hope this helps someone out.

Answer 2

The celery docs mention a few different syntaxes, not sure what you tried is valid. Try the following:
# use a connection string
CELERY_RESULT_BACKEND = 'db+sqlite:///foo.db'
Update:
As in your comment, the docs also mention it is also possible to use the Django ORM/Cache as a result backend. To do this, you must pass the setting you tried into your celery app config:
app.conf.update(
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
)
Alternatively, the docs also explain
If you have connected Celery to your Django settings then you can add
this directly into your settings module (without the app.conf.update
part)
This is a reference to the configuration of the Celery app detailed in the same page. This basically means that if you configured your celery app in a module, and you add the Django settings module as a configuration source for Celery, then setting CELERY_RESULT_BACKEND in your Django settings module, as you did, will also work.
file: proj/proj/celery.py
# important to pass the Django settings to your celery app
app = Celery('proj')
app.config_from_object('django.conf:settings')
file: proj/proj/settings.py
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend'