Django process running tasks instead of Celery - django

Question

I'm running an app with Celery+redis for asynchronous tasks.
I managed to get Celery see the list of tasks. However my tasks aren't executed by Celery workers but by Django process instead.
I tried invoking the tasks with .delay() and .apply_async() without success (actually in these cases the call to the task gets blocked indefinitely and nothing is shown in the logs).
I might be missing something very basic but cannot see where.
Relevant settings follow:
settings.py
CELERY_REDIS_DB = os.environ.get("REDIS_DB_CELERY", 0)
CELERY_REDIS_HOST = os.getenv("REDIS_HOSTNAME", "redis")
CELERY_REDIS_PORT = 6379
CELERY_RESULT_BACKEND = BROKER_URL = (
f'redis://{CELERY_REDIS_HOST}:{CELERY_REDIS_PORT}/{CELERY_REDIS_DB}'
)
CELERY_TIMEZONE = TIME_ZONE
CELERY_RESULT_EXPIRES = 5 * 60 * 60
celery.py
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'maat.settings')
app = Celery('maat')
# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
tasks.py
#shared_task
def remove_task(
environment=None,
payload=None,
**kwargs,
):
LOGGER.info('Received task, now removing')
...
views.py
class MyClass(
LoginRequiredMixin,
PermissionRequiredMixin,
SingleObjectMixin,
APIView
):
return_403 = True
model = models.Environment
slug_field = 'name'
slug_url_kwarg = 'environment'
def delete(self, request, environment):
tasks.remove_task(environment=environment, payload=request.data)
Django 3.2
celery[redis] 5.1.2
redis 3.5.3
django-celery-results 2.0.1

Answer 1

Assuming celery.py lives somewhere that is accessible to views.py, the problem is likely that because you are using a shared task, django does not know how to communicate with the celery broker without more information.
In order to do that, we can follow the advice from our helpful friends at stack overflow and do something like the following in your view:
from path.to.celery import app
app.set_default()