Using RabbitMQ with Celery in Django post_save - django

Question

I am applying asynchronous task handling using Celery on my Django Project.
My project's logic:,
from frontend side, there is a table with rows each having an upload button. User clicks on it and a payload is sent to backend containing a url that contains a file.
File is recieved in django views. And saved into database, table Run. Immediately object is saved a post_save signal is triggered to run a celery task.
The task to be performed is, fetch a list of runs with specific status. For each run, perform a task of downloading the file.
I would like to perform this asynchronously in case there is more than one run. Keeping in mind user can click upload for more than one row from frontend.
I am setting up RabbitMQ as my broker. I have rabbitMQ installed and running. I have set the CELERY_BROKER_URL='amqp://localhost' too in settings.py. I am a little lost on what I should do next in my configurations, could I get some guidance. I think I need to configure celery worker on my tasks.
Below is my code so far :
views.py #view that saves to database
class RunsUploadView(APIView):
serializer_class = RunsURLSerializer
def post(self, request, *args, **kwargs):
crawler_name = self.request.data.get('crawler')
run_id = self.kwargs.get("run_id")
run_url = self.request.data.get("run_url")
run = Run()
run.name = f"{crawler_name}_{run_id}"
run.run = run_id
run.url = run_url
run.save()
return Response(model_to_dict(run))
models.py # run is saved to table Run then a post_save signal is triggered.
from django.db import models
class Run(models.Model):
UPLOAD_STATUS = (
("Pending", "pending"),
("Running", "running"),
("Success", "success"),
("Failed", "failed"),
)
name = models.CharField(max_length=100)
run = models.CharField(max_length=100, unique=True)
url = models.URLField(max_length=1000)
status = models.CharField(
max_length=50, choices=UPLOAD_STATUS, default="Pending")
started_at = models.DateTimeField(null=True)
done_at = models.DateTimeField(null=True)
signals.py #handling the post_save logic after save()
from django.db.models.signals import post_save
from django.dispatch import receiver
from main.models import Run
from main.tasks import DownloadRun
#receiver(post_save, sender=Run)
def download_file(sender, **kwargs):
pending_runs = Run.objects.filter(status='Pending') #all pending runs collected, I would need to handle the runs asynchronously.
for run in pending_runs:
run.status = "Started"
run.save()
DownloadRun(run)
Tasks.py #using a class because I am going to update with more functions.
class DownloadRun:
def __init__(self, run):
run_object = model_to_dict(run)
self.run_url = run_object["url"]
self.download_run()
def download_run(self, dest_folder="runs"):
""Run file is downloaded from url""

Answer 1

I figured out the way forward. I did not configure celery well.
tasks.py
from celery import shared_task #import shared_task decorator from celery
class DownloadRun:
def __init__(self, run):
run_object = model_to_dict(run)
self.run_url = run_object["url"]
self.download_run()
def download_run(self, dest_folder="runs"):
""Run file is downloaded from url""
#shared_task
def celery_task(run_id):
DownloadRun(run_id)
signals.py
from django.db.models.signals import post_save
from django.dispatch import receiver
from main.models import Run
from main.tasks import celery_task #import celery_task from tasks.py
#receiver(post_save, sender=Run)
def download_file(sender, **kwargs):
pending_runs = Run.objects.filter(status='Pending')
for run in pending_runs:
run.status = "Started"
run.save()
celery_task.delay(run.run) #call celery delay() to invoke the task (pass the unique key as parameter, could be id, in my case I chose the run)