Catch django signal sent from celery task. Is it possible? As far as I know they are running in different processes
#celery.task
def my_task():
...
custom_signal.send()
#receiver(custom_signal)
def my_signal_handler():
...
Please take in mind that your async task must be #shared_task decorator. in order to be called from outside since it will not be attached to a concrete app instance. Documentation #shared_task celery
task.py
#shared_task
def send_email(email):
# Do logic for sending email or other task
signal.py
as you can see below this will only execute when post_save (After user executes save) for the model Contract in your case would be anyother model that its being executed.
#receiver(post_save, sender=Contract)
def inflation_signal(sender, **kwargs):
if kwargs['created']:
send_email.delay('mymail#example.com')
Related
I want to receive a signal when user is activated (i.e. when auth_user.is_active becomes 1). I only want to receive this signal once, the very first time that the user is activated.
I have used the answer given to this question, and it works for me:
#receiver(pre_save, sender=User, dispatch_uid='get_active_user_once')
def new_user_activation_handler(sender, instance, **kwargs):
if instance.is_active and User.objects.filter(pk=instance.pk, is_active=False).exists():
logger.info('user is activated')
However this seems to be a customized signal, I believe django has a built-in user_activated signal. I have tried using the built-in signal but it does not fire:
signals.py:
from django_registration.signals import user_activated
#receiver(user_activated, sender=User, dispatch_uid='django_registration.signals.user_activated')
def new_user_activation_handler(sender, instance, **kwargs):
logger.info('user is activated')
Also this is what I have in apps.py:
class MyClassConfig(AppConfig):
name = 'myclass'
def ready(self):
logger.info('ready...')
import myclass.signals # wire up signals ?
Not sure why this signal is not being fired?
In order to get the above code running, I had to install django-registration package.
All the examples that I have seen have:
from registration.signals import user_activated
But in my case I have to use the a diferent namespace:
from django_registration.signals import user_activated
Not sure why...
You have wrong sender. Please see:
Replace:
#receiver(user_activated, sender=User, dispatch_uid='django_registration.signals.user_activated')
def new_user_activation_handler(sender, instance, **kwargs):
logger.info('user is activated')
With:
from django_registration.backends.activation.views import ActivationView
#receiver(user_activated, sender=ActivationView, dispatch_uid='django_registration.signals.user_activated')
def new_user_activation_handler(sender, instance, **kwargs):
logger.info('user is activated')
I have a simply ListCreateApiView class in Django, for example:
class Test(ListCreateApiView):
def list(self, request, *args, **kwargs):
""" Some code which is creating excel file with openpyxl and send as response """
return excel_file
The problem is, when user send a request to get an excel_file, he must wait for 5-10 minutes till the file will return. User can't go to others urls untill file will not download (I have 10k+ objects in my db)
So, how can I add async here? I want that while the file is being formed, a person can follow other links of the application.
Thanks a lot!
Django has some troubles with asynchrony, and async calls won't help you in cpu-intensive tasks, you'd better use some distributed task queue like celery
It will allow you to handle such heavy tasks (i.e. excel generation) in another thread without interrupting the main django thread. Follow the guide for integration with django.
Once installed and configured, create a task and fire it when a user reaches the endpoint:
E.g.:
# tasks.py
from celery import shared_task
#shared_task
def generate_excel(*args, **kwargs):
...
# views.py
from tasks import generate_excel
class Test(ListCreateApiView):
def list(self, request, *args, **kwargs):
generate_excel.delay()
return Response()
I am applying asynchronous task handling using Celery on my Django Project.
My project's logic:,
from frontend side, there is a table with rows each having an upload button. User clicks on it and a payload is sent to backend containing a url that contains a file.
File is recieved in django views. And saved into database, table Run. Immediately object is saved a post_save signal is triggered to run a celery task.
The task to be performed is, fetch a list of runs with specific status. For each run, perform a task of downloading the file.
I would like to perform this asynchronously in case there is more than one run. Keeping in mind user can click upload for more than one row from frontend.
I am setting up RabbitMQ as my broker. I have rabbitMQ installed and running. I have set the CELERY_BROKER_URL='amqp://localhost' too in settings.py. I am a little lost on what I should do next in my configurations, could I get some guidance. I think I need to configure celery worker on my tasks.
Below is my code so far :
views.py #view that saves to database
class RunsUploadView(APIView):
serializer_class = RunsURLSerializer
def post(self, request, *args, **kwargs):
crawler_name = self.request.data.get('crawler')
run_id = self.kwargs.get("run_id")
run_url = self.request.data.get("run_url")
run = Run()
run.name = f"{crawler_name}_{run_id}"
run.run = run_id
run.url = run_url
run.save()
return Response(model_to_dict(run))
models.py # run is saved to table Run then a post_save signal is triggered.
from django.db import models
class Run(models.Model):
UPLOAD_STATUS = (
("Pending", "pending"),
("Running", "running"),
("Success", "success"),
("Failed", "failed"),
)
name = models.CharField(max_length=100)
run = models.CharField(max_length=100, unique=True)
url = models.URLField(max_length=1000)
status = models.CharField(
max_length=50, choices=UPLOAD_STATUS, default="Pending")
started_at = models.DateTimeField(null=True)
done_at = models.DateTimeField(null=True)
signals.py #handling the post_save logic after save()
from django.db.models.signals import post_save
from django.dispatch import receiver
from main.models import Run
from main.tasks import DownloadRun
#receiver(post_save, sender=Run)
def download_file(sender, **kwargs):
pending_runs = Run.objects.filter(status='Pending') #all pending runs collected, I would need to handle the runs asynchronously.
for run in pending_runs:
run.status = "Started"
run.save()
DownloadRun(run)
Tasks.py #using a class because I am going to update with more functions.
class DownloadRun:
def __init__(self, run):
run_object = model_to_dict(run)
self.run_url = run_object["url"]
self.download_run()
def download_run(self, dest_folder="runs"):
""Run file is downloaded from url""
I figured out the way forward. I did not configure celery well.
tasks.py
from celery import shared_task #import shared_task decorator from celery
class DownloadRun:
def __init__(self, run):
run_object = model_to_dict(run)
self.run_url = run_object["url"]
self.download_run()
def download_run(self, dest_folder="runs"):
""Run file is downloaded from url""
#shared_task
def celery_task(run_id):
DownloadRun(run_id)
signals.py
from django.db.models.signals import post_save
from django.dispatch import receiver
from main.models import Run
from main.tasks import celery_task #import celery_task from tasks.py
#receiver(post_save, sender=Run)
def download_file(sender, **kwargs):
pending_runs = Run.objects.filter(status='Pending')
for run in pending_runs:
run.status = "Started"
run.save()
celery_task.delay(run.run) #call celery delay() to invoke the task (pass the unique key as parameter, could be id, in my case I chose the run)
I'm struggling with the following.
I'm trying to create a custom signal that will trigger when the current time will be equal to the value of my model's notify_on DateTimeField.
Something like this:
class Notification(models.Model):
...
notify_on = models.DateTimeField()
def send_email(*args, **kwargs):
# send email
signals.when_its_time.connect(send_email, sender=User)
After I've read through all docs and I found no information on how to implement such a signal.
Any ideas?
UPDATE:
Less naive approach with ability to discard irrelevant tasks: https://stackoverflow.com/a/55337663/9631956
Ok, thanks to comments by #SergeyPugach I've done the following:
Added a post_save signal that calls a function that adds a task to the celery. apply_async let's you pass eta - estimated time of arrival which can accept DateTimeField directly, that's very convenient.
# models.py
from django.db.models import signals
from django.db import models
from .tasks import send_notification
class Notification(models.Model):
...
notify_on = models.DateTimeField()
def notification_post_save(instance, *args, **kwargs):
send_notification.apply_async((instance,), eta=instance.notify_on)
signals.post_save.connect(notification_post_save, sender=Notification)
And the actual task in the tasks.py
import logging
from user_api.celery import app
from django.core.mail import send_mail
from django.template.loader import render_to_string
#app.task
def send_notification(self, instance):
try:
mail_subject = 'Your notification.'
message = render_to_string('notify.html', {
'title': instance.title,
'content': instance.content
})
send_mail(mail_subject, message, recipient_list=[instance.user.email], from_email=None)
except instance.DoesNotExist:
logging.warning("Notification does not exist anymore")
I will not get into details of setting up celery, there's plenty of information out there.
Now I will try to figure out how to update the task after it's notification instance was updated, but that's a completely different story.
In django's documentation there is two interesting signals that may help you on this task: pre_save and post_save.
It depends on your needs, but let's say you want to check if your model's notify_on is equal to the current date after saving your model (actually after calling the save() or create() method). If it's your case you can do:
from datetime import datetime
from django.contrib.auth.models import User
from django.db import models
from django.dispatch import receiver
from django.db.models.signals import post_save
class Notification(models.Model):
...
# Every notification is related to a user
# It depends on your model, but i guess you're doing something similar
user = models.ForeignKey(User, related_name='notify', on_delete=models.DO_NOTHING)
notify_on = models.DateTimeField()
...
def send_email(self, *args, **kwargs):
"""A model method to send email notification"""
...
#receiver(post_save, sender=User)
def create_notification(sender, instance, created, **kwargs):
# check if the user instance is created
if created:
obj = Notification.objects.create(user=instance, date=datetime.now().date())
if obj.notify_on == datetime.now().date():
obj.send_email()
And you should know, that django signals won't work by they own only if there is an action that triggers them. What this mean is that Django signals won't loop over your model's instances and perform an operation, but django signals will trigger when your application is performing an action on the model connected to a signal.
Bonus: To perform a loop over your instances and process an action regulary you may need an asyncworker with a Queue database (mostly, Celery with Redis or RabbitMQ).
My model post processing is using the post_save signal:
from django.core.signals import request_finished
from django.dispatch import receiver
from models import MyModel
from pipeline import this_takes_forever
#receiver(post_save, sender=MyModel)
def my_callback(sender, **kwargs):
this_takes_forever(sender)
The this_takes_forever routine does IO so I want to defer it to avoid blocking the request too much.
I thought this was a great use case for the new asyncio module. But I have a hard time getting my mind around the whole process.
I think I should be able to adapt the signal receiver like this:
#receiver(post_save, sender=MyModel)
def my_callback(sender, **kwargs):
loop = asyncio.get_event_loop()
loop.run_until_complete(this_takes_forever(sender))
loop.close()
Provided this_takes_forever is also adapted to be a coroutine.
#coroutine
def this_takes_forever(instance):
# do something with instance
return instance
This sounds too magical to work. And in fact it halts with an AssertionError:
AssertionError at /new/
There is no current event loop in thread 'Thread-1'.
I don't see where should I start the loop in this context. Anyone tried something like this?
You get no any benefit in your case:
#receiver(post_save, sender=MyModel)
def my_callback(sender, **kwargs):
this_takes_forever(sender)
is equal to
#receiver(post_save, sender=MyModel)
def my_callback(sender, **kwargs):
loop = asyncio.get_event_loop()
loop.run_until_complete(this_takes_forever(sender))
loop.close()
in terms of execution time. loop.run_until_complete waits for end of this_takes_forever(sender) coroutine call, so you get synchronous call in second case as well as in former one.
About AssertionError: you start Django app in multithreaded mode, but asyncio makes default event loop for main thread only -- you should to register new loop for every user-created thread where you need to call asyncio code.
But, say again, asyncio cannot solve your particular problem, it just incompatible with Django.
The standard way for Django is to defer long-running code into celery task (see http://www.celeryproject.org/)