I am applying asynchronous task handling using Celery on my Django Project.
My project's logic:,
from frontend side, there is a table with rows each having an upload button. User clicks on it and a payload is sent to backend containing a url that contains a file.
File is recieved in django views. And saved into database, table Run. Immediately object is saved a post_save signal is triggered to run a celery task.
The task to be performed is, fetch a list of runs with specific status. For each run, perform a task of downloading the file.
I would like to perform this asynchronously in case there is more than one run. Keeping in mind user can click upload for more than one row from frontend.
I am setting up RabbitMQ as my broker. I have rabbitMQ installed and running. I have set the CELERY_BROKER_URL='amqp://localhost' too in settings.py. I am a little lost on what I should do next in my configurations, could I get some guidance. I think I need to configure celery worker on my tasks.
Below is my code so far :
views.py #view that saves to database
class RunsUploadView(APIView):
serializer_class = RunsURLSerializer
def post(self, request, *args, **kwargs):
crawler_name = self.request.data.get('crawler')
run_id = self.kwargs.get("run_id")
run_url = self.request.data.get("run_url")
run = Run()
run.name = f"{crawler_name}_{run_id}"
run.run = run_id
run.url = run_url
run.save()
return Response(model_to_dict(run))
models.py # run is saved to table Run then a post_save signal is triggered.
from django.db import models
class Run(models.Model):
UPLOAD_STATUS = (
("Pending", "pending"),
("Running", "running"),
("Success", "success"),
("Failed", "failed"),
)
name = models.CharField(max_length=100)
run = models.CharField(max_length=100, unique=True)
url = models.URLField(max_length=1000)
status = models.CharField(
max_length=50, choices=UPLOAD_STATUS, default="Pending")
started_at = models.DateTimeField(null=True)
done_at = models.DateTimeField(null=True)
signals.py #handling the post_save logic after save()
from django.db.models.signals import post_save
from django.dispatch import receiver
from main.models import Run
from main.tasks import DownloadRun
#receiver(post_save, sender=Run)
def download_file(sender, **kwargs):
pending_runs = Run.objects.filter(status='Pending') #all pending runs collected, I would need to handle the runs asynchronously.
for run in pending_runs:
run.status = "Started"
run.save()
DownloadRun(run)
Tasks.py #using a class because I am going to update with more functions.
class DownloadRun:
def __init__(self, run):
run_object = model_to_dict(run)
self.run_url = run_object["url"]
self.download_run()
def download_run(self, dest_folder="runs"):
""Run file is downloaded from url""
I figured out the way forward. I did not configure celery well.
tasks.py
from celery import shared_task #import shared_task decorator from celery
class DownloadRun:
def __init__(self, run):
run_object = model_to_dict(run)
self.run_url = run_object["url"]
self.download_run()
def download_run(self, dest_folder="runs"):
""Run file is downloaded from url""
#shared_task
def celery_task(run_id):
DownloadRun(run_id)
signals.py
from django.db.models.signals import post_save
from django.dispatch import receiver
from main.models import Run
from main.tasks import celery_task #import celery_task from tasks.py
#receiver(post_save, sender=Run)
def download_file(sender, **kwargs):
pending_runs = Run.objects.filter(status='Pending')
for run in pending_runs:
run.status = "Started"
run.save()
celery_task.delay(run.run) #call celery delay() to invoke the task (pass the unique key as parameter, could be id, in my case I chose the run)
I have done the below post_save signal in my project.
from django.db.models.signals import post_save
from django.contrib.auth.models import User
# CORE - SIGNALS
# Core Signals will operate based on post
def after_save_handler_attr_audit_obj(sender, **kwargs):
print User.get_profile()
if hasattr(kwargs['instance'], 'audit_obj'):
if kwargs['created']:
kwargs['instance'].audit_obj.create(operation="INSERT", operation_by=**USER.ID**).save()
else:
kwargs['instance'].audit_obj.create(operation="UPDATE").save()
# Connect the handler with the post save signal - Django 1.2
post_save.connect(after_save_handler_attr_audit_obj, dispatch_uid="core.models.audit.new")
The operation_by column, I want to get the user_id and store it. Any idea how can do that?
Can't be done. The current user is only available via the request, which is not available when using purely model functionality. Access the user in the view somehow.
I was able to do it by inspecting the stack and looking for the view then looking at the local variables for the view to get the request. It feels like a bit of a hack, but it worked.
import inspect, os
#receiver(post_save, sender=MyModel)
def get_user_in_signal(sender, **kwargs):
for entry in reversed(inspect.stack()):
if os.path.dirname(__file__) + '/views.py' == entry[1]:
try:
user = entry[0].f_locals['request'].user
except:
user = None
break
if user:
# do stuff with the user variable
Ignacio is right. Django's model signals are intended to notify other system components about events associated with instances and their respected data, so I guess it's valid that you cannot, say, access request data from a model post_save signal, unless that request data was stored on or associated with the instance.
I guess there are lots of ways to handle it, ranging from worse to better, but I'd say this is a prime example for creating class-based/function-based generic views that will automatically handle this for you.
Have your views that inherit from CreateView, UpdateView or DeleteView additionally inherit from your AuditMixin class if they handle verbs that operate on models that need to be audited. The AuditMixin can then hook into the views that successfully create\update\delete objects and create an entry in the database.
Makes perfect sense, very clean, easily pluggable and gives birth to happy ponies. Flipside? You'll either have to be on the soon-to-be-released Django 1.3 release or you'll have to spend some time fiddlebending the function-based generic views and providing new ones for each auditing operation.
You can do that with the help of middleware. Create get_request.py in your app. Then
from threading import current_thread
from django.utils.deprecation import MiddlewareMixin
_requests = {}
def current_request():
return _requests.get(current_thread().ident, None)
class RequestMiddleware(MiddlewareMixin):
def process_request(self, request):
_requests[current_thread().ident] = request
def process_response(self, request, response):
# when response is ready, request should be flushed
_requests.pop(current_thread().ident, None)
return response
def process_exception(self, request, exception):
# if an exception has happened, request should be flushed too
_requests.pop(current_thread().ident, None)
Then add this middleware to your settings:
MIDDLEWARE = [
....
'<your_app>.get_request.RequestMiddleware',
]
Then add import to your signals:
from django.db.models.signals import post_save
from django.contrib.auth.models import User
from <your_app>.get_request import current_request
# CORE - SIGNALS
# Core Signals will operate based on post
def after_save_handler_attr_audit_obj(sender, **kwargs):
print(Current User, current_request().user)
print User.get_profile()
if hasattr(kwargs['instance'], 'audit_obj'):
if kwargs['created']:
kwargs['instance'].audit_obj.create(operation="INSERT", operation_by=**USER.ID**).save()
else:
kwargs['instance'].audit_obj.create(operation="UPDATE").save()
# Connect the handler with the post save signal - Django 1.2
post_save.connect(after_save_handler_attr_audit_obj, dispatch_uid="core.models.audit.new")
Why not adding a middleware with something like this :
class RequestMiddleware(object):
thread_local = threading.local()
def process_request(self, request):
RequestMiddleware.thread_local.current_user = request.user
and later in your code (specially in a signal in that topic) :
thread_local = RequestMiddleware.thread_local
if hasattr(thread_local, 'current_user'):
user = thread_local.current_user
else:
user = None
For traceability add two attributes to your Model(created_by and updated_by), in "updated_by" save the last user who modified the record. Then in your signal you have the user:
models.py:
class Question(models.Model):
question_text = models.CharField(max_length=200)
pub_date = models.DateTimeField('date published')
created_by = models. (max_length=100)
updated_by = models. (max_length=100)
views.py
p = Question.objects.get(pk=1)
p.question_text = 'some new text'
p.updated_by = request.user
p.save()
signals.py
#receiver(pre_save, sender=Question)
def do_something(sender, instance, **kwargs):
try:
obj = Question.objects.get(pk=instance.pk)
except sender.DoesNotExist:
pass
else:
if not obj.user == instance.user: # Field has changed
# do something
print('change: user, old=%s new=%s' % (obj.user, instance.user))
You could also use django-reversion for this purpose, e.g.
from reversion.signals import post_revision_commit
import reversion
#receiver(post_save)
def post_revision_commit(sender, **kwargs):
if reversion.is_active():
print(reversion.get_user())
Read more on their API https://django-reversion.readthedocs.io/en/stable/api.html#revision-api
You can do a small hack by overriding you model save() method and setting the user on the saved instance as additional parameter. To get the user I used get_current_authenticated_user() from django_currentuser.middleware.ThreadLocalUserMiddleware (see https://pypi.org/project/django-currentuser/).
In your models.py:
from django_currentuser.middleware import get_current_authenticated_user
class YourModel(models.Model):
...
...
def save(self, *args, **kwargs):
# Hack to pass the user to post save signal.
self.current_authenticated_user = get_current_authenticated_user()
super(YourModel, self).save(*args, **kwargs)
In your signals.py:
#receiver(post_save, sender=YourModel)
def your_model_saved(sender, instance, **kwargs):
user = getattr(instance, 'current_authenticated_user', None)
PS: Don't forget to add 'django_currentuser.middleware.ThreadLocalUserMiddleware' to your MIDDLEWARE_CLASSES.
I imagine you would have figured this out, but I had the same problem and I realised that all the instances I create had a reference to the user that creates them (which is what you are looking for)
it's possible i guess.
in models.py
class _M(models.Model):
user = models.ForeignKey(...)
in views.py
def _f(request):
_M.objects.create(user=request.user)
in signals.py
#receiver(post_save, sender=_M)
def _p(sender, instance, created, **kwargs):
user = instance.user
No ?
Request object can be obtained from frame record by inspecting.
import inspect
request = [
frame_record[0].f_locals["request"]
for frame_record in inspect.stack()
if frame_record[3] == "get_response"
][0]
def get_requested_user():
import inspect
for frame_record in inspect.stack():
if frame_record[3] == 'get_response':
request = frame_record[0].f_locals['request']
return request.user
else:
return None
context_processors.py
from django.core.cache import cache
def global_variables(request):
cache.set('user', request.user)
----------------------------------
in you model
from django.db.models.signals import pre_delete
from django.dispatch import receiver
from django.core.cache import cache
from news.models import News
#receiver(pre_delete, sender=News)
def news_delete(sender, instance, **kwargs):
user = cache.get('user')
in settings.py
TEMPLATE_CONTEXT_PROCESSORS = (
'web.context_processors.global_variables',
)
I'm struggling with the following.
I'm trying to create a custom signal that will trigger when the current time will be equal to the value of my model's notify_on DateTimeField.
Something like this:
class Notification(models.Model):
...
notify_on = models.DateTimeField()
def send_email(*args, **kwargs):
# send email
signals.when_its_time.connect(send_email, sender=User)
After I've read through all docs and I found no information on how to implement such a signal.
Any ideas?
UPDATE:
Less naive approach with ability to discard irrelevant tasks: https://stackoverflow.com/a/55337663/9631956
Ok, thanks to comments by #SergeyPugach I've done the following:
Added a post_save signal that calls a function that adds a task to the celery. apply_async let's you pass eta - estimated time of arrival which can accept DateTimeField directly, that's very convenient.
# models.py
from django.db.models import signals
from django.db import models
from .tasks import send_notification
class Notification(models.Model):
...
notify_on = models.DateTimeField()
def notification_post_save(instance, *args, **kwargs):
send_notification.apply_async((instance,), eta=instance.notify_on)
signals.post_save.connect(notification_post_save, sender=Notification)
And the actual task in the tasks.py
import logging
from user_api.celery import app
from django.core.mail import send_mail
from django.template.loader import render_to_string
#app.task
def send_notification(self, instance):
try:
mail_subject = 'Your notification.'
message = render_to_string('notify.html', {
'title': instance.title,
'content': instance.content
})
send_mail(mail_subject, message, recipient_list=[instance.user.email], from_email=None)
except instance.DoesNotExist:
logging.warning("Notification does not exist anymore")
I will not get into details of setting up celery, there's plenty of information out there.
Now I will try to figure out how to update the task after it's notification instance was updated, but that's a completely different story.
In django's documentation there is two interesting signals that may help you on this task: pre_save and post_save.
It depends on your needs, but let's say you want to check if your model's notify_on is equal to the current date after saving your model (actually after calling the save() or create() method). If it's your case you can do:
from datetime import datetime
from django.contrib.auth.models import User
from django.db import models
from django.dispatch import receiver
from django.db.models.signals import post_save
class Notification(models.Model):
...
# Every notification is related to a user
# It depends on your model, but i guess you're doing something similar
user = models.ForeignKey(User, related_name='notify', on_delete=models.DO_NOTHING)
notify_on = models.DateTimeField()
...
def send_email(self, *args, **kwargs):
"""A model method to send email notification"""
...
#receiver(post_save, sender=User)
def create_notification(sender, instance, created, **kwargs):
# check if the user instance is created
if created:
obj = Notification.objects.create(user=instance, date=datetime.now().date())
if obj.notify_on == datetime.now().date():
obj.send_email()
And you should know, that django signals won't work by they own only if there is an action that triggers them. What this mean is that Django signals won't loop over your model's instances and perform an operation, but django signals will trigger when your application is performing an action on the model connected to a signal.
Bonus: To perform a loop over your instances and process an action regulary you may need an asyncworker with a Queue database (mostly, Celery with Redis or RabbitMQ).
Catch django signal sent from celery task. Is it possible? As far as I know they are running in different processes
#celery.task
def my_task():
...
custom_signal.send()
#receiver(custom_signal)
def my_signal_handler():
...
Please take in mind that your async task must be #shared_task decorator. in order to be called from outside since it will not be attached to a concrete app instance. Documentation #shared_task celery
task.py
#shared_task
def send_email(email):
# Do logic for sending email or other task
signal.py
as you can see below this will only execute when post_save (After user executes save) for the model Contract in your case would be anyother model that its being executed.
#receiver(post_save, sender=Contract)
def inflation_signal(sender, **kwargs):
if kwargs['created']:
send_email.delay('mymail#example.com')
I'm trying to learn how to use celery to check a date every day on one of my models. One of my models holds an expiration date and a boolean field that says whether their insurance is expired or not.
The model is pretty big so I'm going to post a condensed version. I think I have two options. Either run a celery task on the model method or rewrite the function in my tasks.py. Then I need to use Celery beat to run the schedule to check daily.
I've got the function to work, but I'm passing in the model objects directly which I think is wrong.
I'm also having trouble on how to use the args in the celery beat scheduler inside of my celery.py.
I'm really close to getting this to work, but I think I'm going about executing a task in the wrong way. and I think executing a task on the model method might be the cleanest, I'm just not sure how to accomplish that.
models.py
class CarrierCompany(models.Model):
name = models.CharField(max_length=255, unique=True)
insurance_expiration = models.DateTimeField(null=True)
insurance_active = models.BooleanField()
def insurance_expiration_check(self):
if self.insurance_expiration > datetime.today().date():
self.insurance_active = True
self.save()
print("Insurance Active")
else:
self.insurance_active = False
self.save()
print("Insurance Inactive")
tasks.py
from __future__ import absolute_import, unicode_literals
from celery.decorators import task
from datetime import datetime, date
from django.utils import timezone
from .models import CarrierCompany
#task(name="insurance_expired")
def insurance_date():
carriers = CarrierCompany.objects.all()
for carrier in carriers:
date = datetime.now(timezone.utc)
if carrier.insurance_expiration > date:
carrier.insurance_active = True
carrier.save()
print("Insurance Active")
else:
carrier.insurance_active = False
carrier.save()
print("Insurance Inactive")
celery.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from celery.schedules import crontab
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.local')
app = Celery('POTRTMS')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
app.conf.beat_schedule = {
'check-insurance-daily': {
'task': 'insurance_expired',
'schedule': crontab(hour='8')
},
}
*** updated the beat schedule to reflect when I actually want to run it.
An example for how I might do it would be as follows. Also, instead of using traditional datetime, if you are including timezones in your Django App, you will probably want to use the timezone library instead found here.
models.py
class CarrierCompany(models.Model):
...
#property
def is_insurance_expired(self):
from django.utils import timezone
if self.insurance_expiration > timezone.datetime.today():
print("Insurance Active")
return True
else:
print("Insurance Active")
return False
tasks.py
def insurance_date():
carriers = CarrierCompany.objects.all()
for carrier in carriers:
if carrier.is_insurance_expired:
carrier.insurance_active = True
carrier.save()
else:
carrier.insurance_active = False
carrier.save()
There are other things you could do, like not update it if its False and make the default False, or vice versa in the True case. You could also just do all of that in the models function, though I personally like to keep logic a bit separate (just how I organize my stuff). Hopefully this helps get you past where you are stuck though.