I want to present general statistics about the instances of the class "ModelClass" on a web page for my users. Lets say there are some thousands ModelClass-objects, and that there are a lot of statistics that I need to calculate. I have figured out that I can do this with model managers, and here is a (very) simplified example:
class ModelClassCustomManager(models.Manager):
def get_query_set(self):
return super(ModelClassCustomManager, self).get_query_set().filter(is_complete = True)
class ModelClass(models.Model):
is_complete = models.BooleanField(default = False)
...
objects = models.Manager()
complete_objects = ModelClassCustomManager()
My concern is that this drains a lot of resources if this is calculated when users view the page. I would therefore like to calculate it only when I change or create new ModelClass objects since this is the only time the statistics really change. I guess this can be done by overriding the ModelClass save()-method.
What is the best way to save these results? Should I create another django-model for holding the calculated statistics, or is there another way of storing this information?
EDIT: Thanks to pastylegs for a good answer. Doing it this way, however causes some minor bugs, and I figured I'll explain them here, in case anyone else runs into this issue.
First of all, I incorrectly put the imports in the ModelClass at the top, so I got a circular dependency, which gave some weird results. To save yourself some frustration, put them where pastylegs did. Secondly, in order to overwrite the previous calculations of statistics (and not create a new one every time), just replace
if sender is ModelClass and instance is not Null:
count = ModelClass.objects.all().count()
stat = Stat(name='some_name', value=count).save()
with this:
if sender is ModelClass:
count = Match.objects.all().count()
try:
stat = Stat.objects.get(key="Total")
stat.update(key="Total", value=count) #Update statistic
except:
Stat(key="Total", value=count).save() #Create new
I'd create an app called statistics (or similar) that has a simple model to hold key,value pairs:
class Stat(models.Model):
key = models.CharField(..)
value = models.CharField(...)
then use a signal on your own model to run a function every time a user saves/updates a instance of your model. In models.py:
class ModelClass(models.Model):
...
from django.db.models.signals import post_save
from myapp.signal import updates_stats
post_save.connect(updates_stats, sender=ModelClass)
and then create the receiver function to calculate what ever statistics you need in signals.py
from myapp.models import ModelClass
from stats.models import Stat
def update_stats(sender, instance, signal, *args, **kwargs):
if sender is ModelClass and instance is not Null:
count = ModelClass.objects.all().count()
stat = Stat(name='some_name', value=count).save()
This is a very simple and basic approach and only outlines how you can make use of signals to perform calculations but it is a good start. Ideally you should try to calculate these values outside of the request/response cycle (especially if they are big caculations) as it will hang your server and potentially time out your requests, so you should consider using a queuing/task system like Celery to perform the calculation in the background
Related
I have a medium size Django REST app that I'm looking to add gamification features to.
The application in question is a school webapp where students can create mockup quizzes, participate in exams that teachers publish, and write didactical content that gets voted by other students and teachers.
I want to add some gamification features to make the app more interesting and to incentivize participation and usage of the various features: for example, each student will have a personal "reputation" score, and gain points upon completing certain actions--a student may gain points when completing a quiz with a high score, when submitting some content, or when receiving upvotes to such content.
The tricky part is I want to be able to have this logic be as separate as possible from the existing codebase, for various reasons: separation of concerns, ability to plug the engine in/out if needed, ability to easily deactivate features for certain groups of users, etc.
What I'm looking for here is some software engineering advice that's also Django-specific. Here's a high level description of what I'm thinking of doing--I'd like some advice on the approach.
create a new gamification app. Here I will have models that describe a change in reputation for a user and possibly other related events. The app should also send notifications when gamification-related events occur
from the gamification app, expose a callback-based interface, which the other primary app can call into to dispatch events
use the django-lifecycle package to call the callbacks from gamification when triggers occur.
This way, my existing models would only get touched to register the triggers from django-lifecycle (similar to signals). For example, let's say I want to give students points when they turn in an assignment. Let's say I have an AssignmentSubmission model to handle assignment submissions. With the added lifecycle hook, it'd look like this:
class AssignmentSubmission(models.Model):
NOT_TURNED_IN = 0
TURNED_IN = 1
STATES = ((NOT_TURNED_IN, 'NOT_TURNED_IN'), (TURNED_IN, 'TURNED_IN'))
user = models.ForeignKey(user)
assignment = models.ForeignKey(assignment)
state = models.PositiveSmallIntegerField(choices=STATES, default=NOT_TURNED_IN)
#hook(AFTER_UPDATE, when="state", was=NOT_TURNED_IN, is_now=TURNED_IN)
def on_turn_in(self):
get_gamification_interface().on_assignment_turn_in(self.user)
The on_assignment_turn_in method might look something like:
def on_assignment_turn_in(user):
ReputationIncrease.objects.create(user, points=50)
notifications.notify(user, "You gained 50 points")
This is pretty much just a sketch to give an idea.
I am unsure how get_gamification_interface() would work. Should it return a singleton? Maybe instantiate an object? Or return a class with static methods? I think it'd be best to have a getter like this as opposed to manually importing methods from the gamification app, but maybe it could also create too much overhead.
What's a good way to handle adding "pluggable" features to a project that are inherently entangled with existing models and business logic while also touching those as little as possible?
The foreign key approach is fine. You can easily chain and link queries using information from existing tables and you could even avoid touching the original code by importing your models to the new app. You can use Django signals in your new app and ditch the django-lifecycle extension to avoid adding lines to your core models. I used the following approach to keep track of modified records in a table; take a TrackedModel with fields field_one, field_two, field_n... which will be tracked by one of your new app's model, namely RecordTrackingModel:
from parent_app.models import TrackedModel # The model you want to track from a parent app.
from django.db.models.signals import post_save # I'm choosing post_save just to illustrate.
from django.dispatch import receiver
from datetime import datetime
class RecordTrackingModel(models.Model):
record = models.ForeignKey(TrackedModel, verbose_name=("Tracked Model"), on_delete=models.CASCADE)
field_one = models.TextField(verbose_name=("Tracked Field One"), null=True, blank=True) # Use same field type as original
field_two = models.TextField(("Tracked Field Two"))
field_n = ...
notes = models.TextField(verbose_name=("notes"), null=True, blank=True)
created = models.DateTimeField(verbose_name=("Record creation date"), auto_now=False, auto_now_add=True)
#receiver(post_save, sender=TrackedModel) # Here, listen for the save signal coming from a saved or updated TrackedModel record.
def modified_records(instance, **kwargs):
record = instance
tracked_field_one = instance.field_one
tracked_field_two = instance.field_two
tracked_field_n = another_function(instance.field_n) #an external function that could take a field content as input.
...
note = 'Timestamp: ' + str(datetime.now().isoformat(timespec='seconds'))
track_record = RecordTrackingModel.objects.create(record=record, field_one=tracked_field_one, field_two=tracked_field_two, field_n=tracked_field_n, ..., notes=note)
return track_record
There's no need to add functions to your pre-existing models as the signal dispatcher triggers whenever a save or delete signal appears at TrackedModel. Then you could place "if" statements for wether or not to perform actions based on field values, i.e.: just pass if an AssignmentSubmission record has a "Not Turned In" status.
Check Django signals reference for more information about when they trigger.
Additionally, I would suggest to change the "state" field to boolean type and rename it to something more explicit like "is_turned_in" for ease of use. It will simplify your forms and code. Edit: For more than 2 choices (non-boolean), I prefer using ForeignKey instead. It will let you modify choices over time easily from the admin site.
Edit:
Another approach could be mirroring the original models in your gamification app and call for a mirror record update when a save method is used in the original model.
gamification_app/models.py:
from parent_app.models import OriginalModel # The model you want to track from a parent app.
from django.db.models.signals import post_save # I'm choosing post_save just to illustrate.
from django.dispatch import receiver
from datetime import datetime
def gamification_function(input, parameters):
output = *your gamification logic*
return output
class MirrorModel(models.Model):
original_model = (OriginalModel, verbose_name=("Original Model"), on_delete=models.CASCADE)
field_one = ... #Same type as the original
field_two = ...
field_n = ...
#hook(AFTER_UPDATE, when="field_n", was=OPTION_1, is_now=OPTION_2)
def on_turn_in(self):
gamification_function(self.field, self.other_field)
#receiver(post_save, sender=OriginalModel) # Here, listen for the save signal coming from the original app change record.
def mirror_model_update(instance, **kwargs):
pk = instance.pk
mirror_model = []
if MirrorModel.objects.get(original_model.pk=pk).exists():
mirror_model = MirrorModel.objects.get(original_model.pk=pk)
mirror_model.field_one = instance.field_one # Map field values
*other logic ...*
mirror_model.save() # This should trigger your hook
else:
mirror_model = MirrorModel(original_model = instance, field_one = instance.field_one, ...)
mirror_model.save() #This should trigger your hooks as well, but for a new record
This way you can decouple the gamification app and even choose not to keep a record of all or the same fields as the original model, but fields specific to your functionality needs.
Your idea was good.
In the gamification app add your views, protect it with LoginRequiredMixin, and extend it with a check if a related record in the AssignmentSubmission table exists and is turned on.
In this way you have a 100% separated gamification views, models, logic, ecc...
In the existing views you can add a link to a gamification view.
I have django model of a Task. In my website Tasks can be repeated and not repeated. So I override init method in this way:
def __init__(self, *args, **kwargs):
"""Init new Task
:param period - timedelta object. How often should task be repeated
:param repeated - if true task will be repeated
:param end_date - datetime until which tasks will be repeated or it will be repeated for 10 years
"""
if not self.from_database() # I don't now how to do it
if kwargs.get("repeated"):
period = kwargs["period"]
end_date = kwargs["end_date"] or kwargs["date_time"] + TEN_YEARS
base_task = self
date = base_task.date_time + period
with transaction.atomic():
while date <= end_date:
base_task.clone_task(date)
date += period
try:
del kwargs["repeated"]
del kwargs["period"]
del kwargs["end_date"]
except KeyError:
pass
super().__init__(*args, **kwargs)
But i have problem. How can I check if this instance initing from database or it is new object.
First, I am not sure I would have designed your tasks creation this way. The task object should scope itself and avoid having to deal with other tasks. Maybe, a task creator handling the option of creating repeated tasks might be more appropriate. It is just an opinion without knowing your context but it will avoid your issue without having to do any special stuff.
This being said, as per Django documentation you should avoid the override of the init() method. It proposes 2 other approches. One of them involve overriding the Manager.
https://docs.djangoproject.com/en/3.2/ref/models/instances/
It seams you which to encapsulate some logic from inside your object and in a oop perspective, this is debatable but understandable. However, in the django way of doing things, you should encapsulate this kind of logic in a specialized manager.
I use this way of doing things to maintain fields such as created_at, updated_at, deleted_at and is_deleted in many of my entities and it works fine.
I would not recommand creating objects from inside an object creation. If you want to do so, be sure here, your clone_task method does not include the repeated argument otherwise each repeated task will create many repeated task and you will end up with many duplicates.
This being said, in a model, the following code will return True if object is newly created and False if comming from the db:
self._state.adding
Django documentation about _state
If you can think of a better way to phrase my question, by all means, please change it. Here's the situation I'm facing.
I have 2 models: Agent, Deal. Here's the simplified version of what they look like:
class Agent(models.Model):
name = models.CharField(max_length=100)
price_sum_of_all_deals = models.IntegerField()
class Deal(models.Model):
agent = models.ForeignKey(Agent, on_delete=models.CASCADE)
address = models.CharField(max_length=100)
price = models.IntegerField()
I'm using celery beat to check an API to see if there are any new deals for each agent. With my current configuration, I am searching for a new deal from within a task method and if I find a deal, I add the price of that deal to the price_sum_of_all_deals field of the corresponding agent. The summary of the task looks likes this:
from celery import shared_task
from agents import models
#shared_task
def get_deals():
agents = models.Agent.objects.all()
for agent in agents:
price, address = get_new_deal_from_api(agent.name)
new_deal = models.Deal(agent=agent, address=address, price=price)
new_deal.save()
agent.price_sum_of_all_deals += price
agent.save()
This, however, is not very intuitive and feels like an unnecessary abstraction. Is there a better way of calculating the price_sum_of_all_deals from within the model? What is the best practice here?
I'm relatively new to Django so if there's something glaring that I overlooked, I apologize.
I don't think that's the best way to process, because if one deal is deleted, how do you update price_sum_of_all_deals? You won't be 100% certain that this value is always accurate (thus, it is not acceptable).
Here is one way to do it, without storing the sum value in an attribute of a model:
use your API as usual,
just create the Deal instance and save it,
to get the sum of all deals of an agent at anytime, you can use the following:
Deal.objects.filter(agent=this_agent).aggregate(price_sum_of_all_deals=Sum('price'))
The result of this line is a dict containing a price_sum_of_all_deals with the desired value.
Source:
https://docs.djangoproject.com/en/3.0/topics/db/aggregation/
https://docs.djangoproject.com/en/3.0/ref/models/querysets/
I hope that is what you needed!
I have a model (lets call it Entity) that has an attribute (Attribute) that changes over time, but I want to keep a history of how that attribute changes in the database. I need to be able to filter my Entities by the current value of Attribute in its manager. But because Django (as far as I can tell) won't let me do this in one query natively, I have created a database view that produces the latest value of Attribute for every Entity. So my model structure looks something like this:
class Entity(models.Model):
def set_attribute(self, value):
self.attribute_history.create(value=value)
def is_attribute_positive(self, value):
return self.attribute.value > 0
class AttributeEntry(models.Model):
entity = models.ForeignKey(Entity, related_name='attribute_history')
value = models.IntegerField()
time = models.DateTimeField(auto_now_add=True)
class AttributeView(models.Model)
id = models.IntegerField(primary_key=True, db_column='id',
on_delete=models.DO_NOTHING)
entity = models.OneToOneField(Entity, related_name='attribute')
value = models.IntegerField()
time = models.DateTimeField()
class Meta:
managed = False
My database has the view that produces the current attribute, created with SQL like this:
CREATE VIEW myapp_attributeview AS
SELECT h1.*
FROM myapp_attributehistory h1
LEFT OUTER JOIN myapp_attributehistory h2
ON h1.entity_id = h2.entity_id
AND (h1.time < h2.time
OR h1.time = h2.time
AND h1.id < h2.id)
WHERE h2.id IS NULL;
My problem is that if I set the attribute on a model object using set_attribute() checking it with is_attribute_positive() doesn't always work, because Django may be caching that the related AttributeView object. How I can I make Django update its model, at the very least by requerying the view? Can I mark the attribute property as dirty somehow?
PS: the whole reason I'm doing this is so I can do things like Entity.objects.filter(attribute__value__exact=...).filter(...), so if someone knows an easier way to get that functionality, such an answer will be accepted, too!
I understand that the attribute value is modified by another process (maybe not even Django) accessing the same database. If this is not the case you should take a look at django-reversion.
On the other hand if that is the case, you should take a look at second answer of this. It says that commiting transaction invalidate query cache and offer this snippet.
>>> from django.db import transaction
>>> transaction.enter_transaction_management()
>>> transaction.commit() # Whenever you want to see new data
I never directly solved the problem, but I was able to sidestep it by changing is_attribute_positiive() to directly query the database table, instead of the view.
def is_attribute_positive(self, value):
return self.attribute_history.latest().value > 0
So while the view gives me the flexibility of being able to filter queries on Entity, it seems the best thing to do once the object is received is to operate directly on the table-backed model.
This seems really simple.
On my model save() I want to basically do a get_or_create(). So I want to update the model if it exists or create a new one if not.
This seems like a super simple problem, but I am not getting it right!
class StockLevel(models.Model):
stock_item = models.ForeignKey(StockItem)
current_stock_level = models.IntegerField(blank=True, default=0)
def save(self):
try:
# it exists
a = StockLevel.objects.get(stock_item=self.stock_item)
a.current_stock_level = self.current_stock_level
a.save()
except:
# is does not exist yet
# save as normaly would.
super(StockLevel, self).save()
OR
def save(self):
stock_level_item , created = StockLevel.objects.get_or_create(stock_item=self.stock_item)
stock_level_item.current_stock_level = self.current_stock_level
stock_level_item.save()
This would also go into a infinite loop.
This would just put the save() in an infinite loop. But that is the basic idea of how it should work.
Django uses the same save() method for both creating and updating the object.
User code doesn't need to determine whether to create or update the object, since this is done by the method itself.
Furthermore you can force the save() method to create or update the object by using the methods optional arguments.
This is covered in the Django docs.
This really doesn't sound like the best way to do this. The save method is meant for saving the current instance, not magically querying for an existing one. You should take care of this in the form or view code.
So this is how i solved a similar situation just yesterday,
I created a duplicate model to store the updated information.
let's call the new model "StockLevelUpdates".
I then used signals to insert the saved data from the original model.
I will use your model above as the original model to explain further.
class StockLevelUpdates(models.Model):
stock_item = models.ForeignKey(StockItem)
current_stock_level = models.IntegerField(blank=True, default=0)
#receiver(signals.post_save, sender=StockLevel)
def update_info(sender, instance, **kwargs):
try:
# if it exists
a = StockLevelUpdates.objects.get(stock_item=instance.stock_item)
a.current_stock_level = instance.current_stock_level
a.save()
except:
# if it does not exist yet
# save the new instance
obj = StockLevelUpdates(stock_item=instance.stock_item,
current_stock_level = instance.current_stock_level)
obj.save()
This worked well for me, and you can get all your update reports from the duplicate model.
Perhaps there is a better way to do this kind of thing but this was a quick way out of a sticky situation.