Django : Recalculating mean value in the database after creating a new instance - django

I have informations about companies presented in a table. One of the field of this table is the mean value of each note the company received ('note_moyenne' in models.FicheIdentification).
By clicking on a button, people are able to submit a new note for the company ('note' in models.EvaluationGenerale). I want the mean value of the notes to update in the database each time someone submit a new note.
Here is my models.py :
class FicheIdentification(models.Model):
entreprise=models.ForeignKey(Entreprise, on_delete=models.CASCADE)
note_moyenne=models.IntegerField()
def __str__(self):
return self.entreprise.nom_entreprise
class EvaluationGenerale(models.Model):
entreprise=models.ForeignKey(Entreprise, on_delete=models.CASCADE)
note=models.IntegerField()
commentaires=models.CharField(max_length=1000)
date_evaluation=models.DateField(auto_now_add=True)
def __str__(self):
return self.commentaires
views.py :
class CreerEvaluationGenerale(CreateView):
form_class = FormulaireEvaluationGenerale
model = EvaluationGenerale
def form_valid(self, form):
form.instance.entreprise=Entreprise.objects.filter(siret=self.kwargs['siret']).first()
return super(CreerEvaluationGenerale, self).form_valid(form)
def get_success_url(self):
return reverse('details-evaluations')
Currently I just display the mean value in my table using this
def render_evaluation(self, record):
return (EvaluationGenerale.objects.filter(entreprise=record.entreprise.siret).aggregate(Avg('note'))['note__avg'])
but I really don't like this solution as I want the value to be stored in the database, in FicheIdentification.note_moyenne.
I thought about creating a UpdateView class but couldn't manage to link it with my CreateView.
Any help or documentation would be really appreciated, I'm a bit lost right know...

Typically, you would not store calculated fields. The usual way is not to store the average, but to use an annotation/aggregation in your query.
To centralize this to your model, you would want to write a custom model manager to implement this, so it can be reused anywhere you use your model without rewriting the logic.
class MyModelManager(models.Manager):
def note_average(self, **filter_kwargs):
qs = self.get_queryset()
# replace `...` with your aggregation as needed
return qs.filter(**filter_kwargs).aggregate(...)
class EvaluationGenerale(models.Model):
objects = MyModelManager() # override the default manager
# ... the rest of the model as-is
Then you can use something like the following in your view(s):
EvaluationGenerale.objects.note_average(entreprise=record.entreprise.siret)
See for additional reference: How to add a calculated field to a Django model

I see two ways of doing it.
Either a listener post_save on EvaluationGenerale (doc). You'll be able to compute the new average each time a new EvaluationGenerale is entered in DB.
#receiver(post_save, sender=EvaluationGenerale)
def evaluation_general_note_moyenne_computer_post_save_listener(sender, instance, **kwargs):
entreprise = instance.entreprise
entreprise.note_moyenne = entreprise.evaluationgeneral_set.aggregate(Avg('note')).values()[0])
entreprise.save()
post save listener will only trigger on instance.save() and models.objects.create() not on queryset.update() or model.objects.bulk_create().
Either overriding the save (doc) function of your form to compute the average after the creation of the new EvaluationGenerale
def save(self):
instance = super.save()
entreprise = instance.entreprise
entreprise.note_moyenne = entreprise.evaluationgeneral_set.aggregate(Avg('note')).values()[0]
entreprise.save()
return instance

Assuming there is as single FicheIdentification object per enterprise, you could update the note_moyenne field when you save the EvaluationGenerale object, like:
obj = FicheIdentification(...)
FicheIdentification.objects.filter(entreprise=record.entreprise.siret).update(note_moyenne=obj.aggregate(Avg('note'))['note__avg']
obj.save()
Please let me know if it works.

Related

Django create new instances of a different model using the submitted form data of a model

I have a Topic model which has a ManyToManyField to the Tag model.
Similarly to stack overflow where you can submit new tags at the time of asking a question. I want to be able to create new tags when you create a topic.
A hacky solution from the top of my head.
Class TopicCreateView(CreateView):
model = Topic
template_name = 'blah'
fields = [
'blah'
]
def form_valid(self, form):
tags = self.request.POST.getlist(tags)
for tag in tags:
if not Tag.objects.filter(tag_string__iexact=tag):
try:
new_tag = Tag.objects.create(tag_string=tag)
except IntegrityError:
# stuff
return super().form_valid(form)
I'm new to Django and web frameworks in general, but this seems really hacky to me (if it even works). Given what I've read so far regarding FormSets and such, is there not a better way to achieve this?
Since the model code is not provided, I'm just guessing what you are thinking about:
def form_valid(self, form):
tags = self.request.POST.getlist('tags')
existed_tags = Tag.objects \
.filter(tag_string__iexact=tags) \
.values_list('tag_string', flat=True)
new_tags = set(tags) - set(existed_tags)
for tag in new_tags:
Tag.objects.create(tag_string=tag)
return super().form_valid(form)
Perhaps your original codes can work but just need improvement (and thus I don't think it was 'hacky'). You put the Tag QuerySet within a for loop so the database will be hit in each time of iteration. To get all of the tag values from db first then compare the difference and do further work may be a better way.
Moreover, I think tag creation should not put in form_valid since this is a 'model procedure'. To override the save() method of Topic model or using Signal may be a better choice. Yet this is just my preference and you can still keep this unchanged.
I came up with a more acceptable solution keeping the model creation out of the View() and in the Form(), as defining form_class is still possible with the CreateView().
I took inspiration from how stack overflow handles tags, in that they accept a string and split each tag on '+'.
There may be a way to further improve this but FormSets did not seem like viable option in this instance.
Class TopicCreateForm(forms.ModelForm)
submitted_tags = forms.CharField(max_length=256)
class Meta:
model = Blah
fields = ['blah']
def clean_submitted_tags(self):
tags = self.cleaned_data.get(
'submitted_tags').split("+")
# validation here
return tags
def save(self, *args, **kwargs):
# get author from form instance
author = self.instance.author
tags = self.cleaned_data['submitted_tags']
existing_tags = Tag.objects.values_list(
'tag_string', flat=True)
for tag in [t for t in tags if t not in existing_tags]:
new_tag = Tag.objects.create(
tag_string=tag,
created_by=author)
new_tags = Tag.objects.filter(tag_string__in=tags)
new_tag_uuids = [t.pk for t in new_tags]
self.cleaned_data['tags'] = new_tag_uuids
return super().save(*args, **kwargs)

update django choice field with database results

I am developing an application using django where the UI needs to be updated when user interacts with it. For instance I have a Drop down field where the user selects a drink and submits it then based on that a dropdown with the places that drink is available, price and quantity at each place needs to be displayed. The user will then further submit the form for second process.
From my understanding the Forms in django are pre-defined and I am not able to think of a way using which I could achieve this.
What I could come up was defining a regular form class
class dform(forms.Form):
SOURCES_CHOICES = (
(A, 'A'),
(E, 'E'),
)
drink = forms.ChoiceField(choices = SOURCES_CHOICES)
location = forms.ChoiceField(choices = **GET THIS FROM DATABASE**)
quantity = forms.ChoiceField(choices = **GET THIS FROM DATABASE**)
.
.
.
My view is like,
def getdrink():
if request.method == 'POST':
#code for handling form
drink = dform.cleaned_data['drink']
#code to get values from database
I have no idea how to generate or populate or append the values i get from the database to the choicefield in my form. I did try looking up on SO but none of the solutions here explained properly how to do it. Also, due to certain requirements I am not using the models. So my database is not at all related to the models.
I am at a total loss Please help me out
class MyForm(forms.Form):
my_choice_field = forms.ChoiceField(choices=MY_CHOICES)
So if you want the values to be dynamic(or dependent of some logic) you can simply modify your code to something like this:
either
def get_my_choices():
# you place some logic here
return choices_list
class MyForm(forms.Form):
my_choice_field = forms.ChoiceField(choices=get_my_choices())
or
User_list = [ #place logic here]
class MyForm(forms.Form):
my_choice_field = forms.ChoiceField(choices=get_my_choices())
but once database value is updated, new data value will be popoulated only on restart of server.
So write a function like this in forms:
class MyForm(forms.Form):
def __init__(self, *args, **kwargs):
super(MyForm, self).__init__(*args, **kwargs)
self.fields['my_choice_field'] = forms.ChoiceField( choices=get_my_choices() )
or in place of the get_my_choices u can ad the USER_LIST too.
If you have models for location and quantity, a ModelChoiceField should work:
class dform(forms.Form):
location = forms.ModelChoiceField(queryset = Location.objects.all())
Otherwise, you'll need to query the database directly, for example:
class dform(forms.Form):
location = forms.ChoiceField(choices = get_location_choices())
# elsewhere
from django.db import connection
def get_location_choices():
cursor = connection.cursor()
cursor.execute("select location_id, name from location_table")
return cursor.fetchall()
The SQL query to use here depends on your database engine and table schema.
I think that, based on my understanding of your question, the best solution would be to include JSON objects with your form and load these using jQuery instead of submitting the form over and over. Included in your form, you should add something like:
class MyForm(forms.Form):
CHOICE_DICT = {
'choice_1': [
'option_1',
'option_2',
],
etc...
Then you should include form.CHOICE_DICT in your context, load that with jQuery, and render it depending on changes to other fields.

DRF - How to get WritableField to not load entire database into memory?

I have a very large database (6 GB) that I would like to use Django-REST-Framework with. In particular, I have a model that has a ForeignKey relationship to the django.contrib.auth.models.User table (not so big) and a Foreign Key to a BIG table (lets call it Products). The model can be seen below:
class ShoppingBag(models.Model):
user = models.ForeignKey('auth.User', related_name='+')
product = models.ForeignKey('myapp.Product', related_name='+')
quantity = models.SmallIntegerField(default=1)
Again, there are 6GB of Products.
The serializer is as follows:
class ShoppingBagSerializer(serializers.ModelSerializer):
product = serializers.RelatedField(many=False)
user = serializers.RelatedField(many=False)
class Meta:
model = ShoppingBag
fields = ('product', 'user', 'quantity')
So far this is great- I can do a GET on the list and individual shopping bags, and everything is fine. For reference the queries (using a query logger) look something like this:
SELECT * FROM myapp_product WHERE product_id=1254
SELECT * FROM auth_user WHERE user_id=12
SELECT * FROM myapp_product WHERE product_id=1404
SELECT * FROM auth_user WHERE user_id=12
...
For as many shopping bags are getting returned.
But I would like to be able to POST to create new shopping bags, but serializers.RelatedField is read-only. Let's make it read-write:
class ShoppingBagSerializer(serializers.ModelSerializer):
product = serializers.PrimaryKeyRelatedField(many=False)
user = serializers.PrimaryKeyRelatedField(many=False)
...
Now things get bad... GET requests to the list action take > 5 minutes and I noticed that my server's memory jumps up to ~6GB; why?! Well, back to the SQL queries and now I see:
SELECT * FROM myapp_products;
SELECT * FROM auth_user;
Ok, so that's not good. Clearly we're doing "prefetch related" or "select_related" or something like that in order to get access to all the products; but this table is HUGE.
Further inspection reveals where this happens on Line 68 of relations.py in DRF:
def initialize(self, parent, field_name):
super(RelatedField, self).initialize(parent, field_name)
if self.queryset is None and not self.read_only:
manager = getattr(self.parent.opts.model, self.source or field_name)
if hasattr(manager, 'related'): # Forward
self.queryset = manager.related.model._default_manager.all()
else: # Reverse
self.queryset = manager.field.rel.to._default_manager.all()
If not readonly, self.queryset = ALL!!
So, I'm pretty sure that this is where my problem is; and I need to say, don't select_related here, but I'm not 100% if this is the issue or where to deal with this. It seems like all should be memory safe with pagination, but this is simply not the case. I'd appreciate any advice.
In the end, we had to simply create our own PrimaryKeyRelatedField class to override the default behavior in Django-Rest-Framework. Basically we ensured that the queryset was None until we wanted to lookup the object, then we performed the lookup. This was extremely annoying, and I hope the Django-Rest-Framework guys take note of this!
Our final solution:
class ProductField(serializers.PrimaryKeyRelatedField):
many = False
def __init__(self, *args, **kwargs):
kwarsgs['queryset'] = Product.objects.none() # Hack to ensure ALL products are not loaded
super(ProductField, self).__init__(*args, **kwargs)
def field_to_native(self, obj, field_name):
return unicode(obj)
def from_native(self, data):
"""
Perform query lookup here.
"""
try:
return Product.objects.get(pk=data)
except Product.ObjectDoesNotExist:
msg = self.error_messages['does_not_exist'] % smart_text(data)
raise ValidationError(msg)
except (TypeError, ValueError):
msg = self.error_messages['incorrect_type'] % type(data)
raise ValidationError(msg)
And then our serializer is as follows:
class ShoppingBagSerializer(serializers.ModelSerializer):
product = ProductField()
...
This hack ensures the entire database isn't loaded into memory, but rather performs one-off selects based on the data. It's not as efficient computationally, but it also doesn't blast our server with 5 second database queries loaded into memory!

tastypie with django-simple-history - display model history as rest API

I would like to share django model history (created by django-simple-history) using tastypie.
Problem is, how to prepare ModelResource for this purpose.
Access to model history is by model.history manager. So access to all changes of model we can gain by model.history.all()
What i would like to obtain? For example. I have django model Task and the API endpoints:
http://127.0.0.1/api/v1/task - display all tasks list
http://127.0.0.1/api/v1/task/1 - display details for choosen task
http://127.0.0.1/api/v1/task/1/history - display history of task no. 1
First two links presents default behavior of ModelResource. what i have till now?
class TaskResource(ModelResource):
class Meta:
# it displays all available history entries for all task objects
queryset = Task.history.all()
resource_name = 'task'
def prepend_urls(self):
return [
url(r"^(?P<resource_name>%s)/(?P<pk>\w[\w/-]*)/history$" % (self._meta.resource_name,),
self.wrap_view('get_history'),
name="api_history"),
]
def get_history(self, request, **kwargs):
#...
get_history should return bundle with history entries.. but how this method should look?
I guess, i need to create bundle with needed data, but don't know how exactly should i do that.
Does someeone have experience with simple-history and tastypie to present some simple example?
It seems, solution was simpler than i thought. Maybe someone use this in feature:
class TaskHistoryResource(ModelResource):
class Meta:
queryset = Task.history.all()
filtering = { 'id' = ALL }
class TaskResource(ModelResource):
history = fields.ToManyField(AssetTypeHistoryResource, 'history')
class Meta:
# it displays all available history entries for all task objects
queryset = Task.history.all()
resource_name = 'task'
def prepend_urls(self):
return [
url(r"^(?P<resource_name>%s)/(?P<pk>\w[\w/-]*)/history$" %(self._meta.resource_name,),
self.wrap_view('get_history'),
name="api_history"),
]
def get_history(self, request, **kwargs):
try:
bundle = self.build_bundle(data={'pk': kwargs['pk']}, request=request)
obj = self.cached_obj_get(bundle=bundle, **self.remove_api_resource_names(kwargs))
except ObjectDoesNotExist:
return HttpGone()
except MultipleObjectsReturned:
return HttpMultipleChoices("More than one resource is found at this URI.")
history_resource = TaskHistoryResource()
return history_resource.get_list(request, id=obj.pk)
A bit changed solution from:
http://django-tastypie.readthedocs.org/en/latest/cookbook.html#nested-resources
Basically, there was need to create additional resource with history entries. get_history method creates and returns instance of it with appropriate filter on id field (in django-simple-history id field contain id of major object. Revision primary key names history_id)
Hope, that will help someone.

'private' models, default query sets and chaining methods

I have a private boolean flag on my model, and a custom manager that overwrites the get_query_set method, with a filter, removing private=True:
class myManager(models.Manager):
def get_query_set(self):
qs = super(myManager, self).get_query_set()
qs = qs.filter(private=False)
return qs
class myModel(models.Model):
private = models.BooleanField(default=False)
owner = models.ForeignKey('Profile', related_name="owned")
#...etc...
objects = myManager()
I want the default queryset to exclude the private models be default as a security measure, preventing accidental usage of the model showing private models.
Sometimes, however, I will want to show the private models, so I have the following on the manager:
def for_user(self, user):
if user and not user.is_authenticated():
return self.get_query_set()
qs = super(myManager, self).get_query_set()
qs = qs.filter(Q(owner=user, private=True) | Q(private=False))
return qs
This works excellently, with the limitation that I can't chain the filter. This becomes a problem when I have a fk pointing the myModel and use otherModel.mymodel_set. otherModel.mymodel_set.for_user(user) wont work because mymodel_set returns a QuerySet object, rather than the manager.
Now the real problem starts, as I can't see a way to make the for_user() method work on a QuerySet subclass, because I can't access the full, unfiltered queryset (basically overwriting the get_query_set) form the QuerySet subclass, like I can in the manager (using super() to get the base queryset.)
What is the best way to work around this?
I'm not tied to any particular interface, but I would like it to be as djangoy/DRY as it can be. Obviously I could drop the security and just call a method to filter out private tasks on each call, but I really don't want to have to do that.
Update
manji's answer below is very close, however it fails when the queryset I want isn't a subset of the default queryset. I guess the real question here is how can I remove a particular filter from a chained query?
Define a custom QuerySet (containing your custom filter methods):
class MyQuerySet(models.query.QuerySet):
def public(self):
return self.filter(private=False)
def for_user(self, user):
if user and not user.is_authenticated():
return self.public()
return self.filter(Q(owner=user, private=True) | Q(private=False))
Define a custom manager that will use MyQuerySet (MyQuerySet custom filters will be accessible as if they were defined in the manager[by overriding __getattr__]):
# A Custom Manager accepting custom QuerySet
class MyManager(models.Manager):
use_for_related_fields = True
def __init__(self, qs_class=models.query.QuerySet):
self.queryset_class = qs_class
super(QuerySetManager, self).__init__()
def get_query_set(self):
return self.queryset_class(self.model).public()
def __getattr__(self, attr, *args):
try:
return getattr(self.__class__, attr, *args)
except AttributeError:
return getattr(self.get_query_set(), attr, *args)
Then in the model:
class MyModel(models.Model):
private = models.BooleanField(default=False)
owner = models.ForeignKey('Profile', related_name="owned")
#...etc...
objects = myManager(MyQuerySet)
Now you can:
¤ access by default only public models:
MyModel.objects.filter(..
¤ access for_user models:
MyModel.objects.for_user(user1).filter(..
Because of (use_for_related_fields = True), this same manager wil be used for related managers. So you can also:
¤ access by default only public models from related managers:
otherModel.mymodel_set.filter(..
¤ access for_user from related managers:
otherModel.mymodel_set.for_user(user).filter(..
More informations: Subclassing Django QuerySets & Custom managers with chainable filters (django snippet)
To use the chain you should override the get_query_set in your manager and place the for_user in your custom QuerySet.
I don't like this solution, but it works.
class CustomQuerySet(models.query.QuerySet):
def for_user(self):
return super(CustomQuerySet, self).filter(*args, **kwargs).filter(private=False)
class CustomManager(models.Manager):
def get_query_set(self):
return CustomQuerySet(self.model, using=self._db)
If you need to "reset" the QuerySet you can access the model of the queryset and call the original manager again (to fully reset). However that's probably not very useful for you, unless you were keeping track of the previous filter/exclude etc statements and can replay them again on the reset queryset. With a bit of planning that actually wouldn't be too hard to do, but may be a bit brute force.
Overall manji's answer is definitely the right way to go.
So amending manji's answer you need to replace the existing "model"."private" = False with ("model"."owner_id" = 2 AND "model"."private" = True ) OR "model"."private" = False ). To do that you will need to walk through the where object on the query object of the queryset to find the relevant bit to remove. The query object has a WhereNode object that represents the tree of the where clause, with each node having multiple children. You'd have to call the as_sql on the node to figure out if it's the one you are after:
from django.db import connection
qn = connection.ops.quote_name
q = myModel.objects.all()
print q.query.where.children[0].as_sql(qn, connection)
Which should give you something like:
('"model"."private" = ?', [False])
However trying to do that is probably way more effort than it's worth and it's delving into bits of Django that are probably not API-stable.
My recommendation would be to use two managers. One that can access everything (an escape hatch of sort), the other with the default filtering applied. The default manager is the first one, so you need to play around with the ordering depending on what you need to do. Then restructure your code to know which one to use - so you don't have the problem of having the extra private=False clause in there already.