Django Inline for ManyToMany generate duplicate queries - django

I'm experiencing some major performing issue with my django admin. Lots of duplicate queries based on how many inlines that I have.
models.py
class Setting(models.Model):
name = models.CharField(max_length=50, unique=True)
class Meta:
ordering = ('name',)
def __str__(self):
return self.name
class DisplayedGroup(models.Model):
name = models.CharField(max_length=30, unique=True)
position = models.PositiveSmallIntegerField(default=100)
class Meta:
ordering = ('priority',)
def __str__(self):
return self.name
class Machine(models.Model):
name = models.CharField(max_length=20, unique=True)
settings = models.ManyToManyField(
Setting, through='Arrangement', blank=True
)
class Meta:
ordering = ('name',)
def __str__(self):
return self.name
class Arrangement(models.Model):
machine = models.ForeignKey(Machine, on_delete=models.CASCADE)
setting = models.ForeignKey(Setting, on_delete=models.CASCADE)
displayed_group = models.ForeignKey(
DisplayedGroup, on_delete=models.PROTECT,
default=1)
priority = models.PositiveSmallIntegerField(
default=100,
help_text='Smallest number will be displayed first'
)
class Meta:
ordering = ('priority',)
unique_together = (("machine", "setting"),)
admin.py
class ArrangementInline(admin.TabularInline):
model = Arrangement
extra = 1
class MachineAdmin(admin.ModelAdmin):
inlines = (ArrangementInline,)
If I have 3 settings added on inline form and 1 extra, I have about 10 duplicate queries
SELECT "corps_setting"."id", "corps_setting"."name", "corps_setting"."user_id", "corps_setting"."tagged", "corps_setting"."created", "corps_setting"."modified" FROM "corps_setting" ORDER BY "corps_setting"."name" ASC
- Duplicated 5 times
SELECT "corps_displayedgroup"."id", "corps_displayedgroup"."name", "corps_displayedgroup"."color", "corps_displayedgroup"."priority", "corps_displayedgroup"."created", "corps_displayedgroup"."modified" FROM "corps_displayedgroup" ORDER BY "corps_displayedgroup"."priority" ASC
- Duplicated 5 times.
Could someone please tell me what I'm doing wrong right here? I've spent 3 days trying to figure the problem out myself without luck.
The issue gets worse when I have about 50 settings inlines of a Machine, I will have ~100 queries.
Here is the screenshot

I've assembled a generic solution based on #makaveli's answer that doesn't seem to have problem mentioned in the comments:
class CachingModelChoicesFormSet(forms.BaseInlineFormSet):
"""
Used to avoid duplicate DB queries by caching choices and passing them all the forms.
To be used in conjunction with `CachingModelChoicesForm`.
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
sample_form = self._construct_form(0)
self.cached_choices = {}
try:
model_choice_fields = sample_form.model_choice_fields
except AttributeError:
pass
else:
for field_name in model_choice_fields:
if field_name in sample_form.fields and not isinstance(
sample_form.fields[field_name].widget, forms.HiddenInput):
self.cached_choices[field_name] = [c for c in sample_form.fields[field_name].choices]
def get_form_kwargs(self, index):
kwargs = super().get_form_kwargs(index)
kwargs['cached_choices'] = self.cached_choices
return kwargs
class CachingModelChoicesForm(forms.ModelForm):
"""
Gets cached choices from `CachingModelChoicesFormSet` and uses them in model choice fields in order to reduce
number of DB queries when used in admin inlines.
"""
#property
def model_choice_fields(self):
return [fn for fn, f in self.fields.items()
if isinstance(f, (forms.ModelChoiceField, forms.ModelMultipleChoiceField,))]
def __init__(self, *args, **kwargs):
cached_choices = kwargs.pop('cached_choices', {})
super().__init__(*args, **kwargs)
for field_name, choices in cached_choices.items():
if choices is not None and field_name in self.fields:
self.fields[field_name].choices = choices
All you'll need to do is subclass your model from CachingModelChoicesForm and use CachingModelChoicesFormSet in your inline class:
class ArrangementInlineForm(CachingModelChoicesForm):
class Meta:
model = Arrangement
exclude = ()
class ArrangementInline(admin.TabularInline):
model = Arrangement
extra = 50
form = ArrangementInlineForm
formset = CachingModelChoicesFormSet

EDIT 2020:
Check out the answer by #isobolev below who's taken this answer and improved on it to make it more generic. :)
This is pretty much normal behaviour in Django - it doesn't do the optimization for you, but it gives you decent tools to do it yourself. And don't sweat it, 100 queries isn't really a big problem (I've seen 16k queries on one page) that needs fixing right away. But if your amounts of data are gonna increase rapidly, then it's wise to deal with it of course.
The main weapons you'll be armed with are queryset methods select_related() and prefetch_related(). There's really no point of going too deeply into them since they're very well documented here, but just a general pointer:
use select_related() when the object you're querying has only one related object (FK or one2one)
use prefetch_related() when the object you're querying has multiple related objects (the other end of FK or M2M)
And how to use them in Django admin, you ask? Elementary, my dear Watson. Override the admin page method get_queryset(self, request) so it would look sth like this:
from django.contrib import admin
class SomeRandomAdmin(admin.ModelAdmin):
def get_queryset(self, request):
return super().get_queryset(request).select_related('field1', 'field2').prefetch_related('field3')
EDIT: Having read your comment, I realise that my initial interpretation of your question was absolutely wrong. I do have multiple solutions for your problem as well and here goes that:
The simple one that I use most of the time and recommend: just replace the Django default select widgets with raw_id_field widgets and no queries are made. Just set raw_id_fields = ('setting', 'displayed_group') in the inline admin and be done for.
But, if you don't want to get rid of the select boxes, I can give some half-hacky code that does the trick, but is rather lengthy and not very pretty. The idea is to override the formset that creates the forms and specify choices for these fields in the formset so that they're only queried once from the database.
Here it goes:
from django import forms
from django.contrib import admin
from app.models import Arrangement, Machine, Setting, DisplayedGroup
class ChoicesFormSet(forms.BaseInlineFormSet):
setting_choices = list(Setting.objects.values_list('id', 'name'))
displayed_group_choices = list(DisplayedGroup.objects.values_list('id', 'name'))
def _construct_form(self, i, **kwargs):
kwargs['setting_choices'] = self.setting_choices
kwargs['displayed_group_choices'] = self.displayed_group_choices
return super()._construct_form(i, **kwargs)
class ArrangementInlineForm(forms.ModelForm):
class Meta:
model = Arrangement
exclude = ()
def __init__(self, *args, **kwargs):
setting_choices = kwargs.pop('setting_choices', [((), ())])
displayed_group_choices = kwargs.pop('displayed_group_choices', [((), ())])
super().__init__(*args, **kwargs)
# This ensures that you can still save the form without setting all 50 (see extra value) inline values.
# When you save, the field value is checked against the "initial" value
# of a field and you only get a validation error if you've changed any of the initial values.
self.fields['setting'].choices = [('-', '---')] + setting_choices
self.fields['setting'].initial = self.fields['setting'].choices[0][0]
self.fields['setting'].empty_values = (self.fields['setting'].choices[0][0],)
self.fields['displayed_group'].choices = displayed_group_choices
self.fields['displayed_group'].initial = self.fields['displayed_group'].choices[0][0]
class ArrangementInline(admin.TabularInline):
model = Arrangement
extra = 50
form = ArrangementInlineForm
formset = ChoicesFormSet
def get_queryset(self, request):
return super().get_queryset(request).select_related('setting')
class MachineAdmin(admin.ModelAdmin):
inlines = (ArrangementInline,)
admin.site.register(Machine, MachineAdmin)
If you find something that could be improved or have any questions, let me know.

Nowadays, (kudos to that question), BaseFormset receives a form_kwargs attribute.
The ChoicesFormSet code in the accepted answer could be slightly modified as such:
class ChoicesFormSet(forms.BaseInlineFormSet):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
setting_choices = list(Setting.objects.values_list('id', 'name'))
displayed_group_choices = list(DisplayedGroup.objects.values_list('id', 'name'))
self.form_kwargs['setting_choices'] = self.setting_choices
self.form_kwargs['displayed_group_choices'] = self.displayed_group_choices
The rest of the code stays intact, as desscribed in the accepted answer:
class ArrangementInlineForm(forms.ModelForm):
class Meta:
model = Arrangement
exclude = ()
def __init__(self, *args, **kwargs):
setting_choices = kwargs.pop('setting_choices', [((), ())])
displayed_group_choices = kwargs.pop('displayed_group_choices', [((), ())])
super().__init__(*args, **kwargs)
# This ensures that you can still save the form without setting all 50 (see extra value) inline values.
# When you save, the field value is checked against the "initial" value
# of a field and you only get a validation error if you've changed any of the initial values.
self.fields['setting'].choices = [('-', '---')] + setting_choices
self.fields['setting'].initial = self.fields['setting'].choices[0][0]
self.fields['setting'].empty_values = (self.fields['setting'].choices[0][0],)
self.fields['displayed_group'].choices = displayed_group_choices
self.fields['displayed_group'].initial = self.fields['displayed_group'].choices[0][0]
class ArrangementInline(admin.TabularInline):
model = Arrangement
extra = 50
form = ArrangementInlineForm
formset = ChoicesFormSet
def get_queryset(self, request):
return super().get_queryset(request).select_related('setting')
class MachineAdmin(admin.ModelAdmin):
inlines = (ArrangementInline,)
admin.site.register(Machine, MachineAdmin)

Related

How to exclude django model fields during a save?

I've got a fairly complicated Django model that includes some fields that should only be saved under certain circumstances. As a simple example,
from django.db import models
class MyModel(models.Model):
name = models.CharField(max_length=200)
counter = models.IntegerField(default=0)
def increment_counter(self):
self.counter = models.F('counter') + 1
self.save(update_fields=['counter'])
Here I'm using F expressions to avoid race conditions while incrementing the counter. I'll generally never want to save the value of counter outside of the increment_counter function, as that would potentially undo an increment called from another thread or process.
So the question is, what's the best way to exclude certain fields by default in the model's save function? I've tried the following
def save(self, **kwargs):
if update_fields not in kwargs:
update_fields = set(self._meta.get_all_field_names())
update_fields.difference_update({
'counter',
})
kwargs['update_fields'] = tuple(update_fields)
super().save(**kwargs)
but that results in ValueError: The following fields do not exist in this model or are m2m fields: id. I could of course just add id and any m2m fields in the difference update, but that then starts to seem like an unmaintainable mess, especially once other models start to reference this one, which will add additional names in self._meta.get_all_field_names() that need to be excluded from update_fields.
For what it's worth, I mostly need this functionality for interacting with the django admin site; every other place in the code could relatively easily call model_obj.save() with the correct update_fields.
I ended up using the following:
from django.db import models
class MyModel(models.Model):
name = models.CharField(max_length=200)
counter = models.IntegerField(default=0)
default_save_fields = None
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
if self.default_save_fields is None:
# This block should only get called for the first object loaded
default_save_fields = {
f.name for f in self._meta.get_fields()
if f.concrete and not f.many_to_many and not f.auto_created
}
default_save_fields.difference_update({
'counter',
})
self.__class__.default_save_fields = tuple(default_save_fields)
def increment_counter(self):
self.counter = models.F('counter') + 1
self.save(update_fields=['counter'])
def save(self, **kwargs):
if self.id is not None and 'update_fields' not in kwargs:
# If self.id is None (meaning the object has yet to be saved)
# then do a normal update with all fields.
# Otherwise, make sure `update_fields` is in kwargs.
kwargs['update_fields'] = self.default_save_fields
super().save(**kwargs)
This seems to work for my more complicated model which is referenced in other models as a ForeignKey, although there might be some edge cases that it doesn't cover.
I created a mixin class to make it easy to add to a model, inspired by clwainwright's answer. Though it uses a second mixin class to track which fields have been changed, inspired by this answer.
https://gitlab.com/snippets/1746711

flask ndb.model auto created_by field

I have an ndb.Model
class BaseModel(ndb.Model):
created_on = ndb.DateTimeProperty(auto_now_add=True)
created_by = ndb.KeyProperty(kind='MyUser') # ????
I'm looking for an elegant way create an auto "created_by" field that is populated with g.current_user key upon creation, so that
mymodel = BaseModel()
already have created_on and created_by
any ideas ?
I would recommend defining your own property class:
class OwnerProperty(ndb.KeyProperty):
_auto_current_user_add = False
def __init__(self, auto_current_user_add=False):
super(OwnerProperty, self).__init__(kind='MyUser')
self._auto_current_user_add = auto_current_user_add
def _prepare_for_put(self, entity):
if self._auto_current_user_add and not self._has_value(entity):
value = g.current_user.key
if value is not None:
self._store_value(entity, value)
super(OwnerProperty, self)._prepare_for_put(entity)
I'm not really sure how your g.current_user is setup, but assuming it is an Entity this would work. Note that like ndb.DateTimeProperty, the owner would only get populated once the entity is put for the first time.
This is based on how ndb.UserProperty is written. However, I recommend against using UserProperty directly since user properties do a lot of strange things so it is always better to handle your own user entities.
ended up doing something simple:
class BaseModel(ndb.Model):
created_on = ndb.DateTimeProperty(auto_now_add=True)
created_by = ndb.KeyProperty(kind='TalknetUser')
def __init__(self, *args, **kwargs):
super(BaseModel, self).__init__(parent=talknet_key(), *args, **kwargs)
try:
self.created_by = current_user.ndb.key
except Exception as x:
self.created_by = None
but so far it works

Django forms with prefixes not updating on save()

I iterate through a list of Block objects, instantiate a ModelForm for each of them with a mapping dictionary that links a block_type to a ModelForm model, and then append the form to a list which I pass off to a template for display.
for block in blocks:
block_instance = block_map[block.block_type].objects.get(id=block.id)
new_form = block_forms[block.block_type]
new_form_instance = new_form(
request.user,
request.POST or None,
instance=block_instance,
prefix = block.id
)
form_zones.append(new_form_instance)
Later, while checking request.POST I validate each form
if request.POST.get("save_submit"):
for zone_form_check in story_zones:
for block_form_check in zone_form_check:
if block_form_check.is_valid():
print(block_form_check.cleaned_data.get("content"))
saved = block_form_check.save()
print(saved.content)
valid = True
if valid:
return redirect("Editorial:content", content_id=content_id)
cleaned_data.get("content") produces the updated data, but even after calling save() on the valid form, saved.content produces the object's old content attribute. In other words, a valid form is having save() called upon it, but it is not saving.
One of the forms in question (and currently my only one) is:
class Edit_Text_Block_Form(ModelForm):
content = forms.CharField(widget = forms.Textarea(
attrs = {
"class": "full_tinymce"
}),
label = "",
)
class Meta:
model = TextBlock
fields = []
def __init__(self, user, *args, **kwargs):
self.user = user
super(Edit_Text_Block_Form, self).__init__(*args, **kwargs)
The model in question is a TextBlock, which inherits from a Block objets. Both of those are below:
class Block(models.Model):
zone = models.ForeignKey(Zone)
order = models.IntegerField()
weight = models.IntegerField()
block_type = models.CharField(max_length=32, blank=True)
class Meta:
ordering = ['order']
def delete(self, *args, **kwargs):
# Calling custom delete methods of child blocks
child = block_map[self.block_type].objects.get(id=self.id)
if getattr(child, "custom_delete", None):
child.custom_delete()
# Overriding delete to check if there are any other blocks in the zone.
# If not, the zone itself is deleted
zones = Block.objects.filter(zone=self.zone).count()
if zones <= 1:
self.zone.delete()
# Children of Block Object
class TextBlock(Block):
content = models.TextField(blank=True)
Any ideas for why calling saved = block_form_check.save() isn't updating my model?
Thanks!
I think this is because you've effectively excluded all the model fields from the form by setting fields = [] in the form's Meta class. This means that Django no longer relates the manually-defined content field on the form with the one in the model.
Instead, set fields to ['content'], and it should work as expected.
TL;DR form name cannot start with a number as per html4 specs
Try prefix = "block_%s" % block.id

field choices() as queryset?

I need to make a form, which have 1 select and 1 text input. Select must be taken from database.
model looks like this:
class Province(models.Model):
name = models.CharField(max_length=30)
slug = models.SlugField(max_length=30)
def __unicode__(self):
return self.name
It's rows to this are added only by admin, but all users can see it in forms.
I want to make a ModelForm from that. I made something like this:
class ProvinceForm(ModelForm):
class Meta:
CHOICES = Province.objects.all()
model = Province
fields = ('name',)
widgets = {
'name': Select(choices=CHOICES),
}
but it doesn't work. The select tag is not displayed in html. What did I wrong?
UPDATE:
This solution works as I wanto it to work:
class ProvinceForm(ModelForm):
def __init__(self, *args, **kwargs):
super(ProvinceForm, self).__init__(*args, **kwargs)
user_provinces = UserProvince.objects.select_related().filter(user__exact=self.instance.id).values_list('province')
self.fields['name'].queryset = Province.objects.exclude(id__in=user_provinces).only('id', 'name')
name = forms.ModelChoiceField(queryset=None, empty_label=None)
class Meta:
model = Province
fields = ('name',)
Read Maersu's answer for the method that just "works".
If you want to customize, know that choices takes a list of tuples, ie (('val','display_val'), (...), ...)
Choices doc:
An iterable (e.g., a list or tuple) of
2-tuples to use as choices for this
field.
from django.forms.widgets import Select
class ProvinceForm(ModelForm):
class Meta:
CHOICES = Province.objects.all()
model = Province
fields = ('name',)
widgets = {
'name': Select(choices=( (x.id, x.name) for x in CHOICES )),
}
ModelForm covers all your needs (Also check the Conversion List)
Model:
class UserProvince(models.Model):
user = models.ForeignKey(User)
province = models.ForeignKey(Province)
Form:
class ProvinceForm(ModelForm):
class Meta:
model = UserProvince
fields = ('province',)
View:
if request.POST:
form = ProvinceForm(request.POST)
if form.is_valid():
obj = form.save(commit=True)
obj.user = request.user
obj.save()
else:
form = ProvinceForm()
If you need to use a query for your choices then you'll need to overwrite the __init__ method of your form.
Your first guess would probably be to save it as a variable before your list of fields but you shouldn't do that since you want your queries to be updated every time the form is accessed. You see, once you run the server the choices are generated and won't change until your next server restart. This means your query will be executed only once and forever hold your peace.
# Don't do this
class MyForm(forms.Form):
# Making the query
MYQUERY = User.objects.values_list('id', 'last_name')
myfield = forms.ChoiceField(choices=(*MYQUERY,))
class Meta:
fields = ('myfield',)
The solution here is to make use of the __init__ method which is called on every form load. This way the result of your query will always be updated.
# Do this instead
class MyForm(forms.Form):
class Meta:
fields = ('myfield',)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# Make the query here
MYQUERY = User.objects.values_list('id', 'last_name')
self.fields['myfield'] = forms.ChoiceField(choices=(*MYQUERY,))
Querying your database can be heavy if you have a lot of users so in the future I suggest some caching might be useful.
the two solutions given by maersu and Yuji 'Tomita' Tomita perfectly works, but there are cases when one cannot use ModelForm (django3 link), ie the form needs sources from several models / is a subclass of a ModelForm class and one want to add an extra field with choices from another model, etc.
ChoiceField is to my point of view a more generic way to answer the need.
The example below provides two choice fields from two models and a blank choice for each :
class MixedForm(forms.Form):
speaker = forms.ChoiceField(choices=([['','-'*10]]+[[x.id, x.__str__()] for x in Speakers.objects.all()]))
event = forms.ChoiceField(choices=( [['','-'*10]]+[[x.id, x.__str__()] for x in Events.objects.all()]))
If one does not need a blank field, or one does not need to use a function for the choice label but the model fields or a property it can be a bit more elegant, as eugene suggested :
class MixedForm(forms.Form):
speaker = forms.ChoiceField(choices=((x.id, x.__str__()) for x in Speakers.objects.all()))
event = forms.ChoiceField(choices=(Events.objects.values_list('id', 'name')))
using values_list() and a blank field :
event = forms.ChoiceField(choices=([['','-------------']] + list(Events.objects.values_list('id', 'name'))))
as a subclass of a ModelForm, using the one of the robos85 question :
class MixedForm(ProvinceForm):
speaker = ...

Free-form input for ForeignKey Field on a Django ModelForm

I have two models related by a foreign key:
# models.py
class TestSource(models.Model):
name = models.CharField(max_length=100)
class TestModel(models.Model):
name = models.CharField(max_length=100)
attribution = models.ForeignKey(TestSource, null=True)
By default, a django ModelForm will present this as a <select> with <option>s; however I would prefer that this function as a free form input, <input type="text"/>, and behind the scenes get or create the necessary TestSource object and then relate it to the TestModel object.
I have tried to define a custom ModelForm and Field to accomplish this:
# forms.py
class TestField(forms.TextInput):
def to_python(self, value):
return TestSource.objects.get_or_create(name=value)
class TestForm(ModelForm):
class Meta:
model=TestModel
widgets = {
'attribution' : TestField(attrs={'maxlength':'100'}),
}
Unfortunately, I am getting: invalid literal for int() with base 10: 'test3' when attempting to check is_valid on the submitted form. Where am I going wrong? Is their and easier way to accomplish this?
Something like this should work:
class TestForm(ModelForm):
attribution = forms.CharField(max_length=100)
def save(self, commit=True):
attribution_name = self.cleaned_data['attribution']
attribution = TestSource.objects.get_or_create(name=attribution_name)[0] # returns (instance, <created?-boolean>)
self.instance.attribution = attribution
return super(TestForm, self).save(commit)
class Meta:
model=TestModel
exclude = ('attribution')
There are a few problems here.
Firstly, you have defined a field, not a widget, so you can't use it in the widgets dictionary. You'll need to override the field declaration at the top level of the form.
Secondly get_or_create returns two values: the object retrieved or created, and a boolean to show whether or not it was created. You really just want to return the first of those values from your to_python method.
I'm not sure if either of those caused your actual error though. You need to post the actual traceback for us to be sure.
TestForm.attribution expects int value - key to TestSource model.
Maybe this version of the model will be more convenient for you:
class TestSource(models.Model):
name = models.CharField(max_length=100, primary_key=True)
Taken from:
How to make a modelform editable foreign key field in a django template?
class CompanyForm(forms.ModelForm):
s_address = forms.CharField(label='Address', max_length=500, required=False)
def __init__(self, *args, **kwargs):
super(CompanyForm, self).__init__(*args, **kwargs)
try:
self.fields['s_address'].initial = self.instance.address.address1
except ObjectDoesNotExist:
self.fields['s_address'].initial = 'looks like no instance was passed in'
def save(self, commit=True):
model = super(CompanyForm, self).save(commit=False)
saddr = self.cleaned_data['s_address']
if saddr:
if model.address:
model.address.address1 = saddr
model.address.save()
else:
model.address = Address.objects.create(address1=saddr)
# or you can try to look for appropriate address in Address table first
# try:
# model.address = Address.objects.get(address1=saddr)
# except Address.DoesNotExist:
# model.address = Address.objects.create(address1=saddr)
if commit:
model.save()
return model
class Meta:
exclude = ('address',) # exclude form own address field
This version sets the initial data of the s_address field as the FK from self, during init , that way, if you pass an instance to the form it will load the FK in your char-field - I added a try and except to avoid an ObjectDoesNotExist error so that it worked with or without data being passed to the form.
Although, I would love to know if there is a simpler built in Django override.