Django multiple forms with modelchoicefield -> too many queries - django

Question

I have a table of forms of the same class which contains ModelChoiceField. And each form in one row has the same queryset for this field. Problem is that every time the form is rendered, it is a new query which increases unbearably the number of queries.
The only solution I came up with is to construct the form on the go with js instead of letting django to render it itself. Is there a way to cache these querysets or somewhat preload it at once?
views.py:
shift_table=[]
for project in calendar_projects:
shift_table.append([])
project_branches = project.branches.all()
for i, week in enumerate(month):
for day in week:
shift_table[-1].append(
CreateShiftCalendarForm(initial={'date': day}, branch_choices=project_branches))
forms.py:
CreateShiftCalendarForm(EditShiftCalendarForm):
class Meta(ShiftForm.Meta):
fields = ('project_branch', 'date') + ShiftForm.Meta.fields
widgets = {'date': forms.HiddenInput(), 'length': forms.NumberInput(attrs={'step': 'any'}), 'project_branch': forms.Select()}
def __init__(self, *args, **kwargs):
branch_choices = kwargs.pop('branch_choices', ProjectBranch.objects.none())
super(CreateShiftCalendarForm, self).__init__(*args, **kwargs)
self.fields['project_branch'].queryset = branch_choices
self.fields['project_branch'].empty_label = None

Answer 1

ModelChoiceField is an subclass of ChoiceField in which "normal" choices are replaced with iterator that will iterate through provided queryset. Also there is customized 'to_python' method that will return actual object instead of it's pk. Unfortunately that iterator will reset queryset and hit database once again for each choice field, even if they are sharing queryset
What you need to do is subclass ChoiceField and mimic behaviour of ModelChoiceField with one difference: it will take static choices list instead of queryset. That choices list you will build in your view once for all fields (or forms).

Answer 2

A maybe less invasive hack, using an overload of Django's FormSets and keeping the base form untouched (i.e. keeping the ModelChoiceFields with their dynamic queryset):
from django import forms
class OptimFormSet( forms.BaseFormSet ):
"""
FormSet with minimized number of SQL queries for ModelChoiceFields
"""
def __init__( self, *args, modelchoicefields_qs=None, **kwargs ):
"""
Overload the ModelChoiceField querysets by a common queryset per
field, with dummy .all() and .iterator() methods to avoid multiple
queries when filling the (repeated) choices fields.
Parameters
----------
modelchoicefields_qs : dict
Dictionary of modelchoicefield querysets. If ``None``, the
modelchoicefields are identified internally
"""
# Init the formset
super( OptimFormSet, self ).__init__( *args, **kwargs )
if modelchoicefields_qs is None and len( self.forms ) > 0:
# Store querysets of modelchoicefields
modelchoicefields_qs = {}
first_form = self.forms[0]
for key in first_form.fields:
if isinstance( first_form.fields[key], forms.ModelChoiceField ):
modelchoicefields_qs[key] = first_form.fields[key].queryset
# Django calls .queryset.all() before iterating over the queried objects
# to render the select boxes. This clones the querysets and multiplies
# the queries for nothing.
# Hence, overload the querysets' .all() method to avoid cloning querysets
# in ModelChoiceField. Simply return the queryset itself with a lambda function.
# Django also calls .queryset.iterator() as an optimization which
# doesn't make sense for formsets. Hence, overload .iterator as well.
if modelchoicefields_qs:
for qs in modelchoicefields_qs.values():
qs.all = lambda local_qs=qs: local_qs # use a default value of qs to pass from late to immediate binding (so that the last qs is not used for all lambda's)
qs.iterator = qs.all
# Apply the common (non-cloning) querysets to all the forms
for form in self.forms:
for key in modelchoicefields_qs:
form.fields[key].queryset = modelchoicefields_qs[key]
In your view, you then call:
formset_class = forms.formset_factory( form=MyBaseForm, formset=OptimFormSet )
formset = formset_class()
And then render your template with the formset as described in Django's doc.
Note that on form validation, you will still have 1 query per ModelChoiceField instance, but limited to a single primary key value each time. That is also the case with the accepted answer. To avoid that, the to_python method should use the existing queryset, which would make the hack even hackier.
This works at least for Django 1.11.

Answer 3

I subclassed ChoiceField as suggested by GwynBleidD and it works sufficiently for now.
class ListModelChoiceField(forms.ChoiceField):
"""
special field using list instead of queryset as choices
"""
def __init__(self, model, *args, **kwargs):
self.model = model
super(ListModelChoiceField, self).__init__(*args, **kwargs)
def to_python(self, value):
if value in self.empty_values:
return None
try:
value = self.model.objects.get(id=value)
except self.model.DoesNotExist:
raise ValidationError(self.error_messages['invalid_choice'], code='invalid_choice')
return value
def valid_value(self, value):
"Check to see if the provided value is a valid choice"
if any(value.id == int(choice[0]) for choice in self.choices):
return True
return False