Need some advice for duplicated queries - django

Question

I have a lot of duplicated queries (in django debug toolbar) when i load my menu tabs, im sure i can optimize this but don't find the good way.
Models :
class Categorie(models.Model):
name = models.CharField(max_length=30)
visible = models.BooleanField(default = False)
def __str__(self):
return self.nom
def getscateg(self):
return self.souscategorie_set.all().filter(visible = True)
class SousCategorie(models.Model):
name = models.CharField(max_length=30)
visible = models.BooleanField(default = False)
categorie = models.ForeignKey('Categorie')
def __str__(self):
return self.name
def gettheme(self):
return self.theme_set.all().filter(visible = True)
class Theme(models.Model):
name = models.CharField(max_length=100)
visible = models.BooleanField(default = False)
souscategorie = models.ForeignKey('SousCategorie')
def __str__(self):
return self.name
Views :
def page(request):
categs = Categorie.objects.filter(visible = True)
return render(request, 'page.html', locals())
Templates :
{% for categ in categs %}
<li>
{{categ.name}}
<ul>
{% for scateg in categ.getscateg %}
<li>
{{scateg.name}}
<ul>
{% for theme in scateg.gettheme %}
<li>{{ theme.name }}</li>
{% endfor %}
</ul>
</li>
{% endfor %}
</ul>
</li>
{% endfor %}
I have look at prefetch_related but only work if i want load Categorie from SousCategorie and SousCategorie from Theme, so if i understand i need the contrary of this...

Answer 1

Solved !
If it can help :
from .models import Categorie, SousCategorie, Theme, SousTheme
from django.db.models import Prefetch
pf_souscategorie = Prefetch('souscategorie_set', SousCategorie.objects.filter(visible=True))
pf_theme = Prefetch('souscategorie_set__theme_set', Theme.objects.filter(visible=True))
pf_soustheme = Prefetch('souscategorie_set__theme_set__soustheme_set', SousTheme.objects.filter(visible=True))
categs = Categorie.objects.filter(visible=True).prefetch_related(pf_souscategorie, pf_theme, pf_soustheme)
and call like this in the template :
{% with currscat=categ.souscategorie_set.all %}
{% with currth=souscateg.theme_set.all %}
{% with currsth=theme.soustheme_set.all %}
Bye

Answer 2

In Django every time you evaluate a new queryset a query is executed, so you need to reduce the number of queryset being used. Here is what is happening:
You create a queryset Categorie.objects.filter(visible=True) and it is passed into the view layer, there the first query is executed at the this tag {% for categ in categs %}
Inside the loop, for every category you're calling a method categ.getscateg which returns a new queryset return self.souscategorie_set.all().filter(visible = True), this queryset will be executed at the second loop in your template {% for scateg in categ.getscateg %}
The same thing happens with {% for theme in scateg.gettheme %}
Using prefetch_related was the right move, try something like (didn't test it):
Categorie.objects.filter(visible=True, souscategorie_set__visible=True, souscategorie_set__theme_set__visible=True).prefetch_related('souscategorie_set__theme_set')
prefetch_related works by running a first query to load the categories that satisfy your current filter, then it executed a second query to load all subcategories and so on.
In other cases you can use select_related, but that only works when a single query can be used, as an example, it would work if you needed the category and subcategory of a theme, like:
Theme.objects.filter(pk=1).select_related('souscategorie__categorie')
The difference here is that Theme is the one that has the ForeignKey, so it has only one subcategory, and it can be loaded with a single join, that means you can use select_related only when your queryset is from the Model that points and I think that it also works with OneToOneField.

Answer 3

I have reduce my querysets by 2 by using "with"
It's a good point but i always got a lot of duplicate (like 44 duplicate for 51 queries), for example i hit my database when i do that :
{% for categ in categs %}
{% with currscat=categ.getscateg %}
<li class = "{% if currscat %} has_menu {% endif %}"> {{categ.name}} # This line hit database
{% if currscat %} # This line hit database
<ul class = "menu-1"> ... </ul>
{% endif %}
</li>
{% endwith %}
{% endfor %}
I try to use prefetch_related like this :
categs = Categorie.objects.filter(visible=True).prefetch_related('souscategorie_set')
but thats just add query to database, don't reduce...
Some advice?
Thanks