I have a Django app that tracks publications. Publications have a M2M relationship to authors. Using MySQL.
Simple.
class Publication(models.Model):
slug = models.SlugField(unique=True, max_length=128)
author = models.ManyToManyField(Author, blank=True, null=True, through='Authorship')
title = models.CharField(max_length=128)
def __unicode__(self):
return unicode(self.title)
I have a ListView to show them:
class PubList(ListView):
model = Publication
Most of these are research papers, with several authors. On my template I want to show a list of authors. So I did something like this:
{% for obj in publication_list %}
<tr>
<td>{{ obj.title }}</td>
<td>
{% for a in obj.authorship_set.all %}
{{ a.author.last_name }}, {{ a.author.first_name }}
{% if not forloop.last %}; {% endif %}
{% endfor %}
</td>
</tr>
{% endfor %}
Well, you might guess what my issue is. As the number of Publications grows, the DB calls skyrocket. 119 publications is 500+ queries.
I solved it like so:
In my PubList(ListView) I override get_context_data and set the output of this function to context['authors']:
def get_authors_by_pub():
from django.db import connection
sql = """SELECT p.id,
(
SELECT GROUP_CONCAT(CONCAT(a.last_name, ', ', a.first_name) SEPARATOR '; ')
FROM publication_authorship ap
LEFT JOIN publication_author a ON a.id = ap.author_id
WHERE ap.publication_id = p.id
)
FROM publication_publication p"""
cursor = connection.cursor()
cursor.execute(sql)
rows = cursor.fetchall() or ()
authors = {}
for r in rows:
if r[1]:
authors[r[0]] = r[1]
return authors
Now I have an authors Dictionary like:
{1: 'Tesla, Nikola; Clarke, Aurthur; Hooper, Grace', 2: 'Hopper, Grace; Simpson, Marge'}
Then, on the template, since I can't access a dict by key, I loop through authors to find the ones with the key that is the publication.id:
<td>
{% for key, value in authors.items %}
{% if key == obj.id %}
{{ value }}
{% endif %}
{% endfor %}
</td>
This works, just 2 queries. Even though the authors query is brutal, with nested SELECTs, it's orders of magnitude faster than before.
But I'm wondering if there's a better way. I feel kind of icky looping through the whole dict for each publication on the template. I would love to be able to go authors[obj.id] on the template.
What do you think?
Django covers related queries and lazy loading quite extensively in it's Documentation...Why would you code all of this when django offers:
Publication.objects.prefetch_related('authors').all()
https://docs.djangoproject.com/en/1.6/topics/db/queries/#related-objects
https://docs.djangoproject.com/en/1.6/ref/models/querysets/#prefetch-related
You can use the above queryset inside your ListView:
class PublList(ListView):
queryset = Publication.objects.prefetch_related('authors')
Related
I have two models with one having a foreign key to the other as such:
Models:
class WhoAmI(models.Model):
name = models.CharField(max_length=200)
company = models.CharField(max_length=200)
def __str__(self):
return self.name
class SolarClient(models.Model):
name = models.CharField(max_length=200)
client_owner = models.ForeignKey(WhoAmI, on_delete=models.CASCADE, related_name='solarclients')
addr = models.CharField(max_length=200)
city = models.CharField(max_length=200)
state = models.CharField(max_length=200)
email = models.EmailField()
I am trying to simply display an html table showing each client a salesperson has, with the salesperson listed first with a table of clients below their name.
The only way I could figure out how to do this was to create a dictionary using the code shown below.
class Homeowners(DetailView):
def get(self, request, **kwargs):
salespersons = WhoAmI.objects.all()
homeowners = SolarClient.objects.all().order_by("client_owner") #the name 'objects' is the Manager
rangers = {}
for e in salespersons:
k = salespersons.get(id = e.id)
v = k.solarclients.all()
rangers[k] = v
return render(request, 'homeowners.html', {'homeowners': homeowners, 'salespersons': salespersons, 'rangers': rangers })
I then iterate over the dictionary using:
{% for key, values in rangers.items %}
... display salesperson
{% if values %}
{% for v in values %}
.... display clients
{% endfor %}
{% else %}
... display "No Clients"
{% endif %}
{% endfor %}
Is there a more efficient way to do this? It seems silly to put the data into a dictionary to display it, but after many, many hours of trying different methods, this is the only way I could display the data.
thanks for any suggestions.
views.py
class Homeowners(DetailView):
def get(self, request, **kwargs):
salespersons = WhoAmI.objects.all()
return render(request, 'homeowners.html', {'salespersons': salespersons })
html:
{% for sales in salespersons %}
{% for client in sales.solarclients.all %}
------ Print Client
{% empty %}
---- Client not exist
{% endfor %}
{% endfor %}
There is a nice handy template filter called regroup built in Django which does exactly what you're looking for.
# views.py
homeowners = SolarClient.objects.all().order_by("client_owner").select_related('client_owner')
return render(request, 'homeowners.html', {'homeowners': homeowners})
# homeowners.html
{% regroup homeowners by client_owner as home_owners_list %}
<ul>
{% for client_owner in home_owners_list %}
<b>{{ client_owner.grouper }}</b>
<ul>
{% for client in client_owner.list %}
<li>{{ client.name }}</li>
{% endfor %}
</ul>
{% endfor %}
</ul>
The select_related method is just for performance improvement and omitting it wouldn't affect functionality.
I have two models:
class Status(models.Model):
CHOISES = (
('new', 'New'),
('in_progress', 'InProgress'),
('on_review', 'OnReview'),
('tested', 'Tested'),
('delivered', 'Delivered')
)
status_type = models.CharField(
max_length=11,
choices=CHOISES,
primary_key=True)
class Task(models.Model):
name = models.CharField(max_length=200, blank=False)
status = models.ForeignKey(status)
part of my view:
def task_list(request):
all_tasks = Task.objects.all()
tasks = {}
tasks['new'] = all_tasks.filter(status_id='new')
tasks['in_progress'] = all_tasks.filter(status_id='in_progress')
tasks['on_review'] = all_tasks.filter(status_id='on_review')
tasks['tested'] = all_tasks.filter(status_id='tested')
tasks['delivered'] = all_tasks.filter(status_id='delivered')
....
part of my template:
{% for item in new %}
<div id="{{ item.pk }}">
{{ item.name }}
</div>
{% endfor %}
</div>
{% for item in in_progress %}
<div id="{{ item.pk }}">
{{ item.name }}
</div>
{% endfor %}
.....
The question is, is there any way to optimize all my querysets, or it is imposible to select five filters for one db call?
Like I understand, if I save in view only this call all_tasks = Task.objects.all() and then put logic in template, like this:
{% for item in all_task %}
{% if item.status_id == "new" %}
....
it will be exaclty five calls too
Hope my question is not too broad
Just group your tasks by status in your view:
from itertools import groupby
def task_list(request):
all_tasks = Task.objects.select_related('status').order_by('status')
tasks = {status: list(tasks)
for status, tasks
in groupby(all_tasks, lambda x: x.status)}
And use them in your template like:
{% for task in tasks.new %}
{{ task }}
{% endfor %}
A possibility is to change your CHOICES (typo in there) to map to something that makes sense in ordering, eg integers.
Then perform a single query with orderby and in the template use the ifchanged tag.
For whatever you may try, be careful not to fall in the trap of implementing SQL logic in Django just to avoid an extra query.
In any case, I would recommend you to use something like django-debug-toolbar to have a grasp on times for any alternative.
I have a lot of duplicated queries (in django debug toolbar) when i load my menu tabs, im sure i can optimize this but don't find the good way.
Models :
class Categorie(models.Model):
name = models.CharField(max_length=30)
visible = models.BooleanField(default = False)
def __str__(self):
return self.nom
def getscateg(self):
return self.souscategorie_set.all().filter(visible = True)
class SousCategorie(models.Model):
name = models.CharField(max_length=30)
visible = models.BooleanField(default = False)
categorie = models.ForeignKey('Categorie')
def __str__(self):
return self.name
def gettheme(self):
return self.theme_set.all().filter(visible = True)
class Theme(models.Model):
name = models.CharField(max_length=100)
visible = models.BooleanField(default = False)
souscategorie = models.ForeignKey('SousCategorie')
def __str__(self):
return self.name
Views :
def page(request):
categs = Categorie.objects.filter(visible = True)
return render(request, 'page.html', locals())
Templates :
{% for categ in categs %}
<li>
{{categ.name}}
<ul>
{% for scateg in categ.getscateg %}
<li>
{{scateg.name}}
<ul>
{% for theme in scateg.gettheme %}
<li>{{ theme.name }}</li>
{% endfor %}
</ul>
</li>
{% endfor %}
</ul>
</li>
{% endfor %}
I have look at prefetch_related but only work if i want load Categorie from SousCategorie and SousCategorie from Theme, so if i understand i need the contrary of this...
Solved !
If it can help :
from .models import Categorie, SousCategorie, Theme, SousTheme
from django.db.models import Prefetch
pf_souscategorie = Prefetch('souscategorie_set', SousCategorie.objects.filter(visible=True))
pf_theme = Prefetch('souscategorie_set__theme_set', Theme.objects.filter(visible=True))
pf_soustheme = Prefetch('souscategorie_set__theme_set__soustheme_set', SousTheme.objects.filter(visible=True))
categs = Categorie.objects.filter(visible=True).prefetch_related(pf_souscategorie, pf_theme, pf_soustheme)
and call like this in the template :
{% with currscat=categ.souscategorie_set.all %}
{% with currth=souscateg.theme_set.all %}
{% with currsth=theme.soustheme_set.all %}
Bye
In Django every time you evaluate a new queryset a query is executed, so you need to reduce the number of queryset being used. Here is what is happening:
You create a queryset Categorie.objects.filter(visible=True) and it is passed into the view layer, there the first query is executed at the this tag {% for categ in categs %}
Inside the loop, for every category you're calling a method categ.getscateg which returns a new queryset return self.souscategorie_set.all().filter(visible = True), this queryset will be executed at the second loop in your template {% for scateg in categ.getscateg %}
The same thing happens with {% for theme in scateg.gettheme %}
Using prefetch_related was the right move, try something like (didn't test it):
Categorie.objects.filter(visible=True, souscategorie_set__visible=True, souscategorie_set__theme_set__visible=True).prefetch_related('souscategorie_set__theme_set')
prefetch_related works by running a first query to load the categories that satisfy your current filter, then it executed a second query to load all subcategories and so on.
In other cases you can use select_related, but that only works when a single query can be used, as an example, it would work if you needed the category and subcategory of a theme, like:
Theme.objects.filter(pk=1).select_related('souscategorie__categorie')
The difference here is that Theme is the one that has the ForeignKey, so it has only one subcategory, and it can be loaded with a single join, that means you can use select_related only when your queryset is from the Model that points and I think that it also works with OneToOneField.
I have reduce my querysets by 2 by using "with"
It's a good point but i always got a lot of duplicate (like 44 duplicate for 51 queries), for example i hit my database when i do that :
{% for categ in categs %}
{% with currscat=categ.getscateg %}
<li class = "{% if currscat %} has_menu {% endif %}"> {{categ.name}} # This line hit database
{% if currscat %} # This line hit database
<ul class = "menu-1"> ... </ul>
{% endif %}
</li>
{% endwith %}
{% endfor %}
I try to use prefetch_related like this :
categs = Categorie.objects.filter(visible=True).prefetch_related('souscategorie_set')
but thats just add query to database, don't reduce...
Some advice?
Thanks
I try to display the names of fields from a Model in my template.
This works fine with any type of fields except ManyToManyField.
I use this function in my models.py to return all fields.
def get_all_fields(self):
"""Returns a list of all field names on the instance."""
fields = []
# only display fields with values and skip some fields entirely
if f.editable and value and f.name not in ('id','lastname','firstname') :
fields.append(
{
'label':f.verbose_name,
'name':f.name,
'value':value,
})
return fields
In my template I use this loop to display all fields:
{% for f in modelname.get_all_fields %}
<td>{{f.label|capfirst}}</td><td>{{f.value|escape|urlize|linebreaks}}</td>
{% endfor %}
As mentioned before, this works fine with all fields except ManyToManyFields.
For example one of my M2M relations looks like this:
family = models.ManyToManyField('family', related_name="family", null=True, blank=True)
I'd be thankful for every hint that helps solving this.
Regards
Conrad
Try to specify verbose_name argument for ManytoManyfield
family = models.ManyToManyField('family',verbose_name=u'trampampam', related_name="family", null=True, blank=True)
You write {{f.value|escape|urlize|linebreaks}}, which displays the value of the field. However, the value of a M2M relation is a set of object instances and you need to iterate again over the set (if that is the result you want):
{% load m2m_filter %}
{% for f in modelname.get_all_fields %}
<td>{{f.label|capfirst}}</td>
<td>
{% if f.value|is_m2m %}
{% for object in f.value.objects.all %}
{{ object|escape|urlize|linebreaks }}
{% endfor %}
{% else %}
{{f.value|escape|urlize|linebreaks}}
{% endif %}
</td>
{% endfor %}
and you also have to create the filter
m2m_filter.py
from django import template
from django.db import models
register = template.Library()
def is_m2m(value):
return type(value) == models.ManyToManyField *
register.filter('is_m2m', is_m2m)
* I guess, it's a different type; just check that
I have a model called Subtopic. One of my templates runs a forloop on an object, returning a different field for each cell of a table row.
Two of the table cells look up a field which is a ManytoMany foreign key, both to the same foreign model, Resource. I want each to display different results, based on the value of a boolean field within the Resource model.
What you see below is currently working fine, but doesn't attempt to filter by the boolean field.
models.py:
class ICTResourceManager(models.Manager):
def get_query_set(self):
return super(ICTResourceManager, self).get_query_set().filter('is_ict': True)
class NonICTResourceManager(models.Manager):
def get_query_set(self):
return super(NonICTResourceManager, self).get_query_set().filter('is_ict': False)
class Resource(models.Model):
subtopics = models.ManyToManyField(Subtopic)
external_site = models.ForeignKey(ExternalSite)
link_address = models.URLField(max_length=200, unique=True, verify_exists=False)
requires_login = models.BooleanField()
is_ict = models.BooleanField()
flags = models.ManyToManyField(Flag, blank=True)
comment = models.TextField()
def __unicode__(self):
return u'%s %s' % (self.external_site, self.link_address)
objects = models.Manager()
ict_objects = ICTResourceManager()
nonict_objects = NonICTResourceManager()
class Meta:
ordering = ['external_site', 'link_address']
views.py:
def view_ks5topic(request, modulecode, topicshortname):
listofsubtopics = Subtopic.objects.filter(topic__module__code__iexact = modulecode, topic__shortname__iexact = topicshortname)
themodule = Module.objects.get(code__iexact = modulecode)
thetopic = Topic.objects.get(module__code__iexact = modulecode, shortname__iexact = topicshortname)
return render_to_response('topic_page.html', locals())
My template:
{% for whatever in listofsubtopics %}
<tr>
<td>
{{ whatever.objective_html|safe }}
<p>
{% if request.user.is_authenticated %}
{% with 'objective' as column %}
{% include "edit_text.html" %}
{% endwith %}
{% else %}
{% endif %}
</td>
<td>
{% regroup whatever.resource_set.all by external_site.name as resource_list %}
{% for external_site in resource_list %}
<h4>{{ external_site.grouper }}</h4>
<ul>
{% for item in external_site.list %}
<li>{{ item.comment }}</li>
{% endfor %}
</ul>
{% endfor %}
</td>
</tr>
{% endfor %}
As you can see, I've added extra managers to the model to do the filtering for me, but when I replace the appropriate lines in the template, I just get blanks. I have tried: for external_site.ict_objects in resource_list and for item.ict_objects in resource_list and <a href="{{ item.ict_objects.link_address }}">. If this were in the view I could probably do the filter just by .filter('is_ict': True), but with this being inside a forloop I don't know where to do the filtering.
I also tried writing regroup whatever.resource_set.filter('is_ict': True) in the template, but the syntax for regrouping seems to use resource_set.all rather than resource_set.all() (and I don't know why) so the filter text doesn't work here.
Turns out it was possible to do it using a custom template filter. The original efforts to filter within the template weren't working, given that as is well documented the template language is not a fully-fledged python environment. My original question remains open for anyone who knows an alternative method that more directly addresses the question I was originally asking, but here's how I did it:
myapp_extras.py:
from django import template
register = template.Library()
def ict(value, arg):
"filters on whether the is_ict Boolean is true"
return value.filter(is_ict=arg)
register.filter('ict', ict)
My template, note the use of the custom filter in line 2:
<td>
{% regroup whatever.resource_set.all|ict:1 by external_site.name as resource_list %}
{% for external_site in resource_list %}
<h4>{{ external_site.grouper }}</h4>
<ul>
{% for item in external_site.list %}
<li>{{ item.comment }}</li>
{% endfor %}
</ul>
{% endfor %}
</td>
After this I was able to remove the additional custom managers from the model.
One further question, when filtering for the boolean is_ict field, I found that I had to use filter(is_ict=1) and filter(is_ict=0). Is that the only way to refer to a True or False value?