I am trying to understand the best way to structure queries in django to avoid excessive database hits.
This is similar to the question: Django best practice with foreign key queries, but involves greater 'depth' in the queries.
My situation:
models.py
class Bucket(models.Model):
categories = models.ManyToManyField('Category')
class Category(models.Model):
name = models.CharField(max_length=50)
class SubCategory(models.Model):
category = models.ForeignKey(Category)
class SubSubCategory(models.Model):
subcat = models.ForeignKey(SubCategory)
views.py
def showbucket(request, id):
bucket = Bucket.objects.prefetch_related('categories',).get(pk=id)
cats = bucket.categories.prefetch_related('subcategory_set__subsubcategory_set',)
return render_to_response('showbucket.html', locals(), context_instance=RequestContext(request))
and relevant template:
{% for c in cats %}
{{c}}
<ul>
{% for d in c.subcategory_set.all %}
<li>{{d}}</li>
<ul>
{% for e in d.subsubcategory_set.all %}
<li>{{e}}</li>
{% endfor %}
</ul>
{% endfor %}
</ul>
{% endfor %}
Despite the use of prefetch_related(), I seem to be hitting the database each time the top two for statements are evaluated, e.g. {% for c in cats %}, (at least I believe so from reading the debug_toolbar). Other ways I've tried have ended up with (C x D x E) number of database hits. Is this something inherently wrong with my use of prefetch, queries, or models? What is the best way in Django to access database objects with a "depth > 1" so-to-speak?
Use select_related() instead:
https://docs.djangoproject.com/en/dev/ref/models/querysets/#django.db.models.query.QuerySet.select_related
bucket = Bucket.objects.select_related('categories',).get(id=id)
cats = bucket.categories.select_related('subcategory_set__subsubcategory_set',)
So, i found out there's a couple things going on here:
First, my current understanding on select_related vs prefetch_related:
select_related() follows foreign-key relationships, causing larger result sets but means that later use of FK won't require additional database hits. It is limited to FK and one-to-one relationships.
prefetch_related() does a separate lookup for each relationship and joins them in python, and is means to be used for many-to-many, many-to-one, and GenericRelation and GenericForeignKey.
By the book, I should be using prefetch(), as I was not 'following' the Foreign Keys.
That's what I had understood going into this, but my template seemed to be causing additional queries when evaluating the given for loops in the template, even when I added the use of {with} tags.
At first, I had thought I had discovered something similar to this issue, but I am unable to replicate when I built out my simplified example. I switched from using the debug toolbar to direct checking using the following template code (in the article Tracking SQL Queries for a Request using Django by Karen Tracey, I would link but am link-limited):
{% with sql_queries|length as qcount %}
{{ qcount }} quer{{ qcount|pluralize:"y,ies" }}
{% for qdict in sql_queries %}
{{ qdict.sql }} ({{ qdict.time }} seconds)
{% endfor %}
{% endwith %}
Using this method, I am only seeing 5 queries for using pre-fetch() (7 with debug_toolbar), and queries grow linearly when using select_related() (with +2 for debug_toolbar), which I believe is expected.
I will gladly take any other advice/tools on getting a handle on these types of issues.
Related
I am trying to figure out how to produce the output between two querysets that have similar fields. I have two different models where I keep identical fields, and at times I want to show the differences between these two models. I have researched and successfully used sets and lists in my python code, but I can't quite figure out how to leverage them or determine if I Can. Sets seems to strip out just the field values, and when I try to loop through my sets, it doesn't currently work because it's just the values without the keys.
Example:
class Team(models.Model):
player = models.Charfield
coach = models.Charfield
class TeamCommittee(models.Model):
player = models.Charfield
coach = models.Charfield
I want to be able to query both models at times and be able to exclude data from one model if it exists in the other. I have spent the afternoon trying to use sets, lists, and loops in my django templates but can't quite work this out. I have also tried various querysets using exclude....
I tried something like....
query1 = TeamCommittee.objects.filter(id=self.object.pk).values('coach','player')
query2 = Team.objects.filter(id=self.object.pk).exclude(id__in=query1)
When I use the approach above, I get TypeError: Cannot use multi-field values as a filter value.
I am wondering if I can do this via a query or if I need to dump my querysets and go down a path of manipulating a data dictionary? That seems extreme for what I am trying to do though. This does seem to be a bit more complicated because I am trying to cross reference two different models. If it was the same model this would be a lot easier but it's not an option for this particular use case.
Thanks in advance for any thoughts on the right way to approach this.
If you want to compare on the basis of the ID of both tables, probably you can use this:
teamID = list(Team.objects.all().values_list('id', flat=True))
query1 = TeamCommittee.objects.filter(id__in=teamID)
teamCommitteeID = list(TeamCommittee.objects.all().values_list('id', flat=True))
query2 = Team.objects.filter(id__in=teamCommitteeID)
I was super close...Instead I just did this...
query1 = TeamCommittee.objects.filter(id=self.object.pk).values('coach','player').distinct()
Then in my template I did a very simple....
{% for item in query1.all %}
item
{% endfor %}
Then when I wanted to get the values out of the other queryset I just did the same thing with the loop.
query2 = Team.objects.filter(id=self.object.pk).values('coach','player').distinct()
Then in my template I did a very simple....
{% for item in query2.all %}
item
{% endfor %}
Sometimes simplicity is hard.
The answer above after additional testing only partially worked. I ultimately abandoned that approach and instead incorporated the logic below into my template. I did not need to create separate queries, I just needed to loop through the fields that were already available to me as part of the DetailView I was using....
{% for author in author_detail.author_set.all %}
{% for book in book_detail.book_set.all %}
{% if author.author_name %}
{% if book.book_name in author.book_name %}
{% if author.book_name == book.book_name %}
{% elif author.book_name != book.book_name %}
{{ author.author_name }}
{% endif %}
{% endif %}
{% endif %}
{% endfor %}
{% endfor %}
Thank you to everyone who made suggestions to get me to this point.
I am trying to display ManyToMany field on the template in reversed order.
Here is what I mean:
I managed to display ManyToMany field on template when ManyToMany field was a field in model used so for example:
<br/>{% for tag in post.tag.all %}{{ tag }}<br/>{% endfor %}
will display all of the tags(meaning categories) that the post belongs to based on this model:
class Post(models.Model):
tag = models.ManyToManyField(Tag,blank=True,null=True,related_name='tag')
Now I want something opposite - display authors of the post when ManyToMany field is in the Author model (Post model above stays the same):
class Person(models.Model):
post=models.ManyToManyField(Post,blank=True,null=True,related_name='post')
I am quite sure it has something to do with Related Object Reference ( https://docs.djangoproject.com/en/2.2/ref/models/relations/)
Just can not make it work.
I have tried the following on the template.
{% for post in posts %}
{% for author in post.person_set.all %}{{author}}<br/>{% endfor %}
{% endfor %}
Also, shall I do this kind of searches on the template like above or is it a better practice put this kind of searches in views...resourcewise.
Thanks for help.
You have a misunderstanding on what the related_name= parameter [Django-doc] does. Like the documentation says:
The name to use for the relation from the related object back to this one. (...)
So it is the name of the relation in reverse. In order to make your models "sound", you thus should name it like:
class Person(models.Model):
posts = models.ManyToManyField(Post, blank=True, null=True, related_name='authors')
It also makes sense here to use the plural, so posts instead of post.
In that case, you thus can render this with:
{% for post in posts %}
{% for author in post.authors.all %}{{author}}<br/>{% endfor %}
{% endfor %}
Note that if you want to render all the values for ManyToManyFields, you better use .prefetch_related(..) in the queryset to prefetch the Person,s otherwise rendering the template will result in a lot of extra queries.
I have been succesfully using this brilliant technique to keep my code DRY encapsulating ORM relations in querysets so that code in views is simple and not containing foreign key dependency. But this time I face the following issue best descriped by code:
View:
vendors_qs = vendors_qs.select_related().prefetch_related('agreement_vendors')
Model
class AgreementVendorsQuerySet(models.query.QuerySet):
def some_filter1(self, option):
result = self.filter(.....)
return result
def some_filter2(self, option):
result = self.filter(.....)
return result
And a template:
{% for vendor in vendors_qs %}
<tr>
...
<td>
{% for vend_agr in vendor.agreement_vendors.all %}
{{vend_agr.serial_number}}
{% endfor %}
<td>
</tr>
{% endfor %}
The question is, how and where do I apply the some_filter1 to vendor agreements given that it is fetched as prefetch_related relation. Do I have to apply the filter in the template somehow or in the view itself ?
If I didn't put the question clearly enough, I will ask your questions to clarify further...
UPDATE:
Anna's asnwer looks very much like the truth, but one detail remains unclear. What if I want to apply several filters based on if-condition. For exapmle, if the filters were to apply to vendors, then the code would simply look like:
if (condition_1)
vendors_qs = vendors_qs.filter1()
if (condition_2)
vendors_qs = vendors_qs.filter2()
If I clearly understand your question you need something like this
vendors_qs = vendors_qs.prefetch_related(models.Prefetch('agreement_vendors', queryset=some_filter, to_attr='agreement_vendors_list'))
And then in template you can call it like {% for vend_agr in vendor.agreement_vendors_list %}
Sorry about the strange title, but I couldn't explain the situation in a few words. Let me articulate:
I have a Jobs model whose objects I show in a template. Against each job, I also want to show if the user has already applied for that job.
I have these models
class Job(models.Model):
is_valid = models.BooleanField()
description = models.CharField()
def has_user_applied(self, user):
return jobapplication_set.filter(applicant=user).exists()
class JobApplication(models.Model):
applicant = models.ForeignKey(User)
job = models.ForeignKey(Job)
cover_letter = models.CharField()
And a view in which I fetch all the Jobs:
jobs = Job.objects.filter(is_valid=True)
return HttpResponse( ... {'jobs': jobs} ... )
And a template in which I list them:
{% for j in jobs %}
{{ j.description }} {% if j.has_applied %} (You've already applied) {% endif %}
{% endfor %}
However the "has_applied" function takes "user" as argument and passing arguments in templates is not allowed.
Now I have two issues:
Can I create a "context" so that some functions can assume that a particular user is in question rather than it being passed explicitly and limiting its usage in templates? If it is not possible, what is the elegant way of annotating this information in the model objects in the views?
Secondly, even If I'm able to do this, for each Job object, I still have to execute a separate query to determine if user has already applied. I know this is possible in raw SQL using outer joins, but can I do it using django's ORM?
Easiest and most verbose way of doing it will be just:
view
jobs_applied_by_user = set(JobApplication.objects
.filter(applicant=user)
.distinct()
.values_list('job', flat=True))
template
{% for job in jobs %}
{% if job.pk in jobs_applied_by_user %}
…
One simple solution is to write the custom filter.
{% for j in jobs %}
{{ j.description }} {% if j|has_applied:user %} (You've already applied) {% endif %}
{% endfor %}
Here has_applied is custom filter and it will take user as parameter.
I'm making a database of released music albums
models.py
class Image(models.Model):
image = models.ImageField(....
class Album(models.Model):
title = models.CharField(....
class Release(models.Model):
album = models.ForeignKey(Album)
cover_art = models.ForeignKey(Image, blank=True, null=True, on_delete=models.SET_NULL)
In my template (at the moment I'm using generic views) I have:
{% for a in album_list %}
{% for r in a.release_set.all %}
{% if r.cover_art %}
# display cover art image
{% endif %}
{% endfor %}
{% endfor %}
The problem is that sometimes an album has been released several times with identical cover art, in which case I'd like to display the image only once, with some text listing the releases it pertains to.
I've tried:
{% for i in a.release_set.cover_art %}
{% for i in a.release_set.cover_art_set %}
{% for i in a.release_set.all.cover_art %}
{% for i in a.release_set.all.cover_art_set %}
Or in a simpler case, I'd at least like to display the images smaller if there are more than one of them.
{% if a.release_set.count > 1 %} # works but displays duplicate images
{% if a.release_set.cover_art_set.count > 1 %} # doesn't work (see above)
Is it possible to get a list of objects related by reversing this ForeignKey lookup then asking for the set of their children? The only way I can think of is by assembling some tuples/lists in the view.
I managed this with a new method on the Album model:
class Album(models.Model):
title = models.CharField(....
def distinct_cover_images(self):
"Returns the queryset of distinct images used for this album cover"
pks = self.release_set.all().values_list('cover_art__pk', flat=True)
distinct_cover_images = Images.objects.filter(pk__in=pks).distinct()
return distinct_cover_images
Then the template is much more simple:
{% for i in a.distinct_cover_images %}
Credit to #danilobargen however for his contribution to this code.
If I understood this right:
An album can have several releases
A release has only one cover
You want to loop over all covers of an album
In that case, the following should work:
{% for release in a.release_set.all %}
{{ release.cover_art.image }}
{% endfor %}
If you want to prevent listing identical covers, you can either compare the covers in the loop, or prepare a set with distinct covers in your view, so you can pass it on to the template.
# Solution using a set
context['distinct_coverimages'] = \
set([r.cover_art.image for r in album.release_set.all()])
# Solution using two queries, might perform better
pks = album.release_set.values_list('cover_art__pk', flat=True)
context['distinct_coverimages'] = models.Image.filter(pk__in=pks).distinct()
A third alternative would be creating a custom template filter for your album, to return all distinct release covers.
In any case, I recommend debugging such things in your Django shell. You can issue the shell with ./manage.py shell. If you have installed django-extensions, you can also use ./manage.py shell_plus to autoload all models. All object attributes and functions that don't require arguments (e.g. normal instance attributes or instance functions without arguments like 'string'.isalnum()) can also be used the same way (just without the parentheses) in your template.