Reducing db queries in django - django

Question

I have a view that searches through a database of movie credits, and converts and returns results like so --
# From the following results:
Avatar - James Cameron - director
Avatar - James Cameron - writer
Avatar - James Cameron - editor
Avatar - Julie Jones - writer
Crash - John Smith - director
# ...display in the template as:
Avatar - James Cameron (director, writer, editor)
Avatar - Julie Jones (writer)
Crash - John Smith (director)
However, when I do this conversion and do print connection.queries I am hitting the database about 100 times. Here is what I currently have --
# in models
class VideoCredit(models.Model):
video = models.ForeignKey(VideoInfo)
# if the credit is a current user, FK to his profile,
profile = models.ForeignKey('UserProfile', blank=True, null=True)
# else, just add his name
name = models.CharField(max_length=100, blank=True)
# normalize name for easier searching / pulling of name
normalized_name = models.CharField(max_length=100)
position = models.ForeignKey(Position)
timestamp = models.DateTimeField(auto_now_add=True)
actor_role = models.CharField(max_length=50, blank=True)
class VideoInfo(models.Model):
title = models.CharField(max_length=256, blank=True)
uploaded_by = models.ForeignKey('UserProfile')
...
class Position(models.Model):
position = models.CharField(max_length=100)
ordering = models.IntegerField(max_length=3)
class UserProfile(models.Model):
user = models.ForeignKey(User, unique=True)
...
In my view, I am building a list of three-tuples in the form of (name, video, [list_of_positions]) for the display of credits --
credit_set = VideoCredit.objects.filter(***depends on a previous function***)
list_of_credit_tuples = []
checklist = [] # I am creating a 'checklist' to see whether to append the positions
# list of create a new tuple entry
for credit in credit_set:
if credit.profile: # check to see if the credit has an associated profile
name = credit.profile
else:
name = credit.normalized_name
if (credit.normalized_name, credit.video) in checklist:
list_of_keys = [(name, video) for name, video, positions in list_of_credit_tuples]
index = list_of_keys.index((name, credit.video))
list_of_credit_tuples[index][2].append(credit.position)
else:
list_of_credit_tuples.append((name, credit.video, [credit.position]))
checklist.append((credit.normalized_name, credit.video))
...
And finally, in my template to display the credits (note: if the credit has a profile, provide a link to profile of user) --
{% for name, video, positions in list_of_credit_tuples %}
<p>{% if name.full_name %}
{{name.full_name}}
{% else %}
{{name}}
{% endif %}
{{video}}
({% for position in positions %}{% ifchanged %}{{position}}{% endifchanged %}{% if not forloop.last %}, {% endif %}{% endfor %})
{% endfor %}
Why and where is this view creating so many db queries? How and in which ways could I make this view function more efficient / better? Thank you.

Answer 1

You will want to look into select_related() (https://docs.djangoproject.com/en/1.3/ref/models/querysets/#select-related) to resolve your query leaking issue. If you know ahead of time you're going to be looking at data on models related by a foreignkey you'll want to add select_related. Even better if you know it's only going to be a couple of foreignkeys you can add only the ones you need.
Anytime you see django ran a huge number of queries more than you expected, select_related is almost always the right answer

Answer 2

Try adding this Django snippet which returns the number of queries as well as the queries themselves to your template:
http://djangosnippets.org/snippets/159/
This should easily tell you where your leak is originating from.