Django. How I can optimize db query? - django

I have two models:
class Status(models.Model):
CHOISES = (
('new', 'New'),
('in_progress', 'InProgress'),
('on_review', 'OnReview'),
('tested', 'Tested'),
('delivered', 'Delivered')
)
status_type = models.CharField(
max_length=11,
choices=CHOISES,
primary_key=True)
class Task(models.Model):
name = models.CharField(max_length=200, blank=False)
status = models.ForeignKey(status)
part of my view:
def task_list(request):
all_tasks = Task.objects.all()
tasks = {}
tasks['new'] = all_tasks.filter(status_id='new')
tasks['in_progress'] = all_tasks.filter(status_id='in_progress')
tasks['on_review'] = all_tasks.filter(status_id='on_review')
tasks['tested'] = all_tasks.filter(status_id='tested')
tasks['delivered'] = all_tasks.filter(status_id='delivered')
....
part of my template:
{% for item in new %}
<div id="{{ item.pk }}">
{{ item.name }}
</div>
{% endfor %}
</div>
{% for item in in_progress %}
<div id="{{ item.pk }}">
{{ item.name }}
</div>
{% endfor %}
.....
The question is, is there any way to optimize all my querysets, or it is imposible to select five filters for one db call?
Like I understand, if I save in view only this call all_tasks = Task.objects.all() and then put logic in template, like this:
{% for item in all_task %}
{% if item.status_id == "new" %}
....
it will be exaclty five calls too
Hope my question is not too broad

Just group your tasks by status in your view:
from itertools import groupby
def task_list(request):
all_tasks = Task.objects.select_related('status').order_by('status')
tasks = {status: list(tasks)
for status, tasks
in groupby(all_tasks, lambda x: x.status)}
And use them in your template like:
{% for task in tasks.new %}
{{ task }}
{% endfor %}

A possibility is to change your CHOICES (typo in there) to map to something that makes sense in ordering, eg integers.
Then perform a single query with orderby and in the template use the ifchanged tag.
For whatever you may try, be careful not to fall in the trap of implementing SQL logic in Django just to avoid an extra query.
In any case, I would recommend you to use something like django-debug-toolbar to have a grasp on times for any alternative.

Related

Django templates iterate model fields with indices

I have the follow model:
class UserProfile(...):
...
photo1 = models.URLField()
photo2 = models.URLField()
photo3 = models.URLField()
photo4 = models.URLField()
photo5 = models.URLField()
And in the Create/Update template, I have to write five copies of the following div for file1 to file5:
<div>
Photo 1:
{% if userProfileForm.instance.file1 %}
<a href="{{ userProfileForm.instance.file1 }}" target=_blank>View</a>
{% endif %}
<input type=file name=file1>
{% if userProfileForm.instance.file1 %}
Delete
{% endif %}
</div>
Is there a way to iterate field file<i>?
{% for i in '12345' %}
<div>
Photo {{ forloop.counter }}:
...
</div>
{% endfor %}
in django you have _meta API. So I think this solves your problem if you use get_fields method (and maybe you will filter desired fields out).
hope it helps.
update with example
let me show how you should solve your problem:
desired_fields = []
for field in UserProfile._meta.get_fields()
if "photo" in field.name:
desired_fields.append(field.name)
context.update(fields=desired_fields) # pass it in the context
at this point you have your desired fields which should be used with for loop in the template. And one more thing, you would need to add some template tag to get real field from string representation:
# custom template tag
def from_string_to_field(instance, field_str):
return getattr(instance, field_str, None)
and in the template code will look like this
{% for field in fields %}
{{userProfileForm.instance|from_string_to_field:"field"}}
{% endfor %}

Creating a Hierarchy View in Django

I'm trying to create a hierarchy view in Django, but I'm struggling to make sense of how to use QuerySets effectively.
What I'm aiming for eventually is a html page that displays courses like this:
Main Course 1 --- Child Course 1
--- Child Course 2
Main Course 2 --- Child Course 3
--- Child Course 4
Each group of courses would be wrapped in a div and styled etc.
In my view.py file I have the following:
class HierarchyView(generic.ListView):
template_name = 'curriculum/hierarchy.html'
def get_queryset(self):
return Offering.objects.all()
def get_context_data(self, **kwargs):
context = super(HierarchyView, self).get_context_data(**kwargs)
context['main'] = self.get_queryset().filter(course_type='M')
context['sub'] = self.get_queryset().filter(parent_code__in=context['main'])
return context
The Offering model is set up so that parent_code is a self-referential foreign key (i.e. any course can be a child of any other), like this:
...
parent_code = models.ForeignKey(
'self',
null=True,
blank=True,
on_delete=models.SET_NULL)
...
And in my html template I have:
{% for mainoffering in main %}
<div>
<div>{{ mainoffering.course_name }}</div>
{% for offering in sub %}
<div>{{ offering.course_name }}</div>
{% endfor %}
</div>
{% endfor %}
What this results in, however, is that all child courses appear under all main courses, regardless of whether or not they are actually children of that course, which is obviously not what I'm after.
I'm still learning the ropes in Django, and I'm struggling to find anything that explains in plain English what I need to do. Please help!
I think you would need to change your template to match each of the child courses to their parent courses. Maybe something like:
{% for mainoffering in main %}
<div>
<div>{{ mainoffering.course_name }}</div>
{% for offering in sub %}
{% if offering.parent_code == mainoffering %}
<div>{{ offering.course_name }}</div>
{% endif %}
{% endfor %}
</div>
{% endfor %}
The context['sub'] will return all of them, without any grouping, ordering etc. You can do couple of things to get the desired behavior.
You can do a prefetch related.
from django.db.models import Prefetch
offerings = Offering.objects.filter(course_type='M').prefetch_related(
Prefetch(
"courses_subset",
queryset=Offering.objects.filter(parent_code__in=offerings),
to_attr="sub"
)
)
for o in offerings:
print o.sub
You can actually make this a method in your model and create a template tag (i'd most likely use this).
method in your Offering model
def get_child_courses(self):
child_courses = Offerings.objects.filter(parent_code=self.id)
return child_courses
template tag
#register.simple_tag
def get_child_courses(course):
return course.get_child_courses()
In your template:
{% for mainoffering in main %}
<div>
<div>{{ mainoffering.course_name }}</div>
{% for offering in mainoffering|get_child_course %}
<div>{{ offering.course_name }}</div>
{% endfor %}
</div>
{% endfor %}
You can group them in your template as suggested by accraze. I'd personally go for the second option

Need some advice for duplicated queries

I have a lot of duplicated queries (in django debug toolbar) when i load my menu tabs, im sure i can optimize this but don't find the good way.
Models :
class Categorie(models.Model):
name = models.CharField(max_length=30)
visible = models.BooleanField(default = False)
def __str__(self):
return self.nom
def getscateg(self):
return self.souscategorie_set.all().filter(visible = True)
class SousCategorie(models.Model):
name = models.CharField(max_length=30)
visible = models.BooleanField(default = False)
categorie = models.ForeignKey('Categorie')
def __str__(self):
return self.name
def gettheme(self):
return self.theme_set.all().filter(visible = True)
class Theme(models.Model):
name = models.CharField(max_length=100)
visible = models.BooleanField(default = False)
souscategorie = models.ForeignKey('SousCategorie')
def __str__(self):
return self.name
Views :
def page(request):
categs = Categorie.objects.filter(visible = True)
return render(request, 'page.html', locals())
Templates :
{% for categ in categs %}
<li>
{{categ.name}}
<ul>
{% for scateg in categ.getscateg %}
<li>
{{scateg.name}}
<ul>
{% for theme in scateg.gettheme %}
<li>{{ theme.name }}</li>
{% endfor %}
</ul>
</li>
{% endfor %}
</ul>
</li>
{% endfor %}
I have look at prefetch_related but only work if i want load Categorie from SousCategorie and SousCategorie from Theme, so if i understand i need the contrary of this...
Solved !
If it can help :
from .models import Categorie, SousCategorie, Theme, SousTheme
from django.db.models import Prefetch
pf_souscategorie = Prefetch('souscategorie_set', SousCategorie.objects.filter(visible=True))
pf_theme = Prefetch('souscategorie_set__theme_set', Theme.objects.filter(visible=True))
pf_soustheme = Prefetch('souscategorie_set__theme_set__soustheme_set', SousTheme.objects.filter(visible=True))
categs = Categorie.objects.filter(visible=True).prefetch_related(pf_souscategorie, pf_theme, pf_soustheme)
and call like this in the template :
{% with currscat=categ.souscategorie_set.all %}
{% with currth=souscateg.theme_set.all %}
{% with currsth=theme.soustheme_set.all %}
Bye
In Django every time you evaluate a new queryset a query is executed, so you need to reduce the number of queryset being used. Here is what is happening:
You create a queryset Categorie.objects.filter(visible=True) and it is passed into the view layer, there the first query is executed at the this tag {% for categ in categs %}
Inside the loop, for every category you're calling a method categ.getscateg which returns a new queryset return self.souscategorie_set.all().filter(visible = True), this queryset will be executed at the second loop in your template {% for scateg in categ.getscateg %}
The same thing happens with {% for theme in scateg.gettheme %}
Using prefetch_related was the right move, try something like (didn't test it):
Categorie.objects.filter(visible=True, souscategorie_set__visible=True, souscategorie_set__theme_set__visible=True).prefetch_related('souscategorie_set__theme_set')
prefetch_related works by running a first query to load the categories that satisfy your current filter, then it executed a second query to load all subcategories and so on.
In other cases you can use select_related, but that only works when a single query can be used, as an example, it would work if you needed the category and subcategory of a theme, like:
Theme.objects.filter(pk=1).select_related('souscategorie__categorie')
The difference here is that Theme is the one that has the ForeignKey, so it has only one subcategory, and it can be loaded with a single join, that means you can use select_related only when your queryset is from the Model that points and I think that it also works with OneToOneField.
I have reduce my querysets by 2 by using "with"
It's a good point but i always got a lot of duplicate (like 44 duplicate for 51 queries), for example i hit my database when i do that :
{% for categ in categs %}
{% with currscat=categ.getscateg %}
<li class = "{% if currscat %} has_menu {% endif %}"> {{categ.name}} # This line hit database
{% if currscat %} # This line hit database
<ul class = "menu-1"> ... </ul>
{% endif %}
</li>
{% endwith %}
{% endfor %}
I try to use prefetch_related like this :
categs = Categorie.objects.filter(visible=True).prefetch_related('souscategorie_set')
but thats just add query to database, don't reduce...
Some advice?
Thanks

django filter a regroup within a forloop

I have a model called Subtopic. One of my templates runs a forloop on an object, returning a different field for each cell of a table row.
Two of the table cells look up a field which is a ManytoMany foreign key, both to the same foreign model, Resource. I want each to display different results, based on the value of a boolean field within the Resource model.
What you see below is currently working fine, but doesn't attempt to filter by the boolean field.
models.py:
class ICTResourceManager(models.Manager):
def get_query_set(self):
return super(ICTResourceManager, self).get_query_set().filter('is_ict': True)
class NonICTResourceManager(models.Manager):
def get_query_set(self):
return super(NonICTResourceManager, self).get_query_set().filter('is_ict': False)
class Resource(models.Model):
subtopics = models.ManyToManyField(Subtopic)
external_site = models.ForeignKey(ExternalSite)
link_address = models.URLField(max_length=200, unique=True, verify_exists=False)
requires_login = models.BooleanField()
is_ict = models.BooleanField()
flags = models.ManyToManyField(Flag, blank=True)
comment = models.TextField()
def __unicode__(self):
return u'%s %s' % (self.external_site, self.link_address)
objects = models.Manager()
ict_objects = ICTResourceManager()
nonict_objects = NonICTResourceManager()
class Meta:
ordering = ['external_site', 'link_address']
views.py:
def view_ks5topic(request, modulecode, topicshortname):
listofsubtopics = Subtopic.objects.filter(topic__module__code__iexact = modulecode, topic__shortname__iexact = topicshortname)
themodule = Module.objects.get(code__iexact = modulecode)
thetopic = Topic.objects.get(module__code__iexact = modulecode, shortname__iexact = topicshortname)
return render_to_response('topic_page.html', locals())
My template:
{% for whatever in listofsubtopics %}
<tr>
<td>
{{ whatever.objective_html|safe }}
<p>
{% if request.user.is_authenticated %}
{% with 'objective' as column %}
{% include "edit_text.html" %}
{% endwith %}
{% else %}
{% endif %}
</td>
<td>
{% regroup whatever.resource_set.all by external_site.name as resource_list %}
{% for external_site in resource_list %}
<h4>{{ external_site.grouper }}</h4>
<ul>
{% for item in external_site.list %}
<li>{{ item.comment }}</li>
{% endfor %}
</ul>
{% endfor %}
</td>
</tr>
{% endfor %}
As you can see, I've added extra managers to the model to do the filtering for me, but when I replace the appropriate lines in the template, I just get blanks. I have tried: for external_site.ict_objects in resource_list and for item.ict_objects in resource_list and <a href="{{ item.ict_objects.link_address }}">. If this were in the view I could probably do the filter just by .filter('is_ict': True), but with this being inside a forloop I don't know where to do the filtering.
I also tried writing regroup whatever.resource_set.filter('is_ict': True) in the template, but the syntax for regrouping seems to use resource_set.all rather than resource_set.all() (and I don't know why) so the filter text doesn't work here.
Turns out it was possible to do it using a custom template filter. The original efforts to filter within the template weren't working, given that as is well documented the template language is not a fully-fledged python environment. My original question remains open for anyone who knows an alternative method that more directly addresses the question I was originally asking, but here's how I did it:
myapp_extras.py:
from django import template
register = template.Library()
def ict(value, arg):
"filters on whether the is_ict Boolean is true"
return value.filter(is_ict=arg)
register.filter('ict', ict)
My template, note the use of the custom filter in line 2:
<td>
{% regroup whatever.resource_set.all|ict:1 by external_site.name as resource_list %}
{% for external_site in resource_list %}
<h4>{{ external_site.grouper }}</h4>
<ul>
{% for item in external_site.list %}
<li>{{ item.comment }}</li>
{% endfor %}
</ul>
{% endfor %}
</td>
After this I was able to remove the additional custom managers from the model.
One further question, when filtering for the boolean is_ict field, I found that I had to use filter(is_ict=1) and filter(is_ict=0). Is that the only way to refer to a True or False value?

Django: Refactoring template view to include count?

TEMPLATE:
<ul id="bugs-list">
{% for group in groups %}
<h2>{{ group.name }}</h2> <span></span>
{% for data in group.grab_bugs %}
<li>{{data.name }}</li>
{% endfor %}
{% endfor %}
</ul>
models.py:
class BrowserGroups( models.Model ):
name = models.CharField( max_length=100 )
slug = models.SlugField(unique=True)
browsers = models.ManyToManyField( 'Browser' )
def grab_bugs(self):
bugs = Bug.objects.filter(browser__browsergroups=self,really_bug=True).distinct()
return bugs
def __unicode__(self):
return self.name
class Meta:
verbose_name_plural = 'Browser Groups'
I'm trying to render the number of bugs (data) near the <h2>. What would be an efficient way of including the count of data near the h2? Should I define a separate function in my model class to return the total # of bugs? Or is there a more efficient way?
{% with group.grab_bugs as distinct_bugs %}
<h2>{{ group.name }}</h2> (Count: {{ distinct_bugs.count }})
{% for data in distinct_bugs %}
<li>{{data.name }}</li>
{% endfor %}
{% endwith %}
Explanation: the grab_bugs method of the Group class returns a querset of Bug instances. To get a count of the bugs invoke the count() method on the queryset.
This will cost you two queries (not counting the ones inside the loop). One to get the count and then next to retrieve a list of bugs.