I am trying to find out if it is safe to do the following:
items = MyModel.objects.filter(q)
if items:
list(items)
I know that the QuerySet is being evaluated in the if statement, checking if the database returns an empty set. But I want to know if the value of that evaluation is reused when doing list(items). Is the QuerySet being evaluated here too, or is it using the previously evaluated one?
I know I could just do the following:
items = MyModel.objects.filter(q).all()
if items:
list(items)
And this would result in one evaluation, but I am just trying to find out the behavior the first variation has. I have gone throught these pieces of doc (1 2) but couldn't really find a straight answer to this matter.
No. Both will not execute twice (internally .filter(), .all() and .filter().all() are same). You can check it in django shell itself
from django.db import connection
print connection.queries
items = MyModel.objects.filter(q).all() #or MyModel.objects.filter(q)
if items:
list(items)
print connection.queries
Then here is the magic of .all()
queryset = MyModel.objects.all() #no db hit, hit=0
print list(queryset) #db hit, hit=1
print list(queryset) #no db hit, hit=1
print list(queryset.all()) #db hit, hit=2
print list(queryset.all()) #db hit, hit=3
That means .all() on an evaluated queryset will force db hit.
When a QuerySet is evaluated, it typically caches its results. If the data in the database might have changed since a QuerySet was evaluated, you can get updated results for the same query by calling all() on a previously evaluated QuerySet
It will reuse it's cache, because when you do
if items:
It will call __bool__ method
def __bool__(self):
self._fetch_all()
return bool(self._result_cache)
So as you see inside __bool__ it does call _fetch_all. Which caches data
def _fetch_all(self):
if self._result_cache is None:
self._result_cache = list(self.iterator())
if self._prefetch_related_lookups and not self._prefetch_done:
self._prefetch_related_objects()
For better perfomance do:
items = MyModel.objects.filter(q) # no evaluation
if items.exists(): # evaluates, hits db
# do stuff here # further actions evaluates, hits db
I have a function that search with hatstack, and I need to get the comments of each object that haystack get in the array, I have this:
def search(request):
if 'q' in request.GET and request.GET['q']:
q = request.GET['q']
results = SearchQuerySet().auto_query(q)
things = []
for r in results:
things.append(r.object)
return render_to_response('resultados.html',
{'things': things, 'query': q}, context_instance=RequestContext(request))
How I append to the results the number of comments that each object have?
If I add annotate, debugger throw me: SearchQuerySet has not 'annotate' attribute
SearchQuerySet isn't the ORM query set you're familiar with. It only imitates it. Annotations doesn't make sense with search engines as well. You need to put already prepared data to an index.
Just make another query using ORM.
I read here that Django querysets are lazy, it won't be evaluated until it is actually printed. I have made a simple pagination using the django's built-in pagination. I didn't realize there were apps already such as "django-pagination", and "django-endless" which does that job for.
Anyway I wonder whether the QuerySet is still lazy when I for example do this
entries = Entry.objects.filter(...)
paginator = Paginator(entries, 10)
output = paginator.page(page)
return HttpResponse(output)
And this part is called every time I want to get whatever page I currently I want to view.
I need to know since I don't want unnecessary load to the database.
If you want to see where are occurring, import django.db.connection and inspect queries
>>> from django.db import connection
>>> from django.core.paginator import Paginator
>>> queryset = Entry.objects.all()
Lets create the paginator, and see if any queries occur:
>>> paginator = Paginator(queryset, 10)
>>> print connection.queries
[]
None yet.
>>> page = paginator.page(4)
>>> page
<Page 4 of 788>
>>> print connection.queries
[{'time': '0.014', 'sql': 'SELECT COUNT(*) FROM `entry`'}]
Creating the page has produced one query, to count how many entries are in the queryset. The entries have not been fetched yet.
Assign the page's objects to the variable 'objects':
>>> objects = page.object_list
>>> print connection.queries
[{'time': '0.014', 'sql': 'SELECT COUNT(*) FROM `entry`'}]
This still hasn't caused the entries to be fetched.
Generate the HttpResponse from the object list
>>> response = HttpResponse(page.object_list)
>>> print connection.queries
[{'time': '0.014', 'sql': 'SELECT COUNT(*) FROM `entry`'}, {'time': '0.011', 'sql': 'SELECT `entry`.`id`, <snip> FROM `entry` LIMIT 10 OFFSET 30'}]
Finally, the entries have been fetched.
It is. Django's pagination uses the same rules/optimizations that apply to querysets.
This means it will start evaluating on return HttpResponse(output)
Every time I use Admin to list the entries of a model, the Admin count the rows in the table. Worse yet, it seems to be doing so even when you are filtering your query.
For instance if I want to show only the models whose id is 123, 456, 789 I can do:
/admin/myapp/mymodel/?id__in=123,456,789
But the queries ran (among others) are:
SELECT COUNT(*) FROM `myapp_mymodel` WHERE `myapp_mymodel`.`id` IN (123, 456, 789) # okay
SELECT COUNT(*) FROM `myapp_mymodel` # why???
Which is killing mysql+innodb. It seems that the problem is partially acknowledged in this ticket, but my issue seems more specific since it counts all the rows even if it is not supposed to.
Is there a way to disable that global rows count?
Note: I am using django 1.2.7.
Django 1.8 lets you disable this by setting show_full_result_count = False.
https://docs.djangoproject.com/en/stable/ref/contrib/admin/#django.contrib.admin.ModelAdmin.show_full_result_count
Okay, I think I found a solution. As Peter suggested, the best approach is to work on the count property and it can be done by overriding it with custom query set (as seen in this post) that specialises the count with an approximate equivalent:
from django.db import connections, models
from django.db.models.query import QuerySet
class ApproxCountQuerySet(QuerySet):
"""Counting all rows is very expensive on large Innodb tables. This
is a replacement for QuerySet that returns an approximation if count()
is called with no additional constraints. In all other cases it should
behave exactly as QuerySet.
Only works with MySQL. Behaves normally for all other engines.
"""
def count(self):
# Code from django/db/models/query.py
if self._result_cache is not None and not self._iter:
return len(self._result_cache)
is_mysql = 'mysql' in connections[self.db].client.executable_name.lower()
query = self.query
if (is_mysql and not query.where and
query.high_mark is None and
query.low_mark == 0 and
not query.select and
not query.group_by and
not query.having and
not query.distinct):
# If query has no constraints, we would be simply doing
# "SELECT COUNT(*) FROM foo". Monkey patch so the we
# get an approximation instead.
cursor = connections[self.db].cursor()
cursor.execute("SHOW TABLE STATUS LIKE %s",
(self.model._meta.db_table,))
return cursor.fetchall()[0][4]
else:
return self.query.get_count(using=self.db)
Then in the admin:
class MyAdmin(admin.ModelAdmin):
def queryset(self, request):
qs = super(MyAdmin, self).queryset(request)
return qs._clone(klass=ApproxCountQuerySet)
The approximate function could mess things up on page number 100000, but it is good enough for my case.
I found Nova's answer very helpful, but i use postgres. I modified it slightly to work for postgres with some slight alterations to handle table namespaces, and slightly different "detect postgres" logic.
Here's the pg version.
class ApproxCountPgQuerySet(models.query.QuerySet):
"""approximate unconstrained count(*) with reltuples from pg_class"""
def count(self):
if self._result_cache is not None and not self._iter:
return len(self._result_cache)
if hasattr(connections[self.db].client.connection, 'pg_version'):
query = self.query
if (not query.where and query.high_mark is None and query.low_mark == 0 and
not query.select and not query.group_by and not query.having and not query.distinct):
# If query has no constraints, we would be simply doing
# "SELECT COUNT(*) FROM foo". Monkey patch so the we get an approximation instead.
parts = [p.strip('"') for p in self.model._meta.db_table.split('.')]
cursor = connections[self.db].cursor()
if len(parts) == 1:
cursor.execute("select reltuples::bigint FROM pg_class WHERE relname = %s", parts)
else:
cursor.execute("select reltuples::bigint FROM pg_class c JOIN pg_namespace n on (c.relnamespace = n.oid) WHERE n.nspname = %s AND c.relname = %s", parts)
return cursor.fetchall()[0][0]
return self.query.get_count(using=self.db)
The Nova's solution (ApproxCountQuerySet) works great, however in newer versions of Django queryset method got replaced with get_queryset, so it now should be:
class MyAdmin(admin.ModelAdmin):
def get_queryset(self, request):
qs = super(MyAdmin, self).get_queryset(request)
return qs._clone(klass=ApproxCountQuerySet)
If this is a serious problem you may have to take Drastic Actions™.
Looking at the code for a 1.3.1 install, I see that the admin code is using the paginator returned by get_paginator(). The default paginator class appears to be in django/core/paginator.py. That class has a private value called _count which is set in Paginator._get_count() (line 120 in my copy). This in turn is used to set a property of the Paginator class called count. I think that _get_count() is your target. Now the stage is set.
You have a couple of options:
Directly modify the source. I do not recommend this, but since you seem to be stuck at 1.2.7 you may find that it is the most expedient. Remember to document this change! Future maintainers (including possibly yourself) will thank you for the heads up.
Monkeypatch the class. This is better than direct modification because a) if you don't like the change you just comment out the monkeypatch, and b) it is more likely to work with future versions of Django. I have a monkeypatch going back over 4 years because they still have not fixed a bug in the template variable _resolve_lookup() code that doesn't recognize callables at the top level of evaluation, only at lower levels. Although the patch (which wraps the method of a class) was written against 0.97-pre, it still works at 1.3.1.
I did not spend the time to figure out exactly what changes you would have to make for your problem, but it might be along the lines of adding a _approx_count member to appropriate classes class META and then testing to see if that attr exists. If it does and is None then you do the sql.count() and set it. You might also need to reset it if you are on (or near) the last page of the list. Contact me if you need a little more help on this; my email is in my profile.
It is possible to change the default paginator used by the admin class. Here's one that caches the result for a short period of time: https://gist.github.com/e4c5/6852723
I managed to create a custom paginator that shows the current page numbe, a next button and a show full count link. It allows for the use of the original paginator if needed.
The trick used is to take per_page + 1 elements from db in order to see if we have more elements and then provide a fake count.
Let's say that we want the the third page and the page has 25 elements => We want object_list[50:75]. When calling Paginator.count the queryset will be evaluated for object_list[50:76](note that we take 75+1 elements) and then return either the count as 76 if we got 25+1 elements from db or 50 + the number of elements received if we didn't received 26 elements.
TL;DR:
I've created a mixin for the ModelAdmin:
from django.core.paginator import Paginator
from django.utils.functional import cached_property
class FastCountPaginator(Paginator):
"""A faster paginator implementation than the Paginator. Paginator is slow
mainly because QuerySet.count() is expensive on large queries.
The idea is to use the requested page to generate a 'fake' count. In
order to see if the page is the final one it queries n+1 elements
from db then reports the count as page_number * per_page + received_elements.
"""
use_fast_pagination = True
def __init__(self, page_number, *args, **kwargs):
self.page_number = page_number
super(FastCountPaginator, self).__init__(*args, **kwargs)
#cached_property
def count(self):
# Populate the object list when count is called. As this is a cached property,
# it will be called only once per instance
return self.populate_object_list()
def page(self, page_number):
"""Return a Page object for the given 1-based page number."""
page_number = self.validate_number(page_number)
return self._get_page(self.object_list, page_number, self)
def populate_object_list(self):
# converts queryset object_list to a list and return the number of elements until there
# the trick is to get per_page elements + 1 in order to see if the next page exists.
bottom = self.page_number * self.per_page
# get one more object than needed to see if we should show next page
top = bottom + self.per_page + 1
object_list = list(self.object_list[bottom:top])
# not the last page
if len(object_list) == self.per_page + 1:
object_list = object_list[:-1]
else:
top = bottom + len(object_list)
self.object_list = object_list
return top
class ModelAdminFastPaginationMixin:
show_full_result_count = False # prevents root_queryset.count() call
def changelist_view(self, request, extra_context=None):
# strip count_all query parameter from the request before it is processed
# this allows all links to be generated like this parameter was not present and without raising errors
request.GET = request.GET.copy()
request.GET.paginator_count_all = request.GET.pop('count_all', False)
return super().changelist_view(request, extra_context)
def get_paginator(self, request, queryset, per_page, orphans=0, allow_empty_first_page=True):
# use the normal paginator if we want to count all the ads
if hasattr(request.GET, 'paginator_count_all') and request.GET.paginator_count_all:
return Paginator(queryset, per_page, orphans, allow_empty_first_page)
page = self._validate_page_number(request.GET.get('p', '0'))
return FastCountPaginator(page, queryset, per_page, orphans, allow_empty_first_page)
def _validate_page_number(self, number):
# taken from Paginator.validate_number and adjusted
try:
if isinstance(number, float) and not number.is_integer():
raise ValueError
number = int(number)
except (TypeError, ValueError):
return 0
if number < 1:
number = 0
return number
The pagination.html template:
{% if cl and cl.paginator and cl.paginator.use_fast_pagination %}
{# Fast paginator with only next button and show the total number of results#}
{% load admin_list %}
{% load i18n %}
{% load admin_templatetags %}
<p class="paginator">
{% if pagination_required %}
{% for i in page_range %}
{% if forloop.last %}
{% fast_paginator_number cl i 'Next' %}
{% else %}
{% fast_paginator_number cl i %}
{% endif %}
{% endfor %}
{% endif %}
{% show_count_all_link cl "showall" %}
</p>
{% else %}
{# use the default pagination template if we are not using the FastPaginator #}
{% include "admin/pagination.html" %}
{% endif %}
and templatetags used:
from django import template
from django.contrib.admin.views.main import PAGE_VAR
from django.utils.html import format_html
from django.utils.safestring import mark_safe
register = template.Library()
DOT = '.'
#register.simple_tag
def fast_paginator_number(cl, i, text_display=None):
"""Generate an individual page index link in a paginated list.
Allows to change the link text by setting text_display
"""
if i == DOT:
return '… '
elif i == cl.page_num:
return format_html('<span class="this-page">{}</span> ', i + 1)
else:
return format_html(
'<a href="{}"{}>{}</a> ',
cl.get_query_string({PAGE_VAR: i}),
mark_safe(' class="end"' if i == cl.paginator.num_pages - 1 else ''),
text_display if text_display else i + 1,
)
#register.simple_tag
def show_count_all_link(cl, css_class='', text_display='Show the total number of results'):
"""Generate a button that toggles between FastPaginator and the normal
Paginator."""
return format_html(
'<a href="{}"{}>{}</a> ',
cl.get_query_string({PAGE_VAR: cl.page_num, 'count_all': True}),
mark_safe(f' class="{css_class}"' if css_class else ''),
text_display,
)
You can use it this way:
class MyVeryLargeModelAdmin(ModelAdminFastPaginationMixin, admin.ModelAdmin):
# ...
Or an even simpler version that does not show the Next button and Show the total number of results :
from django.core.paginator import Paginator
from django.utils.functional import cached_property
class FastCountPaginator(Paginator):
"""A faster paginator implementation than the Paginator. Paginator is slow
mainly because QuerySet.count() is expensive on large queries.
The idea is to use the requested page to generate a 'fake' count. In
order to see if the page is the final one it queries n+1 elements
from db then reports the count as page_number * per_page + received_elements.
"""
use_fast_pagination = True
def __init__(self, page_number, *args, **kwargs):
self.page_number = page_number
super(FastCountPaginator, self).__init__(*args, **kwargs)
#cached_property
def count(self):
# Populate the object list when count is called. As this is a cached property,
# it will be called only once per instance
return self.populate_object_list()
def page(self, page_number):
"""Return a Page object for the given 1-based page number."""
page_number = self.validate_number(page_number)
return self._get_page(self.object_list, page_number, self)
def populate_object_list(self):
# converts queryset object_list to a list and return the number of elements until there
# the trick is to get per_page elements + 1 in order to see if the next page exists.
bottom = self.page_number * self.per_page
# get one more object than needed to see if we should show next page
top = bottom + self.per_page + 1
object_list = list(self.object_list[bottom:top])
# not the last page
if len(object_list) == self.per_page + 1:
object_list = object_list[:-1]
else:
top = bottom + len(object_list)
self.object_list = object_list
return top
class ModelAdminFastPaginationMixin:
show_full_result_count = False # prevents root_queryset.count() call
def get_paginator(self, request, queryset, per_page, orphans=0, allow_empty_first_page=True):
page = self._validate_page_number(request.GET.get('p', '0'))
return FastCountPaginator(page, queryset, per_page, orphans, allow_empty_first_page)
def _validate_page_number(self, number):
# taken from Paginator.validate_number and adjusted
try:
if isinstance(number, float) and not number.is_integer():
raise ValueError
number = int(number)
except (TypeError, ValueError):
return 0
if number < 1:
number = 0
return number
As I am an impressed reader of Stack Overflow I want to ask my first question here. Since I encountered a problem with a snippet and I do not know whether I made a mistake or it's a bug in the code I'm using.
I adapted this code for my own site:
http://blog.tkbe.org/archive/django-admin-search-functionality/
It works fine and it's really a great snippet.
But if my search query has length 2, I think that the results are not correct.
So for example if I search for "re" in first name and last name, I get the following results:
Mr. Tom Krem
Ms. Su Ker
Which is pretty strange. For queries with length > 2 I do not encounter this problem.
So maybe this post read somebody who is using the snippet above and can tell me whether he/she encounters the same problem.
If nobody else encounters the problem I know at least that I have a bug somewhere in my code. Maybe in the form I'm using, or something is messed up in the request context.
How can I solve this problem?
Edit 1:
The inclusion tag:
from django import template
from crm.views import SEARCH_VAR
def my_search_form(context):
return {
'context': context,
'search_var': SEARCH_VAR
}
register = template.Library()
register.inclusion_tag('custom_utilities/my_search_form.html')(my_search_form)
The my_search_form.html:
<div id="toolbar"><form
id="changelist-search"
action=""
method="get">
<div><!-- DIV needed for valid HTML -->
<label
for="searchbar"><img src="{{ context.media_url }}/crm/img/search.png"
class="icon"
alt="Search" /></label>
<input
type="text"
size="40"
name="{{ search_var }}"
value="{{ context.query }}"
id="searchbar" />
<input type="submit" value="Search" />
</div>
</form>
</div>
<script
type="text/javascript">document.getElementById("searchbar").focus();
</script>
The view:
#login_required
def crm_contacts(request):
query = request.GET.get('q', '')
#pass additional params to the SortHeaders function
#the additional params will be part of the header <a href...>
#e.g. use it for pagination / use it to provide the query string
additional_params_dict = {'q': query}
foundContacts = search_contact(request,query)
sort_headers = SortHeaders(request, LIST_HEADERS, default_order_field=1, additional_params=additional_params_dict)
if foundContacts is not None:
contact_list = foundContacts.order_by(sort_headers.get_order_by())
else:
contact_list = Contact.objects.order_by(sort_headers.get_order_by())
context = {
'contact_list' : contact_list,
'headers': list(sort_headers.headers()),
'query' : query,
}
return render_to_response("crm/contact_list.html", context,
context_instance=RequestContext(request))
The contact search form:
#models
from crm.models import Contact
from django.db.models import Q
'''
A search form from
http://blog.tkbe.org/archive/django-admin-search-functionality/
adapted to search for contacts.
'''
def search_contact(request,terms=None):
if terms is None:
return Contact.objects.all()
query = Contact.objects
for term in terms:
query = query.filter(
Q(first_name__icontains=term)
| Q(last_name__icontains=term))
return query
Another edit:
I'm using this snippet to sort the table. Probably one should know this in order to understand the code posted above.
Since I can not post links (spam protection) I will try to explain where to find it. Go to Google. Type in: django snippet table sort
Then it should be the second hit. Sort table headers. snippet nr. 308.
Edit: Add the SortHeaders() function
ORDER_VAR = 'o'
ORDER_TYPE_VAR = 'ot'
class SortHeaders:
"""
Handles generation of an argument for the Django ORM's
``order_by`` method and generation of table headers which reflect
the currently selected sort, based on defined table headers with
matching sort criteria.
Based in part on the Django Admin application's ``ChangeList``
functionality.
"""
def __init__(self, request, headers, default_order_field=None,
default_order_type='asc', additional_params=None):
"""
request
The request currently being processed - the current sort
order field and type are determined based on GET
parameters.
headers
A list of two-tuples of header text and matching ordering
criteria for use with the Django ORM's ``order_by``
method. A criterion of ``None`` indicates that a header
is not sortable.
default_order_field
The index of the header definition to be used for default
ordering and when an invalid or non-sortable header is
specified in GET parameters. If not specified, the index
of the first sortable header will be used.
default_order_type
The default type of ordering used - must be one of
``'asc`` or ``'desc'``.
additional_params:
Query parameters which should always appear in sort links,
specified as a dictionary mapping parameter names to
values. For example, this might contain the current page
number if you're sorting a paginated list of items.
"""
if default_order_field is None:
for i, (header, query_lookup) in enumerate(headers):
if query_lookup is not None:
default_order_field = i
break
if default_order_field is None:
raise AttributeError('No default_order_field was specified and none of the header definitions given were sortable.')
if default_order_type not in ('asc', 'desc'):
raise AttributeError('If given, default_order_type must be one of \'asc\' or \'desc\'.')
if additional_params is None: additional_params = {}
self.header_defs = headers
self.additional_params = additional_params
self.order_field, self.order_type = default_order_field, default_order_type
# Determine order field and order type for the current request
params = dict(request.GET.items())
if ORDER_VAR in params:
try:
new_order_field = int(params[ORDER_VAR])
if headers[new_order_field][1] is not None:
self.order_field = new_order_field
except (IndexError, ValueError):
pass # Use the default
if ORDER_TYPE_VAR in params and params[ORDER_TYPE_VAR] in ('asc', 'desc'):
self.order_type = params[ORDER_TYPE_VAR]
def headers(self):
"""
Generates dicts containing header and sort link details for
all defined headers.
"""
for i, (header, order_criterion) in enumerate(self.header_defs):
th_classes = []
new_order_type = 'asc'
if i == self.order_field:
th_classes.append('sorted %sending' % self.order_type)
new_order_type = {'asc': 'desc', 'desc': 'asc'}[self.order_type]
yield {
'text': header,
'sortable': order_criterion is not None,
'url': self.get_query_string({ORDER_VAR: i, ORDER_TYPE_VAR: new_order_type}),
'class_attr': (th_classes and ' class="%s"' % ' '.join(th_classes) or ''),
}
def get_query_string(self, params):
"""
Creates a query string from the given dictionary of
parameters, including any additonal parameters which should
always be present.
"""
params.update(self.additional_params)
return '?%s' % '&'.join(['%s=%s' % (param, value) \
for param, value in params.items()])
def get_order_by(self):
"""
Creates an ordering criterion based on the current order
field and order type, for use with the Django ORM's
``order_by`` method.
"""
return '%s%s' % (
self.order_type == 'desc' and '-' or '',
self.header_defs[self.order_field][1],
)
If you run manage.py shell and then:
>>> from crm.models import Contact
>>> from django.db.models import Q
>>> list=Contact.objects.filter(Q(first_name__icontains='re')|Q(last_name__icontains='re'))
>>> print list
What is the output?
Edit: Right, so if you try:
>>> list=Contact.objects.filter(Q(first_name__icontains='mot')|Q(last_name__icontains='mot'))
>>> print list
(I'm trying to narrow down on the terms that are giving you problem and I saw your last comment)
What is the output?
Edit: If both of the above queries work in the shell, something else is modifying your queryset somewhere and adding some additional criteria...
Are you sure sort_headers() is not modifying the queryset with more than just an order by clause? Could you post sort_headers() to your question?