Is it possible to combine a Django Haystack search with "built-in" QuerySet filter operations, specifically filtering with Q() instances and lookup types not supported by SearchQuerySet? In either order:
haystack-searched -> queryset-filtered
or
queryset-filtered -> haystack-searched
Browsing the Django Haystack documentation didn't give any directions how to do this.
You could filter your queryset based on the results of a Haystack search, using the objects' PKs:
def view(request):
if request.GET.get('q'):
from haystack import ModelSearchForm
form = ModelSearchForm(request.GET, searchqueryset=None, load_all=True)
searchqueryset = form.search()
results = [ r.pk for r in searchqueryset ]
docs = Document.objects.filter(pk__in=results)
# do something with your plain old regular queryset
return render_to_response('results.html', {'documents': docs});
Not sure how this scales, but for small resultsets (a few hundred, in my case), this works fine.
From the docs:
SearchQuerySet.load_all(self)
Efficiently populates the objects in the search results. Without using
this method, DB lookups are done on a per-object basis, resulting in
many individual trips to the database. If load_all is used, the
SearchQuerySet will group similar objects into a single query,
resulting in only as many queries as there are different object types
returned.
http://django-haystack.readthedocs.org/en/latest/searchqueryset_api.html#load-all
Therefore, after you have a filtered SQS, you can do a load_all() on it and just access the database objects via SearchResult.object. E.g.
sqs = SearchQuerySet()
# filter as needed, then load_all
sqs = sqs.load_all()
for result in sqs:
my_obj = result.object
# my_obj is a your model object
If you want to keep up with the pertinence, you have to access the object from the database through "object" :
example in your template:
{% for result in results %}
{{ result.object.title }}
{{ result.objects.author }}
{% endfor %}
But this is really bad since haystack will make an extra request like "SELECT * FROM blah WHERE id = 42" on each results.
Seems you're trying to get those object from your database because you didn't put some extra fields in your index ins't it ? If you add the title AND the author in your SearchIndex, then you can just use your results:
{% for result in results %}
{{ result.title }}
{{ result.author }}
{% endfor %}
and avoid some extra queries.
Related
I have a model specifications which are divided into categories. For the template, in order to display the category as an header in the table, I make a prefetch_related on the Category like this:
categories = Category.objects.distinct().prefetch_related('specifications').filter(filters)
Now I can loop over the categories and show the related specifications like:
{% for category in categories %}
<tr>
<th colspan="7">{{ category.name }} - ({{ category.abbr }})</th>
</tr>
{% for specification in category.specifications.all %}
...
I also want to use the paginator, but now the paginator only counts the categories and not the related specifications. Is it possible to paginate on the specifications with the given query or should I change the query to retrieve all the specifications?
Use Prefetch
categories = Category.objects.distinct().filter(filters)
category_ids = categories.values_list('id', flat=True) # category ids on page
categories = categories.prefetch_related(
Prefetch(
'specifications',
queryset=Specialization.objects.filter(category_id__in=category_ids)
)
)
Here it creates another db request (to fetch category ids) but it will cost less than prefetch all specializations I think. It depends on your data structure but it definitely one of solutions.
Have you tried django-tables2?
With it you could just simply render the table with something like:
Create a CategoryTable class and add it to your view:
class CategoryTable(tables.Table):
class Meta:
model = Category
def your_view(request):
...
categories = Category.objects.distinct()
.prefetch_related('specifications')
.filter(filters)
table = CategoryTable(categories)
table.paginate(page=request.GET.get("page", 1), per_page=25)
return render(request, "category_template.html", {"table": table})
Then, in your template just put:
{% load django_tables2 %}
{% render_table table %}
What you are trying to achieve is an anti-pattern of prefetch_related. Prefetch is to fetch "all" the related rows, but the use case is to paginate the specifications.
Prefetch would be good in cases where the number of related rows per parent row is ~5 (or upto 10, remember that you are wasting DB network bandwidth growing prefetched child rows. So if you are considering pagination, it is best to avoid prefetch)
Note: Child rows = specifications, Parent rows = categories, in your use case.
My answer would be to avoid using prefetch in this case. Just use the following
categories = Category.objects.distinct().filter(filters)
--
{% for category in categories %}
<tr>
<th colspan="7">{{ category.name }} - ({{ category.abbr }})</th>
</tr>
# Use some table lib like django-tables
...
If this is a simple internal project, do go ahead with prefetch, no problems, otherwise you are going to hit DB performance issues.
I am trying to query a table where I have many records with the same name, on purpose. In my example, I'm using the make of the car, and unfortunately I've already ruled out using a foreignkey. Long story. Anyway, I've been able to determine that I can query the table using a ModelChoiceField and using the distinct command as I'm using Postgresql as shown below:
class Vehicle(forms.Form):
dropdown = forms.ModelChoiceField(queryset=Car.objects.none())
def __init__(self, *args, **kwargs):
super(Vehicle, self).__init__(*args, **kwargs)
self.fields['dropdown'].empty_label = ''
qs = Car.objects.distinct('vehicle_make')
The code above does what I need related to the dropdown, it limits the ModelChoiceField to just the unique values of the vehicle_make of the car.
The challenge is when I go to try to display all of the records with that vehicle_make in my template. I've tried to do a Detail View, but it is only showing me that individual record. That make sense since detail view is just for that record, but I'm trying to figure out how to query the table to show me all of the records with that vehicle_make. I've explored the ChoiceField as well, but can't seem to get this to work either. I've tried several variations of the code below in the template, but nothing seems to work.
{% for vehicle_make in car.queryset %}
{{ vehicle_make }}
{% endfor %}
My model is as follows:
Car(models.Model):
vehicle_make = models.Charfield(max_length=264,unique=False)
def __str__(self):
return self.vehicle_make
Thanks in advance for your input and suggestions.
You can query for all Car objects with a given value for vehicle_make with Car.objects.filter(vehicle_make = 'foo'). For example, this ListView would list all cars:
from django.views.generic.list import ListView
from .models import Car
class CarListView(ListView):
context_object_name = "car_list"
queryset = Car.objects.all()
Whereas this view would list all cars of make 'foo':
class FooCarListView(ListView):
context_object_name = "car_list"
queryset = Car.objects.filter(vehicle_make = 'foo')
One easy way to make this more "dynamic" would be to look for a search query in the url, either as a keyword argument or as a querystring. You could use your existing form to create a URL like this, then parse it in a view. For instance, this view would look for URLs appended with ?search=bar:
class SearchableCarListView(ListView):
context_object_name = "car_list"
def get_queryset(self):
search_term = self.request.GET.get('search', None)
if search_term is not None:
return Car.objects.filter(vehicle_make = search_term)
return Car.objects.all()
Your use case may be better served by using an icontains or iexact lookup, or even by making use of Django's full text search.
In the template for any of these views, you can access all of the Car objects in your queryset like so:
{% for car in car_list %}
{{car}}
{% endfor %}
I hope that I have understood and addressed your question. I am not 100% sure how you are using your form right now, so please let me know if I have overlooked anything there.
After our discussion in the comments, I think you are overcomplicating things with your form, and I think that your lookup might not be doing what you want.
We will get a ValuesQuerySet of all distinct vehicle_make values:
qs = Car.objects.values('vehicle_make').distinct()
(See: Select DISTINCT individual columns in django?)
This returns a ValuesQuerySet that looks like this:
<QuerySet [{'vehicle_make': 'vehicle_make_1'}, {'vehicle_make': 'vehicle_make_2'}...]
Where vehicle_make_1, vehicle_make_2 etc. are your distinct values.
We can give this queryset qs to our template and construct a select element. We could also simplify it first to a list of values to make it easier to work with in the template:
values_list = [ c['vehicle_make'] for c in qs.all() ]
Pass that value to our template and use it to make our select element:
<select name='search'>
{% for vehicle_make in values_list %}
<option value = {{vehicle_make}} > {{vehicle_make}} </option>
{% endfor %}
</select>
From here you have several options. In my opinion, the simplest thing would be using this select element in a form that uses the GET method to construct a search URL like we discussed earlier. Check out the first answer to this question for more info: <form method="link" > or <a>? What's the difference? There is a code snippet in the first answer to that question that can easily be adapted to create a GET-based search form with your newly-created select element.
This QuerySet (Let's say Model model has 12 field):
objects = Model.objects.filter(...)
And this template:
{% for object in object %}
<a href='{{ object.get_absolut_url }}'>Foo: {{ object.bar }}</a>
{% endfor %}
perform SQL query which gets unnecessary fields (every 12 fields + relations). I want Django to get only 'bar' field. How can I do this?
By the way I know about values() method, but as it returns dict, I can't call Model methods such as get_absolute_url().
You want to use only():
objects = Model.objects.select_related().only("bar").filter(...)
Keep in mind however if you limit too much of the data down and then use the objects in other ways you can actually cause the ORM to execute extra queries, so make sure to be using something like django-debug-toolbar to ensure you aren't removing these unnecessary fields to only incur the hit of lots of unnecessary queries which is a worse situation.
FYI you can also use defer() to list the fields you don't want loaded if you want to think about it in the other direction.
Is it possible to access annotated values on querysets in templates?
For example I have the following queryset that I'm passing to my template:
context[videos] = Videos.objects.annotate(view_count=Count(views)).order_by(view_count)[:100]
In my template I'm trying to get the view count like this:
{% for video in videos %}
{{ video.view_count }}
{% endfor %}
Which displays nothing.
However if I use:
{{ video.views.count }}
It seems fine - but i believe the second option recalculates the view count. I'd like to use the annotated value since it should already be calculated.
It would make sense that the QuerySet aggregation method per-item, annotate(), has such a name because it annotates (sets) the aggregated value to each item (model instance) it yields, like a normal field, for example:
# Build an annotated queryset
>>> q = Book.objects.annotate(Count('authors'))
# Interrogate the first object in the queryset
>>> q[0]
<Book: The Definitive Guide to Django>
>>> q[0].authors__count
2
# Interrogate the second object in the queryset
>>> q[1]
<Book: Practical Django Projects>
>>> q[1].authors__count
1
About the names:
the name for the annotation is automatically derived from the name of the aggregate function and the name of the field being aggregated. You can override this default name by providing an alias when you specify the annotation:
>>> q = Book.objects.annotate(num_authors=Count('authors'))
>>> q[0].num_authors
2
>>> q[1].num_authors
1
For example with:
context['videos'] = Videos.objects.annotate(view_count=Count('views')).order_by('-view_count')[100:]
You could use:
[video.view_count for video in context['videos']]
Which should be the same as using values_list():
Videos.objects.annotate(view_count=Count('views')).values_list('view_count', flat=True)
And similar to:
{% for video in videos %}
{{ video.view_count }}
{% endfor %}
That said, unlike normal fields, the order in which filters are applied matters, you've been warned B)
Say I have a model:
class Entry(models.Model):
...
tags = TaggableManager()
When I iterate over Entry.objects.all() in a template, entry.tags.all produces one more query to the database. Is it possible to reduce queries number? Using something like select_related() (I know it won't work, since django-taggit uses manytomany relation, but I am sure there should be a way to select all entries with related tags in 1 hit)?
From Django 1.4 onward, you can use prefetch_related to retrieve one-to-many relations on a queryset in a single query. Unfortunately this doesn't work brilliantly with django-taggit, because the 'tags' property is a manager rather than a true relation, and so prefetch_related can't make sense of it. Instead, you need to follow the tagged_items relation:
entries = Entry.objects.prefetch_related('tagged_items__tag')
You then need to go through some similar contortions in the template code to access the prefetched tags, because entry.tags.all will run another query rather than making use of the prefetch:
{% for tagged_item in entry.tagged_items %}
<li>{{ tagged_item.tag.name }}</li>
{% endfor %}
Try using Select Reverse it's designed to grab the entirity of a many2many relationship with a single query.