How can I optimize the following request to eliminate loop? Codes count is several hundred, so I get several hundreds database queries, which is unacceptable.
n = 3
result = []
codes = Target.objects.filter(code__in=['ABC', 'CDE', ...])
for code in codes:
result.append(Data.objects.select_related('target')
.filter(target__code=code)
.values('spec', 'spec_type')
.order_by('-spec')[:n])
Models:
class Data(models.Model):
target = models.ForeignKey(Target)
spec_type = models.CharField()
spec = models.FloatField()
class Target(models.Model):
code = models.TextField(db_index=True)
You do not have to retrieve the codes as a QuerySet to enumerate over. We can directly work with the list of codes.
If you want to construct a QuerySet that contains all the given elements, you can make a QuerySet with union that will fetch these objects. In that case this can be done with .union(…) [Django-doc]:
codes = ['ABC', 'CDE']
n = 3
result = Data.objects.none().union(
*[
Data.objects.filter(target__code=code).values('spec', 'spec_type').order_by('-spec')[:n]
for code in codes
],
all=True
)
Like #Willem Van Onsem said, you don't need to get get a queryset of your Target objects since you already seem to have the codes that you want. Just store the codes in a variable and then you can do a django query using that list.
codes = ['ABC', 'CDE', ...]
result = Data.objects.filter(target__code__in = codes)
This query should return all Data objects of which the related Target object's code is in the list codes.
Related
With the following models:
class Item(models.Model):
name = models.CharField(max_length=255)
attributes = models.ManyToManyField(ItemAttribute)
class ItemAttribute(models.Model):
attribute = models.CharField(max_length=255)
string_value = models.CharField(max_length=255)
int_value = models.IntegerField()
I also have an Item which has 2 attributes, 'color': 'red', and 'size': 3.
If I do any of these queries:
Item.objects.filter(attributes__string_value='red')
Item.objects.filter(attributes__int_value=3)
I will get Item returned, works as I expected.
However, if I try to do a multiple query, like:
Item.objects.filter(attributes__string_value='red', attributes__int_value=3)
All I want to do is an AND. This won't work either:
Item.objects.filter(Q(attributes__string_value='red') & Q(attributes__int_value=3))
The output is:
<QuerySet []>
Why? How can I build such a query that my Item is returned, because it has the attribute red and the attribute 3?
If it's of any use, you can chain filter expressions in Django:
query = Item.objects.filter(attributes__string_value='red').filter(attributes__int_value=3')
From the DOCS:
This takes the initial QuerySet of all entries in the database, adds a filter, then an exclusion, then another filter. The final result is a QuerySet containing all entries with a headline that starts with “What”, that were published between January 30, 2005, and the current day.
To do it with .filter() but with dynamic arguments:
args = {
'{0}__{1}'.format('attributes', 'string_value'): 'red',
'{0}__{1}'.format('attributes', 'int_value'): 3
}
Product.objects.filter(**args)
You can also (if you need a mix of AND and OR) use Django's Q objects.
Keyword argument queries – in filter(), etc. – are “AND”ed together. If you need to execute more complex queries (for example, queries with OR statements), you can use Q objects.
A Q object (django.db.models.Q) is an object used to encapsulate a
collection of keyword arguments. These keyword arguments are specified
as in “Field lookups” above.
You would have something like this instead of having all the Q objects within that filter:
** import Q from django
from *models import Item
#assuming your arguments are kwargs
final_q_expression = Q(kwargs[1])
for arg in kwargs[2:..]
final_q_expression = final_q_expression & Q(arg);
result = Item.objects.filter(final_q_expression)
This is code I haven't run, it's out of the top of my head. Treat it as pseudo-code if you will.
Although, this doesn't answer why the ways you've tried don't quite work. Maybe it has to do with the lookups that span relationships, and the tables that are getting joined to get those values. I would suggest printing yourQuerySet.query to visualize the raw SQL that is being formed and that might help guide you as to why .filter( Q() & Q()) is not working.
I have these classes:
class Keyword(models.Model):
keyword = models.CharField(max_length=100, unique=True)
class Snippet(models.Model):
keywords = models.ManyToManyField(Keyword)
and a list of searched keywords:
searchlist = ['Photo','Cat']
I would like to dynamically query the database for snippets that contain both (AND operation) of the keywords in searchlist.
From that list of snippets, return the list of unique keywords from each of the snippets, excluding the original search terms.
So far I have this:
# Turn list of values into list of Q objects
queries = [Q(keywords__keyword__exact=tag) for tag in searchlist]
# Take one Q object from the list
query = queries.pop()
# Or the Q object with the ones remaining in the list
for item in queries:
query &= item
# Query the model
snippet_list = Snippet.objects.filter(query)
which returns the desired list of snippets but I'm unsure of how to retrieve the combined list of keywords, or if there is a more efficient way to achieve the end result.
EDIT:
I would like to achieve the following pseudo code on snippet_list:
for snippet in snippet_list:
combined_keywords += snippet.keywords
ordered_keyword_list = getOrderedKeywords() # <- I have this function already
final_list = intersection(ordered_keyword_list, combined_keywords)
where combined_keywords is all the keywords in the snippet_list. And final_list is a ordered unique version of combined_keywords.
The final solution looks like this thanks to #Soravux and #Dave-Webb. I'm not sure if this is the most efficient method, but it works.
# Turn list of values into list of Q objects
queries = [Q(keywords__keyword__exact=tag) for tag in valueslist]
query = Q()
for item in queries:
query &= item
# Query the model
snippet_list = Snippet.objects.filter(query)
combined_keywords = []
for snippet in snippet_list:
combined_keywords += snippet.keywords.all()
ordered_keyword_list = Keyword.objects.all().annotate(count=Count('snippet')).order_by('-count')
final_list = [x for x in ordered_keyword_list if x in combined_keywords]
I have the following model.
models.py
class WoCpt(models.Model):
order = models.ForeignKey(Order, null=False)
comp = models.ForeignKey(Comp, null=False)
proc = models.ForeignKey(Proc, null=False)
points = models.IntegerField('Standard Time', null=False)
Sample Database:
[order, comp, proc, points]
[5,4,3,24]
[5,4,1,29]
[5,4,4,23]
[5,3,1,44]
[5,2,1,11]
[7,4,3,24]
[7,4,1,29]
[7,4,4,23]
[7,3,1,44]
[7,2,1,11]
Problem:
Need to calculate total points for order=5 and proc=1
After getting the sum, I need to create a list as below:
(expectation - need to group the similar proc within a given order and prepare a list to pass it to view)
[order, proc, points]
[5,1,84]
[5,3,24]
[5,4,23]
Kindly help.
You need to write some logic your self. I am giving skeleton code for you. In your case you need to use django filters.
order=5
proc=1
new_list = [order, proc, None]
wocpt = WoCpt.objects.filter(order=order, proc=proc)
new_list[2]=sum(row.points for row in wocpt)
print new_list
[5,1,84]
https://docs.djangoproject.com/en/1.6/ref/models/querysets/#filter
No need to get all fancy in Python code. It can just as easily be done in the Django ORM with a single query.
>>> WoCpt.objects.values_list('order', 'proc').annotate(Sum('points'))
[(5,1,84), (5,3,24), (5,4,23)]
This will first group rows by their order and proc columns, and then annotate each row with the sum of points for all contained results. values_list will return a list of tuples, containing the values of the fields as ordered in the values_list parameters, followed by any annotated fields (so ('order', 'proc', 'points_sum') in this case).
If you have these results, and you need the total points for a specific order and proc, you can use a function like this:
def total_points(l, order, proc):
return [x for x in l if x[0] == order and x[1] == proc][2]
i am trying to filter the uniques from a list with this form:
class SpecForm(ModelForm):
a = Doctors_list.objects.values_list('specialty', flat=True)
unique = {z: i for i, z in a}
qs = Doctors_list.objects.filter(id__in=unique.values())
specialty = forms.ModelChoiceField(queryset=qs)
class Meta:
model = Doctors_list
everything seems correct for me, but i get this error: too many values to unpack
any hints?
I think the correct statement should be this:
unique = {z: i for i in a}
Are you specifically trying to put those values into a dictionary? This will yield a list:
unique = [ i for i in a ]
If you go with this, you will have to remove the .values() in qs = Doctors_list.objects.filter(id__in=unique.values()) leaving it like this:
qs = Doctors_list.objects.filter(id__in=unique)
What's going on here is that with brackets in the first approach you're creating a dictionary with just one key and a list as a the value of that key. When you issue .values() you get a list with the dictionary's values. So it's pointless to use a dictionary.
With the second approach you get a list directly.
Hope it helps.
I have such a Book model:
class Book(models.Model):
authors = models.ManyToManyField(Author, ...)
...
In short:
I'd like to retrieve the books whose authors are strictly equal to a given set of authors. I'm not sure if there is a single query that does it, but any suggestions will be helpful.
In long:
Here is what I tried, (that failed to run getting an AttributeError)
# A sample set of authors
target_authors = set((author_1, author_2))
# To reduce the search space,
# first retrieve those books with just 2 authors.
candidate_books = Book.objects.annotate(c=Count('authors')).filter(c=len(target_authors))
final_books = QuerySet()
for author in target_authors:
temp_books = candidate_books.filter(authors__in=[author])
final_books = final_books and temp_books
... and here is what I got:
AttributeError: 'NoneType' object has no attribute '_meta'
In general, how should I query a model with the constraint that its ManyToMany field contains a set of given objects as in my case?
ps: I found some relevant SO questions but couldn't get a clear answer. Any good pointer will be helpful as well. Thanks.
Similar to #goliney's approach, I found a solution. However, I think the efficiency could be improved.
# A sample set of authors
target_authors = set((author_1, author_2))
# To reduce the search space, first retrieve those books with just 2 authors.
candidate_books = Book.objects.annotate(c=Count('authors')).filter(c=len(target_authors))
# In each iteration, we filter out those books which don't contain one of the
# required authors - the instance on the iteration.
for author in target_authors:
candidate_books = candidate_books.filter(authors=author)
final_books = candidate_books
You can use complex lookups with Q objects
from django.db.models import Q
...
target_authors = set((author_1, author_2))
q = Q()
for author in target_authors:
q &= Q(authors=author)
Books.objects.annotate(c=Count('authors')).filter(c=len(target_authors)).filter(q)
Q() & Q() is not equal to .filter().filter(). Their raw SQLs are different where by using Q with &, its SQL just add a condition like WHERE "book"."author" = "author_1" and "book"."author" = "author_2". it should return empty result.
The only solution is just by chaining filter to form a SQL with inner join on same table: ... ON ("author"."id" = "author_book"."author_id") INNER JOIN "author_book" T4 ON ("author"."id" = T4."author_id") WHERE ("author_book"."author_id" = "author_1" AND T4."author_id" = "author_1")
I came across the same problem and came to the same conclusion as iuysal,
untill i had to do a medium sized search (with 1000 records with 150 filters my request would time out).
In my particular case the search would result in no records since the chance that a single record will align with ALL 150 filters is very rare, you can get around the performance issues by verifying that there are records in the QuerySet before applying more filters to save time.
# In each iteration, we filter out those books which don't contain one of the
# required authors - the instance on the iteration.
for author in target_authors:
if candidate_books.count() > 0:
candidate_books = candidate_books.filter(authors=author)
For some reason Django applies filters to empty QuerySets.
But if optimization is to be applied correctly however, using a prepared QuerySet and correctly applied indexes are necessary.