Here's the model:
class ModelA:
title = charfield
m2m = foreignkey, relatedname='m2ms'
This is working:
ModelA.objects.filter(Q(title__icontains='a') & Q(title__icontains='b'))
So it returns all records whose titles contains both letters 'a' and 'b'.
Then same not working for many to many:
ModelA.objects.filter(Q(m2ms__id=1) & Q(m2ms__id=2))
ModelA m2ms list:
for x in ModelA.objects.all():
print x.m2ms.all().values_list('id', Flat=True)
#Output:
1,2,3
1,2
1
1,3,5
4,6,7
1,8
So, expected output of ModelA.objects.filter(Q(m2ms__id=1) & Q(m2ms__id=2)) should be records having these m2m ids: [1,2,3], [1,2]. But its not happening. Why ?
I cant use Q(m2ms__id__in=[1,2]) because it returns same even if I do __in=[1,2,3,4, infinite numbers]
Reason for using Q instead of filter is mentioned in this question - django filter on many to many along with Q
Read this section of the docs.
Particularly this paragraph:
To handle both of these situations, Django has a consistent way of processing filter() and exclude() calls. Everything inside a single filter() call is applied simultaneously to filter out items matching all those requirements. Successive filter() calls further restrict the set of objects, but for multi-valued relations, they apply to any object linked to the primary model, not necessarily those objects that were selected by an earlier filter() call.
I believe if you do ModelA.objects.filter(Q(m2ms__id__in=[1, 2])) or ModelA.objects.filter(m2ms__id__in=[1, 2]) it will work as you expect it to.
Related
So I have 2 query sets that are from the same model but different records. I already made the pythonic way of getting the result. but I want to do it with aggregate and annotate to get this done.
result = [i for i in self.old_searches if round(
abs(same_keys.get(keyword =i.keyword).current_position-i.current_position)) >=1]
so both of self.old_searches and same_keys have the same field called current_position
and for example I need first recprd of the first query be compared with first record second one. but all i need is the current_position.
I always assumed that chaining multiple filter() calls in Django was always the same as collecting them in a single call.
# Equivalent
Model.objects.filter(foo=1).filter(bar=2)
Model.objects.filter(foo=1,bar=2)
but I have run across a complicated queryset in my code where this is not the case
class Inventory(models.Model):
book = models.ForeignKey(Book)
class Profile(models.Model):
user = models.OneToOneField(auth.models.User)
vacation = models.BooleanField()
country = models.CharField(max_length=30)
# Not Equivalent!
Book.objects.filter(inventory__user__profile__vacation=False).filter(inventory__user__profile__country='BR')
Book.objects.filter(inventory__user__profile__vacation=False, inventory__user__profile__country='BR')
The generated SQL is
SELECT "library_book"."id", "library_book"."asin", "library_book"."added", "library_book"."updated" FROM "library_book" INNER JOIN "library_inventory" ON ("library_book"."id" = "library_inventory"."book_id") INNER JOIN "auth_user" ON ("library_inventory"."user_id" = "auth_user"."id") INNER JOIN "library_profile" ON ("auth_user"."id" = "library_profile"."user_id") INNER JOIN "library_inventory" T5 ON ("library_book"."id" = T5."book_id") INNER JOIN "auth_user" T6 ON (T5."user_id" = T6."id") INNER JOIN "library_profile" T7 ON (T6."id" = T7."user_id") WHERE ("library_profile"."vacation" = False AND T7."country" = BR )
SELECT "library_book"."id", "library_book"."asin", "library_book"."added", "library_book"."updated" FROM "library_book" INNER JOIN "library_inventory" ON ("library_book"."id" = "library_inventory"."book_id") INNER JOIN "auth_user" ON ("library_inventory"."user_id" = "auth_user"."id") INNER JOIN "library_profile" ON ("auth_user"."id" = "library_profile"."user_id") WHERE ("library_profile"."vacation" = False AND "library_profile"."country" = BR )
The first queryset with the chained filter() calls joins the Inventory model twice effectively creating an OR between the two conditions whereas the second queryset ANDs the two conditions together. I was expecting that the first query would also AND the two conditions. Is this the expected behavior or is this a bug in Django?
The answer to a related question Is there a downside to using ".filter().filter().filter()..." in Django? seems to indicated that the two querysets should be equivalent.
The way I understand it is that they are subtly different by design (and I am certainly open for correction): filter(A, B) will first filter according to A and then subfilter according to B, while filter(A).filter(B) will return a row that matches A 'and' a potentially different row that matches B.
Look at the example here:
https://docs.djangoproject.com/en/dev/topics/db/queries/#spanning-multi-valued-relationships
particularly:
Everything inside a single filter() call is applied simultaneously to filter out items matching all those requirements. Successive filter() calls further restrict the set of objects
...
In this second example (filter(A).filter(B)), the first filter restricted the queryset to (A). The second filter restricted the set of blogs further to those that are also (B). The entries select by the second filter may or may not be the same as the entries in the first filter.`
These two style of filtering are equivalent in most cases, but when query on objects base on ForeignKey or ManyToManyField, they are slightly different.
Examples from the documentation.
model
Blog to Entry is a one-to-many relation.
from django.db import models
class Blog(models.Model):
...
class Entry(models.Model):
blog = models.ForeignKey(Blog)
headline = models.CharField(max_length=255)
pub_date = models.DateField()
...
objects
Assuming there are some blog and entry objects here.
queries
Blog.objects.filter(entry__headline_contains='Lennon',
entry__pub_date__year=2008)
Blog.objects.filter(entry__headline_contains='Lennon').filter(
entry__pub_date__year=2008)
For the 1st query (single filter one), it match only blog1.
For the 2nd query (chained filters one), it filters out blog1 and blog2.
The first filter restricts the queryset to blog1, blog2 and blog5; the second filter restricts the set of blogs further to blog1 and blog2.
And you should realize that
We are filtering the Blog items with each filter statement, not the Entry items.
So, it's not the same, because Blog and Entry are multi-valued relationships.
Reference: https://docs.djangoproject.com/en/1.8/topics/db/queries/#spanning-multi-valued-relationships
If there is something wrong, please correct me.
Edit: Changed v1.6 to v1.8 since the 1.6 links are no longer available.
As you can see in the generated SQL statements the difference is not the "OR" as some may suspect. It is how the WHERE and JOIN is placed.
Example1 (same joined table) :
(example from https://docs.djangoproject.com/en/dev/topics/db/queries/#spanning-multi-valued-relationships)
Blog.objects.filter(entry__headline__contains='Lennon', entry__pub_date__year=2008)
This will give you all the Blogs that have one entry with both (entry_headline_contains='Lennon') AND (entry__pub_date__year=2008), which is what you would expect from this query.
Result:
Book with {entry.headline: 'Life of Lennon', entry.pub_date: '2008'}
Example 2 (chained)
Blog.objects.filter(entry__headline__contains='Lennon').filter(entry__pub_date__year=2008)
This will cover all the results from Example 1, but it will generate slightly more result. Because it first filters all the blogs with (entry_headline_contains='Lennon') and then from the result filters (entry__pub_date__year=2008).
The difference is that it will also give you results like:
Book with {entry.headline: 'Lennon', entry.pub_date: 2000}, {entry.headline: 'Bill', entry.pub_date: 2008}
In your case
I think it is this one you need:
Book.objects.filter(inventory__user__profile__vacation=False, inventory__user__profile__country='BR')
And if you want to use OR please read: https://docs.djangoproject.com/en/dev/topics/db/queries/#complex-lookups-with-q-objects
From Django docs :
To handle both of these situations, Django has a consistent way of processing filter() calls. Everything inside a single filter() call is applied simultaneously to filter out items matching all those requirements. Successive filter() calls further restrict the set of objects, but for multi-valued relations, they apply to any object linked to the primary model, not necessarily those objects that were selected by an earlier filter() call.
It is clearly said that multiple conditions in a single filter() are applied simultaneously.
That means that doing :
objs = Mymodel.objects.filter(a=True, b=False)
will return a queryset with raws from model Mymodel where a=True AND b=False.
Successive filter(), in some case, will provide the same result. Doing :
objs = Mymodel.objects.filter(a=True).filter(b=False)
will return a queryset with raws from model Mymodel where a=True AND b=False too. Since you obtain "first" a queryset with records which have a=True and then it's restricted to those who have b=False at the same time.
The difference in chaining filter() comes when there are multi-valued relations, which means you are going through other models (such as the example given in the docs, between Blog and Entry models). It is said that in that case (...) they apply to any object linked to the primary model, not necessarily those objects that were selected by an earlier filter() call.
Which means that it applies the successives filter() on the target model directly, not on previous filter()
If I take the example from the docs :
Blog.objects.filter(entry__headline__contains='Lennon').filter(entry__pub_date__year=2008)
remember that it's the model Blog that is filtered, not the Entry. So it will treat the 2 filter() independently.
It will, for instance, return a queryset with Blogs, that have entries that contain 'Lennon' (even if they are not from 2008) and entries that are from 2008 (even if their headline does not contain 'Lennon')
THIS ANSWER goes even further in the explanation. And the original question is similar.
Sometimes you don't want to join multiple filters together like this:
def your_dynamic_query_generator(self, event: Event):
qs \
.filter(shiftregistrations__event=event) \
.filter(shiftregistrations__shifts=False)
And the following code would actually not return the correct thing.
def your_dynamic_query_generator(self, event: Event):
return Q(shiftregistrations__event=event) & Q(shiftregistrations__shifts=False)
What you can do now is to use an annotation count-filter.
In this case we count all shifts which belongs to a certain event.
qs: EventQuerySet = qs.annotate(
num_shifts=Count('shiftregistrations__shifts', filter=Q(shiftregistrations__event=event))
)
Afterwards you can filter by annotation.
def your_dynamic_query_generator(self):
return Q(num_shifts=0)
This solution is also cheaper on large querysets.
Hope this helps.
Saw this in a comment and I thought it was the simplest explanation.
filter(A, B) is the AND ; filter(A).filter(B) is OR
It's true if every linked model satisfies both conditions
I have a Result object that is tagged with "one" and "two". When I try to query for objects tagged "one" and "two", I get nothing back:
q = Result.objects.filter(Q(tags__name="one") & Q(tags__name="two"))
print len(q)
# prints zero, was expecting 1
Why does it not work with Q? How can I make it work?
The way django-taggit implements tagging is essentially through a ManytoMany relationship. In such cases there is a separate table in the database that holds these relations. It is usually called a "through" or intermediate model as it connects the two models. In the case of django-taggit this is called TaggedItem. So you have the Result model which is your model and you have two models Tag and TaggedItem provided by django-taggit.
When you make a query such as Result.objects.filter(Q(tags__name="one")) it translates to looking up rows in the Result table that have a corresponding row in the TaggedItem table that has a corresponding row in the Tag table that has the name="one".
Trying to match for two tag names would translate to looking up up rows in the Result table that have a corresponding row in the TaggedItem table that has a corresponding row in the Tag table that has both name="one" AND name="two". You obviously never have that as you only have one value in a row, it's either "one" or "two".
These details are hidden away from you in the django-taggit implementation, but this is what happens whenever you have a ManytoMany relationship between objects.
To resolve this you can:
Option 1
Query tag after tag evaluating the results each time, as it is suggested in the answers from others. This might be okay for two tags, but will not be good when you need to look for objects that have 10 tags set on them. Here would be one way to do this that would result in two queries and get you the result:
# get the IDs of the Result objects tagged with "one"
query_1 = Result.objects.filter(tags__name="one").values('id')
# use this in a second query to filter the ID and look for the second tag.
results = Result.objects.filter(pk__in=query_1, tags__name="two")
You could achieve this with a single query so you only have one trip from the app to the database, which would look like this:
# create django subquery - this is not evaluated, but used to construct the final query
subquery = Result.objects.filter(pk=OuterRef('pk'), tags__name="one").values('id')
# perform a combined query using a subquery against the database
results = Result.objects.filter(Exists(subquery), tags__name="two")
This would only make one trip to the database. (Note: filtering on sub-queries requires django 3.0).
But you are still limited to two tags. If you need to check for 10 tags or more, the above is not really workable...
Option 2
Query the relationship table instead directly and aggregate the results in a way that give you the object IDs.
# django-taggit uses Content Types so we need to pick up the content type from cache
result_content_type = ContentType.objects.get_for_model(Result)
tag_names = ["one", "two"]
tagged_results = (
TaggedItem.objects.filter(tag__name__in=tag_names, content_type=result_content_type)
.values('object_id')
.annotate(occurence=Count('object_id'))
.filter(occurence=len(tag_names))
.values_list('object_id', flat=True)
)
TaggedItem is the hidden table in the django-taggit implementation that contains the relationships. The above will query that table and aggregate all the rows that refer either to the "one" or "two" tags, group the results by the ID of the objects and then pick those where the object ID had the number of tags you are looking for.
This is a single query and at the end gets you the IDs of all the objects that have been tagged with both tags. It is also the exact same query regardless if you need 2 tags or 200.
Please review this and let me know if anything needs clarification.
first of all, this three are same:
Result.objects.filter(tags__name="one", tags__name="two")
Result.objects.filter(Q(tags__name="one") & Q(tags__name="two"))
Result.objects.filter(tags__name_in=["one"]).filter(tags__name_in=["two"])
i think the name field is CharField and no record could be equal to "one" and "two" at same time.
in python code the query looks like this(always false, and why you are geting no result):
from random import choice
name = choice(["abtin", "shino"])
if name == "abtin" and name == "shino":
we use Q object for implement OR or complex queries
Into the example that works you do an end on two python objects (query sets). That gets applied to any record not necessarily to the same record that has one AND two as tag.
ps: Why do you use the in filter ?
q = Result.objects.filter(tags_name_in=["one"]).filter(tags_name_in=["two"])
add .distinct() to remove duplicates if expecting more than one unique object
I always assumed that chaining multiple filter() calls in Django was always the same as collecting them in a single call.
# Equivalent
Model.objects.filter(foo=1).filter(bar=2)
Model.objects.filter(foo=1,bar=2)
but I have run across a complicated queryset in my code where this is not the case
class Inventory(models.Model):
book = models.ForeignKey(Book)
class Profile(models.Model):
user = models.OneToOneField(auth.models.User)
vacation = models.BooleanField()
country = models.CharField(max_length=30)
# Not Equivalent!
Book.objects.filter(inventory__user__profile__vacation=False).filter(inventory__user__profile__country='BR')
Book.objects.filter(inventory__user__profile__vacation=False, inventory__user__profile__country='BR')
The generated SQL is
SELECT "library_book"."id", "library_book"."asin", "library_book"."added", "library_book"."updated" FROM "library_book" INNER JOIN "library_inventory" ON ("library_book"."id" = "library_inventory"."book_id") INNER JOIN "auth_user" ON ("library_inventory"."user_id" = "auth_user"."id") INNER JOIN "library_profile" ON ("auth_user"."id" = "library_profile"."user_id") INNER JOIN "library_inventory" T5 ON ("library_book"."id" = T5."book_id") INNER JOIN "auth_user" T6 ON (T5."user_id" = T6."id") INNER JOIN "library_profile" T7 ON (T6."id" = T7."user_id") WHERE ("library_profile"."vacation" = False AND T7."country" = BR )
SELECT "library_book"."id", "library_book"."asin", "library_book"."added", "library_book"."updated" FROM "library_book" INNER JOIN "library_inventory" ON ("library_book"."id" = "library_inventory"."book_id") INNER JOIN "auth_user" ON ("library_inventory"."user_id" = "auth_user"."id") INNER JOIN "library_profile" ON ("auth_user"."id" = "library_profile"."user_id") WHERE ("library_profile"."vacation" = False AND "library_profile"."country" = BR )
The first queryset with the chained filter() calls joins the Inventory model twice effectively creating an OR between the two conditions whereas the second queryset ANDs the two conditions together. I was expecting that the first query would also AND the two conditions. Is this the expected behavior or is this a bug in Django?
The answer to a related question Is there a downside to using ".filter().filter().filter()..." in Django? seems to indicated that the two querysets should be equivalent.
The way I understand it is that they are subtly different by design (and I am certainly open for correction): filter(A, B) will first filter according to A and then subfilter according to B, while filter(A).filter(B) will return a row that matches A 'and' a potentially different row that matches B.
Look at the example here:
https://docs.djangoproject.com/en/dev/topics/db/queries/#spanning-multi-valued-relationships
particularly:
Everything inside a single filter() call is applied simultaneously to filter out items matching all those requirements. Successive filter() calls further restrict the set of objects
...
In this second example (filter(A).filter(B)), the first filter restricted the queryset to (A). The second filter restricted the set of blogs further to those that are also (B). The entries select by the second filter may or may not be the same as the entries in the first filter.`
These two style of filtering are equivalent in most cases, but when query on objects base on ForeignKey or ManyToManyField, they are slightly different.
Examples from the documentation.
model
Blog to Entry is a one-to-many relation.
from django.db import models
class Blog(models.Model):
...
class Entry(models.Model):
blog = models.ForeignKey(Blog)
headline = models.CharField(max_length=255)
pub_date = models.DateField()
...
objects
Assuming there are some blog and entry objects here.
queries
Blog.objects.filter(entry__headline_contains='Lennon',
entry__pub_date__year=2008)
Blog.objects.filter(entry__headline_contains='Lennon').filter(
entry__pub_date__year=2008)
For the 1st query (single filter one), it match only blog1.
For the 2nd query (chained filters one), it filters out blog1 and blog2.
The first filter restricts the queryset to blog1, blog2 and blog5; the second filter restricts the set of blogs further to blog1 and blog2.
And you should realize that
We are filtering the Blog items with each filter statement, not the Entry items.
So, it's not the same, because Blog and Entry are multi-valued relationships.
Reference: https://docs.djangoproject.com/en/1.8/topics/db/queries/#spanning-multi-valued-relationships
If there is something wrong, please correct me.
Edit: Changed v1.6 to v1.8 since the 1.6 links are no longer available.
As you can see in the generated SQL statements the difference is not the "OR" as some may suspect. It is how the WHERE and JOIN is placed.
Example1 (same joined table) :
(example from https://docs.djangoproject.com/en/dev/topics/db/queries/#spanning-multi-valued-relationships)
Blog.objects.filter(entry__headline__contains='Lennon', entry__pub_date__year=2008)
This will give you all the Blogs that have one entry with both (entry_headline_contains='Lennon') AND (entry__pub_date__year=2008), which is what you would expect from this query.
Result:
Book with {entry.headline: 'Life of Lennon', entry.pub_date: '2008'}
Example 2 (chained)
Blog.objects.filter(entry__headline__contains='Lennon').filter(entry__pub_date__year=2008)
This will cover all the results from Example 1, but it will generate slightly more result. Because it first filters all the blogs with (entry_headline_contains='Lennon') and then from the result filters (entry__pub_date__year=2008).
The difference is that it will also give you results like:
Book with {entry.headline: 'Lennon', entry.pub_date: 2000}, {entry.headline: 'Bill', entry.pub_date: 2008}
In your case
I think it is this one you need:
Book.objects.filter(inventory__user__profile__vacation=False, inventory__user__profile__country='BR')
And if you want to use OR please read: https://docs.djangoproject.com/en/dev/topics/db/queries/#complex-lookups-with-q-objects
From Django docs :
To handle both of these situations, Django has a consistent way of processing filter() calls. Everything inside a single filter() call is applied simultaneously to filter out items matching all those requirements. Successive filter() calls further restrict the set of objects, but for multi-valued relations, they apply to any object linked to the primary model, not necessarily those objects that were selected by an earlier filter() call.
It is clearly said that multiple conditions in a single filter() are applied simultaneously.
That means that doing :
objs = Mymodel.objects.filter(a=True, b=False)
will return a queryset with raws from model Mymodel where a=True AND b=False.
Successive filter(), in some case, will provide the same result. Doing :
objs = Mymodel.objects.filter(a=True).filter(b=False)
will return a queryset with raws from model Mymodel where a=True AND b=False too. Since you obtain "first" a queryset with records which have a=True and then it's restricted to those who have b=False at the same time.
The difference in chaining filter() comes when there are multi-valued relations, which means you are going through other models (such as the example given in the docs, between Blog and Entry models). It is said that in that case (...) they apply to any object linked to the primary model, not necessarily those objects that were selected by an earlier filter() call.
Which means that it applies the successives filter() on the target model directly, not on previous filter()
If I take the example from the docs :
Blog.objects.filter(entry__headline__contains='Lennon').filter(entry__pub_date__year=2008)
remember that it's the model Blog that is filtered, not the Entry. So it will treat the 2 filter() independently.
It will, for instance, return a queryset with Blogs, that have entries that contain 'Lennon' (even if they are not from 2008) and entries that are from 2008 (even if their headline does not contain 'Lennon')
THIS ANSWER goes even further in the explanation. And the original question is similar.
Sometimes you don't want to join multiple filters together like this:
def your_dynamic_query_generator(self, event: Event):
qs \
.filter(shiftregistrations__event=event) \
.filter(shiftregistrations__shifts=False)
And the following code would actually not return the correct thing.
def your_dynamic_query_generator(self, event: Event):
return Q(shiftregistrations__event=event) & Q(shiftregistrations__shifts=False)
What you can do now is to use an annotation count-filter.
In this case we count all shifts which belongs to a certain event.
qs: EventQuerySet = qs.annotate(
num_shifts=Count('shiftregistrations__shifts', filter=Q(shiftregistrations__event=event))
)
Afterwards you can filter by annotation.
def your_dynamic_query_generator(self):
return Q(num_shifts=0)
This solution is also cheaper on large querysets.
Hope this helps.
Saw this in a comment and I thought it was the simplest explanation.
filter(A, B) is the AND ; filter(A).filter(B) is OR
It's true if every linked model satisfies both conditions
I have a two models:
class Category(models.Model):
pass
class Item(models.Model):
cat = models.ForeignKey(Category)
I am trying to return all Categories for which all of that category's items belong to a given subset of item ids (fixed thanks). For example, all categories for which all of the items associated with that category have ids in the set [1,3,5].
How could this be done using Django's query syntax (as of 1.1 beta)? Ideally, all the work should be done in the database.
Category.objects.filter(item__id__in=[1, 3, 5])
Django creates the reverse relation ship on the model without the foreign key. You can filter on it by using its related name (usually just the model name lowercase but it can be manually overwritten), two underscores, and the field name you want to query on.
lets say you require all items to be in the following set:
allowable_items = set([1,3,4])
one bruteforce solution would be to check the item_set for every category as so:
categories_with_allowable_items = [
category for category in
Category.objects.all() if
set([item.id for item in category.item_set.all()]) <= allowable_items
]
but we don't really have to check all categories, as categories_with_allowable_items is always going to be a subset of the categories related to all items with ids in allowable_items... so that's all we have to check (and this should be faster):
categories_with_allowable_items = set([
item.category for item in
Item.objects.select_related('category').filter(pk__in=allowable_items) if
set([siblingitem.id for siblingitem in item.category.item_set.all()]) <= allowable_items
])
if performance isn't really an issue, then the latter of these two (if not the former) should be fine. if these are very large tables, you might have to come up with a more sophisticated solution. also if you're using a particularly old version of python remember that you'll have to import the sets module
I've played around with this a bit. If QuerySet.extra() accepted a "having" parameter I think it would be possible to do it in the ORM with a bit of raw SQL in the HAVING clause. But it doesn't, so I think you'd have to write the whole query in raw SQL if you want the database doing the work.
EDIT:
This is the query that gets you part way there:
from django.db.models import Count
Category.objects.annotate(num_items=Count('item')).filter(num_items=...)
The problem is that for the query to work, "..." needs to be a correlated subquery that looks up, for each category, the number of its items in allowed_items. If .extra had a "having" argument, you'd do it like this:
Category.objects.annotate(num_items=Count('item')).extra(having="num_items=(SELECT COUNT(*) FROM app_item WHERE app_item.id in % AND app_item.cat_id = app_category.id)", having_params=[allowed_item_ids])