Different results on query changing Q object order - django

Question

I have a problem when making a queryset using Q objects. I'm getting different results depending on how i order some Q conditions. I will simplify my models a little so as to describe my problem in a clean way.
class D(models.Model):
one_attr = models.BooleanField()
s_attr = models.ManyToManyField(S, through='DRelatedToS')
d_to_p = models.ForeignKey(P)
class S(models.Model):
other_attr = models.BooleanField()
s_to_p = models.ForeignKey(P)
class DRelatedToS(models.Model):
to_s = models.ForeignKey(S)
to_d = models.ForeignKey(D)
date = models.DateField()
class P(models.Model):
the_last_attr = models.PositiveIntegerField()
Summary of the relations:
D <-- DRelatedToS --> S --> P
| ^
| |
-------->------->------>----^
With these models and relations, i get two different results depending on how i arrange Q conditions:
First query, that gives one result
D.objects.filter(
Q(one_attr=True, s_attr__s_to_p__the_last_attr=5)
|
Q(one_attr=False, d_to_p__the_last_attr=10)
)
Second query, giving another result, different from first query
D.objects.filter(
Q(one_attr=False, d_to_p__the_last_attr=10)
|
Q(one_attr=True, s_attr__s_to_p__the_last_attr=5)
)
My question is: why is this happening? Is there any problem on how i am doing my query?
When i watch the SQL statements derived from these queries, i get two different statements: one that make a LEFT OUTER JOIN and a lot of INNER JOINs and the second that makes all INNER JOINs. The one that actually return what i want is the one that makes a LEFT OUTER JOIN. This make me feel that all my queries can return bad results depending on how i arrange its conditions. Is this a bug or i am doing anything (or everything) wrong?

Answer 1

Different order should return equal result in your example.
Nevertheless I tested your code (using corrections I made in the question's codes) but can't generate the error you describe. Maybe you had introduced other errors and missed when simplified the code, could you post the sample data you used? (see my data below).
First your sample code is buggy I had edited your question to suggest following corrections to fix problems, simplify and improve it for tests, but I don't see the changes updated so I repeat here:
Correction 1: Model changes in a diff format:
3,4c6,10
< s_attr = models.ManyToMany(S, through='DRelatedToS')
< d_to_p = models.ForeignKey(P)
---
> s_attr = models.ManyToManyField('S', through='DRelatedToS')
> d_to_p = models.ForeignKey('P')
>
> def __unicode__(self):
> return 'D:(%s,%s,%s)' % (self.id, self.one_attr, self.d_to_p.the_last_attr)
8,9c14
< other_attr = models.BooleanField()
< s_to_p = models.ForeignKey(P)
---
> s_to_p = models.ForeignKey('P')
13d17
< to_p = models.ForeignKey(P)
15c19
< date = models.DateField()
---
> to_s = models.ForeignKey(S)
19a24
>
So after apply corrections models look like this:
class D(models.Model):
one_attr = models.BooleanField()
s_attr = models.ManyToManyField('S', through='DRelatedToS')
d_to_p = models.ForeignKey('P')
def __unicode__(self):
return 'D:(%s,%s,%s)' % (self.id, self.one_attr, self.d_to_p.the_last_attr)
class S(models.Model):
s_to_p = models.ForeignKey('P')
class DRelatedToS(models.Model):
to_d = models.ForeignKey(D)
to_s = models.ForeignKey(S)
class P(models.Model):
the_last_attr = models.PositiveIntegerField()
Correction 2: Your lookup fields in queries are wrong (Fixed in answer).
Following is what I did:
Create project and app named testso:
django-admin.py startproject marianobianchi
cd marianobianchi
python manage.py startapp testso
Add your models & adjust project settings (database settings, add testso to INSTALLED_APPS)
Add sample data:
mkdir testso/fixtures
cat > testso/fixtures/initial_data.json
[
{"pk": 1, "model": "testso.d", "fields": {"one_attr": true, "d_to_p": 3}},
{"pk": 2, "model": "testso.d", "fields": {"one_attr": true, "d_to_p": 4}},
{"pk": 3, "model": "testso.d", "fields": {"one_attr": false, "d_to_p": 5}},
{"pk": 4, "model": "testso.d", "fields": {"one_attr": false, "d_to_p": 5}},
{"pk": 1, "model": "testso.s", "fields": {"s_to_p": 1}},
{"pk": 2, "model": "testso.s", "fields": {"s_to_p": 2}},
{"pk": 3, "model": "testso.s", "fields": {"s_to_p": 3}},
{"pk": 1, "model": "testso.drelatedtos", "fields": {"to_d": 2, "to_s": 1}},
{"pk": 2, "model": "testso.drelatedtos", "fields": {"to_d": 1, "to_s": 2}},
{"pk": 3, "model": "testso.drelatedtos", "fields": {"to_d": 1, "to_s": 3}},
{"pk": 1, "model": "testso.p", "fields": {"the_last_attr": 5}},
{"pk": 2, "model": "testso.p", "fields": {"the_last_attr": 5}},
{"pk": 3, "model": "testso.p", "fields": {"the_last_attr": 3}},
{"pk": 4, "model": "testso.p", "fields": {"the_last_attr": 4}},
{"pk": 5, "model": "testso.p", "fields": {"the_last_attr": 10}}
]
python manage.py syncdb
python manage.py shell
In the shell:
>>> from testso.models import *
>>> from django.db.models import Q
>>> D.objects.filter(Q(one_attr=True, s_attr__s_to_p__the_last_attr=5) | Q(one_attr=False, d_to_p__the_last_attr=10))
[<D: D:(1,True,3)>, <D: D:(2,True,4)>, <D: D:(3,False,10)>, <D: D:(4,False,10)>]
>>> D.objects.filter(Q(one_attr=False, d_to_p__the_last_attr=10) | Q(one_attr=True, s_attr__s_to_p__the_last_attr=5))
[<D: D:(1,True,3)>, <D: D:(2,True,4)>, <D: D:(3,False,10)>, <D: D:(4,False,10)>]
Same result! ...as expected.

Answer 2

I cannot answer your question directly, but there may be another way of doing what you want to do which may yield more consistent results:
subset_a = D.objects.filter(one_attr=False, d_to_p__the_last_attr=10)
subset_b = D.objects.filter(one_attr=True, s_attr__p__the_last_attr=5)
union_set = subset_a | subset_b
union_set = union_set.distinct()
The | operator on two querysets does a union, and the distinct will make sure you do not get dupe rows. I would be interested in hearing whether the arrangement here matters as well.

Answer 3

Django seems to have a weak ORM implementation. I would suggest using "filter" or some other way to query the database and see if you get the same inconsistencies.
Or perhaps you should look for alternative ORM implementations, like peewee.