how to merge two annotated querysets into one result - django

Question

Model:
class Foo(models.model):
name = models.CharField(max_length = 50, blank = True, unique = True)
class Bar1(models.Model):
foo = models.ForeignKey('Foo')
value = models.DecimalField(max_digits=10,decimal_places=2)
class Bar2(models.Model):
foo = models.ForeignKey('Foo')
value = models.DecimalField(max_digits=10,decimal_places=2)
Clasess Bar1 and Bar2 are unrelated, so I can't do it as one class what would solve the problem. But this is only example to show the problem as pure as possible.
first = Foo.objects.all().annotate(Sum("bar1__value"))
second = Foo.objects.all().annotate(Sum("bar2__value"))
each of this querysets contains correct values.
I can't merge it into:
both = Foo.objects.all().annotate(Sum("bar1__value")).annotate(Sum("bar2__value"))
Because the sum value multiplicates - this is unfortunately expected behaviour - because of JOINS
And now the problem - how to merge/join first and second to get the both?
Example:
Bar 1:
foo | value
--------------
A | 10
B | 20
B | 20
Bar 2:
foo | value
--------------
A | -0.10
A | -0.10
B | -0.25
both (value differs depends on order of entering bar1 and bar2)
foo | bar1__value__sum | bar2__value__sum
---------------------------------
A | 20 | -0.20
B | 40 | -0.50
expected result:
foo | bar1__value__sum | bar2__value__sum
---------------------------------
A | 10 | -0.20
B | 40 | -0.25
I couldn't use itertools.chains because the result is:
foo | bar1__value__sum | bar2__value__sum
---------------------------------
A | null | -0.20
B | null | -0.25
A | 10 | null
B | 40 | null

Answer 1

Your problem is a known limitation of Django's ORM: https://code.djangoproject.com/ticket/10060.
If you're ok with doing two queries, here's one option:
result = Foo.objects.annotate(b1_sum=Sum("bar1__value"))
bar2_sums = Foo.objects.annotate(b2_sum=Sum("bar2__value")).in_bulk()
for foo in result:
foo.b2_sum = bar2_sums.get(foo.pk).b2_sum

Answer 2

According to answer of #emulbreh i read the ticket and found some solution. I go this way and made this:
models.py:
from django.db.models.expressions import RawSQL
from django.db.models.query import QuerySet
(...)
class NewManager(models.Manager):
"""A re-usable Manager to access a custom QuerySet"""
def __getattr__(self, attr, *args):
try:
return getattr(self.__class__, attr, *args)
except AttributeError:
# don't delegate internal methods to the queryset
if attr.startswith('__') and attr.endswith('__'):
raise
return getattr(self.get_query_set(), attr, *args)
def get_query_set(self):
return self.model.QuerySet(self.model, using=self._db)
class Foo(models.Model):
name = models.CharField(max_length = 50, blank = True, unique = True)
objects =NewManager()
def __str__(self):
return self.name
class QuerySet(QuerySet):
def annotate_sum(self, modelClass, field_name):
annotation_name="%s__%s__%s" % (modelClass._meta.model_name,field_name,'sum')
raw_query = "SELECT SUM({field}) FROM {model2} WHERE {model2}.{model3}_id = {model1}.id".format(
field = field_name,
model3 = self.model._meta.model_name,
model2 = modelClass._meta.db_table,
model1 = self.model._meta.db_table
)
debug.debug("%s" % raw_query)
annotation = {annotation_name: RawSQL(raw_query, [])}
return self.annotate(**annotation)
And views.py:
both = Foo.objects.annotate_sum(Bar1, 'value').annotate_sum( Bar2, 'value')
the sql result is exact what I want:
SELECT "app_foo"."id", "app_foo"."name", (SELECT SUM(value) FROM app_bar1 WHERE app_bar1.foo_id = app_foo.id) AS "bar1__value__sum", (SELECT SUM(value) FROM app_bar2 WHERE app_bar2.foo_id = app_foo.id) AS "bar2__value__sum" FROM "app_foo"
Of course it isn't perfect - it needs some error checking (e.g. double quotes) or aliases, but i think this is the right direction

Answer 3

I landed on this page after having a similar problem, but with Count instead of Sum.
The simplest solution is to use Count(<field>, distinct=True) on the 2nd Count, i.e.
both = Foo.objects.all().annotate(Count("bar1__value")
).annotate(Count("bar2__value", distinct=True))
References:
ticket 10060/comment:60 linked by #emulbreh answer
django 2.0 docs / Aggregation # Combining multiple aggregations