Aggregate grouped annotation - django

Question

I'd like to sum all the event durations per day. This is my model:
class Event(models.Model):
start = models.DateTimeField()
end = models.DateTimeField()
Sample data:
import datetime
from random import randint
for i in range(0, 1000):
start = datetime.datetime(
year=2016,
month=1,
day=randint(1, 10),
hour=randint(0, 23),
minute=randint(0, 59),
second=randint(0, 59)
)
end = start + datetime.timedelta(seconds=randint(30, 1000))
Event.objects.create(start=start, end=end)
I can get the event count per day like so:
(I know extra is bad, but I'm using 1.9 at the moment. When I upgrade I'll move to using TruncDate)
Event.objects.extra({'date': 'date(start)'}).order_by('date').values('date').annotate(count=Count('id'))
[{'count': 131, 'date': datetime.date(2016, 1, 1)},
{'count': 95, 'date': datetime.date(2016, 1, 2)},
{'count': 99, 'date': datetime.date(2016, 1, 3)},
{'count': 85, 'date': datetime.date(2016, 1, 4)},
{'count': 87, 'date': datetime.date(2016, 1, 5)},
{'count': 94, 'date': datetime.date(2016, 1, 6)},
{'count': 97, 'date': datetime.date(2016, 1, 7)},
{'count': 111, 'date': datetime.date(2016, 1, 8)},
{'count': 97, 'date': datetime.date(2016, 1, 9)},
{'count': 104, 'date': datetime.date(2016, 1, 10)}]
I can annotate to add the duration:
In [3]: Event.objects.annotate(duration=F('end') - F('start')).first().duration
Out[3]: datetime.timedelta(0, 470)
But I can't figure out how to sum this annotation the same way I can count events. I've tried the following but I get a KeyError on 'duration'.
Event.objects.annotate(duration=F('end') - F('start')).extra({'date': 'date(start)'}).order_by('date').values('date').annotate(total_duration=Sum('duration'))
And If I add duration to the values clause then it no longer groups by date.
Is this possible in a single query and without adding a duration field to the model?

Answer 1

I was about to write an answer that Django ORM does not support this. And yes, then I spent another hour on this problem (in addition to the 1,5 hours already spent before starting to write this answer), but as it turns out, Django does support it. And without hacking. Good news!
import datetime as dt
from django.db import models
from django.db.models import F, Sum, When, Case
from django.db.models.functions import TruncDate
from app.models import Event
a = Event.objects.annotate(date=TruncDate('start')).values('date').annotate(
day_duration=Sum(Case(
When(date=TruncDate(F('start')), then=F('end') - F('start')),
default=dt.timedelta(), output_field=models.DurationField()
))
)
And some preliminary tests to (hopefully) prove that this stuff actually does what you asked.
In [71]: a = Event.objects.annotate(date=TruncDate('start')).values('date').annotate(day_duration=Sum(Case(
...: When(date=TruncDate(F('start')), then=F('end') - F('start')),
...: default=dt.timedelta(), output_field=models.DurationField()
...: ))
...: )
In [72]: for e in a:
...: print(e)
...:
{'day_duration': datetime.timedelta(0, 41681), 'date': datetime.date(2016, 1, 10)}
{'day_duration': datetime.timedelta(0, 46881), 'date': datetime.date(2016, 1, 3)}
{'day_duration': datetime.timedelta(0, 48650), 'date': datetime.date(2016, 1, 1)}
{'day_duration': datetime.timedelta(0, 52689), 'date': datetime.date(2016, 1, 8)}
{'day_duration': datetime.timedelta(0, 45788), 'date': datetime.date(2016, 1, 5)}
{'day_duration': datetime.timedelta(0, 49418), 'date': datetime.date(2016, 1, 7)}
{'day_duration': datetime.timedelta(0, 45984), 'date': datetime.date(2016, 1, 9)}
{'day_duration': datetime.timedelta(0, 51841), 'date': datetime.date(2016, 1, 2)}
{'day_duration': datetime.timedelta(0, 63770), 'date': datetime.date(2016, 1, 4)}
{'day_duration': datetime.timedelta(0, 57205), 'date': datetime.date(2016, 1, 6)}
In [73]: q = dt.timedelta()
In [74]: o = Event.objects.filter(start__date=dt.date(2016, 1, 7))
In [75]: p = Event.objects.filter(start__date=dt.date(2016, 1, 10))
In [76]: for e in o:
...: q += (e.end - e.start)
In [77]: q
Out[77]: datetime.timedelta(0, 49418) # Matches 2016.1.7, yay!
In [78]: q = dt.timedelta()
In [79]: for e in p:
...: q += (e.end - e.start)
In [80]: q
Out[80]: datetime.timedelta(0, 41681) # Matches 2016.1.10, yay!
NB! This works from version 1.9, I don't think you can do this with the earlier versions because the TruncDate function is missing. And before 1.8 you don't have the Case and When thingies as well of course.