Django : how to optimize this query on a large table - django

Question

To give the context, I have a lot of temperature measurements taken at different stations and I want to check if it is in accordance with what was forecast.
My model is :
class Station(models.Model):
station_id = models.CharField(max_length = 18 ,primary_key = True)
sector = models.CharField(max_length = 40)
class Weather(models.Model):
station = models.ForeignKey(Station)
temperature = models.FloatField()
date = models.DateField()
class Forecast(models.Model):
station = models.ForeignKey(Station)
date = models.DateField()
score = models.IntegerField()
For each temperature measurement, I would like to know the average of the forecasting scores for the station over the last 7 days, unless there is another temperature measurement in this time frame, in which case it is the starting point. The following code does what I want but is much too slow to execute (~10minutes !) :
observations = Weather.objects.all().order_by('station','date')
for obs in observations:
try :
if obs.station == previous.station:
date_inf = min(obs.date- timedelta(days=7), previous.date)
else :
date_inf = obs.date- timedelta(days=7)
except UnboundLocalError :
date_inf = obs.date- timedelta(days=7)
forecast = Forecast.objects.filter(
station=obs.station
).filter(
date__gte = date_inf
).filter(
date__lte = obs.date - timedelta(days=1)
).aggregate(average_score=Avg('score'))
if forecast["average_score"] is not None:
print(forecast["average_score"],obs.rating)
# Some more code....
previous = obs
How can I optimize the execution time ? Is there a way to do it with a single query ?
Thanks !

Answer 1

For every measurement, you re-compute the average of last 7 days. If your measurements are closer together than 7 days, you will have overlap. E.g. if your measurements are 1 day apart then you re-calculate average on each object 6 times in the database which is SLOW.
Your best bet is to grab all measurements, then all forecasts that match, then do the averaging in memory in Python. Sure, more python code, but it will run faster.