I have a Ticket booking model
class Movie(models.Model):
name = models.CharField(max_length=254, unique=True)
class Show(models.Model):
day = models.ForeignKey(Day)
time = models.TimeField(choices=CHOICE_TIME)
movie = models.ForeignKey(Movie)
class MovieTicket(models.Model):
show = models.ForeignKey(Show)
user = models.ForeignKey(User)
booked_at = models.DateTimeField(default=timezone.now)
I would like to filter MovieTicket with its user field and group them according to its show field, and order them by the recent booked time. And respond back with json data using Django REST framework like this:
[
{
show: 4,
movie: "Lion king",
time: "07:00 pm",
day: "23 Apr 2017",
total_tickets = 2
},
{
show: 7,
movie: "Gone girl",
time: "02:30 pm",
day: "23 Apr 2017",
total_tickets = 1
}
]
I tried this way:
>>> MovieTicket.objects.filter(user=23).order_by('-booked_at').values('show').annotate(total_tickets=Count('show'))
<QuerySet [{'total_tickets': 1, 'show': 4}, {'total_tickets': 1, 'show': 4}, {'total_tickets': 1, 'show': 7}]>
But its not grouping according to the show. Also how can I add other related fields (i.e., show__movie__name, show__day__date, show__time)
I explain it more generally on the graph of the database model. It can be applied to any "GROUP BY" with an extra contents.
+-------------------------+
| MovieTicket (booked_at) |
+-----+--------------+----+
| |
+---------+--------+ +--+---+
| Show (time) | | User |
++----------------++ +------+
| |
+------+-------+ +-----+------+
| Movie (name) | | Day (date) |
+--------------+ +------------+
The question is: How to summarize MovieTicket (the topmost object) grouped by Show (one related object) filtered by User (other related object) with reporting details from some related deeper objects (Movie and Day) and sorting these results by some field aggregated from the topmost model by the group (by the booked time of the recent MovieTicket in the group):
Answer explained by more general steps:
Start with the topmost model:
(MovieTicket.objects ...)
Apply filters:
.filter(user=user)
It is important to group by pk of the nearest related models (at least models those which are not made constant by the filter) - It is only "Show" (because "User" object is still filtered to one user)
.values('show_id')
Even if all other fields would be unique together (show__movie__name, show__day__date, show__time) it is better for the database engine optimizer to group the query by show_id because all these other fields depend on show_id and can not impact the number of groups.
Annotate necessary aggregation functions:
.annotate(total_tickets=Count('show'), last_booking=Max('booked_at'))
Add required dependent fields:
.values('show_id', 'show__movie__name', 'show__day__date', 'show__time')
Sort what is necessary:
.order_by('-last_booking') (descending from the latest to the oldest)
It is very important to not output or sort any field of the topmost model without encapsulating it by aggregation function. (Min and Max functions are good for sampling something from a group. Every field not encapsulated by aggregation would be added to "group by" list and that will break intended groups. More tickets to the same show for friend could be booked gradually but should be counted together and reported by the latest booking.)
Put it together:
from django.db.models import Max
qs = (MovieTicket.objects
.filter(user=user)
.values('show_id', 'show__movie__name', 'show__day__date', 'show__time')
.annotate(total_tickets=Count('show'), last_booking=Max('booked_at'))
.order_by('-last_booking')
)
The queryset can be easily converted to JSON how demonstrated zaphod100.10 in his answer, or directly for people not interested in django-rest framework this way:
from collections import OrderedDict
import json
print(json.dumps([
OrderedDict(
('show', x['show_id']),
('movie', x['show__movie__name']),
('time', x['show__time']), # add time formatting
('day': x['show__day__date']), # add date formatting
('total_tickets', x['total_tickets']),
# field 'last_booking' is unused
) for x in qs
]))
Verify the query:
>>> print(str(qs.query))
SELECT app_movieticket.show_id, app_movie.name, app_day.date, app_show.time,
COUNT(app_movieticket.show_id) AS total_tickets,
MAX(app_movieticket.booked_at) AS last_booking
FROM app_movieticket
INNER JOIN app_show ON (app_movieticket.show_id = app_show.id)
INNER JOIN app_movie ON (app_show.movie_id = app_movie.id)
INNER JOIN app_day ON (app_show.day_id = app_day.id)
WHERE app_movieticket.user_id = 23
GROUP BY app_movieticket.show_id, app_movie.name, app_day.date, app_show.time
ORDER BY last_booking DESC
Notes:
The graph of models is similar to ManyToMany relationship, but MovieTickets are individual objects and probably hold seat numbers.
It would be easy to get a similar report for more users by one query. The field 'user_id' and the name would be added to "values(...)".
The related model Day is not intuitive, but it is clear that is has a field date and hopefully also some non trivial fields, maybe important for scheduling shows with respect to events like cinema holidays. It would be useful to set the field 'date' as the primary key of Day model and spare a relationship lookup frequently in many queries like this.
(All important parts of this answer could be found in the oldest two answers: Todor and zaphod100.10. Unfortunately these answers have not been combined together and then not up-voted by anyone except me, even that the question has many up-votes.)
I would like to filter MovieTicket with its user field and group them
according to its show field, and order them by the recent booked time.
This queryset will give you exactly what you want:
tickets = (MovieTicket.objects
.filter(user=request.user)
.values('show')
.annotate(last_booking=Max('booked_at'))
.order_by('-last_booking')
)
And respond back with json data using Django rest framework like this:
[
{
show: 4,
movie: "Lion king",
time: "07:00 pm",
day: "23 Apr 2017",
total_tickets = 2
},
{
show: 7,
movie: "Gone girl",
time: "02:30 pm",
day: "23 Apr 2017",
total_tickets = 1
}
]
Well this json data is not the same as the query you described. You can add total_tickets by extending the annotation and show__movie__name into the .values clause: this will change the grouping to show+movie_name, but since show only has one movie_name it wont matter.
However, you cannot add show__day__date and show__time, because one show have multiple date-times, so which one would you want from a group? You could for example fetch the maximum day and time but this does not guarantee you that at this day+time there will be a show, because these are different fields, not related by each other. So the final attempt may look like:
tickets = (MovieTicket.objects
.filter(user=request.user)
.values('show', 'show__movie__name')
.annotate(
last_booking=Max('booked_at'),
total_tickets=Count('pk'),
last_day=Max('show__day'),
last_time=Max('show__time'),
)
.order_by('-last_booking')
)
You have to group by show and then count the total number of movie tickets.
MovieTicket.objects.filter(user=23).values('show').annotate(total_tickets=Count('show')).values('show', 'total_tickets', 'show__movie__name', 'show__time', 'show__day__date'))
Use this serilizer class for the above queryset. It will give the required json output.
class MySerializer(serializers.Serializer):
show = serailizer.IntegerField()
movie = serializer.StringField(source='show__movie__name')
time = serializer.TimeField(source='show__time')
day = serializer.DateField(source='show__day__date')
total_tickets = serializer.IntegerField()
It is not possible to order_by booked_at since that information gets lost when we group by show. If we order by booked_at group by will happen on unique booked_at times and show ids and that is why the ticket count was coming 1. Without order_by you will get correct count.
EDIT:
use this query:
queryset = (MovieTicket.objects.filter(user=23)
.order_by('booked_at').values('show')
.annotate(total_tickets=Count('show'))
.values('show', 'total_tickets', 'show__movie__name',
'show__time', 'show__day__date')))
You cannot annotate on an annotated field. So you will to find the total tickets count in python. To calculate total_tickets count for unique show ids:
tickets = {}
for obj in queryset:
if obj['show'] not in tickets.keys():
tickets[obj['show']] = obj
else:
tickets[obj['show']]['total_tickets'] += obj['total_tickets']
the final list of objects you need is tickets.values()
The same serializer above can be used with these objects.
You can try this.
Show.objects.filter(movieticket_sets__user=23).values('id').annotate(total_tickets=Count('movieticket_set__user')).values('movie__name', 'time', 'day').distinct()
OR
Show.objects.filter(movieticket_sets__user=23).values('id').annotate(total_tickets=Count('id')).values('movie__name', 'time', 'day').distinct()
I have this model for employee overtime hours
class Overtime(IncomeBase):
day = models.DateField(verbose_name="Date")
value = models.FloatField(default=1)
I need to extract the total value for each month. Now I am using a daily QuerySet in the manager.
class OvertimeManager(models.Manager):
def daily_report(self):
return self.values('day').annotate(hours=models.Sum('value')).order_by('-day')
However now I need a monthly report that will get the Sum of value for each month.
I tried to extract the month first but then I lose the values.
Note: the month should not have the total sum for all years, so specifically I need to group by month,year
If you are using Postgresql you can do this. Ofc there is similar fuctions.
Overtime.objects.extra({'month': "to_char(day, 'Mon')", "year": "extract(year from day)"}).values('month', 'year').annotate(Sum('value'))
More info:
http://www.postgresql.org/docs/7.4/static/functions-formatting.html
http://www.postgresql.org/docs/9.1/static/functions-datetime.html
Or django way:
from django.db import connection
truncate_month = connection.ops.date_trunc_sql('month','day')
Overtime.objects.extra({'month': truncate_month}).values('month').annotate(Sum('value'))
I think this will help you.
Hi I am writing a Django view which ouputs data for graphing on the client side (High Charts). The data is climate data with a given parameter recorded once per day.
My query is this:
format = '%Y-%m-%d'
sd = datetime.datetime.strptime(startdate, format)
ed = datetime.datetime.strptime(enddate, format)
data = Climate.objects.filter(recorded_on__range = (sd, ed)).order_by('recorded_on')
Now, as the range is increased the dataset obviously gets larger and this does not present well on the graph (aside from slowing things down considerably).
Is there an way to group my data as averages in time periods - specifically average for each month or average for each year?
I realize this could be done in SQL as mentioned here: django aggregation to lower resolution using grouping by a date range
But I would like to know if there is a handy way in Django itself.
Or is it perhaps better to modify the db directly and use a script to populate month and year fields from the timestamp?
Any help much appreciated.
Have you tried using django-qsstats-magic (https://github.com/kmike/django-qsstats-magic)?
It makes things very easy for charting, here is a timeseries example from their docs:
from django.contrib.auth.models import User
import datetime, qsstats
qs = User.objects.all()
qss = qsstats.QuerySetStats(qs, 'date_joined')
today = datetime.date.today()
seven_days_ago = today - datetime.timedelta(days=7)
time_series = qss.time_series(seven_days_ago, today)
print 'New users in the last 7 days: %s' % [t[1] for t in time_series]
I have such model and query
class Employer(Models.model)
name = ...
class JobTitle(Models.model)
name = ...
employer = models.ForeignKey(Employer)
and query is
Employer.objects.select_related('jobtitle')
.filter(jtt__activatedate__range=[startdate,enddate])
.annotate(jtt_count=Count('jobtitle'))
.order_by('-jtt_count')[:5]
As you see it returns 5 employer list which has maximum number of jobtitles which are related to that employer and whose activation date is in some certain range.
However, I also want to get the total number of jobtitles of each employer in that query.
Of course I may loop over each employer and make such query JobTitle.objects.filter(employer = emp) and taking length of that query but it is bad solution.
How can I achive this in that query?
Although it may not be possible to get both total number and filtered number of job titles, I may get the jobttiles of each emplyoer such that len(emp.jobtitle) however it also didn't work.
Thanks
Try the extra lookup. So, in your case it may be like this:
.extra(
select={
'jobtitle_count': 'SELECT COUNT(*) FROM YOURAPP_jobtitle WHERE YOURAPP_jobtitle.employer_id = YOURAPP_employer.id'
},
)
I have the following models, Art and ArtScore:
class Art(models.Model):
title = models.CharField()
class ArtScore(models.Model):
art = models.ForeignKey(Art)
date = models.DateField(auto_now_add = True)
amount = models.IntegerField()
Certain user actions results in an ArtScore entry, for instance whenever you click 'I like this art', I save a certain amount of ArtScore for that Art.
Now I'm trying to show a page for 'most popular this week', so I need a query aggregating only ArtScore amounts for that time range.
I built the below query but it's flawed...
popular = Art.objects.filter(
artscore__date__range=(weekago, today)
).annotate(
score=Sum('artscore__amount')
).order_by('-score')
... because it only excludes Art that doesn't have an ArtScore record in the date range, but does not exclude the ArtScore records outside the date range.
Any pointers how to accomplish this would be appreciated!
Thanks,
Martin
it looks like according to this:
http://docs.djangoproject.com/en/dev/topics/db/aggregation/#order-of-annotate-and-filter-clauses what you have should do what you want. is the documentation wrong? bug maybe?
"the second query will only include
good books in the annotated count."
in regards to:
>>> Publisher.objects.filter(book__rating__gt=3.0).annotate(num_books=Count('book'))
change this to your query (forget about the order for now):
Art.objects.filter(
artscore__date__range=(weekago, today)
).annotate(
score=Sum('artscore__amount')
)
now we can say
"the query will only include artscores
in the date range in the annotated
sum."