How to do SELECT MAX in Django? - django

I have a list of objects how can I run a query to give the max value of a field:
I'm using this code:
def get_best_argument(self):
try:
arg = self.argument_set.order_by('-rating')[0].details
except IndexError:
return 'no posts'
return arg
rating is an integer

See this. Your code would be something like the following:
from django.db.models import Max
# Generates a "SELECT MAX..." query
Argument.objects.aggregate(Max('rating')) # {'rating__max': 5}
You can also use this on existing querysets:
from django.db.models import Max
args = Argument.objects.filter(name='foo') # or whatever arbitrary queryset
args.aggregate(Max('rating')) # {'rating__max': 5}
If you need the model instance that contains this max value, then the code you posted is probably the best way to do it:
arg = args.order_by('-rating')[0]
Note that this will error if the queryset is empty, i.e. if no arguments match the query (because the [0] part will raise an IndexError). If you want to avoid that behavior and instead simply return None in that case, use .first():
arg = args.order_by('-rating').first() # may return None

Django also has the 'latest(field_name = None)' function that finds the latest (max. value) entry. It not only works with date fields but also with strings and integers.
You can give the field name when calling that function:
max_rated_entry = YourModel.objects.latest('rating')
return max_rated_entry.details
Or you can already give that field name in your models meta data:
from django.db import models
class YourModel(models.Model):
#your class definition
class Meta:
get_latest_by = 'rating'
Now you can call 'latest()' without any parameters:
max_rated_entry = YourModel.objects.latest()
return max_rated_entry.details

I've tested this for my project, it finds the max/min in O(n) time:
from django.db.models import Max
# Find the maximum value of the rating and then get the record with that rating.
# Notice the double underscores in rating__max
max_rating = App.objects.aggregate(Max('rating'))['rating__max']
return App.objects.get(rating=max_rating)
This is guaranteed to get you one of the maximum elements efficiently, rather than sorting the whole table and getting the top (around O(n*logn)).

sol 01:
from .models import MyMODEL
max_rating = MyMODEL.objects.order_by('-rating').first()
sol 02:
from django.db.models import Max
from .models import MyMODEL
max_rating = MyMODEL.objects.aggregate(Max('rating'))

If you also want to get a value other than None in case the table is empty (e.g. 0), combine Max with Coalesce:
from django.db.models import Max, Value
from django.db.models.functions import Coalesce
max_rating = SomeModel.objects.aggregate(
max_rating=Coalesce(Max('rating'), Value(0))
)['max_rating']

To maybe improve on #afahim answer with regards to #Raydel Miranda comment, if you want a random comment. If you want all, then use just the filter
from django.db.models import Max
# Find the maximum value of the rating and then get the record with that rating.
# Notice the double underscores in rating__max
max_rating = App.objects.aggregate(Max('rating'))['rating__max']
return App.objects.filter(rating=max_rating).first()

maybe it will help someone's trouble, in CBV
def get_queryset(self):
sorgu = Sunum.objects.values('id', 'firma', 'projeadi', 'sunumdurum__durum', 'sunumdurum__aciklama'
).annotate(max_rank=Max('sunumdurum__kayittarihi'))
szlk={}
for sor in sorgu :
ana = sor['id'], sor['firma'], sor['projeadi']
dana = sor['sunumdurum__durum'], sor['sunumdurum__aciklama'], sor['max_rank']
szlk.setdefault(ana, dana)
return szlk

Related

In Django how can I get result from two tables in a single queryset without having relationship? [duplicate]

I'm trying to build the search for a Django site I am building, and in that search, I am searching in three different models. And to get pagination on the search result list, I would like to use a generic object_list view to display the results. But to do that, I have to merge three querysets into one.
How can I do that? I've tried this:
result_list = []
page_list = Page.objects.filter(
Q(title__icontains=cleaned_search_term) |
Q(body__icontains=cleaned_search_term))
article_list = Article.objects.filter(
Q(title__icontains=cleaned_search_term) |
Q(body__icontains=cleaned_search_term) |
Q(tags__icontains=cleaned_search_term))
post_list = Post.objects.filter(
Q(title__icontains=cleaned_search_term) |
Q(body__icontains=cleaned_search_term) |
Q(tags__icontains=cleaned_search_term))
for x in page_list:
result_list.append(x)
for x in article_list:
result_list.append(x)
for x in post_list:
result_list.append(x)
return object_list(
request,
queryset=result_list,
template_object_name='result',
paginate_by=10,
extra_context={
'search_term': search_term},
template_name="search/result_list.html")
But this doesn't work. I get an error when I try to use that list in the generic view. The list is missing the clone attribute.
How can I merge the three lists, page_list, article_list and post_list?
Concatenating the querysets into a list is the simplest approach. If the database will be hit for all querysets anyway (e.g. because the result needs to be sorted), this won't add further cost.
from itertools import chain
result_list = list(chain(page_list, article_list, post_list))
Using itertools.chain is faster than looping each list and appending elements one by one, since itertools is implemented in C. It also consumes less memory than converting each queryset into a list before concatenating.
Now it's possible to sort the resulting list e.g. by date (as requested in hasen j's comment to another answer). The sorted() function conveniently accepts a generator and returns a list:
result_list = sorted(
chain(page_list, article_list, post_list),
key=lambda instance: instance.date_created)
If you're using Python 2.4 or later, you can use attrgetter instead of a lambda. I remember reading about it being faster, but I didn't see a noticeable speed difference for a million item list.
from operator import attrgetter
result_list = sorted(
chain(page_list, article_list, post_list),
key=attrgetter('date_created'))
Try this:
matches = pages | articles | posts
It retains all the functions of the querysets which is nice if you want to order_by or similar.
Please note: this doesn't work on querysets from two different models.
Related, for mixing querysets from the same model, or for similar fields from a few models, starting with Django 1.11 a QuerySet.union() method is also available:
union()
union(*other_qs, all=False)
New in Django 1.11. Uses SQL’s UNION operator to combine the results of two or more QuerySets. For example:
>>> qs1.union(qs2, qs3)
The UNION operator selects only distinct values by default. To allow duplicate values, use the all=True
argument.
union(), intersection(), and difference() return model instances of
the type of the first QuerySet even if the arguments are QuerySets of
other models. Passing different models works as long as the SELECT
list is the same in all QuerySets (at least the types, the names don’t
matter as long as the types in the same order).
In addition, only LIMIT, OFFSET, and ORDER BY (i.e. slicing and
order_by()) are allowed on the resulting QuerySet. Further, databases
place restrictions on what operations are allowed in the combined
queries. For example, most databases don’t allow LIMIT or OFFSET in
the combined queries.
You can use the QuerySetChain class below. When using it with Django's paginator, it should only hit the database with COUNT(*) queries for all querysets and SELECT() queries only for those querysets whose records are displayed on the current page.
Note that you need to specify template_name= if using a QuerySetChain with generic views, even if the chained querysets all use the same model.
from itertools import islice, chain
class QuerySetChain(object):
"""
Chains multiple subquerysets (possibly of different models) and behaves as
one queryset. Supports minimal methods needed for use with
django.core.paginator.
"""
def __init__(self, *subquerysets):
self.querysets = subquerysets
def count(self):
"""
Performs a .count() for all subquerysets and returns the number of
records as an integer.
"""
return sum(qs.count() for qs in self.querysets)
def _clone(self):
"Returns a clone of this queryset chain"
return self.__class__(*self.querysets)
def _all(self):
"Iterates records in all subquerysets"
return chain(*self.querysets)
def __getitem__(self, ndx):
"""
Retrieves an item or slice from the chained set of results from all
subquerysets.
"""
if type(ndx) is slice:
return list(islice(self._all(), ndx.start, ndx.stop, ndx.step or 1))
else:
return islice(self._all(), ndx, ndx+1).next()
In your example, the usage would be:
pages = Page.objects.filter(Q(title__icontains=cleaned_search_term) |
Q(body__icontains=cleaned_search_term))
articles = Article.objects.filter(Q(title__icontains=cleaned_search_term) |
Q(body__icontains=cleaned_search_term) |
Q(tags__icontains=cleaned_search_term))
posts = Post.objects.filter(Q(title__icontains=cleaned_search_term) |
Q(body__icontains=cleaned_search_term) |
Q(tags__icontains=cleaned_search_term))
matches = QuerySetChain(pages, articles, posts)
Then use matches with the paginator like you used result_list in your example.
The itertools module was introduced in Python 2.3, so it should be available in all Python versions Django runs on.
In case you want to chain a lot of querysets, try this:
from itertools import chain
result = list(chain(*docs))
where: docs is a list of querysets
The big downside of your current approach is its inefficiency with large search result sets, as you have to pull down the entire result set from the database each time, even though you only intend to display one page of results.
In order to only pull down the objects you actually need from the database, you have to use pagination on a QuerySet, not a list. If you do this, Django actually slices the QuerySet before the query is executed, so the SQL query will use OFFSET and LIMIT to only get the records you will actually display. But you can't do this unless you can cram your search into a single query somehow.
Given that all three of your models have title and body fields, why not use model inheritance? Just have all three models inherit from a common ancestor that has title and body, and perform the search as a single query on the ancestor model.
This can be achieved by two ways either.
1st way to do this
Use union operator for queryset | to take union of two queryset. If both queryset belongs to same model / single model than it is possible to combine querysets by using union operator.
For an instance
pagelist1 = Page.objects.filter(
Q(title__icontains=cleaned_search_term) |
Q(body__icontains=cleaned_search_term))
pagelist2 = Page.objects.filter(
Q(title__icontains=cleaned_search_term) |
Q(body__icontains=cleaned_search_term))
combined_list = pagelist1 | pagelist2 # this would take union of two querysets
2nd way to do this
One other way to achieve combine operation between two queryset is to use itertools chain function.
from itertools import chain
combined_results = list(chain(pagelist1, pagelist2))
You can use Union:
qs = qs1.union(qs2, qs3)
But if you want to apply order_by on the foreign models of the combined queryset... then you need to Select them beforehand this way... otherwise it won't work.
Example
qs = qs1.union(qs2.select_related("foreignModel"), qs3.select_related("foreignModel"))
qs.order_by("foreignModel__prop1")
where prop1 is a property in the foreign model.
DATE_FIELD_MAPPING = {
Model1: 'date',
Model2: 'pubdate',
}
def my_key_func(obj):
return getattr(obj, DATE_FIELD_MAPPING[type(obj)])
And then sorted(chain(Model1.objects.all(), Model2.objects.all()), key=my_key_func)
Quoted from https://groups.google.com/forum/#!topic/django-users/6wUNuJa4jVw. See Alex Gaynor
Requirements:
Django==2.0.2, django-querysetsequence==0.8
In case you want to combine querysets and still come out with a QuerySet, you might want to check out django-queryset-sequence.
But one note about it. It only takes two querysets as it's argument. But with python reduce you can always apply it to multiple querysets.
from functools import reduce
from queryset_sequence import QuerySetSequence
combined_queryset = reduce(QuerySetSequence, list_of_queryset)
And that's it. Below is a situation I ran into and how I employed list comprehension, reduce and django-queryset-sequence
from functools import reduce
from django.shortcuts import render
from queryset_sequence import QuerySetSequence
class People(models.Model):
user = models.OneToOneField(User, on_delete=models.CASCADE)
mentor = models.ForeignKey('self', null=True, on_delete=models.SET_NULL, related_name='my_mentees')
class Book(models.Model):
name = models.CharField(max_length=20)
owner = models.ForeignKey(Student, on_delete=models.CASCADE)
# as a mentor, I want to see all the books owned by all my mentees in one view.
def mentee_books(request):
template = "my_mentee_books.html"
mentor = People.objects.get(user=request.user)
my_mentees = mentor.my_mentees.all() # returns QuerySet of all my mentees
mentee_books = reduce(QuerySetSequence, [each.book_set.all() for each in my_mentees])
return render(request, template, {'mentee_books' : mentee_books})
Here's an idea... just pull down one full page of results from each of the three and then throw out the 20 least useful ones... this eliminates the large querysets and that way you only sacrifice a little performance instead of a lot.
The best option is to use the Django built-in methods:
# Union method
result_list = page_list.union(article_list, post_list)
That will return the union of all the objects in those querysets.
If you want to get just the objects that are in the three querysets, you will love the built-in method of querysets, intersection.
# intersection method
result_list = page_list.intersection(article_list, post_list)
This will do the work without using any other libraries:
result_list = page_list | article_list | post_list
You can use "|"(bitwise or) to combine the querysets of the same model as shown below:
# "store/views.py"
from .models import Food
from django.http import HttpResponse
def test(request):
# ↓ Bitwise or
result = Food.objects.filter(name='Apple') | Food.objects.filter(name='Orange')
print(result)
return HttpResponse("Test")
Output on console:
<QuerySet [<Food: Apple>, <Food: Orange>]>
[22/Jan/2023 12:51:44] "GET /store/test/ HTTP/1.1" 200 9
And, you can use |= to add the queryset of the same model as shown below:
# "store/views.py"
from .models import Food
from django.http import HttpResponse
def test(request):
result = Food.objects.filter(name='Apple')
# ↓↓ Here
result |= Food.objects.filter(name='Orange')
print(result)
return HttpResponse("Test")
Output on console:
<QuerySet [<Food: Apple>, <Food: Orange>]>
[22/Jan/2023 12:51:44] "GET /store/test/ HTTP/1.1" 200 9
Be careful, if adding the queryset of a different model as shown below:
# "store/views.py"
from .models import Food, Drink
from django.http import HttpResponse
def test(request):
# "Food" model # "Drink" model
result = Food.objects.filter(name='Apple') | Drink.objects.filter(name='Milk')
print(result)
return HttpResponse("Test")
There is an error below:
AssertionError: Cannot combine queries on two different base models.
[22/Jan/2023 13:40:54] "GET /store/test/ HTTP/1.1" 500 96025
But, if adding the empty queryset of a different model as shown below:
# "store/views.py"
from .models import Food, Drink
from django.http import HttpResponse
def test(request):
# "Food" model # Empty queryset of "Drink" model
result = Food.objects.filter(name='Apple') | Drink.objects.none()
print(result)
return HttpResponse("Test")
There is no error below:
<QuerySet [<Food: Apple>]>
[22/Jan/2023 13:51:09] "GET /store/test/ HTTP/1.1" 200 9
Again be careful, if adding the object by get() as shown below:
# "store/views.py"
from .models import Food
from django.http import HttpResponse
def test(request):
result = Food.objects.filter(name='Apple')
# ↓↓ Object
result |= Food.objects.get(name='Orange')
print(result)
return HttpResponse("Test")
There is an error below:
AttributeError: 'Food' object has no attribute '_known_related_objects'
[22/Jan/2023 13:55:57] "GET /store/test/ HTTP/1.1" 500 95748
This recursive function concatenates array of querysets into one queryset.
def merge_query(ar):
if len(ar) ==0:
return [ar]
while len(ar)>1:
tmp=ar[0] | ar[1]
ar[0]=tmp
ar.pop(1)
return ar

How to query in django model having fields as foreign keys?

Models.py :.
class Match(models.Model):
rep_broker = models.ForeignKey('account.User', related_name='rep_broker',on_delete=models.CASCADE,blank=True,null=True)
boat = models.ForeignKey('boat.Boat', related_name='matches', on_delete=models.CASCADE)
How to query the Match model to get the output like "rep_broker have 3 boats matching"
table example:
broker1 | boat1
broker1 | boat2
broker2 | boat3
output : broker1 have 2 boats matching
broker2 have 1 boat matching
Output should be list of dictionary
[{"name":"broker1","no_of_boats":"2"},{"name":"broker2","no_of_boats":"1"}}
You can load the Match objects and the corresponding User (broker) and Boats with a query and the post-process that query with .groupby(…) [Python-doc]:
from itertools import groupby
from operator import attrgetter
qs = Match.objects.select_related('rep_broker', 'boat').order_by('rep_broker')
result = {
k: [v.boat for v in vs]
for k, vs in groupby(qs, attrgetter('rep_broker'))
}
This will produce a dictionary that maps the User objects to lists of Boat objects, given the User of course matches with at least one Boat.
In case you want the number of Boats, you can create a query to annotate the User objects with:
from django.db.models import Count, Value
from django.db.models.functions import Coalesce
result = list(User.objects.values(
'name'
).annotate(
no_of_boats=Colaesce(Count('rep_broker'), Value(0))
))
This will construct a list of dictionaries. This is however a primitive obsession antipattern [refactoring.guru]. You might want to annotate the User objects itself:
from django.db.models import Count, Value
from django.db.models.functions import Coalesce
qs = User.objects.annotate(
no_of_boats=Coalesce(Count('rep_broker'), Value(0))
))
The User objects that arise from this queryset will simply have an extra attribute .no_of_boats, but thus the logical layer the User model offers still persists.
You can filter out Users with no match to a boat in this case with:
from django.db.models import Count
qs = User.objects.filter(
rep_broker__isnull=False
).annotate(
no_of_boats=Count('rep_broker'))
))

Django: Combine a date and time field and filter

I have a django model that has a date field and a separate time field. I am trying to use a filter to find a value on the latest record by date/time that is less than the current record's date time.
How do I use annotate/aggregate to combine the date and time fields into one and then do a filter on it?
models.py
class Note(models.model):
note_date = models.DateField(null=True)
note_time = models.TimeField(null=True)
note_value = models.PositiveIntegerField(null=True)
def get_last(n):
"""
n: Note
return: Return the note_value of the most recent Note prior to given Note.
"""
latest = Note.objects.filter(
note_date__lte=n.note_date
).order_by(
'-note_date', '-note_time'
).first()
return latest.note_value if latest else return 0
This will return any notes from a previous date, but if I have a two notes on the same date, one at 3pm and one at 1pm, and I send the 3pm note to the function, I want to get the value of the 1pm note. Is there a way to annotate the two fields into one for comparison, or do I have to perform a raw SQL query? Is there a way to convert the date and time component into one, similar to how you could use Concat for strings?
Note.objects.annotate(
my_dt=Concat('note_date', 'note_time')
).filter(
my_dt__lt=Concat(models.F('note_date'), models.F('note_time')
).first()
I am too late but here is what I did
from django.db.models import DateTimeField, ExpressionWrapper, F
notes = Note.objects.annotate(my_dt=ExpressionWrapper(F('note_date') + F('note_time'), output_field=DateTimeField()))
Now we have added a new field my_dt of datetime type and can add a filter further to do operations
Found an answer using models.Q here: filter combined date and time in django
Note.objects.filter(
models.Q(note_date__lt=n.note_date) | models.Q(
note_date=n.note_date,
note_time__lt=n.note_time
)
).first()
I guess I just wasn't searching by the right criteria.
Here is another Approach which is more authentic
from django.db.models import Value, DateTimeField
from django.db.models.functions import Cast, Concat
notes = Note.objects.annotate(my_dt=Cast(
Concat('note_date', Value(" "), 'note_time', output_field=DateTimeField()),
output_field=DateTimeField()
).filter(my_dt__lte=datetime.now())
Here is another solution following others.
def get_queryset(self):
from django.db import models
datetime_wrapper = models.ExpressionWrapper(models.F('note_date') + models.F('note_time'), output_field=models.DateTimeField())
return Note.objects.annotate(
note_datetime=datetime_wrapper
).filter(note_datetime__gt=timezone.now()).order_by('note_datetime')

How to annotate a difference of datetime in days

I have a Booking model that has start and end datetime fields. I want to know how many days a booking covers. I can do this in Python but I need this value for further annotations.
Here's what I've tried:
In [1]: Booking.objects.annotate(days=F('end')-F('start'))[0].days
Out[1]: datetime.timedelta(16, 50400)
There are a few problems here:
I want an integer (or other number type I can use in calculations) of days as the output, not a timedelta. Setting output_field doesn't do anything meaningful here.
My sums are based on datetimes. Subtractions like this, without removing the time could lead to the whole number of days being off.
In Python I would do (end.date() - start.date()).days + 1. How can I do that in-database, preferably through the ORM (eg database functions), but a RawSQL would suffice to get this out the door?
I've written a couple of database functions to cast and truncate the dates to solve both problems under PostgreSQL. The DATE_PART and DATE_TRUNC internal function I'm using are DB-specific ☹
from django.db.models import Func
class DiffDays(Func):
function = 'DATE_PART'
template = "%(function)s('day', %(expressions)s)"
class CastDate(Func):
function = 'date_trunc'
template = "%(function)s('day', %(expressions)s)"
Then I can:
In [25]: Booking.objects.annotate(days=DiffDays(CastDate(F('end'))-CastDate(F('start'))) + 1)[0].days
Out[25]: 18.0
There is another, easy solution of this problem. You can use:
from django.db.models import F
from django.db.models.functions import ExtractDay
and then:
Booking.objects.annotate(days=(ExtractDay(F('end')-F('start'))+1))[0].days
If you are using MYSQL database, You could do it using Custom DB Function as,
from django.db.models.functions import Func
class TimeStampDiff(Func):
class PrettyStringFormatting(dict):
def __missing__(self, key):
return '%(' + key + ')s'
def __init__(self, *expressions, **extra):
unit = extra.pop('unit', 'day')
self.template = self.template % self.PrettyStringFormatting({"unit": unit})
super().__init__(*expressions, **extra)
function = 'TIMESTAMPDIFF'
template = "%(function)s(%(unit)s, %(expressions)s)"
Usage
from django.db.models import F, IntegerField
booking_queryset = Booking.objects.annotate(
days=TimeStampDiff(F('start'), F('end'), output_field=IntegerField()))
if booking_queryset.exist():
print(booking_queryset[0].__dict__)

How do I convert a django QuerySet to numpy record array?

How do I convert a django QuerySet to numpy record array?
PS: I know you can iterate and construct it and but is there any other cleaner solution?
import numpy as np
qs = MyModel.objects.all()
vlqs = qs.values_list()
r = np.core.records.fromrecords(vlqs, names=[f.name for f in MyModel._meta.fields])
This uses the QuerySet iterator directly and avoids the time-and-garbage-wasting step of creating a python list. It also uses MyModel._meta.fields to get the actual field names from the model, as explained at Get model's fields in Django
If you just want a single field (e.g. the 'votes' field of the model) extracted into a one-dimensional array, you can do:
vlqs = qs.values_list('votes', flat=True)
votes = np.fromiter(vlqs, numpy.dtype('int_'))
This is like asking "how do I convert the contents of my fridge into dinner?". It depends on what you have in your fridge and what you'd like to eat. The short answer (equivalent to saying "by cooking") is to iterate over the queryset, constructing objects of whatever composite data types you'd like to instantiate the array with (generally an iterable and a dictionary). The long answer depends on what you'd actually like to accomplish.
If you want to get all of your objects and create a numpy array with objects as elements of array:
import numpy as np
qs = MyModel.objects.all()
numpy_array = np.array(list(qs))
According to my work, I use something as below:
import numpy as np
qs = MyModel.objects.values_list('id','first_name','last_name').filter(gender='male').order_by('id')
numpy_array = np.array(list(qs))
Rows of array corresponds to records and columns of array corresponds to values that I defined above (id, first name, last name).
What I was looking for:
From QuerySet qs get vlqs (django.db.models.query.ValuesListQuerySet)
vlqs = qs.values_list()
Covert vlqs to list
mylist = list(vlqs)
Create numpy record array
# Names are the model fields
r = np.core.records.array(mylist, names='field1, field2, field3')
And to put it into a neat little function to which you just pass any Django Queryset:
import pandas as pd
import numpy as np
def qs_to_df(qs):
""" QuerySet to DataFrame """
Model = qs.model
np_array = np.core.records.fromrecords(qs.values_list(), names=[f.name for f in Model._meta.fields])
return pd.DataFrame(np_array)
what you could do is:
[index[0] for index in qs.values_list('votes')]
and ready...XD
Going off of #CpILL 's answer you can turn most querysets into a numpy record array like so:
def qs_to_ra(qs, *args):
"""
Turn most querysets directly into a numpy record array
:param qs: django queryset
:param args: takes a list of field names to specify
:return: numpy.recarray
"""
model = qs.model
if args:
return np.core.records.fromrecords(qs.values_list(*args), names=args)
return np.core.records.fromrecords(qs.values_list(), names=[f.name for f in model._meta.fields])
You can also turn them directly into a pandas dataframe like so:
def qs_to_df(qs, *args):
"""
Turn most querysets directly into a pandas dataframe.
:param qs: django queryset
:param args: takes a list of field names to specify
:return: pandas.DataFrame
"""
model = qs.model
if args:
return pd.DataFrame.from_records(list(qs.values_list(*args)), columns=args)
return pd.DataFrame.from_records(list(qs.values_list()), columns=[f.name for f in model._meta.fields])