Here is the raw query set Django ORM:
ob = Shop.objects.raw('SELECT * from shops GROUP BY
(duplicate_field_name) having COUNT(*) = 1 ORDER BY some_field')
listorder = ["check_in","check_out","location"]
This listorder part is dynamic. I don't know how it ll be. It ll change the ordering from time to time & one more thing can't apply ordering on raw query sets because I want the whole data for other purpose.After that only i can apply ordering.
Here want ordering by the list "listorder".
mObj = ob.order_by[*listorder].
In above facing error like can't apply ordering to raw query sets.
Anyone having any idea?
If you want a raw queryset to be ordered by different fields, you can add them to the ORDER BY clause.
ob = Shop.objects.raw('SELECT * from shops GROUP BY
(duplicate_field_name) having COUNT(*) = 1 ORDER BY check_in, check_out, location')
if you want the order to be reversed for a particular field you can change it as
ob = Shop.objects.raw('SELECT * from shops GROUP BY
(duplicate_field_name) having COUNT(*) = 1 ORDER BY check_in, check_out DESC, location')
If the ordering is going to be dynamic, you can create the querystring dynamically.
qs = ''SELECT * from shops GROUP BY
(duplicate_field_name) having COUNT(*) = 1'
# some other code here to decide what your ordering is example
order_fields = ['id','location','check_in','check_out']
qs = qs + "ORDER BY " + ",".join(order_fields)
Then you can query as before
Shop.objects.raw(qs)
Related
I have an Article model like this
from django.contrib.contenttypes.fields import GenericRelation
from django.db import models
from hitcount.models import HitCountMixin, HitCount
class Article(models.Model):
title = models.CharField(max_length=250)
hit_count_generic = GenericRelation(
HitCount, object_id_field='object_pk',
related_query_name='hit_count_generic_relation')
when I do Article.objects.order_by('hit_count_generic__hits'), I am getting results.but when I do
articles_by_id = Article.objects.filter(id__in=ids).annotate(qs_order=models.Value(0, models.IntegerField()))
articles_by_name = Article.objects.filter(title__icontains='sports').annotate(qs_order=models.Value(1, models.IntegerField()))
articles = articles_by_id.union(articles_by_name).order_by('qs_order', 'hit_count_generic__hits')
getting error
ORDER BY term does not match any column in the result set
How can i achieve union like this? I had to use union instead of AND and OR because i need to preserve order. ie; articles_by_id should come first and articles_by_name should come second.
using Django hitcount for hitcount https://github.com/thornomad/django-hitcount. Hitcount model is given below.
class HitCount(models.Model):
"""
Model that stores the hit totals for any content object.
"""
hits = models.PositiveIntegerField(default=0)
modified = models.DateTimeField(auto_now=True)
content_type = models.ForeignKey(
ContentType, related_name="content_type_set_for_%(class)s", on_delete=models.CASCADE)
object_pk = models.TextField('object ID')
content_object = GenericForeignKey('content_type', 'object_pk')
objects = HitCountManager()
As suggested by #Angela tried prefetch related.
articles_by_id = Article.objects.prefetch_related('hit_count_generic').filter(id__in=[1, 2, 3]).annotate(qs_order=models.Value(0, models.IntegerField()))
articles_by_name = Article.objects.prefetch_related('hit_count_generic').filter(title__icontains='date').annotate(qs_order=models.Value(1, models.IntegerField()))
the query of the prefetch_related when checked is not selecting the hitcount at all see.
SELECT "articles_article"."id", "articles_article"."created", "articles_article"."last_changed_date", "articles_article"."title", "articles_article"."title_en", "articles_article"."slug", "articles_article"."status", "articles_article"."number_of_comments", "articles_article"."number_of_likes", "articles_article"."publish_date", "articles_article"."short_description", "articles_article"."description", "articles_article"."cover_image", "articles_article"."page_title", "articles_article"."category_id", "articles_article"."author_id", "articles_article"."creator_id", "articles_article"."article_type", 0 AS "qs_order" FROM "articles_article" WHERE "articles_article"."id" IN (1, 2, 3)
From Django's official documentation:
Further, databases place restrictions on what operations are allowed in the combined queries. For example, most databases don’t allow LIMIT or OFFSET in the combined queries.
So, make sure that your database allows combining queries like this.
ORDER BY term does not match any column in the result set
You are getting this error, because that's exactly what's happening. Your final result-set for articles does not contain the hits column from the hitcount table , due to which the result-set cannot order using this column.
Before delving into the answer, let's look at what's happening with your django querysets under the hood.
Retrieve a particular set of articles and include an extra ordering field qs_order set to 0.
articles_by_id = Article.objects.filter(id__in=ids).annotate(qs_order=models.Value(0, models.IntegerField()))
SQL Query for the above
Select id, title,....., 0 as qs_order from article where article.id in (Select ....) # whatever you did to get your ids or just a flat list
Retrieve another set of articles and include an extra ordering field qs_order set to 1
articles_by_name = Article.objects.filter(title__icontains='sports').annotate(qs_order=models.Value(1, models.IntegerField()))
SQL Query for the above
Select id, title, ...1 as qs_order from article where title ilike '%sports%'
Original queryset and order_by hit_count_generic__hits
Article.objects.order_by('hit_count_generic__hits')
This will actually perform an inner join and fetch the hitcount table to order by the hits column.
Query
Select id, title,... from article inner join hitcount on ... order by hits ASC
Union
So when you do your union, the result-set of the above 2 queries is combined and then ordered using your qs_order and then hits ...where it fails.
Solution
Use prefetch_related to get your hitcount table in the initial queryset filtering, so you can then use the hits column in the union to order.
articles_by_id = Article.objects.prefetch_related('hit_count_generic').filter(id__in=ids).annotate(qs_order=models.Value(0, models.IntegerField()))
articles_by_name = Article.objects.prefetch_related('hit_count_generic').filter(title__icontains='sports').annotate(qs_order=models.Value(1, models.IntegerField()))
Now as you have the desired table and its columns in both your SELECT queries, your union should work the way you have defined.
articles = articles_by_id.union(articles_by_name).order_by('qs_order', 'hit_count_generic__hits')
Just replacing prefetch_related with select_related works for me.
https://docs.djangoproject.com/en/3.2/ref/models/querysets/#select-related
I would want to do a group by with a pair of values in django.
Consider I have the following model,
class Vehicle:
vehicle_id = models.PositiveIntegerField()
version_id = models.PositiveIntegerField(default=1)
name=models.CharField(max_length=8096)
description=models.CharField(max_length=8096)
vehicle_id is not the primary key since there can be multiple rows in the table with the same vehicle_id but with different version_id
Now I would want to get the latest versions of all the vehicles.
select * from (
select vehicle_id, MAX(version_id) as MaxVersion from Vehicle group by vehicle_id
) s1
JOIN Vehicle s2
on s2.vehicle_id = s1.vehicle_id AND s2.version_id=s1.MaxVersion;
Tried to represent this in Django ORM like below,
feed_max_version = (
self
.values("vehicle_id")
.annotate(max_version=Max("version_id"))
.order_by("vehicle_id")
)
q_statement = Q()
for pair in feed_max_version:
q_statement |= Q(vehicle_id__exact=pair["vehicle_id"]) & Q(
version_id=pair["max_version"]
)
return self.filter(q_statement)
But this seems less efficient and takes long time to load. I am not very keen on having raw SQL since I would not be able to add any more queryset methods on top of it.
I'm trying to add an annotation to a QuerySet that is True/False when the value of a field on one related object is less than the value of a field on a different related object.
Here are some models for an example:
class RobotManager(models.Manager):
queryset = super(RobotManager, self).get_queryset()
queryset = queryset.annotate(canteen_empty=UNKNOWN CODE)
return queryset
class Robot(models.Model):
# Has some other unrelated stuff
objects = RobotManager()
class CanteenLevel(models.Model):
time = models.DateTimeField()
robot = models.ForeignKey("SomeApp.Robot")
gallons = models.IntegerField()
class RobotConfiguration(models.Model):
time = models.DateTimeField()
robot = models.ForeignKey("SomeApp.Robot")
canteen_empty_level = models.IntegerField()
With the above models, as the Robot's Configuration or CanteenLevel change, we create new records and save the historicals.
What I would like to do is add an annotation to a Robot QuerySet that states if the Robot's Canteen is considered empty (Robot's latest CanteenLevel.gallons is less than the Robot's latest Configuration.canteen_empty_level).
The aim is to allow for a statement like this using the annotation in the QuerySet:
bad_robots = Robot.objects.filter(canteen_empty=True)
I had tried something like this in the annotation:
canteen_empty=ExpressionWrapper(CanteenLevel.objects.filter(robot=OuterRef('pk')).order_by('-time').values('gallons')[:1] <= RobotConfiguration.objects.filter(robot=OuterRef('robot')).order_by('-time').values('canteen_empty_level')[:1], output_field=models.BooleanField))
But obviously the "<=" operator isn't allowed.
I also tried this:
canteen_empty=Exists(CanteenLevel.objects.filter(robot=OuterRef('pk')).order_by('-time').values('gallons')[:1].filter(gallons__lte=Subquery(RobotConfiguration.objects.filter(robot=OuterRef('robot')).order_by('-time').values('canteen_empty_level')[:1]))))
But you can't filter after taking a slice of a QuerySet.
Any help would be appreciated!
We can make two annotations here:
from django.db.models import Subquery, OuterRef
latest_gallons = Subquery(CanteenLevel.objects.filter(
robot=OuterRef('pk')
).order_by('-time').values('gallons')[:1])
latest_canteen = Subquery(RobotConfiguration.objects.filter(
robot=OuterRef('pk')
).order_by('-time').values('canteen_empty_level')[:1])
then we can first annotate the Robot objects with these, and filter:
from django.db.models import F
Robot.objects.annotate(
latest_gallons=latest_gallons,
latest_canteen=latest_canteen
).filter(latest_gallons__lte=F('latest_canteen'))
This will construct a query that looks like:
SELECT robot.*,
(SELECT U0.gallons
FROM canteenlevel U0
WHERE U0.robot_id = robot.id
ORDER BY U0.time DESC
LIMIT 1) AS latest_gallons,
(SELECT U0.canteen_empty_level
FROM robotconfiguration U0
WHERE U0.robot_id = robot.id
ORDER BY U0.time DESC
LIMIT 1) AS latest_canteen
FROM robot
WHERE
(SELECT U0.gallons
FROM canteenlevel U0
WHERE U0.robot_id = robot.id
ORDER BY U0.time DESC
LIMIT 1
) <= (
SELECT U0.canteen_empty_level
FROM robotconfiguration U0
WHERE U0.robot_id = robot.id
ORDER BY U0.time DESC
LIMIT 1
)
Note however that if a Robot has no related CanteenLevel, or RobotConfiguration (one of them, or both), then that Robot will not be included in the queryset.
I want to get a list of max ids for a filter I have in Django
class Foo(models.Model):
name = models.CharField()
poo = models.CharField()
Foo.objects.filter(name__in=['foo','koo','too']).latest_by_id()
End result a queryset having only the latest objects by id for each name. How can I do that in Django?
Edit: I want multiple objects in the end result. Not just one object.
Edit1: Added __in. Once again I need only latest( as a result distinct) objects for each name.
Something like this.
my_id_list = [Foo.objects.filter(name=name).latest('id').id for name in ['foo','koo','too']]
Foo.objects.filter(id__in=my_id_list)
The above works. But I want a more concise way of doing it. Is it possible to do this in a single query/filter annotate combination?
you can try:
qs = Foo.objects.filter(name__in=['foo','koo','too'])
# Get list of max == last pk for your filter objects
max_pks = qs.annotate(mpk=Max('pk')).order_by().values_list('mpk', flat=True)
# after it filter your queryset by last pk
result = qs.filter(pk__in=max_pks)
If you are using PostgreSQL you can do the following
Foo.objects.order_by('name', '-id').distinct('name')
MySQL is more complicated since is lacks a DISTINCT ON clause. Here is the raw query that is very hard to force Django to generate from ORM function calls:
Foo.objects.raw("""
SELECT
*
FROM
`foo`
GROUP BY `foo`.`name`
ORDER BY `foo`.`name` ASC , `foo`.`id` DESC
""")
I happen on some query like this below:
bphoto = B_Photo.objects.filter(q_user).order_by('-pub_date')
bactivity = B_Activity.objects.filter(q_user).order_by('-pub_date')
bpart = B_Part.objects.filter(q_user).order_by('-pub_date')
q_user is a Q object,and what i want is to sort all results in the three tables,use the 'pub_date' field.
how can i simplify this kind of query?
B_Photo, B_Activity and B_Part don't have the same table structure, right? I think you cannot make a query with three different table. UNION can do this but it requires all the sub-queries return data with the same structure.
It seems that you want to display a timeline mixed with photos, activities and parts. The most reasonable way is to sort the results in python.
bphoto = B_Photo.objects.filter(q_user).order_by('-pub_date')
bactivity = B_Activity.objects.filter(q_user).order_by('-pub_date')
bpart = B_Part.objects.filter(q_user).order_by('-pub_date')
timeline = sorted(bphoto + bactivity + bpart, key=lambda x:x.pub_date)
UPDATE:
I see what you mean. If you have too much data in these 3 tables and you only want to show the most recent, say, 20 records, you can run a raw UNION sql on the 3 tables like this:
cursor = connection.cursor()
cursor.execute("SELECT id, type FROM (
SELECT id, 'photo' AS type, pub_date FROM b_photo UNION
SELECT id, 'activity' AS type, pub_date FROM b_activity UNION
SELECT id, 'part' AS type, pub_date FROM b_part) AS my_query
ORDER BY pub_date DESC LIMIT 20")
results = cursor.fetchall()
# results will be something like ((id, type), (id, type), ...)
# where type indicates what the id is (photo, activity or part)
Then use individual B_XXX.objects.get(id=id) to get each object in ids.
for result in results:
if result[1] == 'photo':
obj = B_Photo.objects.get(id=result[0])
elif result[1] == 'activity':
obj = B_Activity.objects.get(id=result[0])
elif result[1] == 'part':
obj = B_Part.objects.get(id=result[0])
# do sth with obj...