How to create an annotation in Django that references two related models - django

I'm trying to add an annotation to a QuerySet that is True/False when the value of a field on one related object is less than the value of a field on a different related object.
Here are some models for an example:
class RobotManager(models.Manager):
queryset = super(RobotManager, self).get_queryset()
queryset = queryset.annotate(canteen_empty=UNKNOWN CODE)
return queryset
class Robot(models.Model):
# Has some other unrelated stuff
objects = RobotManager()
class CanteenLevel(models.Model):
time = models.DateTimeField()
robot = models.ForeignKey("SomeApp.Robot")
gallons = models.IntegerField()
class RobotConfiguration(models.Model):
time = models.DateTimeField()
robot = models.ForeignKey("SomeApp.Robot")
canteen_empty_level = models.IntegerField()
With the above models, as the Robot's Configuration or CanteenLevel change, we create new records and save the historicals.
What I would like to do is add an annotation to a Robot QuerySet that states if the Robot's Canteen is considered empty (Robot's latest CanteenLevel.gallons is less than the Robot's latest Configuration.canteen_empty_level).
The aim is to allow for a statement like this using the annotation in the QuerySet:
bad_robots = Robot.objects.filter(canteen_empty=True)
I had tried something like this in the annotation:
canteen_empty=ExpressionWrapper(CanteenLevel.objects.filter(robot=OuterRef('pk')).order_by('-time').values('gallons')[:1] <= RobotConfiguration.objects.filter(robot=OuterRef('robot')).order_by('-time').values('canteen_empty_level')[:1], output_field=models.BooleanField))
But obviously the "<=" operator isn't allowed.
I also tried this:
canteen_empty=Exists(CanteenLevel.objects.filter(robot=OuterRef('pk')).order_by('-time').values('gallons')[:1].filter(gallons__lte=Subquery(RobotConfiguration.objects.filter(robot=OuterRef('robot')).order_by('-time').values('canteen_empty_level')[:1]))))
But you can't filter after taking a slice of a QuerySet.
Any help would be appreciated!

We can make two annotations here:
from django.db.models import Subquery, OuterRef
latest_gallons = Subquery(CanteenLevel.objects.filter(
robot=OuterRef('pk')
).order_by('-time').values('gallons')[:1])
latest_canteen = Subquery(RobotConfiguration.objects.filter(
robot=OuterRef('pk')
).order_by('-time').values('canteen_empty_level')[:1])
then we can first annotate the Robot objects with these, and filter:
from django.db.models import F
Robot.objects.annotate(
latest_gallons=latest_gallons,
latest_canteen=latest_canteen
).filter(latest_gallons__lte=F('latest_canteen'))
This will construct a query that looks like:
SELECT robot.*,
(SELECT U0.gallons
FROM canteenlevel U0
WHERE U0.robot_id = robot.id
ORDER BY U0.time DESC
LIMIT 1) AS latest_gallons,
(SELECT U0.canteen_empty_level
FROM robotconfiguration U0
WHERE U0.robot_id = robot.id
ORDER BY U0.time DESC
LIMIT 1) AS latest_canteen
FROM robot
WHERE
(SELECT U0.gallons
FROM canteenlevel U0
WHERE U0.robot_id = robot.id
ORDER BY U0.time DESC
LIMIT 1
) <= (
SELECT U0.canteen_empty_level
FROM robotconfiguration U0
WHERE U0.robot_id = robot.id
ORDER BY U0.time DESC
LIMIT 1
)
Note however that if a Robot has no related CanteenLevel, or RobotConfiguration (one of them, or both), then that Robot will not be included in the queryset.

Related

Django Nested ManyToManyField objects count query

we have Project as main model, which contains 2 fields of M2M relationship.
class First(models.Model):
first_results_M2M = models.ManyToManyField(First_Results)
class Second(models.Model):
second_results_M2M = models.ManyToManyField(Second_Results)
class Project(models.Model):
project_first_M2M = models.ManyToManyField(First)
project_second_M2M = models.ManyToManyField(Second)
I m trying to count all the objects present in first_results_M2M of all the project_first_M2M objects within each Project object.
Here's the below example to count all the objects of first_results_M2M for Project object 1.
total_first_all = First_Results.objects.filter(first__project__id=1).count()
I want to render the total count of total_first_all and total_second_all in the template.
Project_Query = Project.objects.all()
for each_proj in Project_Query:
print(each_proj.total_first_all) ## should print the count the `first_resuls_M2M` for each project obj.
Please let me know how to do achieve it in more effecient/fast way besides annotate.
annotate.total_first_all=Count('project_first_M2M__first_results_M2M')
You .annotate(..) [Django-doc] your queryset, like:
from django.db.models import Count
project_query = Project.objects.annotate(
total_first_all=Count('project_first_M2M__first_results_M2M')
)
for project in project_query:
print(project.total_first_all)
This will not make a query per Project object, but calculate the counts for all Projects in "bulk".
For multiple ones, you can make use of subqueries to reduce the amount of nested JOIN:
from django.db.models import Count, OuterRef, Subquery
project_query = Project.objects.annotate(
total_first_all=Subquery(
First_Results.objects.filter(first__project=OuterRef('pk')).values('first__project').values(cnt=Count('*')).order_by('first__project')
),
total_second_all=Subquery(
Second_Results.objects.filter(second__project=OuterRef('pk')).values('second__project').values(cnt=Count('*')).order_by('second__project')
)
)

Applying union() on same model is not recognising ordering using GenericRelation

I have an Article model like this
from django.contrib.contenttypes.fields import GenericRelation
from django.db import models
from hitcount.models import HitCountMixin, HitCount
class Article(models.Model):
title = models.CharField(max_length=250)
hit_count_generic = GenericRelation(
HitCount, object_id_field='object_pk',
related_query_name='hit_count_generic_relation')
when I do Article.objects.order_by('hit_count_generic__hits'), I am getting results.but when I do
articles_by_id = Article.objects.filter(id__in=ids).annotate(qs_order=models.Value(0, models.IntegerField()))
articles_by_name = Article.objects.filter(title__icontains='sports').annotate(qs_order=models.Value(1, models.IntegerField()))
articles = articles_by_id.union(articles_by_name).order_by('qs_order', 'hit_count_generic__hits')
getting error
ORDER BY term does not match any column in the result set
How can i achieve union like this? I had to use union instead of AND and OR because i need to preserve order. ie; articles_by_id should come first and articles_by_name should come second.
using Django hitcount for hitcount https://github.com/thornomad/django-hitcount. Hitcount model is given below.
class HitCount(models.Model):
"""
Model that stores the hit totals for any content object.
"""
hits = models.PositiveIntegerField(default=0)
modified = models.DateTimeField(auto_now=True)
content_type = models.ForeignKey(
ContentType, related_name="content_type_set_for_%(class)s", on_delete=models.CASCADE)
object_pk = models.TextField('object ID')
content_object = GenericForeignKey('content_type', 'object_pk')
objects = HitCountManager()
As suggested by #Angela tried prefetch related.
articles_by_id = Article.objects.prefetch_related('hit_count_generic').filter(id__in=[1, 2, 3]).annotate(qs_order=models.Value(0, models.IntegerField()))
articles_by_name = Article.objects.prefetch_related('hit_count_generic').filter(title__icontains='date').annotate(qs_order=models.Value(1, models.IntegerField()))
the query of the prefetch_related when checked is not selecting the hitcount at all see.
SELECT "articles_article"."id", "articles_article"."created", "articles_article"."last_changed_date", "articles_article"."title", "articles_article"."title_en", "articles_article"."slug", "articles_article"."status", "articles_article"."number_of_comments", "articles_article"."number_of_likes", "articles_article"."publish_date", "articles_article"."short_description", "articles_article"."description", "articles_article"."cover_image", "articles_article"."page_title", "articles_article"."category_id", "articles_article"."author_id", "articles_article"."creator_id", "articles_article"."article_type", 0 AS "qs_order" FROM "articles_article" WHERE "articles_article"."id" IN (1, 2, 3)
From Django's official documentation:
Further, databases place restrictions on what operations are allowed in the combined queries. For example, most databases don’t allow LIMIT or OFFSET in the combined queries.
So, make sure that your database allows combining queries like this.
ORDER BY term does not match any column in the result set
You are getting this error, because that's exactly what's happening. Your final result-set for articles does not contain the hits column from the hitcount table , due to which the result-set cannot order using this column.
Before delving into the answer, let's look at what's happening with your django querysets under the hood.
Retrieve a particular set of articles and include an extra ordering field qs_order set to 0.
articles_by_id = Article.objects.filter(id__in=ids).annotate(qs_order=models.Value(0, models.IntegerField()))
SQL Query for the above
Select id, title,....., 0 as qs_order from article where article.id in (Select ....) # whatever you did to get your ids or just a flat list
Retrieve another set of articles and include an extra ordering field qs_order set to 1
articles_by_name = Article.objects.filter(title__icontains='sports').annotate(qs_order=models.Value(1, models.IntegerField()))
SQL Query for the above
Select id, title, ...1 as qs_order from article where title ilike '%sports%'
Original queryset and order_by hit_count_generic__hits
Article.objects.order_by('hit_count_generic__hits')
This will actually perform an inner join and fetch the hitcount table to order by the hits column.
Query
Select id, title,... from article inner join hitcount on ... order by hits ASC
Union
So when you do your union, the result-set of the above 2 queries is combined and then ordered using your qs_order and then hits ...where it fails.
Solution
Use prefetch_related to get your hitcount table in the initial queryset filtering, so you can then use the hits column in the union to order.
articles_by_id = Article.objects.prefetch_related('hit_count_generic').filter(id__in=ids).annotate(qs_order=models.Value(0, models.IntegerField()))
articles_by_name = Article.objects.prefetch_related('hit_count_generic').filter(title__icontains='sports').annotate(qs_order=models.Value(1, models.IntegerField()))
Now as you have the desired table and its columns in both your SELECT queries, your union should work the way you have defined.
articles = articles_by_id.union(articles_by_name).order_by('qs_order', 'hit_count_generic__hits')
Just replacing prefetch_related with select_related works for me.
https://docs.djangoproject.com/en/3.2/ref/models/querysets/#select-related

How can I filter a Django queryset by the latest of a related model?

Imagine I have the following 2 models in a contrived example:
class User(models.Model):
name = models.CharField()
class Login(models.Model):
user = models.ForeignKey(User, related_name='logins')
success = models.BooleanField()
datetime = models.DateTimeField()
class Meta:
get_latest_by = 'datetime'
How can I get a queryset of Users, which only contains users whose last login was not successful.
I know the following does not work, but it illustrates what I want to get:
User.objects.filter(login__latest__success=False)
I'm guessing I can do it with Q objects, and/or Case When, and/or some other form of annotation and filtering, but I can't suss it out.
We can use a Subquery here:
from django.db.models import OuterRef, Subquery
latest_login = Subquery(Login.objects.filter(
user=OuterRef('pk')
).order_by('-datetime').values('success')[:1])
User.objects.annotate(
latest_login=latest_login
).filter(latest_login=False)
This will generate a query that looks like:
SELECT auth_user.*, (
SELECT U0.success
FROM login U0
WHERE U0.user_id = auth_user.id
ORDER BY U0.datetime DESC
LIMIT 1
) AS latest_login
FROM auth_user
WHERE (
SELECT U0.success
FROM login U0
WHERE U0.user_id = auth_user.id
ORDER BY U0.datetime
DESC LIMIT 1
) = False
So the outcome of the Subquery is the success of the latest Login object, and if that is False, we add the related User to the QuerySet.
You can first annotate the max dates, and then filter based on success and the max date using F expressions:
User.objects.annotate(max_date=Max('logins__datetime'))\
.filter(logins__datetime=F('max_date'), logins__success=False)
for check bool use success=False and for get latest use latest()
your filter has been look this:
User.objects.filter(success=False).latest()

Django How To Query ManyToMany Relationship Where All Objects Match

I have the following models:
## Tags for issues
class issueTags(models.Model):
name = models.CharField(max_length=400)
class issues(models.Model):
tags = models.ManyToManyField(issueTags,blank = True)
In my view I get an array from some client side JavaScript i.e.
(Pdb) array_data = request.POST['arr']
(Pdb) array_data
'["2","3"]'
How should I filter my issues object to find all issues which match all tags in the array? (the 2,3 are the ID values for tag__id.
If there is a better way to arrange the objects that would also work so I can search in this fashion.
At the time of writing this, the existing answers are either incorrect (e.g. filtering matching all Issues that have any of the specified tags and the correct tag count) or inefficient (e.g. attaching filters in a loop).
For the following models:
class IssueTag(models.Model):
name = models.CharField(max_length=400, blank=True)
class Issue(models.Model):
label = models.CharField(max_length=50, blank=True)
tags = models.ManyToManyField(IssueTag, related_name='issues')
I suggest using Django Annotation in conjunction with a filter like so:
from django.db.models import Count, Q
tags_to_match = ['tag1', 'tag2']
issues_containing_all_tags = Issue.objects \
.annotate(num_correct_tags=Count('tags',
filter=Q(tags__name__in=tags_to_match))) \
.filter(num_correct_tags=2)
to get all Issues that have all required tags (but may have additional tags, as is required in the question).
This will produce the following SQL query, that resolves all tag matching in a single IN clause:
SELECT "my_app_issue"."id", "my_app_issue"."label",
COUNT("my_app_issue_tags"."issuetag_id")
FILTER (WHERE "my_app_issuetag"."name" IN ('tag1', 'tag2'))
AS "num_correct_tags"
FROM "my_app_issue"
LEFT OUTER JOIN "my_app_issue_tags" ON ("my_app_issue"."id" = "my_app_issue_tags"."issue_id")
LEFT OUTER JOIN "my_app_issuetag" ON ("my_app_issue_tags"."issuetag_id" = "my_app_issuetag"."id")
GROUP BY "my_app_issue"."id", "my_app_issue"."label"
HAVING COUNT("my_app_issue_tags"."issuetag_id")
FILTER (WHERE ("my_app_issuetag"."name" IN ('tag1', 'tag2'))) = 2;
args=('tag1', 'tag2', 'tag1', 'tag2', 2)
I haven't tested this, but I think you could do the following:
from django.db.models import Q
array_data = array_data.split(',')
issues.objects.filter(
tags__in=array_data,
).exclude(
# Exclude any that aren't in array_data
~Q(tags__in=array_data)
).annotate(
matches=Count(tags, distinct=True)
).filter(
# Make sure the number found is right.
matches=len(array_data)
)
FYI, you should be using Issue, IssueTag for your model names to follow Django's naming pattern.
It isn't most elegant solution or pythonic but I ended up just looping around the resulting filter.
def filter_on_category(issue_object,array_of_tags):
#keep filtering to make an and
i = 0
current_filter = issue_object
while (i < (len(array_of_tags))):
#lets filter again
current_filter=current_filter.filter(tags__id__in=array_of_tags[i])
i=i+1
return current_filter
Django field lookups argument (__) for many-to-many fields needs list argument. I have created a dummy list for each array element of IssueTags and pass it to lookups argument and it works as expected.
Let you have this models:
class IssueTags(models.Model):
name = models.CharField(max_length=400)
class Issues(models.Model):
tags = models.ManyToManyField(IssueTags,blank = True)
You want to get Issues which contains all of these IssueTags = ["1","2","3"]
issue_tags_array = ["1","2","3"]
#First initialize queryset
queryset = Issues.objects.all()
i = 0
while i < len(issue_tags_array):
#dummy issue_tag list
issue_tag = [issue_tags_array[i]]
#lets filter again
queryset = queryset.filter(tags__id__in=issue_tag)
i=i+1
return queryset

Annotate django query if filtered row exists in second table

I have two tables (similar to the ones below):
class Piece(models.Model):
cost = models.IntegerField(default=50)
piece = models.CharField(max_length=256)
class User_Piece (models.Model):
user = models.ForeignKey(User)
piece = models.ForeignKey(Piece)
I want to do a query that returns all items in Piece, but annotates each row with whether or not the logged in user owns that piece (so there exists a row in User_Piece where user is the logged in user).
I tried:
pieces = Piece.objects.annotate(owned=Count('user_piece__id'))
But it puts a count > 0 for any piece that is owned by any user. I'm not sure where/how I put in the condition that the user_piece must have the specified user I want. If I filter on user__piece__user=user, then I don't get all the rows from Piece, only those that are owned.
You could use Exist subquery wrapper:
from django.db.models import Exists, OuterRef
subquery = User_Piece.objects.filter(user=user, piece=OuterRef('pk'))
Piece.objects.annotate(owned=Exists(subquery))
https://docs.djangoproject.com/en/dev/ref/models/expressions/#exists-subqueries
In newer versions of Django, you can do:
from django.db.models import Exists, OuterRef
pieces = Piece.objects.annotate(
owned=Exists(UserPiece.objects.filter(piece=OuterRef('id'), user=request.user))
)
for piece in pieces:
print(piece.owned) # prints True or False
Of course, you can replace the name owned with any name you want.
Easy approach, be careful with performance:
pk_pices = ( User_Piece
.objects
.filter(user=user)
.distinct()
.values_list( 'id', flat=True)
)
pieces = pieces.objects.filter( id__in = pk_pieces )
Also, notice that you have a n:m relation ship, you can rewrite models as:
class Piece(models.Model):
cost = models.IntegerField(default=50)
piece = models.CharField(max_length=256)
users = models.ManyToManyField(User, through='User_Piece', #<- HERE!
related_name='Pieces') #<- HERE!
And get user pieces as:
pieces = currentLoggedUser.pieces.all()