I have a model with two foreign keys on another one.
DB structure:
class LightUsers(models.Model):
admin = models.ForeignKey(User, db_index=True, related_name="lightuser_admin")
user = models.ForeignKey(User, db_index=True, related_name="lightuser_user")
class User(models.Model):
...
I want to perform query like:
SELECT * FROM lightusers INNER JOIN user ON (lightusers.admin_id = user.id) WHERE lightusers.user_id IN ( 1, 2, 3)
To achieve this I wrote next code:
light_users = LightUsers.objects.filter(user__in=[1,2,3]).select_related('user')
But Django translates this code in query with join by another foreign key field. Like this:
SELECT * FROM lightusers INNER JOIN user ON (lightusers.user_id = user.id) WHERE lightusers.user_id IN ( 1, 2, 3)
Is there a way to force Django use Foreign Key that I want. Or raw query is the only option?
I need to get data from user table by ids in admin column
IF you want this, then you just have to use the admin field:
light_users = LightUsers.objects.filter(admin__in=[1,2,3])
but that's not the same as:
I want to perform query like:
SELECT * FROM lightusers INNER JOIN user ON (lightusers.admin_id = user.id) WHERE lightusers.user_id IN ( 1, 2, 3)
This basically means give me all lightusers with user_id IN ( 1, 2, 3) and having an admin. In an ORM fashion this can be achieved like so:
light_users = LightUsers.objects.filter(
user_id__in=[1,2,3], #this will filter lightusers.user_id IN ( 1, 2, 3)
admin__isnull=False #this will INNER JOIN users on admin and get row having an admin
)
Well in the end i gave up to make this in one query. And divided it into two:
relations = LightUsers.objects.filter(user__in=[1, 2, 3]).values_list('admin_id', 'user_id')
light_users = User.objects.filter(id__in=[relation[0] for relation in relations])
And then combined result.
Related
I'm trying to optimize the fired queries of an API. I have four models namely User, Content, Rating, and UserRating with some relations to each other. I want the respective API returns all of the existing contents alongside their rating count as well as the score given by a specific user to that.
I used to do something like this: Content.objects.all() as a queryset, but I realized that in the case of having a huge amount of data tons of queries will be fired. So I've done some efforts to optimize the fired queries using select_related() and prefetch_related(). However, I'm dealing with an extra python searching, that I hope to remove that, using a controlled prefetch_related() — applying a filter just for a specific prefetch in a nested prefetch and select.
Here are my models:
from django.db import models
from django.conf import settings
class Content(models.Model):
title = models.CharField(max_length=50)
class Rating(models.Model):
count = models.PositiveBigIntegerField(default=0)
content = models.OneToOneField(Content, on_delete=models.CASCADE)
class UserRating(models.Model):
user = models.ForeignKey(
settings.AUTH_USER_MODEL, blank=True, null=True, on_delete=models.CASCADE
)
score = models.PositiveSmallIntegerField()
rating = models.ForeignKey(
Rating, related_name="user_ratings", on_delete=models.CASCADE
)
class Meta:
unique_together = ["user", "rating"]
Here's what I've done so far:
contents = (
Content.objects.select_related("rating")
.prefetch_related("rating__user_ratings")
.prefetch_related("rating__user_ratings__user")
)
for c in contents: # serializer like
user_rating = c.rating.user_ratings.all()
for u in user_rating: # how to remove this dummy search?
if u.user_id == 1:
print(u.score)
Queries:
(1) SELECT "bitpin_content"."id", "bitpin_content"."title", "bitpin_rating"."id", "bitpin_rating"."count", "bitpin_rating"."content_id" FROM "bitpin_content" LEFT OUTER JOIN "bitpin_rating" ON ("bitpin_content"."id" = "bitpin_rating"."content_id"); args=(); alias=default
(2) SELECT "bitpin_userrating"."id", "bitpin_userrating"."user_id", "bitpin_userrating"."score", "bitpin_userrating"."rating_id" FROM "bitpin_userrating" WHERE "bitpin_userrating"."rating_id" IN (1, 2); args=(1, 2); alias=default
(3) SELECT "users_user"."id", "users_user"."password", "users_user"."last_login", "users_user"."is_superuser", "users_user"."first_name", "users_user"."last_name", "users_user"."email", "users_user"."is_staff", "users_user"."is_active", "users_user"."date_joined", "users_user"."user_name" FROM "users_user" WHERE "users_user"."id" IN (1, 4); args=(1, 4); alias=default
As you can see on the above fired queries I've only three queries rather than too many queries which were happening in the past. However, I guess I can remove the python searching (the second for loop) using a filter on my latest query — users_user"."id" IN (1,) instead. According to this post and my efforts, I couldn't apply a .filter(rating__user_ratings__user_id=1) on the third query. Actually, I couldn't match my problem using Prefetch(..., queryset=...) instance given in this answer.
I think you are looking for Prefetch object:
https://docs.djangoproject.com/en/4.0/ref/models/querysets/#prefetch-objects
Try this:
from django.db.models import Prefetch
contents = Content.objects.select_related("rating").prefetch_related(
Prefetch(
"rating__user_ratings",
queryset=UserRating.objects.filter(user__id=1),
to_attr="user_rating_number_1",
)
)
for c in contents: # serializer like
print(c.rating.user_rating_number_1[0].score)
I have an Article model like this
from django.contrib.contenttypes.fields import GenericRelation
from django.db import models
from hitcount.models import HitCountMixin, HitCount
class Article(models.Model):
title = models.CharField(max_length=250)
hit_count_generic = GenericRelation(
HitCount, object_id_field='object_pk',
related_query_name='hit_count_generic_relation')
when I do Article.objects.order_by('hit_count_generic__hits'), I am getting results.but when I do
articles_by_id = Article.objects.filter(id__in=ids).annotate(qs_order=models.Value(0, models.IntegerField()))
articles_by_name = Article.objects.filter(title__icontains='sports').annotate(qs_order=models.Value(1, models.IntegerField()))
articles = articles_by_id.union(articles_by_name).order_by('qs_order', 'hit_count_generic__hits')
getting error
ORDER BY term does not match any column in the result set
How can i achieve union like this? I had to use union instead of AND and OR because i need to preserve order. ie; articles_by_id should come first and articles_by_name should come second.
using Django hitcount for hitcount https://github.com/thornomad/django-hitcount. Hitcount model is given below.
class HitCount(models.Model):
"""
Model that stores the hit totals for any content object.
"""
hits = models.PositiveIntegerField(default=0)
modified = models.DateTimeField(auto_now=True)
content_type = models.ForeignKey(
ContentType, related_name="content_type_set_for_%(class)s", on_delete=models.CASCADE)
object_pk = models.TextField('object ID')
content_object = GenericForeignKey('content_type', 'object_pk')
objects = HitCountManager()
As suggested by #Angela tried prefetch related.
articles_by_id = Article.objects.prefetch_related('hit_count_generic').filter(id__in=[1, 2, 3]).annotate(qs_order=models.Value(0, models.IntegerField()))
articles_by_name = Article.objects.prefetch_related('hit_count_generic').filter(title__icontains='date').annotate(qs_order=models.Value(1, models.IntegerField()))
the query of the prefetch_related when checked is not selecting the hitcount at all see.
SELECT "articles_article"."id", "articles_article"."created", "articles_article"."last_changed_date", "articles_article"."title", "articles_article"."title_en", "articles_article"."slug", "articles_article"."status", "articles_article"."number_of_comments", "articles_article"."number_of_likes", "articles_article"."publish_date", "articles_article"."short_description", "articles_article"."description", "articles_article"."cover_image", "articles_article"."page_title", "articles_article"."category_id", "articles_article"."author_id", "articles_article"."creator_id", "articles_article"."article_type", 0 AS "qs_order" FROM "articles_article" WHERE "articles_article"."id" IN (1, 2, 3)
From Django's official documentation:
Further, databases place restrictions on what operations are allowed in the combined queries. For example, most databases don’t allow LIMIT or OFFSET in the combined queries.
So, make sure that your database allows combining queries like this.
ORDER BY term does not match any column in the result set
You are getting this error, because that's exactly what's happening. Your final result-set for articles does not contain the hits column from the hitcount table , due to which the result-set cannot order using this column.
Before delving into the answer, let's look at what's happening with your django querysets under the hood.
Retrieve a particular set of articles and include an extra ordering field qs_order set to 0.
articles_by_id = Article.objects.filter(id__in=ids).annotate(qs_order=models.Value(0, models.IntegerField()))
SQL Query for the above
Select id, title,....., 0 as qs_order from article where article.id in (Select ....) # whatever you did to get your ids or just a flat list
Retrieve another set of articles and include an extra ordering field qs_order set to 1
articles_by_name = Article.objects.filter(title__icontains='sports').annotate(qs_order=models.Value(1, models.IntegerField()))
SQL Query for the above
Select id, title, ...1 as qs_order from article where title ilike '%sports%'
Original queryset and order_by hit_count_generic__hits
Article.objects.order_by('hit_count_generic__hits')
This will actually perform an inner join and fetch the hitcount table to order by the hits column.
Query
Select id, title,... from article inner join hitcount on ... order by hits ASC
Union
So when you do your union, the result-set of the above 2 queries is combined and then ordered using your qs_order and then hits ...where it fails.
Solution
Use prefetch_related to get your hitcount table in the initial queryset filtering, so you can then use the hits column in the union to order.
articles_by_id = Article.objects.prefetch_related('hit_count_generic').filter(id__in=ids).annotate(qs_order=models.Value(0, models.IntegerField()))
articles_by_name = Article.objects.prefetch_related('hit_count_generic').filter(title__icontains='sports').annotate(qs_order=models.Value(1, models.IntegerField()))
Now as you have the desired table and its columns in both your SELECT queries, your union should work the way you have defined.
articles = articles_by_id.union(articles_by_name).order_by('qs_order', 'hit_count_generic__hits')
Just replacing prefetch_related with select_related works for me.
https://docs.djangoproject.com/en/3.2/ref/models/querysets/#select-related
I have the following model used to store a bidirectional relationship between two users. The records are always inserted where the smaller user id is user_a while the larger user id is user_b.
Is there a way to retrieve all records belonging to a reference user and the correct value of the status (apply negative transformation to relationship_type if user_a) based on whether the reference user id is larger or smaller than the other user id?
Perhaps two separate queries, one where reference user = user_a and another where reference user = user_b, followed by a join?
class Relationship(models.Model):
RELATIONSHIP_CHOICES = (
(0, 'Blocked'),
(1, 'Allowed'),
(-2, 'Pending_A'),
(2, 'Pending_B'),
(-3, 'Blocked_A'),
(3, 'Blocked_B'),
)
user_a = models.ForeignKey(CustomUser, on_delete=models.SET_NULL, related_name='user_a',null=True)
user_b = models.ForeignKey(CustomUser, on_delete=models.SET_NULL, related_name='user_b',null=True)
relationship_type = models.SmallIntegerField(choices=RELATIONSHIP_CHOICES, default=0)
A SQL query of what I'm trying to achieve:
(SELECT user_b as user_select, -relationship_type as type_select WHERE user_a='reference_user') UNION (SELECT user_a as user_select, relationship_type as type_select WHERE user_b='reference_user')
Given you have the id of the user user_id, you can filter with:
from django.db.models import Q
Relationship.objects.filter(Q(user_a_id=user_id) | Q(user_b_id=user_id))
If you have a CustomUser object user, it is almost the same:
from django.db.models import Q
Relationship.objects.filter(Q(user_a=user) | Q(user_b=user))
If you are looking to obtain Relationships with a given type, we can do the following:
from django.db.models import Q
rel_type = 2 # example rel_type
Relationship.objects.filter(
Q(user_a=user, relationship_type=rel_type) |
Q(user_b=user, relationship_type=-rel_type)
)
Here we thus retrieve Relationship objects with user_a the given user and relationship_type=2, or Relationship objects with user_b the given user, and relationship_type=-2.
We could annotate the querysets, and then take the union, like:
qs1 = Relationship.objects.filter(
user_a=user, relationship_type=rel_type
).annotate(
user_select=F('user_b'),
rel_type=F('relationship_type')
)
qs2 = Relationship.objects.filter(
user_a=user, relationship_type=rel_type
).annotate(
user_select=F('user_a'),
rel_type=-F('relationship_type')
)
qs = qs1.union(qs2)
Although I do not know if that is a good idea: the annotations are not "writable" (so you can not update these).
It might be better to implement some sort of "proxy object" that can swap user_a and user_b, and negate the relationship type, and thus is able to act as if it is a real Relationship object.
As you said, id in user_a is always smaller than user_b. So if you query with user_b=user then you should always get the references where user_id in the reference is always higher than other user_id. So I think you can use following querysets:
user = CustomUser.objects.get(id=1)
user_a_references = Relationship.objects.filter(user_a=user)
user_b_references = Relationship.objects.filter(user_b=user)
all_relation_ships = user_a_reference.union(user_b_references)
I have following models:
class Dictionary(models.Model):
word = models.CharField(unique=True)
class ProcessedText(models.Model):
text_id = models.ForeiginKey('Text')
word_id = models.ForeignKey('Dictionary')
class UserDictionary(models.Model):
word_id = models.ForeignKey('Dictionary')
user_id = models.ForeignKye('User')
I want to make query using django ORM same with next sql
SELECT * FROM ProcessedText, UserDictionary WHERE
ProcessedText.text_id = text_id
AND ProcessedText.word_id = UserDictionary.word_id
AND UserDictionary.user_id = user_id
How to do it in one query without using cycles?
This might help you:
How do I select from multiple tables in one query with Django?
And also you may have to restructure your models to enable select_related concept of django.
I have a model with simple relation
class Tasks(models.Model):
initiator = models.ForeignKey(User, on_delete = models.CASCADE)
class TaskResponsiblePeople(models.Model):
task = models.ForeignKey('Tasks')
auth_user = models.ForeignKey(User)
And I need to write an analogue of an SQL query as follows:
select a.initiator, b.auth_user
from Tasks a
inner join TaskResponsiblePeople b
on TaskResponsiblePeople.task_id = task.id
where Tasks.initiator = 'value A' OR TaskResponsiblePeople.auth_user = 'value B'
The problem is that the OR statement deals with two different tables and I've got no idea about the right Django syntax to mimique the above-stated raw-SQL query. Help me out please !
UPDATE 1
According to the below-stated answer, I use the following code:
people = TaskResponsiblePeople.objects.filter(Q(task__initiator = request.user.id)|Q(auth_user = request.user.id)).select_related('auth_user')
print people.query
# The result of the print copy-pasted from console
# SELECT * FROM `task_responsible_people`
# LEFT OUTER JOIN `tasks` ON (`task_responsible_people`.`task_id` = `tasks`.`id`)
# LEFT OUTER JOIN `auth_user` T4
# ON (`task_responsible_people`.`auth_user_id` = T4.`id`)
# WHERE (`tasks`.`initiator_id` = 7 OR
# 'task_responsible_people`.`auth_user_id` = 7)
tasks = Tasks.objects.prefetch_related(
Prefetch('task_responsible_people', queryset=people, to_attr='people'))
However, in the final resultset I can still see records where neither initiator nor auth_user are equal to request.user (equal to 7 in this case)
I avoid using ".values" because of the potential need to serialize and transform the queryset into json.
I think you can do it this way if you just want those specific columns:
from django.db.models import Q
qs = Tasks.objects.filter(Q(initiator=userA) | Q(taskresponsiblepeople__auth_user=userB))\
.values('initiator', 'taskresponsiblepeople__auth_user')
To examine the generated query you can look at:
print(qs.query)
I don't have the models in my database but it should generate a query similar to following:
SELECT "tasks"."initiator_id", "taskresponsiblepeople"."auth_user_id"
FROM "tasks" LEFT OUTER JOIN "taskresponsiblepeople"
ON ( "tasks"."id" = "taskresponsiblepeople"."tasks_id" )
WHERE ("tasks"."initiator_id" = userA_id
OR "taskresponsiblepeople"."auth_user_id" = userB_id))