I'm trying to make a query in django grouping by a field with choices. I wanna get all choices values, algo choices with no records in the database.
My model:
CHOICES = (
('new', 'New'),
('in_process', 'In process'),
('finished', 'Finished')
)
class Task(models.Model):
...
status = models.CharField(max_length=10, choices=CHOICES)
My current query is:
qs = Task.objects\
.values('status')\
.annotate(total=models.Count('status'))
At this moment, I only have finished task in db, but I wanna get all choices values, with zero if it doesn't have records.
Any idea?
You can do this with:
Task._meta.get_field('status').choices
The choices on a model field are only enforced at the application (Python) level - the DB doesn't know what choices are available.
Related
I'm trying to optimize the fired queries of an API. I have four models namely User, Content, Rating, and UserRating with some relations to each other. I want the respective API returns all of the existing contents alongside their rating count as well as the score given by a specific user to that.
I used to do something like this: Content.objects.all() as a queryset, but I realized that in the case of having a huge amount of data tons of queries will be fired. So I've done some efforts to optimize the fired queries using select_related() and prefetch_related(). However, I'm dealing with an extra python searching, that I hope to remove that, using a controlled prefetch_related() — applying a filter just for a specific prefetch in a nested prefetch and select.
Here are my models:
from django.db import models
from django.conf import settings
class Content(models.Model):
title = models.CharField(max_length=50)
class Rating(models.Model):
count = models.PositiveBigIntegerField(default=0)
content = models.OneToOneField(Content, on_delete=models.CASCADE)
class UserRating(models.Model):
user = models.ForeignKey(
settings.AUTH_USER_MODEL, blank=True, null=True, on_delete=models.CASCADE
)
score = models.PositiveSmallIntegerField()
rating = models.ForeignKey(
Rating, related_name="user_ratings", on_delete=models.CASCADE
)
class Meta:
unique_together = ["user", "rating"]
Here's what I've done so far:
contents = (
Content.objects.select_related("rating")
.prefetch_related("rating__user_ratings")
.prefetch_related("rating__user_ratings__user")
)
for c in contents: # serializer like
user_rating = c.rating.user_ratings.all()
for u in user_rating: # how to remove this dummy search?
if u.user_id == 1:
print(u.score)
Queries:
(1) SELECT "bitpin_content"."id", "bitpin_content"."title", "bitpin_rating"."id", "bitpin_rating"."count", "bitpin_rating"."content_id" FROM "bitpin_content" LEFT OUTER JOIN "bitpin_rating" ON ("bitpin_content"."id" = "bitpin_rating"."content_id"); args=(); alias=default
(2) SELECT "bitpin_userrating"."id", "bitpin_userrating"."user_id", "bitpin_userrating"."score", "bitpin_userrating"."rating_id" FROM "bitpin_userrating" WHERE "bitpin_userrating"."rating_id" IN (1, 2); args=(1, 2); alias=default
(3) SELECT "users_user"."id", "users_user"."password", "users_user"."last_login", "users_user"."is_superuser", "users_user"."first_name", "users_user"."last_name", "users_user"."email", "users_user"."is_staff", "users_user"."is_active", "users_user"."date_joined", "users_user"."user_name" FROM "users_user" WHERE "users_user"."id" IN (1, 4); args=(1, 4); alias=default
As you can see on the above fired queries I've only three queries rather than too many queries which were happening in the past. However, I guess I can remove the python searching (the second for loop) using a filter on my latest query — users_user"."id" IN (1,) instead. According to this post and my efforts, I couldn't apply a .filter(rating__user_ratings__user_id=1) on the third query. Actually, I couldn't match my problem using Prefetch(..., queryset=...) instance given in this answer.
I think you are looking for Prefetch object:
https://docs.djangoproject.com/en/4.0/ref/models/querysets/#prefetch-objects
Try this:
from django.db.models import Prefetch
contents = Content.objects.select_related("rating").prefetch_related(
Prefetch(
"rating__user_ratings",
queryset=UserRating.objects.filter(user__id=1),
to_attr="user_rating_number_1",
)
)
for c in contents: # serializer like
print(c.rating.user_rating_number_1[0].score)
There is a race condition situation, when I want to create a new instance of model Order.
There is a daily_id field that everyday for any category starts from one. It means every category has its own daily id.
class Order(models.Model):
daily_id = models.SmallIntegerField(default=0)
category = models.ForeignKey(Categoty, on_delete=models.PROTECT, related_name="orders")
declare_time = models.DateField()
...
}
daily_id field of new record is being calculated using this method:
def get_daily_id(category, declare_time):
try:
last_order = Order.objects.filter(declare_time=declare_time,
category=category).latest('daily_id')
return last_order.daily_id + 1
except Order.DoesNotExist:
# If no order has been registered in declare_time date.
return 1
The problem is that when two different users are registering orders in the same category at the same time, it is highly likely that the orders have the repetitive daily_id values.
I have tried #transaction.atomic decorator for post method of DRF APIView and it didn't work!
You must use an auto increment and add a view that computes your semantic order like :
SELECT *, ROW_NUMBER() OVER(PARTITION BY MyDayDate ORDER BY id_autoinc) AS daily_id
Related model
class AbstractTask(models.Model):
user = models.ForeignKey(User, on_delete=models.CASCADE)
issued_at = models.DateTimeField(auto_now_add=True)
Problem
I need to show some User statistics per days in the admin panel. Lets say I just need the number of issued tasks. And I need to be able to filter it by issue date (how many were issued yesterday, the day before yesterday, etc).
How I am trying to do it
I use User proxy models to register ModelAdmin for different statistics pages.
I use slightly modified (changed date ranges) DateFieldListFilter on task__issued_at field:
list_filter = [
('task__issued_at', DateFieldListFilter),
'username',
]
Filters on date field don't work
Filters don't work because they end up generating query similar to this:
queryset = (User.objects
.annotate(
# Different statistics.
num_tasks=Count('task'),
)
.filter(
# DateFieldListFilter.
task__issued_at__gte='2020-01-01',
task__issued_at__lt='2020-01-02',
)
.values('id', 'num_tasks')
)
SQL:
SELECT "auth_user"."id",
COUNT("task"."id") AS "num_tasks"
FROM "auth_user"
LEFT OUTER JOIN "task" ON ("auth_user"."id" = "task"."user_id")
INNER JOIN "task" T3 ON ("auth_user"."id" = T3."user_id")
WHERE (T3."issued_at" >= 2020-01-01 00:00:00+03:00
AND T3."issued_at" < 2020-01-02 00:00:00+03:00)
GROUP BY "auth_user"."id"
The problem is that filter adds second join on table "task" when I need just one.
Forcing first inner join by adding .filter(task__isnull=False) doesn't help. It just keeps performing two identical inner joins.
It is the same behavior in django 2 and 3.
Can It be done in Django?
Preferably as simple as possible: without raw sql, without much magic and with continuing using DateFieldListFilter.
But any solution would help.
The alternative QuerySet below gives the same result without any additional joins:
(queryset = User.objects
.annotate(
# Different statistics.
num_tasks=Count(
'task',
filter=models.Q(
Q(task__issued_at__gte='2020-01-01') &
Q(task__issued_at__lt='2020-01-02')
)
),
)
.values('id', 'num_tasks')
)
SQL:
SELECT "auth_user"."id", COUNT("task"."id")
FILTER (WHERE ("task"."issued_at" >= 2020-01-01 00:00:00+03:00 AND "task"."issed_at" < 2020-01-02 00:00:00+03:00)) AS "num_tasks"
FROM "auth_user"
LEFT OUTER JOIN "task" ON ("auth_user"."id" = "task"."user_id")
GROUP BY "auth_user"."id"
but not sure about the performance compared with yours.
Anyway, to make it work with the DateFieldListFilter you just need to override the queryset method:
class CustomDateFieldListFilter(DateFieldListFilter):
def queryset(self, request, queryset):
# Compare the requested value to decide how to filter the queryset.
q_objects = models.Q()
for key, value in self.used_parameters.items():
q_objects &= models.Q(**{key: value})
return queryset.annotate(num_tasks=Count('task', filter=models.Q(q_objects))).values('id', 'num_tasks')
and specify the new class:
list_filter = [
('task__issued_at', CustomDateFieldListFilter),
...
]
That's it.
My DB query is below:
select distinct m.topic
from education.sessions s, education.modules m
where s.subject_id = 'Biology'
and s.module_id = m.id
and m.status = 'Active'
order by m.topic
I am trying the equivalent of the above in QuerySet and I do not seem to get it.First off, I do not know where to include the status='active' check and the "Order By".
Session and Module are my 2 tables with Session having a FK on module. Module has 2 columns Subject and Topic and I need the unique topics for the given subject if status = 'Active'
In Views.py, I have the below:
def load_topics(request):
subject = request.GET.get('subject')
topics = Session.objects.filter(subject=subject).values_list('module__topic', flat=True).distinct()
return render(request, 'evaluation/topic_dropdown_list_options.html', {'topics': topics})
Something like this should help.
If the FK is on Module - then you can access the modules through the related name - it should be module_set if you did not define this on the field.
If I was setting this project up, I would define the related name as modules.
session = models.ForeignKey(
"Session",
null=True,
on_delete=models.CASCADE,
related_name="modules",
)
You can filter with module_set or if you change the related name to modules like so - replace modules with module_set or the related name you set up if you do not want to use modules
topics = Session.objects.filter(
subject=subject,
modules__status='Active'
).order_by(
'modules__topic'
).values_list(
'modules__topic', flat=True
).distinct()
I would like to augment one of my model admins with an interesting value. Given a model like this:
class Participant(models.Model):
pass
class Registration(models.Model):
participant = models.ForeignKey(Participant)
is_going = models.BooleanField(verbose_name='Is going')
Now, I would like to show the number of other Registrations for this Participant where is_going is False. So, something akin to this SQL query:
SELECT reg.*, COUNT(past.id) AS not_going_num
FROM registrations AS reg, registrations AS past
WHERE past.participant_id = reg.participant_id AND
past.is_going = False
I think I can extend the Admin's queryset() method according to Django Admin, Show Aggregate Values From Related Model, by annotating it with the extra Count, but I still cannot figure out how to work the self-join and filter into this.
I looked at Self join with django ORM and Django self join , How to convert this query to ORM query, but the former is doing SELECT * AND the latter seems to have data model problems.
Any suggestions on how to solve this?
See edit history for previous version of the answer.
The admin implementation below will display "Not Going Count" for each Registration model. The "Not Going Count" is the count of is_going=False for the registration's participant.
#admin.register(Registration)
class RegistrationAdmin(admin.ModelAdmin):
list_display = ['id', 'participant', 'is_going', 'ng_count']
def ng_count(self, obj):
return obj.not_going_count
ng_count.short_description = 'Not Going Count'
def get_queryset(self, request):
qs = super(RegistrationAdmin, self).get_queryset(request)
qs = qs.filter(participant__registration__isnull=False)
qs = qs.annotate(not_going_count=Sum(
Case(
When(participant__registration__is_going=False, then=1),
default=0,
output_field=models.IntegerField())
))
return qs
Below is a more thorough explanation of the QuerySet:
qs = qs.filter(participant__registration__isnull=False)
The filter causes Django to perform two joins - an INNER JOIN to participant table, and a LEFT OUTER JOIN to registration table.
qs = qs.annotate(not_going_count=Sum(
Case(
When(participant__registration__is_going=False, then=1),
default=0,
output_field=models.IntegerField())
)
))
This is a standard aggregate, which will be used to SUM up the count of is_going=False. This translates into the SQL
SUM(CASE WHEN past."is_going" = False THEN 1 ELSE 0 END)
The sum is generated for each registration model, and the sum belongs to the registration's participant.
I might misunderstood, but you can do for single participant:
participant = Participant.objects.get(id=1)
not_going_count = Registration.objects.filter(participant=participant,
is_going=False).count()
For all participants,
from django.db.models import Count
Registration.objects.filter(is_going=False).values('participant') \
.annotate(not_going_num=Count('participant'))
Django doc about aggregating for each item in a queryset.