I have a Q object that queries my database that works something like this:
class EventSearchManager(models.Manager):
q_objects = []
terms = [term.strip() for term in search_terms.split()]
today = date.today()
if timeselect == "Today":
first_day = today
last_day = None
for term in terms:
search = (
Q(name__icontains=term) |
Q(tags__label__icontains=term),
)
if first_day is not None:
operators.update({'start_date__gte': first_day})
if last_day is not None:
operators.update({'start_date__lte': last_day})
q_objects.append(Q(*search, **operators))
qs = self.get_queryset()
return qs.filter(reduce(operator.or_, q_objects))
It works well, but I've just refactored the Events so that start_date exists in a separate EventInstance model (this way an event can have an indefinite amount of start dates).
Now I would like to adapt this search to return Event objects so that operators.update({'start_date__gte': first_day}) references the start_date of all associated EventInstance objects. Is there an easy syntax adjustment I can make, or will I need to reconstruct this process entirely? Or am I simply asking too much of the Q object?
This is my EventInstance model which establishes the relationship:
class EventInstance(models.Model):
event = models.ForeignKey(Event)
start = models.DateTimeField()
duration = models.TimeField()
recurring = models.CharField(max_length=2)
Q objects are exactly the same as a normal filter condition. Since you can follow relationships in a filter condition, you can do it in a Q as well.
You don't show your models, but assuming the relation is just called eventinstance, you can do:
operators.update({'eventinstance__start_date__gte': first_day})
Related
I'm stuck on a seemingly simple issue. I want to do a different queryset if sold_data is empty. Is there an effective way to do this? Or do I need to use a for loop and loop over all the listing objects and check each one?
class Listing(models.Model):
list_price = models.IntegerField()
sold_price = models.IntegerField(null=True, blank=True)
... other fields
data = Listing.objects.filter(...) # Note: I had already made other queries
if sold_price == None:
data = data.filter(list_price__gte=1)
else:
data = data.filter(sold_price__gte=1)
You can do it using Q object.
from django.db.models import Q
# your filtered queryset is in 'data' varibale
data.filter(Q(sold_price__isnull=False, sold_price__gte=1) | Q(sold_price__isnull=True, list_price__gte=1))
if you wanna check if an object is None use the is operator
I'm not sure if I did understand your question here is what I get : you wanna filter list_price if the data contains an object with empty value else filter sold_price
You can try this
if data.filter(sold_price__isnull=True).exists():
data = data.filter(list_price__gte=1)
else:
data = data.filter(sold_price__gte=1)
I have a History model like below
class History(models.Model):
class Meta:
app_label = 'subscription'
ordering = ['-start_datetime']
subscription = models.ForeignKey(Subscription, related_name='history')
FREE = 'free'
Premium = 'premium'
SUBSCRIPTION_TYPE_CHOICES = ((FREE, 'Free'), (Premium, 'Premium'),)
name = models.CharField(max_length=32, choices=SUBSCRIPTION_TYPE_CHOICES, default=FREE)
start_datetime = models.DateTimeField(db_index=True)
end_datetime = models.DateTimeField(db_index=True, blank=True, null=True)
cancelled_datetime = models.DateTimeField(blank=True, null=True)
Now i have a queryset filtering like below
users = get_user_model().objects.all()
queryset = users.exclude(subscription__history__end_datetime__lt=timezone.now())
The issue is that in the exclude above it is checking end_datetime for all the rows for a particular history object. But i only want to compare it with first row of history object.
Below is how a particular history object looks like. So i want to write a queryset filter which can do datetime comparison on first row only.
You could use a Model Manager method for this. The documentation isn't all that descriptive, but you could do something along the lines of:
class SubscriptionManager(models.Manager):
def my_filter(self):
# You'd want to make this a smaller query most likely
subscriptions = Subscription.objects.all()
results = []
for subscription in subscriptions:
sub_history = subscription.history_set.first()
if sub_history.end_datetime > timezone.now:
results.append(subscription)
return results
class History(models.Model):
subscription = models.ForeignKey(Subscription)
end_datetime = models.DateTimeField(db_index=True, blank=True, null=True)
objects = SubscriptionManager()
Then: queryset = Subscription.objects().my_filter()
Not a copy-pastable answer, but shows the use of Managers. Given the specificity of what you're looking for, I don't think there's a way to get it just via the plain filter() and exclude().
Without knowing what your end goal here is, it's hard to say whether this is feasible, but have you considered adding a property to the subscription model that indicates whatever you're looking for? For example, if you're trying to get everyone who has a subscription that's ending:
class Subscription(models.Model):
#property
def ending(self):
if self.end_datetime > timezone.now:
return True
else:
return False
Then in your code: queryset = users.filter(subscription_ending=True)
I have tried django's all king of expressions(aggregate, query, conditional) but was unable to solve the problem so i went with RawSQL and it solved the problem.
I have used the below SQL to select the first row and then compare the end_datetime
SELECT (end_datetime > %s OR end_datetime IS NULL) AS result
FROM subscription_history
ORDER BY start_datetime DESC
LIMIT 1;
I will select my answer as accepted if not found a solution with queryset filter chaining in next 2 days.
Consider a Hotel with Room objects and Reservation objects for those rooms. I want to find which rooms are available at a given period or (particularly in the example below) from which date onward.
A reservation can be 'deleted', which is obtained by settings the "Live" field. So they're not actually deleted but just inactive, and this would need to remain this way.
>>> indate = "20141225"
>>> Room.objects.exclude(
(models.Q(reservation__live=True , reservation__date_out__gt=indate)
| models.Q(reservation__date_out__isnull=True, reservation__live=True))
)
Problem statement: The above code has an unfortunate side effect: when there is a reservation with live=True that is outside the period, and when there's another reservation live=False within the period for the same room, then that Room will be excluded. This should not be the case: since the reservation within the period I'm asking for is not set to live=True, it should be not be taken into account.
It looks like my query above is not considering the same reservation when doing the (live=True and date_out__gt=indate) comparison through the room-reservation relation.
Question: is there a way, within the exclude(), to ensure the same reservation is considered in the comparison?
I tried playing around with negative models.Q (~Models.Q) to no avail.
Note the above is just a code extract (where the issue resides) out of a much larger query. Therefore I cannot simply do qs.filter(reservation__live=True). Moving over the query to filter instead of exclude doesn't seem to be an option.
Edit: adding my simplified models on request, below.
Edit2: adding some test data explaining the issue, and adding a chained exclude() as per #knbk's suggestion.
class AvailableRoomManager(models.Manager):
def available_with_Q(self, indate, outdate):
qs = super(AvailableRoomManager, self).get_queryset()
if indate and outdate:
qs = qs.exclude(models.Q(reservation__date_out__gt=indate, reservation__date_in__lt=outdate, reservation__live=True)
| models.Q(reservation__date_out__isnull=True, reservation__date_in__isnull=False, reservation__date_in__lt=outdate, reservation__live=True))
elif indate and not outdate:
qs = qs.exclude((models.Q(reservation__date_out__gt=indate, reservation__date_in__isnull=False, reservation__live=True)
| models.Q(reservation__date_out__isnull=True, reservation__date_in__isnull=False, reservation__live=True)))
return qs
def available_with_chained_excludes(self, indate, outdate):
qs = super(AvailableRoomManager, self).get_queryset()
if indate and outdate:
qs = qs.exclude(reservation__date_out__gt=indate, reservation__date_in__lt=outdate, reservation__live=True) \
.exclude(reservation__date_out__isnull=True, reservation__date_in__isnull=False, reservation__date_in__lt=outdate, reservation__live=True)
elif indate and not outdate:
qs = qs.exclude(reservation__date_out__gt=indate, reservation__date_in__isnull=False, reservation__live=True) \
.exclude(reservation__date_out__isnull=True, reservation__date_in__isnull=False, reservation__live=True)
return qs
class Room(models.Model):
name = models.CharField(max_length=30, unique=True)
objects = models.Manager()
available_rooms = AvailableRoomManager()
def __str__(self):
return self.name
class Reservation(models.Model):
date_in = models.DateField()
date_out = models.DateField(blank=True, null=True)
room = models.ForeignKey(Room)
live = LiveField() # See django-livefield; to do deletion. Basically adds a field "live" to the model.
objects = LiveManager()
all_objects = LiveManager(include_soft_deleted=True)
The problem pops up in the exclude() statements above when there is an active (live=True) outside the period searching for, and when there is an inactive (live != True) inside the period I'm searching for.
Some simple test data using the above models, showing what the issue is about:
# Let's make two rooms, R001 and R002
>>> room1 = Room.objects.get_or_create(name="R001")[0]
>>> room2 = Room.objects.get_or_create(name="R002")[0]
# First reservation, with no date_out, is created but then "deleted" by setting field 'live' to False
>>> res1 = Reservation.objects.get_or_create(date_in="2014-12-01", date_out=None, room=room1)[0]
>>> res1.live = False
>>> res1.save()
# Second reservation in same room is created with date_out set to Dec 15th
>>> res2 = Reservation.objects.get_or_create(date_in="2014-12-01", date_out="2014-12-15", room=room1)[0]
# Here I'd expect to have R001 listed as well... this is not the case
>>> Room.available_rooms.available_with_Q("2014-12-16", "")
[<Room: R002>]
>>> Room.available_rooms.available_with_chained_excludes("2014-12-16", "")
[<Room: R002>]
# As a test, when changing "deleted" res1's Room to room2, the manager does return R001
>>> res1.room = room2
>>> res1.save()
>>> Room.available_rooms.available_with_Q("2014-12-16", "")
[<Room: R001>, <Room: R002>]
>>> Room.available_rooms.available_with_chained_excludes("2014-12-16", "")
[<Room: R001>, <Room: R002>]
I tested it with your github project, and I got the same results as you did. It seems that, while filter correctly converts multiple related-object filters to an INNER JOIN, exclude converts it in some sort of subquery where each filter (even within the same call) is checked on its own.
I've found a workaround for this, and that is to explicitly create the subquery of reservations:
elif indate and not outdate:
ress = Reservation.objects.filter(Q(live=True, date_in__isnull=False), Q(date_out__gt=indate) | Q(date_out__isnull=True))
rooms = Room.objects.exclude(reservation__in=ress)
etc...
Btw, if you ever need to use filter instead of exclude, the following queries are always identical, and in fact, this is what Django does internally:
Room.objects.exclude(<some_filter>)
Room.objects.filter(~Q(<some_filter>))
You want to not allow a checkin if there is already a reservation in that time, or a user would like to checkout when there is already a reservation placed:
first case : R_IN User_IN R_out
second case: R_IN User_IN (the room is considered busy, so no more check ins)
third case : User_IN R_IN User_OUT
Room.objects.exclude(
models.Q(reservation__live=False, reservation__date_in__lte=indate, reservation__date_out__gte=indate)
|
models.Q(reservation__live=False, reservation__date_in__lte=indate, reservation__date_out__isnull=True)
|
models.Q(reservation__live=False, reservation__date_in__gte=indate, reservation__date_in__lte=outdate)
)
if you need to take into account the case where outdate is not available you could simply factor out the last case, handling how it's best for you, maybe removing that room altogether because of possible conflicts or forcing the user to leave a day before the reservation you already have checks in
I am trying to build a messaging application with Django. The reason I don’t use postman is that I need messaging between other objects than users and I don’t need most of the postman’s features.
Here are my models:
class Recipient(models.Model):
...
def get_unread_threads():
see below...
class Thread(models.Model):
author = models.ForeignKey(Recipient, related_name='initiated_threads')
recipients = models.ManyToManyField(
Tribe,
related_name='received_thread',
through='ThreadReading')
subject = models.CharField(max_length=64)
class Meta:
app_label = 'game'
class Message(models.Model):
author = models.ForeignKey(Recipient, related_name='sent_messages')
date_add = models.DateTimeField(auto_now_add=True)
date_edit = models.DateTimeField(blank=True)
content = models.TextField(max_length=65535)
thread = models.ForeignKey(Thread)
class Meta:
get_latest_by = 'date'
class ThreadReading(models.Model):
thread = models.ForeignKey(Thread)
recipient = models.ForeignKey(Recipient)
date_last_reading = models.DateTimeField(auto_now=True)
My problem is about get_unread_threads. I can’t really find out how to do that. Here is a first try:
def get_unread_threads(self):
"""
return a queryset corresponding to the threads
which at least one message is unread by the recipient
"""
try:
query = self.received_thread.filter(
message__latest__date_add__gt=\
self.threadreading_set.get(thread_id=F('id')).date_last_reading)
except ObjectDoesNotExist:
query = None
return query
But obviously it doesn’t work because lookup can’t follow method latest.
Here you go:
# Get all the readings of the user
thread_readings = recipient.threadreading_set.all()
# Build a query object including all messages who's last reading is after the
# last edit that was made AND whose thread is the thread of the current
# iteration's thread reading
q = models.Q()
for thread_reading in thread_readings:
q = q | models.Q(
models.Q(
date_edit__lte=thread_reading.date_last_reading
& models.Q(
thread=thread_reading.thread
)
)
)
# Get a queryset of all the messages, including the threads (via a JOIN)
queryset = Message.objects.select_related('thread')
# Now, exclude from the queryset every message that matches the above query
# (edited prior to last reading) OR that is in the list of last readings
queryset = queryset.exclude(
q | models.Q(
thread__pk__in=[thread_reading.pk for thread_reading in thread_readings]
)
)
# Make an iterator (to pretend that this is efficient) and return a generator of it
iterator = queryset.iterator()
return (message.thread for message in iterator)
:)
Now, don't ever actually do this - rethink your models. I would read a book called "Object Oriented Analysis and Design with Applications". It'll teach you a great deal about how to think when you're data modelling.
EDIT:
It turns out the real question is - how do I get select_related to follow the m2m relationships I have defined? Those are the ones that are taxing my system. Any ideas?
I have two classes for my django app. The first (Item class) describes an item along with some functions that return information about the item. The second class (Itemlist class) takes a list of these items and then does some processing on them to return different values. The problem I'm having is that returning a list of items from Itemlist is taking a ton of queries, and I'm not sure where they're coming from.
class Item(models.Model):
# for archiving purposes
archive_id = models.IntegerField()
users = models.ManyToManyField(User, through='User_item_rel',
related_name='users_set')
# for many to one relationship (tags)
tag = models.ForeignKey(Tag)
sub_tag = models.CharField(default='',max_length=40)
name = models.CharField(max_length=40)
purch_date = models.DateField(default=datetime.datetime.now())
date_edited = models.DateTimeField(auto_now_add=True)
price = models.DecimalField(max_digits=6, decimal_places=2)
buyer = models.ManyToManyField(User, through='Buyer_item_rel',
related_name='buyers_set')
comments = models.CharField(default='',max_length=400)
house_id = models.IntegerField()
class Meta:
ordering = ['-purch_date']
def shortDisplayBuyers(self):
if len(self.buyer_item_rel_set.all()) != 1:
return "multiple buyers"
else:
return self.buyer_item_rel_set.all()[0].buyer.name
def listBuyers(self):
return self.buyer_item_rel_set.all()
def listUsers(self):
return self.user_item_rel_set.all()
def tag_name(self):
return self.tag
def sub_tag_name(self):
return self.sub_tag
def __unicode__(self):
return self.name
and the second class:
class Item_list:
def __init__(self, list = None, house_id = None, user_id = None,
archive_id = None, houseMode = 0):
self.list = list
self.house_id = house_id
self.uid = int(user_id)
self.archive_id = archive_id
self.gen_balancing_transactions()
self.houseMode = houseMode
def ret_list(self):
return self.list
So after I construct Itemlist with a large list of items, Itemlist.ret_list() takes up to 800 queries for 25 items. What can I do to fix this?
Try using select_related
As per a question I asked here
Dan is right in telling you to use select_related.
select_related can be read about here.
What it does is return in the same query data for the main object in your queryset and the model or fields specified in the select_related clause.
So, instead of a query like:
select * from item
followed by several queries like this every time you access one of the item_list objects:
select * from item_list where item_id = <one of the items for the query above>
the ORM will generate a query like:
select item.*, item_list.*
from item a join item_list b
where item a.id = b.item_id
In other words: it will hit the database once for all the data.
You probably want to use prefetch_related
Works similarly to select_related, but can deal with relations selected_related cannot. The join happens in python, but I've found it to be more efficient for this kind of work than the large # of queries.
Related reading on the subject