I need to find data within a certain set of parameters
I am building a small booking system, that lets user see what vehicles are available for booking for their little safari trip.
The system has bookings that have been entered previously or made previously by a client.
If a booking's pickup_date = 2011-03-01 and dropoff_date = 2011-03-15 and I run a query with pickup=2011-03-09 and dropoff=2011-03-14 in my views as below, it doesn't return any results to see if a booking within that timeframe has been made.
views.py
def dates(request, template_name='groups/_dates.html'):
pickup=request.GET.get('pickup','None');
dropoff=request.GET.get('dropoff','None');
order = Order.objects.filter(pick_up__lte=pickup).filter(drop_off__gte=dropoff)
context = {'order':order,}
return render_to_response(template_name,context,
context_instance=RequestContext(request))
Any suggestions on how to do this?
Or should I be looking at an alternate way of running this query?
Could it be posible that as your passing the raw string to the queryset is not on the correct format, try converting the strings to datetime objects.
Later you can try using the range lookup is more efficient on some DB engines and more simple to read and code.
from django.db.models import Q
start_date = datetime.date(2005, 1, 1)
end_date = datetime.date(2005, 3, 31)
orders = Order.objects.filter(drop_off__gte=start_date, pick_up__lte=end_date)
# Or maybe better
orders = Order.objects.filter(Q(drop_off__gte=start_date), Q(pick_up__lte=end_date))
Can you try this :
order = Order.objects.filter(pick_up**__date__**lte=pickup).filter(drop_off**__date__**gte=dropoff)
https://docs.djangoproject.com/fr/2.0/ref/models/querysets/#date
Related
I am trying to find a better way to loop over orders for the next seven days including today, what I have already:
unfilled_orders_0 = Model.objects.filter(delivery_on__date=timezone.now() + timezone.timedelta(0))
context['todays_orders'] = unfilld_orders_0.aggregate(field_1_sum=Sum('field_1'), field_2_sum=Sum('field_2'),field_3_sum=Sum('field_3'), field_4_sum=Sum('field_4'),field_5_sum=Sum('field_5'))
I'm wondering if I can somehow avoid having to do this seven times--one for each day. I assume there is a more efficient way to do this.
You can do this with a single ORM / db query, by providing Sum with an extra filter argument:
days_ahead = 7
fields = ["field_1", "field_2", ...]
aggregate_kwargs = {
f"s_{field}_{day}": Sum(field, filter=Q(delivery_on__date=now+timedelta(days=day)))
for field in fields
for day in range(days_ahead)
}
unfilled_orders = Model.objects.filter(delivery_on__date__lt=now+timedelta(days=days_ahead)
context.update(unfilled_orders.aggregate(**aggregate_kwargs))
You can approach it with a for loop and store the data in the context like so
from django.utils import timezone
from django.db.models import Sum
context = {}
for i in range(7):
qs = Model.objects.filter(delivery_on__date=(timezone.now() + timezone.timedelta(i)).date())
context = {}
context[f'orders_{i}'] = qs.aggregate(
field_1_sum=Sum('field_1'),
field_2_sum=Sum('field_2'),
field_3_sum=Sum('field_3'),
field_4_sum=Sum('field_4'),
field_5_sum=Sum('field_5'))
This query will hit 7 times the database, otherwise you can use another approach which will hit the db only once
context = {}
qs = Model.objects.filter(delivery_on__date__lte=timezone.now()-timezone.timedelta(days=7)).order_by('delivery_on')
dates = qs.values('delivery_on__date', flat=True).distinct()
for i in dates:
_qs = qs.filter(create_ts__date=i)
context[f'orders_{i}'] = _qs.aggregate(
field_1_sum=Sum('field_1'),
field_2_sum=Sum('field_2'),
field_3_sum=Sum('field_3'),
field_4_sum=Sum('field_4'),
field_5_sum=Sum('field_5'))
You define how many days backwards the qs will be including all orders then distinct the dates and filter the already filtered qs for the dates.
I am suffering to get a query working despite all I have been trying based on my web search, and I think I need some help before becoming crazy.
I have four models:
class Series(models.Model):
puzzles = models.ManyToManyField(Puzzle, through='SeriesElement', related_name='series')
...
class Puzzle(models.Model):
puzzles = models.ManyToManyField(Puzzle, through='SeriesElement', related_name='series')
...
class SeriesElement(models.Model):
puzzle = models.ForeignKey(Puzzle,on_delete=models.CASCADE,verbose_name='Puzzle',)
series = models.ForeignKey(Series,on_delete=models.CASCADE,verbose_name='Series',)
puzzle_index = models.PositiveIntegerField(verbose_name='Order',default=0,editable=True,)
class Play(models.Model):
puzzle = models.ForeignKey(Puzzle, on_delete=models.CASCADE, related_name='plays')
user = models.ForeignKey(settings.AUTH_USER_MODEL, blank=True,null=True, on_delete=models.SET_NULL, related_name='plays')
series = models.ForeignKey(Series, blank=True, null=True, on_delete=models.SET_NULL, related_name='plays')
puzzle_completed = models.BooleanField(default=None, blank=False, null=False)
...
each user can play any puzzle several times, each time creating a Play record.
that means that for a given set of (user,series,puzzle) we can have several Play records,
some with puzzle_completed = True, some with puzzle_completed = False
What I am trying (unsuccesfully) to achieve, is to calculate for each series, through an annotation, the number of puzzles nb_completed_by_user and nb_not_completed_by_user.
For nb_completed_by_user, I have something which works in almost all cases (I have one glitch in one of my test that I cannot explain so far):
Series.objects.annotate(nb_completed_by_user=Count('puzzles',
filter=Q(puzzles__plays__puzzle_completed=True,
puzzles__plays__series_id=F('id'),puzzles__plays__user=user), distinct=True))
For nb_not_completed_by_user, I was able to make a query on Puzzle that gives me the good answer, but I am not able to transform it into a Subquery expression that works without throwing up an error, or to get a Count expression to give me the proper answer.
This one works:
puzzles = Puzzle.objects.filter(~Q(plays__puzzle_completed=True,
plays__series_id=1, plays__user=user),series=s)
but when trying to move to a subquery, I cannot find the way to use the following expression not to throw the error:ValueError: This queryset contains a reference to an outer query and may only be used in a subquery.
pzl_completed_by_user = Puzzle.objects.filter(plays__series_id=OuterRef('id')).exclude(
plays__puzzle_completed=True,plays__series_id=OuterRef('id'), plays__user=user)
and the following Count expression doesn't give me the right result:
Series.objects.annotate(nb_not_completed_by_user=Count('puzzles', filter=~Q(
puzzle__plays__puzzle_completed=True, puzzle__plays__series_id=F('id'),
puzzle__plays__user=user))
Could anybody explain me how I could obtain both values ?
and eventually to propose me a link which explains clearly how to use subqueries for less-obvious cases than those in the official documentation
Thanks in advance
Edit March 2021:
I recently found two posts which guided me through one potential solution to this specific issue:
Django Count and Sum annotations interfere with each other
and
Django 1.11 Annotating a Subquery Aggregate
I implemented the proposed solution from https://stackoverflow.com/users/188/matthew-schinckel and https://stackoverflow.com/users/1164966/benoit-blanchon
having help classes: class SubqueryCount(Subquery) and class SubquerySum(Subquery)
class SubqueryCount(Subquery):
template = "(SELECT count(*) FROM (%(subquery)s) _count)"
output_field = PositiveIntegerField()
class SubquerySum(Subquery):
template = '(SELECT sum(_sum."%(column)s") FROM (%(subquery)s) _sum)'
def __init__(self, queryset, column, output_field=None, **extra):
if output_field is None:
output_field = queryset.model._meta.get_field(column)
super().__init__(queryset, output_field, column=column, **extra)
It works extremely well ! and is far quicker than the conventional Django Count annotation.
... at least in SQlite, and probably PostgreSQL as stated by others.
But when I tried in a MariaDB environnement ... it crashed !
MariaDB is apparently not able / not willing to handle correlated subqueries as those are considered sub-optimal.
In my case, as I try to get from the database multiple Count/distinct annotations for each record at the same time, I really see a tremendous gain in performance (in SQLite)
that I would like to replicate in MariaDB.
Would anyone be able to help me figure out a way to implement those helper functions for MariaDB ?
What should template be in this environnement?
matthew-schinckel ?
benoit-blanchon ?
rktavi ?
Going a bit deeper and analysis the Django docs a bit more in details, I was finally able to produce a satisfying way to produce a Count or Sum based on subquery.
For simplifying the process, I defined the following helper functions:
To generate the subquery:
def get_subquery(app_label, model_name, reference_to_model_object, filter_parameters={}):
"""
Return a subquery from a given model (work with both FK & M2M)
can add extra filter parameters as dictionary:
Use:
subquery = get_subquery(
app_label='puzzles', model_name='Puzzle',
reference_to_model_object='puzzle_family__target'
)
or directly:
qs.annotate(nb_puzzles=subquery_count(get_subquery(
'puzzles', 'Puzzle','puzzle_family__target')),)
"""
model = apps.get_model(app_label, model_name)
# we need to declare a local dictionary to prevent the external dictionary to be changed by the update method:
parameters = {f'{reference_to_model_object}__id': OuterRef('id')}
parameters.update(filter_parameters)
# putting '__id' instead of '_id' to work with both FK & M2M
return model.objects.filter(**parameters).order_by().values(f'{reference_to_model_object}__id')
To count the subquery generated through get_subquery:
def subquery_count(subquery):
"""
Use:
qs.annotate(nb_puzzles=subquery_count(get_subquery(
'puzzles', 'Puzzle','puzzle_family__target')),)
"""
return Coalesce(Subquery(subquery.annotate(count=Count('pk', distinct=True)).order_by().values('count'), output_field=PositiveIntegerField()), 0)
To sum the subquery generated through get_subquery on the field field_to_sum:
def subquery_sum(subquery, field_to_sum, output_field=None):
"""
Use:
qs.annotate(total_points=subquery_sum(get_subquery(
'puzzles', 'Puzzle','puzzle_family__target'),'points'),)
"""
if output_field is None:
output_field = queryset.model._meta.get_field(column)
return Coalesce(Subquery(subquery.annotate(result=Sum(field_to_sum, output_field=output_field)).order_by().values('result'), output_field=output_field), 0)
The required imports:
from django.db.models import Count, Subquery, PositiveIntegerField, DecimalField, Sum
from django.db.models.functions import Coalesce
I spent so many hours on solving this ...
I hope that this will save many of you all the frustration I went through figuring out the right way to proceed.
I have two models:
Model A is an AbstractUserModel and Model B
class ModelB:
user = ForeignKey(User, related_name='modelsb')
timestamp = DateTimeField(auto_now_add=True)
What I want to find is how many users have at least one ModelB object created at least in 3 of the 7 past days.
So far, I have found a way to do it but I know for sure there is a better one and that is why I am posting this question.
I basically split the query into 2 parts.
Part1:
I added a foo method inside the User Model that checks if a user meets the above conditions
def foo(self):
past_limit = starting_date - timedelta(days=7)
return self.modelsb.filter(timestamp__gte=past_limit).order_by('timestamp__day').distinct('timestamp__day').count() > 2
Part 2:
In the Custom User Manager, I find the users that have more than 2 modelsb objects in the last 7 days and iterate through them applying the foo method for each one of them.
By doing this I narrow down the iterations of the required for loop. (basically its a filter function but you get the point)
def boo(self):
past_limit = timezone.now() - timedelta(days=7)
candidates = super().get_queryset().annotate(rc=Count('modelsb', filter=Q(modelsb__timestamp__gte=past_limit))).filter(rc__gt=2)
return list(filter(lambda x: x.foo(), candidates))
However, I want to know if there is a more efficient way to do this, that is without the for loop.
You can use conditional annotation.
I haven't been able to test this query, but something like this should work:
from django.db.models import Q, Count
past_limit = starting_date - timedelta(days=7)
users = User.objects.annotate(
modelsb_in_last_seven_days=Count('modelsb__timestap__day',
filter=Q(modelsb__timestamp__gte=past_limit),
distinct=True))
.filter(modelsb_in_last_seven_days__gte = 3)
EDIT:
This solution did not work, because the distinct option does specify what field makes an entry distinct.
I did some experimenting on my own Django instance, and found a way to make this work using SubQuery. The way this works is that we generate a subquery where we make the distinction ourself.
counted_modelb = ModelB.objects
.filter(user=OuterRef('pk'), timestamp__gte=past_limit)
.values('timestamp__day')
.distinct()
.annotate(count=Count('timestamp__day'))
.values('count')
query = User.objects
.annotate(modelsb_in_last_seven_days=Subquery(counted_modelb, output_field=IntegerField()))
.filter(modelsb_in_last_seven_days__gt = 2)
This annotates each row in the queryset with the count of all distinct days in modelb for the user, with a date greater than the selected day.
In the subquery I use values('timestamp__day') to make sure I can do distinct() (Because a combination of distinct('timestamp__day') and annotate() is unsupported.)
I have
Class Profile(models.Model)
turn_off_date = models.DateTimeField(null= True, blank = True,auto_now = False)
I need to find all profile records, that have expiration date= Exactly x days from now.
How do I do this?
I can think of iterating through all profiles and manually comparing dates, but it seems not effective
Update:
I neeed to filter by date, not by datetime field as in duplicate question
Right now I am doing it like this:
profiles = Profile.objects.all()
for profile in profiles:
if(profile.days_left() == x):
print(profile.days_left())
And in my Profile model I defined a method:
def days_left(self):
turn_off_date = self.turn_off_date
days_left = (turn_off_date - datetime.now()).days
if days_left < 0:
days_left = 0
return days_left
You can just query by __date with a given date
For datetime fields, casts the value as date. Allows chaining additional field lookups. Takes a date value.
.filter(turn_off_date__date=(datetime.now()+timedelta(days=x)).date())
You can use range to achieve this,
Example:
Profile.objects.filter(turn_off_date__range=(start_date, end_date))
I guess something like this will work fine:
From Django 1.9 you can use the __date field lookup, exactly as you
have mentioned in your question. For older versions, you will have to
do with the other methods.
Thanks to https://stackoverflow.com/a/45735324/2950593
On o postgresql db based Django, how can I filter by time for a datetimefield as below?
class Foo(models.Model):
start_date = models.DateTimeField()
end_date = models.DateTimeField()
IE: I want to filter Foo objects with "16:30" start_date and "19:00" end_date.
Thanks.
What about adding in a TimeField?
http://docs.djangoproject.com/en/dev/ref/models/fields/#timefield
Otherwise you would need to write a custom query to take advantage of the database´s time capabilites since DateTimeFields don't have that capability natively in Django.
You could consider writing a function to denormalize hours and minutes from start_date to a new start_time field and then query the start_time field.
Solution to my own q:
def query_by_times(start_hour, start_min, end_hour, end_min):
query = 'EXTRACT(hour from start_date) = %i and EXTRACT(minute from start_date) = %i and EXTRACT(hour from end_date) = %i and EXTRACT(minute from end_date) = %i' % (start_hour, start_min, end_hour, end_min)
return Foo.objects.extra(where=[query])
In your situation, database normalization looks like thebest solution, since execution time and system load will rise as your related database table keeps more records...
Another solution to do this without using database filer functions is using filter alongside lambda:
from datetime import time
records = Foo.objects.all()
filtered = filter(lambda x: (time(16,30)==x.start_date.time() and time(19,0)==x.end_date.time()), records)
But using this on a large dataset will need too much time and system resource.
Try
Foo.objects.filter(start_date = MyDate1, end_date = MyDate2)
where MyDate1 and MyDate2 are your defined datetime objects. It should work, right?
I think which you need uses the range[1].
[1]http://docs.djangoproject.com/en/1.2/ref/models/querysets/#range