I'm attempting query a table and filter the results by date on a datetime field:
.filter(bucket__gte = start_date) where bucket is a datetimefield and start_date is a date object.
However django converts the start_date to a timestamp in the raw sql ex 2020-02-01 00:00:00 when I want it just be a date ex 2020-02-01.
For some reason casting bucket to a date or casting start_time to a timestamp makes the query 30x slower.
When I manually write a query and compare bucket directly to a date ex bucket >= '2020-02-01' the query is blazing fast.
How can I get the django orm to do this?
Seems that its most efficient to convert your date to a datetime in python then do the lookup on the orm since you are filtering on a DateTimeField
from datetime import datetime
.filter(bucket__gte=datetime.combine(start_date, datetime.min.time()))
If the field bucket is indexed, the explain of this query should indicate an Index Scan which would provide desired and most efficient execution plan.
If not it should still be faster since you avoid casting
For some reason casting bucket to a date or casting start_time to a timestamp makes the query 30x slower.
Yes, casting bucket to a date would prevent use of an index (unless the index was over the casted column). But casting start_time to a timestamp would not. What is it 30 times slower than? You just said python automatically converts it, so, how is that different than casting it? As in, what is your actual code?
When I manually write a query and compare bucket directly to a date ex bucket >= '2020-02-01' the query is blazing fast.
OK, but what is it actually doing?
explain select * from foo where bucket > '2021-03-01';
QUERY PLAN
-------------------------------------------------------------------------
Seq Scan on foo (cost=0.00..38.25 rows=753 width=8)
Filter: (bucket > '2021-03-01 00:00:00-05'::timestamp with time zone)
(2 rows)
PostgreSQL is also converting it to a timestamp. Does it give the right answer or the wrong answer?
Try convet your datetime to date in filter:
.filter(bucket__date__gte = start_date)
Related
Let's say I have TIME_ZONE variable in settings set to 'Europe/Prague' and also USE_TZ set to True. I also have some data stored in Example:
id timestamp
1 2012-07-27T00:00:00+02:00
2 2018-03-11T02:00:00+01:00
3 2013-11-04T14:08:40+01:00
This is what I'm trying to achieve:
Extract all dates from those data
Filter those data date by date and perform some other action on them
For extracting dates I use either Example.dates('timestamp', 'day') or Example.annotate(date=TruncDay('timestamp')).values('date').
Now here is the difference: for first object from example above (with timestamp=2012-07-27T00:00:00+02:00), date returned by first approach is 2012-07-27, whereas for second approach it is 2012-07-26.
I would like filter to be timezone aware, so I'm currently sticking with the first one.
For filtering I am using Example.filter(timestamp__date=date). And there's a problem - it seems that __date filters by date converted to UTC. For date 2012-07-27 it returns empty QuerySet and for 2012-07-26 it returns first object.
Is there any way to achieve filtering by timezone aware date?
how can I translate query like this in django orm?
select id, year, month
where (year*100 + month) between 201703 and 201801
Thanks in advance for any help
You can first create an annotation, and then filter on that:
from django.db.models import F
(Modelname.objects
.annotate(yearmonth=F('year')*100+F('month'))
.filter(yearmonth__range=(201703, 201801)))
So here we construct an annotation yearmonth (you can use another name if you like), and make it equal to the year column times 100 plus the month column. Next we can filter on that annotation, and do so by specifying a __range here with two bounds.
Normally this will work for any database system that performs the operations you here perform (multiplying a column with a constant number, adding two values together), as well as do the __range filter (in MySQL this is translated into <var> BETWEEN <min> AND <max>). Since we use Django ORM however, if we later decide to use another database, the query will be translated in the other database query language (given of course that is possible).
How about using something similar to this.
Did you try filter and __range
Created_at will be the field in your DB
ModelName.objects.filter(created_at__range=(start_date, end_date))
Later you can do calculation in your view this is just a workaround.
If you want to run the exactly same query then probably you can run using.
ModelName.objects.raw("select id, year, month
where (year*100 + month) between 201703 and 201801")
Why doesn't Django in SQL table django_admin_log use TIMESTAMP instead of DATETIME for attribute action_time? I mean according to this answer it should be using TIMESTAMP.
LogEntry's action_time field is a models.DateTimeField, which (perhaps unsurprisingly) is stored as a DATETIME field.
I think you've misunderstood the answer you linked - it recommends using DATETIME, not TIMESTAMP:
If you meant that you want to decide between using a UNIX timestamp or a native MySQL datetime field, go with the native format. You can do calculations within MySQL that way ("SELECT DATE_ADD(my_datetime, INTERVAL 1 DAY)") and it is simple to change the format of the value to a UNIX timestamp ("SELECT UNIX_TIMESTAMP(my_datetime)") when you query the record if you want to operate on it with PHP.
I have used this query in my view..
temp2=transaction.objects.filter(user_id=client_obj,Date_of_trans.date()=rec_var1)[0].Trans_Amount
I need to compare a datetime field present in my model named Date_of_trans to a variable received from user but the code is not working... what query should be written?
This is basically a sql query. So you should think like that. How can you do this in sql. I mean what condition will you apply in sql. For finding records of particular date with datetime field you should check records between start of the day to end of the day.
try this
from datetime import datetime, time
temp2=transaction.objects.filter(user_id=client_obj,Date_of_trans>datetime.combine(rec_var1, time(0,0,0)), Date_of_trans <= datetime.combine(rec_var1, time(23,59,59)) )[0].Trans_Amount
The above code is written taking into consideration that rec_var1 is a date() object.
Here you check all transactions between start of the day, till end of the day. I think this will resolve your problem.
I've use datetime.combine function which combines date and time object to form datetime object which is required here.
Thanks
I encountered a model like this:
class Task(models.Model):
timespan = models.IntegerField(null=True, blank=True)
class Todo(models.Model):
limitdate = models.DateTimeField(null=True, blank=True)
task = models.ForeignKey(Task)
I need to extract all Todos with a limitdate that is lower or equal to today's date + a timespan defined in the related Task model.
Something like (dummy example):
today = datetime.datetime.now()
Todo.objects.filter(limitdate__lte=today + F('task__timespan'))
Now, I can do that with a loop but I'm looking for a way to do it with F(), and I can't find one.
I'm starting to wonder if I can do that with F(). Maybe I should use extra ?
Please note that I don't have the luxury of changing the model code.
The main issue is that DB does not support date + integer and its hard to write ORM query to date + integer::interval, for PostgreSQL for example, where integer is the value of the task_timespan column, in days count.
However, as
limitdate <= today + task__timespan equals to
limitdate - today <= task__timespan
We could transform the query to
Todo.objects.filter(task__timespan__gte=F('limitdate') - today).distinct()
thus the SQL becomes something like integer >= date - date, that should work in PostgreSQL because date - date outputs interval which could be compared w/ integer days count.
In other DBs such as SqLite, it's complicated because dates need to be cast w/ julianday() at first...and I think you need to play w/ extra() or even raw() to get the correct SQL.
Also, as Chris Pratt suggests, if you could use timestamp in all relative fields, the query task might become easier because of less limited add and subtract operations.
P.S. I don't have env to verify it now, you could try it first.
The problem is that there's no TIMESPAN type on a database. So, F cannot return something that you can actually work with in this context. I'm not sure what type of field you actually used in your database, but the only way I can think of to do this is to the store the timespan as an integer consisting of seconds, add that to "today" as a timestamp, and then convert it back into a datetime which you can use to compare with limitdate. However, I'm unsure if Django will accept such complex logic with F.