Using Django model syntax, if I do this:
ThatModel.objects.filter(
last_datetime__lte=now + datetime.timedelta(seconds=F("interval")))
I get:
TypeError: unsupported type for timedelta days component: ExpressionNode
Is there a way to make this work with pure Django syntax (and not parsing all the results with Python)?
Just avoid timedelta's F-ignorance
filter knows about F, but timedelta does not.
The trick is to keep the F out of the timedelta argument list:
ThatModel.objects.filter(
last_datetime__lte=now + datetime.timedelta(seconds=1)*F("interval"))
This will work with PostgreSQL, but, alas, not with SQlite.
From django docs:
Django provides F expressions to allow such comparisons. Instances
of F() act as a reference to a model field within a query. These
references can then be used in query filters to compare the values of
two different fields on the same model instance.
That means you can use F() for comparing within queries. F() returns reference so when you use it as parameter for timedelta object, you get the error ExpressionNode. You can check the documentation. You might check the source code of F()
For your solution, you can check this: DateModifierNode, or just save the value of interval elsewhere and then pass it as parameter of timedelta.
Related
I have some table ports(switch_ip, slot_number, port_number, many, more, columns) and would like to achieve the following PostgreSQL query using Django:
SELECT switch_ip, array_agg((slot_number, port_number, many, more, columns) ORDER BY slot_number, port_number) info
FROM ports
GROUP BY switch_ip
ORDER BY switch_ip
Using django.contrib.postgres.aggregates here's what I got so far:
Port.objects \
.values('switch_ip') \
.annotate(
info=ArrayAgg('slot_number', ordering=('slot_number', 'port_number'))
) \
.order_by('switch_ip')
I am unable to include more than one column in the ArrayAgg. None of ArrayAgg(a, b, c), ArrayAgg((a, b, c)), ArrayAgg([a, b, c]) seem to work. A workaround could involve separate ArrayAggs for each column and each with the same ordering. I would despise this because I have many columns. Is there any nicer workaround, possibly more low-level?
I suspect this is no issue with ArrayAgg itself but rather with tuple expressions in general. Is there any way to have tuples at all in Django queries? For example, what would be the corresponding Django of:
SELECT switch_ip, (slot_number, port_number, many, more, columns) info
FROM ports
If this is not yet possible in Django, how feasible would it be to implement?
I have spent lot of time searching for a working solution and here is a full recipe with code example.
You need to define Array "function" with square brackets in template
from django.db.models.expressions import Func
class Array(Func):
template = '%(function)s[%(expressions)s]'
function = 'ARRAY'
You need to define output field format (it must be array of some django field). For example an array of strings
from django.contrib.postgres.fields import ArrayField
from django.db.models.fields import CharField
out_format = ArrayField(CharField(max_length=200))
Finally make an ArrayAgg expression
from django.db.models import F
annotate = {'2-fields': ArrayAgg(Array(F('field1'), F('field2'), output_field=out_format), distinct=True) }
model.objects.all().annotate(**annotate)
(Optional) If field1 or field2 are not CharFields, you may include Cast as an argument of Array
from django.db.models.functions import Cast
annotate = {'2-fields': ArrayAgg(Array(Cast(F('field1'), output_field=CharField(max_length=200)), F('field2'), output_field=out_format), distinct=True) }
Having done a bit more research I guess one could add the missing tuple functionality as follows:
Create a new model field type named TupleField. The implementation might look kind of similar to django.contrib.postgres.fields.ArrayField. TupleField would be rather awkward because I don't think any RDBMS allows for composite types to be used as column types so usage of TupleField would be limited to (possibly intermediate?) query results.
Create a new subclass of django.db.models.Expression which wraps multiple expressions on its own (like Func in general, so looking at Func's implementation might be worthwile) and evaluates to a TupleField. Name this subclass TupleExpression for example.
Then I could simply annotate with ArrayAgg(TupleExpression('slot_number', 'port_number', 'many', 'more', 'columns'), ordering=('slot_number', 'port_number')) to solve my original problem. This would annotate each switch_ip with correctly-ordered arrays of tuples where each tuple represents one switch port.
I have a rather complex query that's generating a Django RawQuerySet. This specific query returns some fields that aren't part of the model that the RawQuerySet is based on, so I'm using .annotate(field_name=models.Value('field_name')) to attach it as an attribute to individual records in the RawQuerySet. The most important custom field is actually a uuid, which I use to compose URLs using Django's {% url %} functionality.
Here's the problem: I'm not using standard uuids inside my app, I'm using SmallUUIDs (compressed UUIDs.) These are stored in the database as native uuidfields then converted to shorter strings in python. So I need to somehow convert the uuid returned as part of the RawQuerySet to a SmallUUID for use inside a template to generate a URL.
My code looks somewhat like this:
query = "SELECT othertable.uuid_field as my_uuid FROM myapp_mymodel
JOIN othertable o ON myapp_mymodel.x = othertable.x"
MyModel.objects.annotate(
my_uuid=models.Value('my_uuid'),
).raw(query)
Now there is a logical solution here, there's an optional kwarg for models.Value called output_field, making the code look like this:
MyModel.objects.annotate(
my_uuid=models.Value('my_uuid', output_field=SmallUUIDField()),
).raw(query)
But it doesn't work! That kwarg is completely ignored and the type of the attribute is based on the type returned from the database and not what's in output_field. In my case, I'm getting a uuid output because Postgres is returning a UUID type, but if I were to change the query to SELECT cast othertable.uuid_field as text) as my_uuid I'd get the attribute in the format of a string. It appears that Django (at least version 1.11.12) doesn't actually care what is in that kwarg in this instance.
So here's what I'm thinking are my potential solutions, in no particular order:
Change the way the query is formatted somehow (either in Django or in the SQL)
Change the resulting RawQuerySet in some way before it's passed to the view
Change something inside the templates to convert the UUID to a smalluuid for use in the URL reverse process.
What's my best next steps here?
A couple of issues with your current approach:
Value() isn't doing what you think it is - your annotation is literally just annotating each row with the value "my_uuid" because that is what you have passed to it. It isn't looking up the field of that name (to do that you need to use F expressions).
Point 1 above doesn't matter anyway because as soon as you use raw() then the annotation is ignored - which is why you see no effect coming from it.
Bottom line is that trying to annotate a RawQuerySet isn't going to be easy. There is a translations argument that it accepts, but I can't think of a way to get that to work with the type of join you are using.
The next best suggestion that I can think of is that you just manually convert the field into a SmallUUID object when you need it - something like this:
from smalluuid import SmallUUID
objects = MyModel.objects.raw(query)
for o in objects:
# Take the hex string obtained from the database and convert it to a SmallUUID object.
# If your database has a built-in UUID type you will need to do
# SmallUUID(small=o.my_uuid) instead.
my_uuid = SmallUUID(hex=o.my_uuid)
(I'm doing this in a loop just to illustrate - depending on where you need this you can do it in a template tag or view).
I am using Django, with mongoengine. I have a model Classes with an inscriptions list, And I want to get the docs that have an id in that list.
classes = Classes.objects.filter(inscriptions__contains=request.data['inscription'])
Here's a general explanation of querying ArrayField membership:
Per the Django ArrayField docs, the __contains operator checks if a provided array is a subset of the values in the ArrayField.
So, to filter on whether an ArrayField contains the value "foo", you pass in a length 1 array containing the value you're looking for, like this:
# matches rows where myarrayfield is something like ['foo','bar']
Customer.objects.filter(myarrayfield__contains=['foo'])
The Django ORM produces the #> postgres operator, as you can see by printing the query:
print Customer.objects.filter(myarrayfield__contains=['foo']).only('pk').query
>>> SELECT "website_customer"."id" FROM "website_customer" WHERE "website_customer"."myarrayfield_" #> ['foo']::varchar(100)[]
If you provide something other than an array, you'll get a cryptic error like DataError: malformed array literal: "foo" DETAIL: Array value must start with "{" or dimension information.
Perhaps I'm missing something...but it seems that you should be using .filter():
classes = Classes.objects.filter(inscriptions__contains=request.data['inscription'])
This answer is in reference to your comment for rnevius answer
In Django ORM whenever you make a Database call using ORM, it will generally return either a QuerySet or an object of the model if using get() / number if you are using count() ect., depending on the functions that you are using which return other than a queryset.
The result from a Queryset function can be used to implement further more refinement, like if you like to perform a order() or collecting only distinct() etc. Queryset are lazy which means it only hits the database when they are actually used not when they are assigned. You can find more information about them here.
Where as the functions that doesn't return queryset cannot implement such things.
Take time and go through the Queryset Documentation more in depth explanation with examples are provided. It is useful to understand the behavior to make your application more efficient.
I'm trying to achive an Aggregation Query and that's my code:
TicketGroup.objects.filter(event=event).aggregate(
total_group=Sum(F('total_sold')*F('final_price')))
I have 'total_sold' and 'final_price' in TicketGroup object and all what I want to do is sum and multiply values to get the total sold of all TicketGroups together.
All I get is this error:
Expression contains mixed types. You must set output_field
What I am doing wrong, since I'm calling 'total_group' as my output field?
Thanks!
By output_field Django means to provide field type for the result of the Sum.
from django.db.models import FloatField, F
total_group=Sum(F('total_sold')*F('final_price'), output_field=FloatField())
should do the trick.
I had to use something different in order to make my query work. Just output_field wont solve it. I needed a simple division between two aliases. These are output of two annotations.
from django.db.models import FloatField, ExpressionWrapper, F
distinct_people_with_more_than_zero_bill = Task.objects.filter(
billable_efforts__gt=0).values('report__title').annotate(
Count('assignee', distinct=True)).annotate(
Sum('billable_efforts'))
annotate(yy=ExpressionWrapper(F('billable_efforts__sum') / F('assignee__count'), output_field=FloatField()))
The key here is ExpressionWrapper.
Without this, you will get an error: received non-expression(s)
The hint came for Django documentation itself, which says:
If the fields that you’re combining are of different types you’ll need
to tell Django what kind of field will be returned. Since F() does not
directly support output_field you will need to wrap the expression
with ExpressionWrapper
Link: https://docs.djangoproject.com/en/2.2/ref/models/expressions/
I have the following model:
class Process(models.Model):
title = models.Charfield(max_length=255)
date_up = models.DateTimeField(auto_now_add=True)
days_activation = models.PositiveSmallIntegerField(default=0)
Now I need to query for all Process objects that have expired, according to their value of days_activation.
I tried
from datetime import datetime, timedelta
Process.objects.filter(date_up__lte=datetime.now()-timedelta(days=F('days_activation')))
and received the following error message:
TypeError: unsupported type for timedelta days component: F
I can of course do it in Python:
filter (lambda x: x.date_up<=datetime.now() - timedelta(days=x.days_activation),
Process.objects.all ()),
but I really need to produce a django.db.models.query.QuerySet.
7 days == 1 day * 7
F is deep-black Django magic and the objects that encounter it
must belong to the appropriate magical circles to handle it.
In your case, django.db.models.query.filter knows about F, but datetime.timedelta does not.
Therefore, you need to keep the F out of the timedelta argument list.
Fortunately, multiplication of timedelta * int is supported by F,
so the following can work:
Process.objects.filter(date_up__lte=datetime.now()-timedelta(days=1)*F('days_activation'))
As it turns out, this will work with PostgreSQL, but will not work with SQlite (for which Django 1.11 only supports + and - for timedelta,
perhaps because of a corresponding SQlite limitation).
You are mixing two layers: run-time layer and the database layer. F function is just a helper which allows you to build slightly more complex queries with django ORM. You are using timedelta and Ftogether and expecting that django ORM will be smart enough to convert these things to raw SQL, but it can't, as I see. Maybe I am wrong and do not know something about django ORM.
Anyway, you can rewrite you ORM call with extra extra and build the WHERE clause manually using native SQL functions which equals to datetime.now() and timedelta.
You have to extend Aggregate. Do like below:
from django.db import models as DM
class BaseSQL(object):
function = 'DATE_SUB'
template = '%(function)s(NOW(), interval %(expressions)s day)'
class DurationAgr(BaseSQL, DM.Aggregate):
def __init__(self, expression, **extra):
super(DurationAgr, self).__init__(
expression,
output_field=DM.DateTimeField(),
**extra
)
Process.objects.filter(date_up__lte=DurationAgr('days_activation'))
Hopefully, It will work for you. :)
I tried to use solution by Lutz Prechelt above, but got MySQL syntax error.
It's because we can't perform arithmetic operations with INTERVAL in MySQL.
So, for MySQL my solution is create a custom DB function:
class MysqlSubDate(Func):
function = 'SUBDATE'
output_field = DateField()
Example of usage:
.annotate(remainded_days=MysqlSubDate('end_datetime', F('days_activation')))
Also you can use timedelta, it will be converted into INTERVAL
.annotate(remainded_days=MysqlSubDate('end_datetime', datetime.timedelta(days=10)))