My initial question was here and was related to the postgres backend. Postgres subquery, ordering by subquery
Now my problem has moved onwards to the Django ORM layer. I essentially want to order a query by a postgres function ('idx', taken from the above stackoverflow work)
I've gone through trying to use model.objects.extra(order_by ) or simply order_by but I believe both of these need the order_by parameter to be an attribute or a field known to Django.
I'm trying to think how to solve this without having to revert to using an entirely raw SQL query through a model manager.
You can use extra to add the function result to your query and then order by it. Something like:
MyModel.objects.extra(select={'idx': 'idx(foo, bar)'}, order_by=['idx'])
Related
How do I batch update a field with a function in Django and The function argument is the field value . such as
Data.objects.filter(need_add=1).update(need_add=Myfunc(F('need_add')))
You can't. .update does its work at the SQL query level, i.e. inside the Database engine. Unless you can encode your function in SQL, you have to fetch the objects, update them, and save them. There is .bulk_update (link) as an optimization compared to a simple iteration over a queryset saving objects one-by-one.
Database functions here
I have a concern with django subqueries using the django ORM. When we fetch a queryset or perform a DB operation, I have the option of bypassing all assumptions that django might make for the database that needs to be used by forcing usage of the specific database that I want.
b_det = Book.objects.using('some_db').filter(book_name = 'Mark')
The above disregards any database routers I might have set and goes straight to 'some_db'.
But if my models approximately look like so :-
class Author(models.Model):
author_name=models.CharField(max_length=255)
author_address=models.CharField(max_length=255)
class Book(models.Model):
book_name=models.CharField(max_length=255)
author=models.ForeignKey(Author, null = True)
And I fetch a QuerySet representing all books that are called Mark like so:-
b_det = Book.objects.using('some_db').filter(book_name = 'Mark')
Then later if somewhere in the code I trigger a subquery by doing something like:-
if b_det:
auth_address = b_det[0].author.author_address
Then this does not make use of the original database 'some_db' that I had specified early on for the main query. This again goes through the routers and picks up (possibly) the incorrect database.
Why does django do this. IMHO , if I had selected forced usage of database for the original query then even for the subquery the same database needs to be used. Why must the database routers come into picture for this at all?
This is not a subquery in the strict SQL sense of the word. What you are actually doing here is to execute one query and use the result of that to find related items.
You can chain filters and do lots of other operations on a queryset but it will not be executed until you take a slice on it or call .values() but here you are actually taking a slice
auth_address = b_det[0].#rest of code
So you have a materialized query and you are now trying to find the address of the related author and that requires another query but you are not using with so django is free to choose which database to use. You cacn overcome this by using select_related
I have a query like: Foo.objects.extra(select={'extra_column':'SELECT ...'}), and I want to filter on the extra_column, but I cannot get this alias on filter. I am thinking of using annotate(), but I don't know annotate() can using on selectclause, all the tutorial is on SUM etc. I also tried HAVING clause, but it seems that Django doesn't support HAVING on extra().
Any suggestions will be very appreciated.
Edit:
Django 1.6, MySQL
I am query on some complicated condition, so I must using extra() to write the subquery. But I cannot using filter() to access the alias of the subquery. If I turned to annotate(), which can be accessed in later filter(), but seems that it only works with SUM, MAX, MIN, AVG, thus I cannot use it on my custom SELECT subquery.
I read the similar question here: Django ORM: Filter by extra attribute. It said that when query on a alias on MySQL, you need to use HAVING clause.
I don't know if this issue is also on other SQL backends like PostgreSQL, or on higher version of Django. Maybe there is no built-in solution on this question in Django1.6 with MySQL?
The as the correct marked answer, of the similar question, mentions; it is not possible to use a filter (because filter() only inspects model definitions). It suggest to use another extra, containing a where clause.
So something like;
Foo.objects\
.extra(select={'extra_column':'SELECT ...'})\
.extra(where=["extra_column = %s"], params=[value])
As of Django 1.7 you can work with lookups:
https://docs.djangoproject.com/en/1.7/howto/custom-lookups/
I started investigating why my Django Model.objects.filter(condition = variable).order_by(textcolumn) queries do not yield objects in correct order. And found out that it is database (Postgresql) issue.
In my earlier question (Postgresql sorting language specific characters (collation)) i figured out (with a lot of help from zero323 in actually getting it to work) that i can specify collation per database query like this:
SELECT nimi COLLATE "et_EE" FROM test ORDER BY nimi ASC;
But as much as i can see, order_by only accepts field names as arguments.
I was wondering, that if it is somehow possible to extend that functionality to include also the collation parameter? Is it possible to hack it in somehow using mixins or whatnot? Or is feature request the only way to do this right now?
I wish it would work something like this:
Model.objects.filter(condition = variable).order_by(*fieldnames, collation = 'et_EE')
Edit1:
Apparently im not the only one to ask for this:
https://groups.google.com/forum/#!msg/django-developers/0iESVnawNAY/JefMfAm7nQMJ
Alan
As #olau menioned in the comment, since Django 3.2 Collate utility is available. For older Django versions see the original information below the following code sample:
# Starting with Django 3.2:
Test.objects.order_by(Collate('nimi', 'et_EE'))
Since Django 1.8 order_by() accepts not only field names but also query expressions.
In another answer I gave an example of how you can override the default collation for a column. The useful query expression here is Func(), which you may subclass or use directly:
nimi_et = Func(
'nimi',
function='et_EE',
template='(%(expressions)s) COLLATE "%(function)s"')
Test.objects.order_by(nimi_et.asc())
Yet, note that the resulting SQL will be more like:
SELECT nimi FROM test ORDER BY nimi COLLATE "et_EE" ASC;
That is, the collation is overridden in ORDER BY clause rather than in SELECT clause. However, if you need to use it in a WHERE clause, you can use Func() in annotate().
allright. It seems that right now the Raw Queries are the only way to do this.
But There is django ticket open which will hopefully be closed/resolved sometime soon.
everybody.
I work with Django 1.3 and Postgres 9.0. I have very complex sql query which extends simple model table lookup with some extra fields. And it wrapped in table function since it is parameterized.
A month before I managed to make it work with the help of raw query but RawQuerySet lacks a lot of features which I really need (filters, count() and clone() methods, chainability).
The idea looks simple. QuerySet lets me to perform this query:
SELECT "table"."field1", ... ,"table"."fieldN" FROM "table"
whereas I need to do this:
SELECT "table"."field1", ... ,"table"."fieldN" FROM proxy(param1, param2)
So the question is: How can I do this? I've already started to create custom manager but can't substitute model.db_table with custom string (because it's being quoted and database stops to recognize the function call).
If there's no way to substitute table with function I would like to know if I can create QuerySet from RawQuerySet (Not the cleanest solution but simple RawQuerySet brings so much pain in... one body part).
If your requirement is that you want to use Raw SQL for efficiency purposes and still have access to model methods, what you really need is a library to map what fields of the SQL are the columns of what models.
And that library is, Unjoinify