this is a simple question. I'd like to know if it is the same to write:
queryset = Model.objects.filter(field=None)
than:
queryset = Model.objects.filter(field__isnull=True)
I'm using django 1.8
They are equal:
>>> str(Person.objects.filter(age__isnull=True).query) == str(Person.objects.filter(age=None).query)
True
>>> print(Person.objects.filter(age=None).query)
SELECT "person_person"."id", "person_person"."name", "person_person"."yes", "person_person"."age" FROM "person_person" WHERE "person_person"."age" IS NULL
>>> print(Person.objects.filter(age__isnull=True).query)
SELECT "person_person"."id", "person_person"."name", "person_person"."yes", "person_person"."age" FROM "person_person" WHERE "person_person"."age" IS NULL
Exclusion: the Postgres JSON field (see the answer of #cameron-lee)
It depends on the type of field. As mentioned in other answers, they are usually equivalent but in general, this isn't guaranteed.
For example, the Postgres JSON field uses =None to specify that the json has the value null while __isnull=True means there is no json:
https://docs.djangoproject.com/en/3.0/ref/contrib/postgres/fields/#jsonfield
Just to keep in mind that you cannot reverse the condition with your first solution:
# YOU CANNOT DO THIS
queryset = Model.objects.filter(field!=None)
However you can do this:
queryset = Model.objects.filter(field__isnull=False)
Related
Is it possible to filter a Django queryset by model property?
i have a method in my model:
#property
def myproperty(self):
[..]
and now i want to filter by this property like:
MyModel.objects.filter(myproperty=[..])
is this somehow possible?
Nope. Django filters operate at the database level, generating SQL. To filter based on Python properties, you have to load the object into Python to evaluate the property--and at that point, you've already done all the work to load it.
I might be misunderstanding your original question, but there is a filter builtin in python.
filtered = filter(myproperty, MyModel.objects)
But it's better to use a list comprehension:
filtered = [x for x in MyModel.objects if x.myproperty()]
or even better, a generator expression:
filtered = (x for x in MyModel.objects if x.myproperty())
Riffing off #TheGrimmScientist's suggested workaround, you can make these "sql properties" by defining them on the Manager or the QuerySet, and reuse/chain/compose them:
With a Manager:
class CompanyManager(models.Manager):
def with_chairs_needed(self):
return self.annotate(chairs_needed=F('num_employees') - F('num_chairs'))
class Company(models.Model):
# ...
objects = CompanyManager()
Company.objects.with_chairs_needed().filter(chairs_needed__lt=4)
With a QuerySet:
class CompanyQuerySet(models.QuerySet):
def many_employees(self, n=50):
return self.filter(num_employees__gte=n)
def needs_fewer_chairs_than(self, n=5):
return self.with_chairs_needed().filter(chairs_needed__lt=n)
def with_chairs_needed(self):
return self.annotate(chairs_needed=F('num_employees') - F('num_chairs'))
class Company(models.Model):
# ...
objects = CompanyQuerySet.as_manager()
Company.objects.needs_fewer_chairs_than(4).many_employees()
See https://docs.djangoproject.com/en/1.9/topics/db/managers/ for more.
Note that I am going off the documentation and have not tested the above.
Looks like using F() with annotations will be my solution to this.
It's not going to filter by #property, since F talks to the databse before objects are brought into python. But still putting it here as an answer since my reason for wanting filter by property was really wanting to filter objects by the result of simple arithmetic on two different fields.
so, something along the lines of:
companies = Company.objects\
.annotate(chairs_needed=F('num_employees') - F('num_chairs'))\
.filter(chairs_needed__lt=4)
rather than defining the property to be:
#property
def chairs_needed(self):
return self.num_employees - self.num_chairs
then doing a list comprehension across all objects.
I had the same problem, and I developed this simple solution:
objects = [
my_object
for my_object in MyModel.objects.all()
if my_object.myProperty == [...]
]
This is not a performatic solution, it shouldn't be done in tables that contains a large amount of data. This is great for a simple solution or for a personal small project.
PLEASE someone correct me, but I guess I have found a solution, at least for my own case.
I want to work on all those elements whose properties are exactly equal to ... whatever.
But I have several models, and this routine should work for all models. And it does:
def selectByProperties(modelType, specify):
clause = "SELECT * from %s" % modelType._meta.db_table
if len(specify) > 0:
clause += " WHERE "
for field, eqvalue in specify.items():
clause += "%s = '%s' AND " % (field, eqvalue)
clause = clause [:-5] # remove last AND
print clause
return modelType.objects.raw(clause)
With this universal subroutine, I can select all those elements which exactly equal my dictionary of 'specify' (propertyname,propertyvalue) combinations.
The first parameter takes a (models.Model),
the second a dictionary like:
{"property1" : "77" , "property2" : "12"}
And it creates an SQL statement like
SELECT * from appname_modelname WHERE property1 = '77' AND property2 = '12'
and returns a QuerySet on those elements.
This is a test function:
from myApp.models import myModel
def testSelectByProperties ():
specify = {"property1" : "77" , "property2" : "12"}
subset = selectByProperties(myModel, specify)
nameField = "property0"
## checking if that is what I expected:
for i in subset:
print i.__dict__[nameField],
for j in specify.keys():
print i.__dict__[j],
print
And? What do you think?
i know it is an old question, but for the sake of those jumping here i think it is useful to read the question below and the relative answer:
How to customize admin filter in Django 1.4
It may also be possible to use queryset annotations that duplicate the property get/set-logic, as suggested e.g. by #rattray and #thegrimmscientist, in conjunction with the property. This could yield something that works both on the Python level and on the database level.
Not sure about the drawbacks, however: see this SO question for an example.
I have a queryset like:
qs = MyModel.objects.filter(name='me').values_list('activation_date')
here activation_date is DateTimeField in models.
When I download excel sheet from this qs I am not getting activation date in string format.
How can I convert this field('activation_date') in string or how to typecast it in qs?
https://docs.djangoproject.com/en/2.2/ref/models/fields/#datetimefield
A date and time, represented in Python by a datetime.datetime instance.
You can get a string representation of a DateTimeField casting it directly:
str(obj)
# obj = qs[0][0] ? or qs[0][1] ?
You'll get result like this (in this example I use datetime.datetime.now() since a DateTimeField is represented by datetime.datetime is the same behavior):
>>> now = datetime.datetime.now()
>>> str(now)
'2013-06-26 00:14:26.260524'
if you want less information or formatted in other mode you can use strftime() function for format them. see:
>>> now.strftime('%Y-%m-%d %H:%M')
'2013-06-26 00:14'
extra() is an old API that Django aims to deprecate at some point in the future. I would avoid using it.
Try the following instead:
from django.db.models import F, Func, Value, CharField
qs.annotate(
formatted_date=Func(
F('date'),
Value('dd.MM.yyyy hh:mm'),
function='to_char',
output_field=CharField()
)
)
This works only with a database that supports the to_char date type formatting function. Postgres provides this function by default.
If you use a MSSQL backend you could swap to_char with FORMAT.
For MySQL use DATE_FORMAT.
For Oracle consult their
documentation, etc.
After the queryset is evaluated this will add the annotation formatted_date to each object in the queryset that is returned.
extra is deprecated in Django 2.0
That's why I think the best solution to get a stringified datetime is:
foo_bar = FooBarModel.objects.annotate(
str_datetime=Cast(
TruncSecond('some_datetime_field', DateTimeField()), CharField()
)
).values('str_datetime').first()
The result is:
foo_bar.str_datetime:
(str)'2014-03-28 15:36:55'
Also I'd like to mention that you can format it as well in any way you want like:
from django.db.models import Value
foo_bar = FooBarModel.objects.annotate(
day=Cast(ExtractDay('some_datetime_field'), CharField()),
hour=Cast(ExtractHour('some_datetime_field'), CharField()),
str_datetime=Concat(
Value('Days: '), 'day', Value(' Hours: '), 'hour',
output_field=CharField()
)
).values('str_datetime').first()
The result is:
foo_bar.str_datetime:
(str)'Days: 28 Hours: 15'
If you are using Postgres, you can do it like this (date format options here). The solution is database dependent, but it sure beats looping though a long list in Python land after your perform the query.
qs = MyModel.objects.filter(name='me')
qs = qs.extra(select={'datestr':"to_char(activation_date, 'YYYY-MM-DD HH24:MI:SS')"})
qs = qs.values_list('datestr')
I am sure MySQL has some equivalent function as Postgres's to_char, but you'll have to find that on your own as I am not a MySQL guy.
Very surprised to see that no one suggested the cast to a simple TextField (note, I'm using Postgres so I can't confirm for other RDBMSes):
from django.db.models.functions import Cast
from django.db.models import TextField
queryset = FooBarModel.objects.values(my_datetime=Cast('some_datetime_field', TextField()))
foo_bar = queryset.first()
foo_bar['my_datetime']
>>> u'2019-10-03 17:59:37.979578+00'
It similarly also works fine for nested fields:
queryset = FooBarModel.objects.values(Cast('baz__some_datetime_field', TextField()))
Alternatively, a custom Func can also be used (also specific to Postgres here, but can be modified for any other RDBMS):
class FullDateTimeCast(Func):
"""
Coerce an expression to a new field type.
"""
function = 'TO_CHAR'
template = '%(function)s(%(expressions)s, \'FMDay, Month DD, YYYY at HH12:MI:SS AM\')'
queryset = FooBarModel.objects.values(my_datetime=FullDateTimeCast('some_datetime_field', TextField()))
foo_bar = queryset.first()
foo_bar['my_datetime']
>>> u' Thursday, October 03, 2019 at 17:59:37 PM'
qs = MyModel.objects.filter(name='me')
qs = qs.extra(select={'datestr':"DATE_FORMAT(activation_date, '%Y-%m-%d')"})
qs = qs.values_list('datestr')
You can also convert the date in queryset to string using map function. Example:
qs = MyModel.objects.filter(name='me').values_list('activation_date', flat=True)
data = map(str, qs)
I did it this way
.annotate(date_str=ExpressionWrapper(
Func(F('date'), Value('%d/%m/%Y %H:%i'), function='DATE_FORMAT'), output_field=CharField()
))
If you are doing this once, refer to Yannic Hamann's answer. However if you find yourself converting to str from the database a lot, you can define the Func as a class to avoid having to type output_field and function a bunch.
class DateToChar(models.Func):
"""
Custom Func expression to convert datetimes to str's in database query
Params for initializer
------
expression_1
expression resulting in a date: ex: F('date')
expression_2
Format string as an expression: Value('YYYY-MM-DD')
"""
arity = 2
function = 'to_char'
output_field = models.CharField()
Note that function will change depending on the database backend. This is written for Postgres.
This can be used like
qs = qs.annotate(date_str=DateToChar(F('date'), Value('YYYY-MM-DD'))
or with any expression that results in a date/datetime/Integer/Float/Decimal field (See Postgres to_char. Varies by database) and an expression resulting in a CharField or TextField.
See Func documentation for more information
I had a similar issue then I solved this in the following way:
from django.db.models.functions import Substr
list( Model.objects.values('when_date').annotate(date= Substr('when_date',1,10), total=Sum('amount')) )
(Django 4.1, python 3.10)
I want to do pretty much the same like in this ticket at djangoproject.com, but with some additonal formatting. From this query
>>> MyModel.objects.values('cryptic_value_name')
[{'cryptic_value_name': 1}, {'cryptic_value_name': 2}]
I want to get something like that:
>>> MyModel.objects.values(renamed_value='cryptic_value_name')
[{'renamed_value': 1}, {'renamed_value': 2}]
Is there another, more builtin way or do I have to do this manually?
From django>=1.8 you can use annotate and F object
from django.db.models import F
MyModel.objects.annotate(renamed_value=F('cryptic_value_name')).values('renamed_value')
Also extra() is going to be deprecated, from the django docs:
This is an old API that we aim to deprecate at some point in the future. Use it only if you cannot express your query using other queryset methods. If you do need to use it, please file a ticket using the QuerySet.extra keyword with your use case (please check the list of existing tickets first) so that we can enhance the QuerySet API to allow removing extra(). We are no longer improving or fixing bugs for this method.
Without using any other manager method (tested on v3.0.4):
from django.db.models import F
MyModel.objects.values(renamed_value=F('cryptic_value_name'))
Excerpt from Django docs:
An F() object represents the value of a model field or annotated
column. It makes it possible to refer to model field values and
perform database operations using them without actually having to pull
them out of the database into Python memory.
It's a bit hacky, but you could use the extra method:
MyModel.objects.extra(
select={
'renamed_value': 'cryptic_value_name'
}
).values(
'renamed_value'
)
This basically does SELECT cryptic_value_name AS renamed_value in the SQL.
Another option, if you always want the renamed version but the db has the cryptic name, is to name your field with the new name but use db_column to refer to the original name in the db.
I am working with django 1.11.6
( And the key:value pair is opposite to that of accepted answer )
This is how i am making it work for my project
def json(university):
address = UniversityAddress.objects.filter(university=university)
address = address.extra(select={'city__state__country__name': 'country', 'city__state__name': 'state', 'city__name': 'city'})
address = address.values('country', 'state', "city", 'street', "postal_code").get()
return address
Note that adding simultanous objects.filter().extra().values() is same as above.
Try passing as kwargs:
MyModel.objects.annotate(**{'A B C':F('profile_id')}).values('A B C')
In my case, there were spaces and other special characters included in the key of each value in the result set so this did the trick.
Its more than simple if you want to rename few fields of the mode.
Try
projects = Project.objects.filter()
projects = [{'id': p.id, 'name': '%s (ID:%s)' % (p.department, p.id)} for p in projects]
Here i do not have a name field in the table, but i can get that after tweaking a little bit.
I have a problem with Django queryset ordering.
My model contains a field named position, a PositiveSmallIntegerField which I'd like to used to order query results.
I use order_by('position'), which works great.
Problem : my position field is nullable (null=True, blank=True), because I don't wan't to specify a position for every 50000 instances of my model. When some instances have a NULL position, order_by returns them in the top of the list: I'd like them to be at the end.
In raw SQL, I used to write things like:
IF(position IS NULL or position='', 1, 0)
(see http://www.shawnolson.net/a/730/mysql-sort-order-with-null.html). Is it possible to get the same result using Django, without writing raw SQL?
You can use the annotate() from django agrregation to do the trick:
items = Item.objects.all().annotate(null_position=Count('position')).order_by('-null_position', 'position')
As of Django 1.8 you can use Coalesce() to convert NULL to 0.
Sample:
import datetime
from django.db.models.functions import Coalesce, Value
from app import models
# Coalesce works by taking the first non-null value. So we give it
# a date far before any non-null values of last_active. Then it will
# naturally sort behind instances of Box with a non-null last_active value.
the_past = datetime.datetime.now() - datetime.timedelta(days=10*365)
boxes = models.Box.objects.all().annotate(
new_last_active=Coalesce(
'last_active', Value(the_past)
)
).order_by('-new_last_active')
It's a shame there are a lot of questions like this on SO that are not marked as duplicate. See (for example) this answer for the native solution for Django 1.11 and newer. Here is a short excerpt:
Added the nulls_first and nulls_last parameters to Expression.asc() and desc() to control the ordering of null values.
Example usage (from comment to that answer):
from django.db.models import F
MyModel.objects.all().order_by(F('price').desc(nulls_last=True))
Credit goes to the original answer author and commenter.
Using extra() as Ignacio said optimizes a lot the end query. In my aplication I've saved more than 500ms (that's a lot for a query) in database processing using extra() instead of annotate()
Here is how it would look like in your case:
items = Item.objects.all().extra(
'select': {
'null_position': 'CASE WHEN {tablename}.position IS NULL THEN 0 ELSE 1 END'
}
).order_by('-null_position', 'position')
{tablename} should be something like {Item's app}_item following django's default tables name.
I found that the syntax in Pablo's answer needed to be updated to the following on my 1.7.1 install:
items = Item.objects.all().extra(select={'null_position': 'CASE WHEN {name of Item's table}.position IS NULL THEN 0 ELSE 1 END'}).order_by('-null_position', 'position')
QuerySet.extra() can be used to inject expressions into the query and order by them.
I have first_name, last_name & alias (optional) which I need to search for. So, I need a query to give me all the names that have an alias set.
Only if I could do:
Name.objects.filter(alias!="")
So, what is the equivalent to the above?
You could do this:
Name.objects.exclude(alias__isnull=True)
If you need to exclude null values and empty strings, the preferred way to do so is to chain together the conditions like so:
Name.objects.exclude(alias__isnull=True).exclude(alias__exact='')
Chaining these methods together basically checks each condition independently: in the above example, we exclude rows where alias is either null or an empty string, so you get all Name objects that have a not-null, not-empty alias field. The generated SQL would look something like:
SELECT * FROM Name WHERE alias IS NOT NULL AND alias != ""
You can also pass multiple arguments to a single call to exclude, which would ensure that only objects that meet every condition get excluded:
Name.objects.exclude(some_field=True, other_field=True)
Here, rows in which some_field and other_field are true get excluded, so we get all rows where both fields are not true. The generated SQL code would look a little like this:
SELECT * FROM Name WHERE NOT (some_field = TRUE AND other_field = TRUE)
Alternatively, if your logic is more complex than that, you could use Django's Q objects:
from django.db.models import Q
Name.objects.exclude(Q(alias__isnull=True) | Q(alias__exact=''))
For more info see this page and this page in the Django docs.
As an aside: My SQL examples are just an analogy--the actual generated SQL code will probably look different. You'll get a deeper understanding of how Django queries work by actually looking at the SQL they generate.
Name.objects.filter(alias__gt='',alias__isnull=False)
Firstly, the Django docs strongly recommend not using NULL values for string-based fields such as CharField or TextField. Read the documentation for the explanation:
https://docs.djangoproject.com/en/dev/ref/models/fields/#null
Solution:
You can also chain together methods on QuerySets, I think. Try this:
Name.objects.exclude(alias__isnull=True).exclude(alias="")
That should give you the set you're looking for.
1. When using exclude, keep the following in mind to avoid common mistakes:
Should not add multiple conditions into an exclude() block like filter(). To exclude multiple conditions, you should use multiple exclude().
Example: (NOT a AND NOT b)
Entry.objects.exclude(title='').exclude(headline='')
equal to
SELECT... WHERE NOT title = '' AND NOT headline = ''
======================================================
2. Only use multiple when you really know about it:
Example: NOT (a AND b)
Entry.objects.exclude(title='', headline='')
equal to
SELECT.. WHERE NOT (title = '' AND headline = '')
If you want to exclude null (None), empty string (""), as well as a string containing white spaces (" "), you can use the __regex along with __isnull filter option
Name.objects.filter(
alias__isnull = False,
alias__regex = r"\S+"
)
alias__isnull=False excludes all the columns null columns
aliax__regex = r"\S+" makes sure that the column value contains at least one or more non whitespace characters.
From Django 1.8,
from django.db.models.functions import Length
Name.objects.annotate(alias_length=Length('alias')).filter(alias_length__gt=0)
You can simply do this:
Name.objects.exclude(alias="").exclude(alias=None)
It's really just that simple. filter is used to match and exclude is to match everything but what it specifies. This would evaluate into SQL as NOT alias='' AND alias IS NOT NULL.
Another approach using a generic isempty lookup, that can be used with any field.
It can also be used by django rest_framework or other apps that use django lookups:
from distutils.util import strtobool
from django.db.models import Field
from django.db.models.lookups import BuiltinLookup
#Field.register_lookup
class IsEmpty(BuiltinLookup):
lookup_name = 'isempty'
prepare_rhs = False
def as_sql(self, compiler, connection):
sql, params = compiler.compile(self.lhs)
condition = self.rhs if isinstance(self.rhs, bool) else bool(strtobool(self.rhs))
if condition:
return "%s IS NULL or %s = ''" % (sql, sql), params
else:
return "%s <> ''" % sql, params
You can then use it like this:
Name.objects.filter(alias__isempty=False)
this is another simple way to do it .
Name.objects.exclude(alias=None)