I want to check if given field is SELECTed in given queryset. Assuming there is Queryset.has() method, this should run without AssertionError:
qs = Author.objects.all()
assert not qs.has('articles_count')
qs = qs.annotate(articles_count=Count('articles'))
assert qs.has('articles_count')
qs = qs.values('pk')
assert not qs.has('articles_count')
(assume model Author has no field articles_count by default)
I am late to the party, but wanted to share my solution if anyone needs it in the future :)
Columns are not present on the QuerySet, but you can check annotated columns on the Query:
authors = Author.objects.all().annotate(articles_count=Count('articles'))
assert 'articles_count' in authors.query.annotations
Or you could just get one author from QuerySet and check if it has required field:
author = Author.objects.annotate(articles_count=Count('articles')).first()
assert hasattr(author, 'articles_count')
I would prefer the second method since it's not diving deep into Django's internal system.
If you are planning to use complete QuerySet, but don't want to execute two queries you could evaluate QuerySet and then slice it like so:
authors = Author.objects.annotate(articles_count=Count('articles'))
bool(authors) # Just here to evaluate the queryset
assert hasattr(authors[0], 'articles_count') # Doesn't query the database again
I had a similar issue and used the try - except logic :
try:
queryset.columnname
except:
print("Nope, columnname is not in queryset")
Hope it helps somebody.
Cheers
You can try Python's hasattr method:
In [3]: hasattr(qs, "articles_count")
Out[3]: False
Or if you want to check this attr of every model instance:
all([hasattr(x, "articles_count") for x in qs])
Related
I have two models:
Model A is an AbstractUserModel and Model B
class ModelB:
user = ForeignKey(User, related_name='modelsb')
timestamp = DateTimeField(auto_now_add=True)
What I want to find is how many users have at least one ModelB object created at least in 3 of the 7 past days.
So far, I have found a way to do it but I know for sure there is a better one and that is why I am posting this question.
I basically split the query into 2 parts.
Part1:
I added a foo method inside the User Model that checks if a user meets the above conditions
def foo(self):
past_limit = starting_date - timedelta(days=7)
return self.modelsb.filter(timestamp__gte=past_limit).order_by('timestamp__day').distinct('timestamp__day').count() > 2
Part 2:
In the Custom User Manager, I find the users that have more than 2 modelsb objects in the last 7 days and iterate through them applying the foo method for each one of them.
By doing this I narrow down the iterations of the required for loop. (basically its a filter function but you get the point)
def boo(self):
past_limit = timezone.now() - timedelta(days=7)
candidates = super().get_queryset().annotate(rc=Count('modelsb', filter=Q(modelsb__timestamp__gte=past_limit))).filter(rc__gt=2)
return list(filter(lambda x: x.foo(), candidates))
However, I want to know if there is a more efficient way to do this, that is without the for loop.
You can use conditional annotation.
I haven't been able to test this query, but something like this should work:
from django.db.models import Q, Count
past_limit = starting_date - timedelta(days=7)
users = User.objects.annotate(
modelsb_in_last_seven_days=Count('modelsb__timestap__day',
filter=Q(modelsb__timestamp__gte=past_limit),
distinct=True))
.filter(modelsb_in_last_seven_days__gte = 3)
EDIT:
This solution did not work, because the distinct option does specify what field makes an entry distinct.
I did some experimenting on my own Django instance, and found a way to make this work using SubQuery. The way this works is that we generate a subquery where we make the distinction ourself.
counted_modelb = ModelB.objects
.filter(user=OuterRef('pk'), timestamp__gte=past_limit)
.values('timestamp__day')
.distinct()
.annotate(count=Count('timestamp__day'))
.values('count')
query = User.objects
.annotate(modelsb_in_last_seven_days=Subquery(counted_modelb, output_field=IntegerField()))
.filter(modelsb_in_last_seven_days__gt = 2)
This annotates each row in the queryset with the count of all distinct days in modelb for the user, with a date greater than the selected day.
In the subquery I use values('timestamp__day') to make sure I can do distinct() (Because a combination of distinct('timestamp__day') and annotate() is unsupported.)
django 2.0.2 python 3.4
models.py
Post(models.Model):
Id = pk
content = text
Reply(models.Model):
Id = pk
PostId = Fk(Post)
content = text
view.py
Post.objects.all().annotate(lastreply=F("Reply__content__last"))
can use last query in F() ?
As far as I know, latest cannot be used with F().
One possible solution is including a timestamp in the reply class
Post(models.Model):
Id = pk
content = text
Reply(models.Model):
Id = pk
PostId = Fk(Post)
content = text
timestamp = DateTime(auto)
Then you can use a query of this format to get the latest reply for each post.
Reply.objects.annotate(max_time=Max('Post__Reply__timestamp')).filter(timestamp=F('max_time'))
Please note that this is really time consuming for large number of records.
If you are using a Postgres DB you can use distinct()
Reply.objects.order_by('Post__Id','-timestamp').distinct('Post__Id')
F expression has no way to do that.
but Django has another way to handle it.
https://docs.djangoproject.com/en/2.0/ref/models/expressions/#subquery-expressions
for this problem, the code below can solve this:
from django.db.models import OuterRef, Subquery
sub_qs = Reply.objects.filter(
PostId=OuterRef('pk')
).order_by('timestamp')
qs = Post.objects.annotate(
last_reply_content=Subquery(
sub_qs.values('content')[:1]))
how does it work?
sub_qs is the related model queryset, where you want to take only the last reply for each post, to do that, we use the OuterRef, it will take care to get replies related to this post, and finally the order_by that will order by the timestamp, the first is the most recent, and the last is the eldest.
sub_qs = Reply.objects.filter(
PostId=OuterRef('pk')
).order_by('timestamp')
the second part is the Post queryset with a annotate, we wanna apply the sub_qs in an extra field, and using subquery will allow us to insert another queryset inside of annotate
we use .values('content') to get only the content field, and slice the sub_qs with [:1] to get only the first occurrence.
qs = Post.objects.annotate(
last_reply_content=Subquery(
sub_qs.values('content')[:1]))
While creating a front end for a Django module I faced the following problem inside Django core:
In order to display a link to the next/previous object from a model query, we can use the extra-instance-methods of a model instance: get_next_by_FIELD() or get_previous_by_FIELD(). Where FIELD is a model field of type DateField or DateTimeField.
Lets explain it with an example
from django.db import models
class Shoe(models.Model):
created = models.DateTimeField(auto_now_add=True, null=False)
size = models.IntegerField()
A view to display a list of shoes, excluding those where size equals 4:
def list_shoes(request):
shoes = Shoe.objects.exclude(size=4)
return render_to_response(request, {
'shoes': shoes
})
And let the following be a view to display one shoe and the corresponding
link to the previous and next shoe.
def show_shoe(request, shoe_id):
shoe = Shoe.objects.get(pk=shoe_id)
prev_shoe = shoe.get_previous_by_created()
next_shoe = shoe.get_next_by_created()
return render_to_response('show_shoe.html', {
'shoe': shoe,
'prev_shoe': prev_shoe,
'next_shoe': next_shoe
})
Now I have the situation that the show_shoe view displays the link to the previous/next regardless of the shoes size. But I actually wanted just shoes whose size is not 4.
Therefore I tried to use the **kwargs argument of the get_(previous|next)_by_created() methods to filter out the unwanted shoes, as stated by the documentation:
Both of these methods will perform their queries using the default manager for the model. If you need to emulate filtering used by a custom manager, or want to perform one-off custom filtering, both methods also accept
optional keyword arguments, which should be in the format described in Field lookups.
Edit: Keep an eye on the word "should", because then also (size_ne=4) should work, but it doesn't.
The actual problem
Filtering using the lookup size__ne ...
def show_shoe(request, shoe_id):
...
prev_shoe = shoe.get_previous_by_created(size__ne=4)
next_shoe = shoe.get_next_by_created(size__ne=4)
...
... didn't work, it throws FieldError: Cannot resolve keyword 'size_ne' into field.
Then I tried to use a negated complex lookup using Q objects:
from django.db.models import Q
def show_shoe(request, shoe_id):
...
prev_shoe = shoe.get_previous_by_created(~Q(size=4))
next_shoe = shoe.get_next_by_created(~Q(size=4))
...
... didn't work either, throws TypeError: _get_next_or_previous_by_FIELD() got multiple values for argument 'field'
Because the get_(previous|next)_by_created methods only accept **kwargs.
The actual solution
Since these instance methods use the _get_next_or_previous_by_FIELD(self, field, is_next, **kwargs) I changed it to accept positional arguments using *args and passed them to the filter, like the **kwargs.
def my_get_next_or_previous_by_FIELD(self, field, is_next, *args, **kwargs):
"""
Workaround to call get_next_or_previous_by_FIELD by using complext lookup queries using
Djangos Q Class. The only difference between this version and original version is that
positional arguments are also passed to the filter function.
"""
if not self.pk:
raise ValueError("get_next/get_previous cannot be used on unsaved objects.")
op = 'gt' if is_next else 'lt'
order = '' if is_next else '-'
param = force_text(getattr(self, field.attname))
q = Q(**{'%s__%s' % (field.name, op): param})
q = q | Q(**{field.name: param, 'pk__%s' % op: self.pk})
qs = self.__class__._default_manager.using(self._state.db).filter(*args, **kwargs).filter(q).order_by('%s%s' % (order, field.name), '%spk' % order)
try:
return qs[0]
except IndexError:
raise self.DoesNotExist("%s matching query does not exist." % self.__class__._meta.object_name)
And calling it like:
...
prev_shoe = shoe.my_get_next_or_previous_by_FIELD(Shoe._meta.get_field('created'), False, ~Q(state=4))
next_shoe = shoe.my_get_next_or_previous_by_FIELD(Shoe._meta.get_field('created'), True, ~Q(state=4))
...
finally did it.
Now the question to you
Is there an easier way to handle this? Should shoe.get_previous_by_created(size__ne=4) work as expected or should I report this issue to the Django guys, in the hope they'll accept my _get_next_or_previous_by_FIELD() fix?
Environment: Django 1.7, haven't tested it on 1.9 yet, but the code for _get_next_or_previous_by_FIELD() stayed the same.
Edit: It is true that complex lookups using Q object is not part of "field lookups", it's more part of the filter() and exclude() functions instead. And I am probably wrong when I suppose that get_next_by_FIELD should accept Q objects too. But since the changes involved are minimal and the advantage to use Q object is high, I think these changes should get upstream.
tags: django, complex-lookup, query, get_next_by_FIELD, get_previous_by_FIELD
(listing tags here, because I don't have enough reputations.)
You can create custom lookup ne and use it:
.get_next_by_created(size__ne=4)
I suspect the method you've tried first only takes lookup arg for the field you're basing the get_next on. Meaning you won't be able to access the size field from the get_next_by_created() method, for example.
Edit : your method is by far more efficient, but to answer your question on the Django issue, I think everything is working the way it is supposed to. You could offer an additional method such as yours but the existing get_next_by_FIELD is working as described in the docs.
You've managed to work around this with a working method, which is OK I guess, but if you wanted to reduce the overhead, you could try a simple loop :
def get_next_by_field_filtered(obj, field=None, **kwargs):
next_obj = getattr(obj, 'get_next_by_{}'.format(field))()
for key in kwargs:
if not getattr(next_obj, str(key)) == kwargs[str(key)]:
return get_next_by_field_filtered(next_obj, field=field, **kwargs)
return next_obj
This isn't very efficient but it's one way to do what you want.
Hope this helps !
Regards,
I have a third-party function which gives me a filtered queryset (e.g. records with 'valid'=True) but I want to remove a particular condition (e.g. to have all records, both valid and invalid).
Is there a way to remove a filter condition to an already-filtered queryset?
E.g.
only_valid = MyModel.objects.filter(valid=True)
all_records = only_valid.**remove_filter**('valid')
(I know that it would be better to define 'all_records' before 'only_valid', but this is just an example...)
Although there is no official way to do this using filter notation, you may easily do it with Q-notation.
For example, if you ensure that third-part function returns a Q object, not a filtered QuerySet, you may do the following:
q = ThirdParty()
q = q | Q(valid=False)
And the resulting SQL conditions will be joined using OR operator.
From the docs:
Each time you refine a QuerySet, you get a brand-new QuerySet that is in no way bound to the previous QuerySet. Each refinement creates a separate and distinct QuerySet that can be stored, used and reused.
I doubt therefore, that there is a standard way to do it. You could dig into the code, see, what filter() does and try a bit. If that doesn't help, my assumption is, you're out of luck and need to re-build the query yourself.
Use this function
from django.db.models import Q
def remove_filter(lookup, queryset):
"""
Remove filter lookup in queryset
```
>>> queryset = User.objects.filter(email='user#gmail.com')
>>> queryset.count()
1
>>> remove_filter('email', queryset)
>>> queryset.count()
1000
```
"""
query = queryset.query
q = Q(**{lookup: None})
clause, _ = query._add_q(q, self.used_aliases)
def filter_lookups(child):
return child.lhs.target != clause.children[0].lhs.target
query.where.children = list(filter(filter_lookups, query.where.children))
Here's what I did in a similar case.
all_records = MyModel.objects.all()
only_valid = MyModel.objects.filter(valid=True)
only_valid.original = all_records
...
all_records = only_valid.original
Obviously this clears any other filters too so it won't be right for every case.
original_query_set = MyModel.objects.filter(**conditions)
model_class = orginal_query_set.model
new_query_set = model_class.objects.filter(**new_conditions)
You can use the .model attribute on a QuerySet to get the model class, then use the model class to make a brand new QuerySet.
Thanks for making me check the source code Boldewyn. So, it seems there's a new method right below filter() with the same parameters... called exclude() (as of commit mini-hash ef6c680, for when it loses its line number)
Return a new QuerySet instance with NOT (args) ANDed to the existing set.
So, to answer the original question:
only_valid = MyModel.objects.filter(valid=True)
filtered_results = only_valid.exclude(the_condition_to_remove=True)
I have these models:
def Foo(Models.model):
size = models.IntegerField()
# other fields
def is_active(self):
if check_condition:
return True
else:
return False
def Bar(Models.model):
foo = models.ForeignKey("Foo")
# other fields
Now I want to query Bars that are having active Foo's as such:
Bar.objects.filter(foo.is_active())
I am getting error such as
SyntaxError at /
('non-keyword arg after keyword arg'
How can I achieve this?
You cannot query against model methods or properties. Either use the criteria within it in the query, or filter in Python using a list comprehension or genex.
You could also use a custom manager. Then you could run something like this:
Bar.objects.foo_active()
And all you have to do is:
class BarManager(models.Manager):
def foo_active(self):
# use your method to filter results
return you_custom_queryset
Check out the docs.
I had similar problem: I am using class-based view object_list and I had to filter by model's method. (storing the information in database wasn't an option because the property was based on time and I would have to create a cronjob and/or... no way)
My answer is ineffective and I don't know how it's gonna scale on larger data; but, it works:
q = Model.objects.filter(...)...
# here is the trick
q_ids = [o.id for o in q if o.method()]
q = q.filter(id__in=q_ids)
You can't filter on methods, however if the is_active method on Foo checks an attribute on Foo, you can use the double-underscore syntax like Bar.objects.filter(foo__is_active_attribute=True)