Context
I really want to, but I don't understand how I can limit an already existing Prefetch object
Models
class MyUser(AbstractUser):
pass
class Absence(Model):
employee = ForeignKey(MyUser, related_name='absences', on_delete=PROTECT)
start_date = DateField()
end_date = DateField()
View
class UserAbsencesListAPIView(ListAPIView):
queryset = MyUser.objects.order_by('first_name')
serializer_class = serializers.UserWithAbsencesSerializer
filterset_class = filters.UserAbsencesFilterSet
Filter
class UserAbsencesFilterSet(FilterSet):
first_name = CharFilter(lookup_expr='icontains', field_name='first_name')
from_ = DateFilter(method='filter_from', distinct=True)
to = DateFilter(method='filter_to', distinct=True)
What do I need
With the Request there are two arguments from_ and to. I should return Users with their Absences, which (Absences) are bounded by from_ and/or to intervals. It's very simple for a single argument, i can limit the set using Prefetch object:
def filter_from(self, queryset, name, value):
return queryset.prefetch_related(
Prefetch(
'absences',
Absence.objects.filter(Q(start_date__gte=value) | Q(start_date__lte=value, end_date__gte=value)),
)
)
Similarly for to.
But what if I want to get a limit by two arguments at once?
When the from_ attribute is requested - 'filter_from' method is executed; for the to argument, another method filter_to is executed.
I can't use prefetch_related twice, I get an exception ValueError: 'absences' lookup was already seen with a different queryset. You may need to adjust the ordering of your lookups..
I've tried using to_attr, but it looks like I can't access it in an un-evaluated queryset.
I know that I can find the first defined Prefetch in the _prefetch_related_lookups attribute of queryset, but is there any way to apply an additional filter to it or replace it with another Prefetch object so that I can end up with a query similar to:
queryset.prefetch_related(
Prefetch(
'absences',
Absence.objects.filter(
Q(Q(start_date__gte=from_) | Q(start_date__lte=from_, end_date__gte=from_))
& Q(Q(end_date__lte=to) | Q(start_date__lte=to, end_date__gte=to))
),
)
)
django-filter seems to have its own built-in filter for range queries:
More info here and here
So probably just easier to use that instead:
def filter_date_range(self, queryset, name, value):
if self.lookup_expr = "range":
#return queryset with specific prefetch
if self.lookup_expr = "lte":
#return queryset with specific prefetch
if self.lookup_expr = "gte":
#return queryset with specific prefetch
I haven't tested this and you may need to play around with the unpacking of value but it should get you most of the way there.
Related
I am trying to create a filter search bar that I can customize. For example, if I type a value into a search bar, then it will query a model and retrieve a list of instances that match the value. For example, here is a view:
class StudentListView(FilterView):
template_name = "leads/student_list.html"
context_object_name = "leads"
filterset_class = StudentFilter
def get_queryset(self):
return Lead.objects.all()
and here is my filters.py:
class
StudentFilter(django_filters.FilterSet):
class Meta:
model = Lead
fields = {
'first_name': ['icontains'],
'email': ['exact'],
}
Until now, I can only create a filter search bar that can provide a list of instances that match first_name or email(which are fields in the Lead model). However, this does now allow me to do more complicated tasks. Lets say I added time to the filter fields, and I would like to not only filter the Lead model with the time value I submitted, but also other Lead instances that have a time value that is near the one I submitted. Basically, I want something like the def form_valid() used in the views where I can query, calculate, and even alter the values submitted.
Moreover, if possible, I would like to create a filter field that is not necessarily an actual field in a model. Then, I would like to use the submitted value to do some calculations as I filter for the list of instances. If you have any questions, please ask me in the comments. Thank you.
You can do just about anything by defining a method on the filterset to map the user's input onto a queryset. Here's one I did earlier. Code much cut down ...
The filter coat_info_contains is defined as a CharFilter, but it is further parsed by the method which splits it into a set of substrings separated by commas. These substrings are then used to generate Q elements (OR logic) to match a model if the substring is contained in any of three model fields coating_1, coating_2 and coating_3
This filter is not implicitly connected to any particular model field. The connection is through the method= specification of the filter to the filterset's method, which can return absolutely any queryset on the model that can be programmed.
Hope I haven't cut out anything vital.
import django_filters as FD
class MemFilter( FD.FilterSet):
class Meta:
model = MyModel
# fields = [fieldname, ... ] # default filters created for these. Not required if all declarative.
# fields = { fieldname: [lookup_expr_1, ...], ...} # for specifying possibly multiple lookup expressions
fields = {
'ft':['gte','lte','exact'], 'mt':['gte','lte','exact'],
...
}
# declarative filters. Lots and lots of
...
coat_info_contains = FD.CharFilter( field_name='coating_1',
label='Coatings contain',
method='filter_coatings_contains'
)
...
def filter_coatings_contains( self, qs, name, value):
values = value.split(',')
qlist = []
for v in values:
qlist.append(
Q(coating_1__icontains = v) |
Q(coating_2__icontains = v) |
Q(coating_3__icontains = v) )
return qs.filter( *qlist )
So I have a database of books, and I want to search it based on filters and keywords so I've overridden the get_queryset method in my BookSearch view:
class BookSearch(generics.ListAPIView):
serializer_class = ProductDetailViewSerializer
model = ProductDetailView
def get_queryset(self):
queryset = None
categories = self.kwargs['categories'].rstrip()
keywords = self.kwargs['keywords'].rstrip()
if isinstance(categories, str) and isinstance(keywords, str):
book_filter = BookFilter(categories)
sql = self.get_sql(categories, keywords, book_filter)
queryset = ProductDetailView.objects.filter(
id__in=RawSQL(sql, book_filter.params)
)
message = f"{queryset.query}"
log_to_file('BookSearch.log', 'BookSearch.get_queryset', message)
return queryset
That log_to_file call logs the query that django uses, which I've abbreviated here
but is as follows:
SELECT `jester_productdetailview`.`id`,
`jester_productdetailview`.`isbn`,
`jester_productdetailview`.`title`
FROM `jester_productdetailview`
WHERE `jester_productdetailview`.`id` IN (
select id from jester_productdetailview
where ( authors like '%Beatrix%' or
illustrators like '%Beatrix%' or
title like '%Beatrix%' ) )
ORDER BY `jester_productdetailview`.`title` ASC
If I run that query in my database manually, I get 186 rows:
'119371','9780723259572','A Beatrix Potter Treasury'
'130754','9780241293348','A Christmas Wish'
'117336','9780241358740','A Pumpkin for Peter' ...
To get the query above, I call the view through the API, yet by the time the queryset is returned, there are no results ???
http://127.0.0.1:8000/api/book-search/{"filter": "all"}/Beatrix/
returns []
You are returning queryset only within the if condition only. By default, the function is sending None. Return the queryset outside the if condition as well.
First time with Django. Trying to add an annotation to queryset:
class EnrollmentManager(models.Manager.from_queryset(EnrollmentCustomQuerySet)):
COURSE_DURATION = datetime.timedelta(days=183)
def get_queryset(self):
"""Overrides the models.Manager method"""
lookback = make_aware(datetime.datetime.today() - self.COURSE_DURATION)
qs = super(EnrollmentManager, self).get_queryset().annotate( \
is_expired=(Value(True)), output_field=models.BooleanField())
return qs
At the moment I am just trying to add an extra 'calculated' field on the returned queryset, which is hard-coded to True and the attribute/field should be called is_expired.
If I can get that to work, then Value(True) needs to be a derived value based on this expression:
F('enrolled') < lookback
But since 'enrolled' is a database field and lookback is calculated, how will I be able to do that?
Note
I tried this, which executes without throwing the error:
qs = super(EnrollmentManager, self).get_queryset().annotate( \
is_expired=(Value(True, output_field=models.BooleanField())))
and in the shell I can see it:
Enrollment.objects.all()[0].is_expired -> returns True
and I can add it to the serializer:
class EnrollmentSerializer(serializers.ModelSerializer):
is_active = serializers.SerializerMethodField()
is_current = serializers.SerializerMethodField()
is_expired = serializers.SerializerMethodField()
COURSE_DURATION = datetime.timedelta(days=183)
class Meta:
model = Enrollment
fields = ('id', 'is_active', 'is_current', 'is_expired')
def get_is_expired(self, obj):
return obj.is_expired
So it is possible...but how can I replace my hard-coded 'True" with a calculation?
UPDATE
Reading the documentation, it states:
"Annotates each object in the QuerySet with the provided list of query expressions. An expression may be a simple value, a reference to a field on the model (or any related models), or an aggregate expression (averages, sums, etc.) that has been computed over the objects that are related to the objects in the QuerySet."
A simple value - so, not a simple COMPUTED value then?
That makes me think this is not possible...
It seems like a pretty good use-case for a Case expression. I suggest getting as familiar as you can with these expression tools, they're very helpful!
I haven't tested this, but it should work. I'm assuming enrolled is a tz-aware datetime for when they first enrolled...
from django.db.models import Case, When, Value
def get_queryset(self):
"""Overrides the models.Manager method"""
lookback = make_aware(datetime.datetime.today() - self.COURSE_DURATION)
qs = super(EnrollmentManager, self).get_queryset().annotate(
is_expired=Case(
When(
enrolled__lt=lookback,
then=Value(True)
),
default=Value(False),
output_field=models.BooleanField()
)
)
You also don't have to pre-calculate the lookback variable. Check out ExpressionWrappers and this StackOverflow answer that addresses this.
ExpressionWrapper(
TruncDate(F('date1')) + datetime.timedelta(days=365),
output_field=DateField(),
)
Of course, I don't mean to do what prefetch_related does already.
I'd like to mimic what it does.
What I'd like to do is the following.
I have a list of MyModel instances.
A user can either follows or doesn't follow each instance.
my_models = MyModel.objects.filter(**kwargs)
for my_model in my_models:
my_model.is_following = Follow.objects.filter(user=user, target_id=my_model.id, target_content_type=MY_MODEL_CTYPE)
Here I have n+1 query problem, and I think I can borrow what prefetch_related does here. Description of prefetch_related says, it performs the query for all objects and when the related attribute is required, it gets from the pre-performed queryset.
That's exactly what I'm after, perform query for is_following for all objects that I'm interested in. and use the query instead of N individual query.
One additional aspect is that, I'd like to attach queryset rather than attach the actual value, so that I can defer evaluation until pagination.
If that's too ambiguous statement, I'd like to give the my_models queryset that has is_following information attached, to another function (DRF serializer for instance).
How does prefetch_related accomplish something like above?
A solution where you can get only the is_following bit is possible with a subquery via .extra.
class MyModelQuerySet(models.QuerySet):
def annotate_is_follwing(self, user):
return self.extra(
select = {'is_following': 'EXISTS( \
SELECT `id` FROM `follow` \
WHERE `follow`.`target_id` = `mymodel`.id \
AND `follow`.`user_id` = %s)' % user.id
}
)
class MyModel(models.Model):
objects = MyModelQuerySet.as_manager()
usage:
my_models = MyModel.objects.filter(**kwargs).annotate_is_follwing(request.user)
Now another solution where you can get a whole list of following objects.
Because you have a GFK in the Follow class you need to manually create a reverse relation via GenericRelation. Something like:
class MyModelQuerySet(models.QuerySet):
def with_user_following(self, user):
return self.prefetch_related(
Prefetch(
'following',
queryset=Follow.objects.filter(user=user) \
.select_related('user'),
to_attr='following_user'
)
)
class MyModel(models.Model):
following = GenericRelation(Follow,
content_type_field='target_content_type',
object_id_field='target_id'
related_query_name='mymodels'
)
objects = MyModelQuerySet.as_manager()
def get_first_following_object(self):
if hasattr(self, 'following_user') and len(self.following_user) > 0:
return self.following_user[0]
return None
usage:
my_models = MyModel.objects.filter(**kwargs).with_user_following(request.user)
Now you have access to following_user attribute - a list with all follow objects per mymodel, or you can use a method like get_first_following_object.
Not sure if this is the best approach, and I doubt this is what prefetch_related does because I'm joining here.
I found there's way to select extra columns in your query.
extra_select = """
EXISTS(SELECT * FROM follow_follow
WHERE follow_follow.target_object_id = myapp_mymodel.id AND
follow_follow.target_content_type_id = %s AND
follow_follow.user_id = %s)
"""
qs = self.extra(
select={'is_following': extra_select},
select_params=[CONTENT_TYPE_ID, user.id]
)
So you can do this with join.
prefetch_related way of doing it would be separate queryset and look it up in queryset for the attribute.
I am working with django-rest-framework and I have an API that returns me the info with a filter like this:
http://example.com/api/products?category=clothing&in_stock=True
--this returns me 10 items
But it also returns the whole Model data if I dont put the filters, this is the default way.
http://example.com/api/products/
--this returns me more than 100 (all the Model Table)
How can I disable this default operation, I mean, how can I make a filter to be necesary to make this api works? or even better! how can I make the last URL to return an empty json response?
UPDATE
Here is some code:
serializers.py
class OEntradaDetalleSerializer(serializers.HyperlinkedModelSerializer):
item = serializers.RelatedField(source='producto.item')
descripcion = serializers.RelatedField(source='producto.descripcion')
unidad = serializers.RelatedField(source='producto.unidad')
class Meta:
model = OEntradaDetalle
fields = ('url','item','descripcion','unidad','cantidad_ordenada','cantidad_recibida','epc')
views.py
class OEntradaDetalleViewSet(BulkUpdateModelMixin,viewsets.ModelViewSet):
filter_backends = (filters.DjangoFilterBackend,)
filter_fields = ('cantidad_ordenada','cantidad_recibida','oentrada__codigo_proveedor','oentrada__folio')
queryset = OEntradaDetalle.objects.all()
serializer_class = OEntradaDetalleSerializer
urls.py
router2 = BulkUpdateRouter()
router2.register(r'oentradadetalle', OEntradaDetalleViewSet)
urlpatterns = patterns('',
url(r'^api/',include(router2.urls)),
)
URL EXAMPLE
http://localhost:8000/api/oentradadetalle/?oentrada__folio=E01
THIS RETURNS ONLY SOME FILTERED VALUES
http://localhost:8000/api/oentradadetalle/
THIS RETURNS EVERYTHING IN THE MODEL (I need to remove this or make it return some empty data)
I would highly recommend using pagination, to prevent anyone from being able to return all of the results (which likely takes a while).
If you can spare the extra queries being made, you can always check if the filtered and unfiltered querysets match, and just return an empty queryset if that is the case. This would be done in the filter_queryset method on your view.
def filter_queryset(self, queryset):
filtered_queryset = super(ViewSet, self).filter_queryset(queryset)
if queryset.count() === len(filtered_queryset):
return queryset.model.objects.none()
return filtered_queryset
This will make one additional query for the count of the original queryset, and if it is the same as the filtered queryset, an empty queryset will be returned. If the queryset was actually filtered, it will be returned and the results will be what you are expecting.