Django REST Framework - Filtering - django

I want to filter multiple fields with multiple queries like this:
api/listings/?subburb=Subburb1, Subburb2&property_type=House,Apartment,Townhouse,Farm .. etc
Are there any built in ways, I looked at django-filters but it seems limited, and I think I would have to do this manually in my api view, but its getting messy, filtering on filters on filters

filtering on filters on filters is not messy it is called chained filters.
And chain filters are necessary because sometime there is going to be property_type some time not:
if property_type:
qs = qs.filter(property_type=property_type)
If you are thinking there is going to be multiple queries then not, it will still executed in one query because queryset are lazy.
Alternatively you can build a dict and pass it just one time:
d = {'property_type:': property_type, 'subburb': subburb}
qs = MyModel.objects.filter(**d)

Complex filters are not out of the box supported by DRF or even by django-filter plugin. For simple cases you can define your own get_queryset method
This is straight from the documentation
def get_queryset(self):
queryset = Purchase.objects.all()
username = self.request.query_params.get('username', None)
if username is not None:
queryset = queryset.filter(purchaser__username=username)
return queryset
However this can quickly become messy if you are supported multiple filters and even some of them complex.
The solution is to define a custom filterBackend class and a ViewSet Mixin. This mixins tells the viewset how to understand a typical filter backend and this backend can understand very complex filters all defined explicitly, including rules when those filters should be applied.
A sample filter backend is like this (I have defined three different filters on different query parameters in the increasing order of complexity:
class SomeFiltersBackend(FiltersBackendBase):
"""
Filter backend class to compliment GenericFilterMixin from utils/mixin.
"""
mapping = {'owner': 'filter_by_owner',
'catness': 'filter_by_catness',
'context': 'filter_by_context'}
def rule(self):
return resolve(self.request.path_info).url_name == 'pet-owners-list'
Straight forward filter on ORM lookups.
def filter_by_catness(self, value):
"""
A simple filter to display owners of pets with high catness, canines excuse.
"""
catness = self.request.query_params.get('catness')
return Q(owner__pet__catness__gt=catness)
def filter_by_owner(self, value):
if value == 'me':
return Q(owner=self.request.user.profile)
elif value.isdigit():
try:
profile = PetOwnerProfile.objects.get(user__id=value)
except PetOwnerProfile.DoesNotExist:
raise ValidationError('Owner does not exist')
return Q(owner=profile)
else:
raise ValidationError('Wrong filter applied with owner')
More complex filters :
def filter_by_context(self, value):
"""
value = {"context_type" : "context_id or context_ids separated by comma"}
"""
import json
try:
context = json.loads(value)
except json.JSONDecodeError as e:
raise ValidationError(e)
context_type, context_ids = context.items()
context_ids = [int(i) for i in context_ids]
if context_type == 'default':
ids = context_ids
else:
ids = Context.get_ids_by_unsupported_contexts(context_type, context_ids)
else:
raise ValidationError('Wrong context type found')
return Q(context_id__in=ids)
To understand fully how this works, you can read up my detailed blogpost : http://iank.it/pluggable-filters-for-django-rest-framework/
All the code is there in a Gist as well : https://gist.github.com/ankitml/fc8f4cf30ff40e19eae6

Related

How to share validation and schemas between a DRF FilterBackend and a Serializer

I am implementing some APIs using Django Rest Framework, and using the generateschema command to generate the OpenApi 3.0 specs afterwards.
While working on getting the schema to generate correctly, I realized that my code seemed to be duplicating a fair bit of logic between the FilterBackend and Serializer I was using. Both of them were accessing and validating the query parameters from the request.
I like the way of specifying the fields in the Serializer (NotesViewSetGetRequestSerializer in my case), and I would like to use that in my FilterBackend (NoteFilterBackend in my case). It would be nice to have access to the validated_data within the filter, and also be able to use the serializer to implement the filtering schemas.
Are there good solutions out there for only needing to specify your request query params once, and re-using with the filter and serializer?
I've reproduced a simplified version of my code below. I'm happy to provide more info on ResourceURNRelatedField if it's needed (it extends RelatedField and uses URNs instead of primary keys), but I think this would apply to any kind of field.
class NotesViewSet(generics.ListCreateAPIView, mixins.UpdateModelMixin):
allowed_methods = ("GET")
queryset = Note.objects.all()
filter_backends = [NoteFilterBackend]
serializer_class = NotesViewSetResponseSerializer
def get(self, request, *args, **kwargs):
query_params_dict = request.query_params
request_serializer = NotesViewSetGetRequestSerializer(data=query_params_dict)
request_serializer.is_valid(raise_exception=True)
validated_data = request_serializer.validated_data
member = validated_data.get("member_urn")
team = validated_data.get("team_urn")
if not provider_can_view_member(request.user, member, team):
return custom404(
request,
HttpResponseNotFound(
"Member does not exist!. URN={}".format(member.urn())
),
)
return super(NotesViewSet, self).list(request, *args, **kwargs)
class NotesViewSetGetRequestSerializer(serializers.Serializer):
member_urn = ResourceURNRelatedField(queryset=User.objects.all(), required=True)
team_urn = ResourceURNRelatedField(queryset=Team.objects.all(), required=True)
privacy_scope = serializers.CharField(required=False)
def validate_privacy_scope(self, value):
choices = dict(Note.PRIVACY_SCOPE_CHOICES)
if value and value not in choices:
raise serializers.ValidationError(
"bad privacy scope {}. Supported values are: {}".format(value, choices)
)
else:
return value
class NoteFilterBackend(BaseFilterBackend):
def filter_queryset(self, request, queryset, view):
member_urn = request.query_params.get("member_urn")
customer_uuid = URN.from_urn(member_urn).id
privacy_scope = request.query_params.get("privacy_scope")
team_urn = request.query_params.get("team_urn")
team_uuid = URN.from_urn(team_urn).id
queryset = queryset.filter(customer__uuid=customer_uuid)
if privacy_scope == Note.PRIVACY_SCOPE_TEAM_PROVIDERS:
queryset = queryset.filter(team__uuid=team_uuid)
return queryset

How to use the django-admin-search with search_FieldNameHere ? I am unable to use the same

The django-admin-search plugin version 0.3.5 has beem installed with django 3.0.5 on linux. How exactly to use the search_field query, I fail to understand. I am not great working with OOPS yet.
What I tried:
def search_mfrom(request, field_value, param_values):
"""
intercept query filter for description field
"""
query = Q()
#print(request, field_value, param_values, 'llllllllllllll')
query = Q(mfrom='sowmiya#abc.com')
print(query, 'qqqqqqqqqqqq')
return query
The print(query, 'qqqqqqqqqqq') never executes. Am I passing in something wrong? The form field is mfrom.
Secondly, wish to send in login specific records in queryset which this tool is overriding. How to do that? I am able to override the queryset based on login domain part of the request.user, but since this tool is also overriding the queryset object of the modeladmin, I don't know how to go about. The code snippet for the same:
def get_queryset(self, request):
domain_list = []
temp = []
qs = super(LogSearchFormAdmin, self).get_queryset(request)
if request.user.is_superuser:
return qs
else:
domain_list = Domain.objects.filter(
customer__in=Customer.objects.filter(
email=request.user.username)).values_list(
'name', flat=True)
#print(list(domain_list))
dom_names = list(domain_list)
#print(dom_names)
qs = MailLogs.objects.none()
if dom_names:
for d in dom_names:
qs = MailLogs.objects.filter(
Q(mfrom__icontains=d)|Q(mto__icontains=d))
# qs |= MailLogs.objects.filter(mfrom__icontains=d)
# qs |= MailLogs.objects.filter(mto__icontains=d)
print(qs.query)
#print(qs)
return qs
I believe it is a mistake in the documentation for this case, I will update it coming soon.
I am not near my development laptop in this week, for this reasson I try to help you without testing code, sorry for this.
Another details is: I create this lib for Django 2, and in this pandemic a lot has happened, international change, visa details, search for house, new job, and more, it's crazy days. And I not have certain if works with Django 3. In next month I check this (I believe I will have solved everything)
But if works, all your need is:
Install a lib (check GitHub page)
Create your search form
from django.forms import Form
from django.forms import CharField
class YourFormSearch(Form):
mfrom = CharField() ## this type is just for registry will be ignored because your override this
Override in the admin
class YourAdmin(AdvancedSearchAdmin):
def search_mfrom(self, field, field_value, form_field, request, param_values):
"""
intercept mfrom filter and override
"""
query = Q()
# your Q logic here
return query

Customize queryset in django-filter ModelChoiceFilter (select) and ModelMultipleChoiceFilter (multi-select) menus based on request

I'm using django-filter in 2 places: My Django Rest Framework API, and in my FilterViews (Django Filter's Generic ListViews.) In the case of my FilterViews I'm showing both select boxes (ModelChoiceFilter) and multi-select boxes (ModelMultipleChoiceFilter) to be filtered on.
I need to be able to limit what's in those select and multi-select inputs based on a field inside the request.
It's relatively simple to change what's listed as a kwarg in the relevant field in the FilterSet. For example, here's my FilterSet where the queryset is set as a kwarg:
class FieldFilter(django_filters.FilterSet):
"""Filter for the field list in the API"""
dataset = ModelChoiceFilter(queryset=Dataset.objects.all())
class Meta(object):
"""Meta options for the filter"""
model = Field
fields = ['dataset']
And it's relatively straightforward to limit what the result is in DRF inside the get_queryset() method. For example, here's my DRF ViewSet:
class FieldViewSet(viewsets.ReadOnlyModelViewSet):
"""A ViewSet for viewing dataset fields"""
queryset = Field.objects.all()
serializer_class = FieldSerializer
filter_class = FieldFilter
def get_queryset(self):
"""Get the queryset"""
queryset = super(FieldViewSet, self).get_queryset()
queryset = queryset.filter(
dataset__organization=self.request.organization)
return queryset
I just can't find anywhere to edit the Dataset field in the filter_class when the view is being displayed.
This is super straightforward in Django FormView generic views, but it doesn't appear that FieldViewSet follows the same get_form() structure as generic views. It's also relatively straightforward to do in the admin, but DRF/Django-Filter don't seem to follow that structure either.
Is there any way to customize the queryset in those inputs on a per-request basis? Preferably both on FilterViews and in the HTML API browser, but just in FilterViews would be fine if it's too complicated for the HTML API browser.
After hours of search, I found the solution in the official documentation here!
The queryset argument for ModelChoiceFilter and ModelMultipleChoiceFilter supports callable behavior. If a callable is passed, it will be invoked with the request as its only argument.
import django_filters as filters
from django.utils.translation import gettext as _
def ourBranches(request):
if request is None:
return Branch.objects.none()
company = request.user.profile.company
return Branch.objects.filter(company=company)
class UnitFilter(filters.FilterSet):
branch = filters.ModelChoiceFilter(
queryset=ourBranches, empty_label=_("All Branches"))
class Meta:
model = Unit
fields = ('branch', )
and in the view, I made sure to pass the request as well
qs = Unit.objects.all()
filter = UnitFilter(self.request.GET, request=self.request, queryset=qs)
table = UnitTable(filter.qs)
I also had problems finding a resolution to this.
I solved it (I think) via the following:
views.py
table_filter = ExampleFilter(request.GET, kwarg_I_want_to_pass=request.user, queryset=qs)
filters.py
class ExampleFilter(django_filters.FilterSet):
def __init__(self, *args, **kwargs):
self.user = kwargs.pop('kwarg_I_want_to_pass', None)
super(ExampleFilter, self).__init__(*args, **kwargs)
self.filters['field_to_filter'].extra.update({
'queryset': Supplier.objects.filter(related_user=self.user),
'empty_label': '',
'help_text': False
})
class Meta:
model = ExampleModel
fields = ['related_user', 'field_to_filter', ... other fields]

How to use serializer result as queryset on another serializer

Is possible to create a dependence between serializers, like the code below?
class ProSerializer(serializers.ModelSerializer):
entity = serializers.PrimaryKeyRelatedField(many=False,queryset=Entity.objects.all())
foo = serializers.PrimaryKeyRelatedField(many=True,queryset=Foo.objects.filter(entity=entity))
class Meta:
model = ..............
What I want to do is to limit the queryset on Foo to just the ones from the chosen entity. Is there a way to do that?
Django Rest Framework does not make this easy, at least in version 2.x – and I am not sure whether there are/were any plans to make it better in version 3.
I hacked this fixed in various places with try catches in serializer inits filtering any applicable field's queryset by the parent property passed in the data dictionary before making an attempt at standardising the problem – the following is what I came up with.
SlugRelatedDependentField
class SlugRelatedDependentField(SlugRelatedField):
def __init__(self, depends_on=None, **kwargs):
assert depends_on is not None, 'The `depends_on` argument is required.'
self.depends_on = depends_on # archive_unit__organization or organization
self.depends_segments = self.depends_on.split('__')
self.depends_parent = self.depends_segments.pop(0)
self.depends_field = SimpleLazyObject(lambda: self.parent.parent.fields[self.depends_parent])
self.depends_queryset = SimpleLazyObject(lambda: self.depends_field.queryset)
self.depends_model = SimpleLazyObject(lambda: self.depends_queryset.model)
super(SlugRelatedDependentField, self).__init__(**kwargs)
def contextualize(self, instance, data):
self.data = data
self.instance = instance
def get_queryset(self):
try:
return self.queryset.filter(**{self.depends_on: reduce(getattr, self.depends_segments, self.get_relation())})
except self.depends_model.DoesNotExist:
# if parent was absent or invalid, empty the queryset
return self.queryset.none()
except TypeError:
# if parent was a Page instance, use the full queryset, it's only a list view
return self.queryset.all()
def get_relation(self):
try:
# if an allowed parent was passed, filter by it
return self.depends_queryset.get(**{self.depends_field.slug_field: self.data[self.depends_parent]})
except (KeyError, TypeError):
# if data was empty or no parent was passed, try and grab it off of the model instance
if isinstance(self.instance, self.parent.parent.Meta.model):
return getattr(self.instance, self.depends_parent)
elif self.instance is None:
raise self.depends_model.DoesNotExist
else:
raise TypeError
Usage
class RepositorySerializer(ModelSerializer):
organization = SlugRelatedField(queryset=Organization.objects.all(), slug_field='slug')
teams = SlugRelatedDependentField(allow_null=True, depends_on='organization', many=True, queryset=Team.objects.all(), required=False, slug_field='slug')
def __init__(self, instance=None, data=empty, **kwargs):
f = self.fields['teams']
# assign instance and data for get_queryset
f.child_relation.contextualize(instance, data)
# inject relation values from instance if they were omitted so they are validated regardless
if data is not empty and instance and name not in data:
data[name] = [getattr(relation, f.child_relation.slug_field) for relation in getattr(instance, name).all()]
super(RepositorySerializer, self).__init__(instance=instance, data=data, **kwargs)
Summary
SlugRelatedDependentField expands on the regular SlugRelatedField to accept a depends_on kwarg which accepts a string describing the field's relation to another – in this example, the usage describes that the teams assigned to this repository must belong to the organization.
A few gotchas
I empty the queryset with .none() if the parent does not exist, this avoids choice leak, which may be otherwise exposed via OPTIIONS requests and validation messages, and is usually undesirable.
I used data when querying for the parent record, IIRC the reason I did this was because data is consistently available whilst the parent field's object may not be e.g. in the case of PATCH requests.
You'll notice I inject any omitted relation values in the latter portion of the serializer init, this serves the purpose of forcing validation to run on the many field – useful e.g. if the user changed the organization of the record in a PATCH request, meaning the assigned teams no longer apply.
Support for distant relations
Another problem this solution caters for is referencing distant relations, this can be done by passing a __ delimited string to depends_on e.g. repository__organization, I don't have a great example use case for this, but it's there if you need it.

Django multiple forms with modelchoicefield -> too many queries

I have a table of forms of the same class which contains ModelChoiceField. And each form in one row has the same queryset for this field. Problem is that every time the form is rendered, it is a new query which increases unbearably the number of queries.
The only solution I came up with is to construct the form on the go with js instead of letting django to render it itself. Is there a way to cache these querysets or somewhat preload it at once?
views.py:
shift_table=[]
for project in calendar_projects:
shift_table.append([])
project_branches = project.branches.all()
for i, week in enumerate(month):
for day in week:
shift_table[-1].append(
CreateShiftCalendarForm(initial={'date': day}, branch_choices=project_branches))
forms.py:
CreateShiftCalendarForm(EditShiftCalendarForm):
class Meta(ShiftForm.Meta):
fields = ('project_branch', 'date') + ShiftForm.Meta.fields
widgets = {'date': forms.HiddenInput(), 'length': forms.NumberInput(attrs={'step': 'any'}), 'project_branch': forms.Select()}
def __init__(self, *args, **kwargs):
branch_choices = kwargs.pop('branch_choices', ProjectBranch.objects.none())
super(CreateShiftCalendarForm, self).__init__(*args, **kwargs)
self.fields['project_branch'].queryset = branch_choices
self.fields['project_branch'].empty_label = None
ModelChoiceField is an subclass of ChoiceField in which "normal" choices are replaced with iterator that will iterate through provided queryset. Also there is customized 'to_python' method that will return actual object instead of it's pk. Unfortunately that iterator will reset queryset and hit database once again for each choice field, even if they are sharing queryset
What you need to do is subclass ChoiceField and mimic behaviour of ModelChoiceField with one difference: it will take static choices list instead of queryset. That choices list you will build in your view once for all fields (or forms).
A maybe less invasive hack, using an overload of Django's FormSets and keeping the base form untouched (i.e. keeping the ModelChoiceFields with their dynamic queryset):
from django import forms
class OptimFormSet( forms.BaseFormSet ):
"""
FormSet with minimized number of SQL queries for ModelChoiceFields
"""
def __init__( self, *args, modelchoicefields_qs=None, **kwargs ):
"""
Overload the ModelChoiceField querysets by a common queryset per
field, with dummy .all() and .iterator() methods to avoid multiple
queries when filling the (repeated) choices fields.
Parameters
----------
modelchoicefields_qs : dict
Dictionary of modelchoicefield querysets. If ``None``, the
modelchoicefields are identified internally
"""
# Init the formset
super( OptimFormSet, self ).__init__( *args, **kwargs )
if modelchoicefields_qs is None and len( self.forms ) > 0:
# Store querysets of modelchoicefields
modelchoicefields_qs = {}
first_form = self.forms[0]
for key in first_form.fields:
if isinstance( first_form.fields[key], forms.ModelChoiceField ):
modelchoicefields_qs[key] = first_form.fields[key].queryset
# Django calls .queryset.all() before iterating over the queried objects
# to render the select boxes. This clones the querysets and multiplies
# the queries for nothing.
# Hence, overload the querysets' .all() method to avoid cloning querysets
# in ModelChoiceField. Simply return the queryset itself with a lambda function.
# Django also calls .queryset.iterator() as an optimization which
# doesn't make sense for formsets. Hence, overload .iterator as well.
if modelchoicefields_qs:
for qs in modelchoicefields_qs.values():
qs.all = lambda local_qs=qs: local_qs # use a default value of qs to pass from late to immediate binding (so that the last qs is not used for all lambda's)
qs.iterator = qs.all
# Apply the common (non-cloning) querysets to all the forms
for form in self.forms:
for key in modelchoicefields_qs:
form.fields[key].queryset = modelchoicefields_qs[key]
In your view, you then call:
formset_class = forms.formset_factory( form=MyBaseForm, formset=OptimFormSet )
formset = formset_class()
And then render your template with the formset as described in Django's doc.
Note that on form validation, you will still have 1 query per ModelChoiceField instance, but limited to a single primary key value each time. That is also the case with the accepted answer. To avoid that, the to_python method should use the existing queryset, which would make the hack even hackier.
This works at least for Django 1.11.
I subclassed ChoiceField as suggested by GwynBleidD and it works sufficiently for now.
class ListModelChoiceField(forms.ChoiceField):
"""
special field using list instead of queryset as choices
"""
def __init__(self, model, *args, **kwargs):
self.model = model
super(ListModelChoiceField, self).__init__(*args, **kwargs)
def to_python(self, value):
if value in self.empty_values:
return None
try:
value = self.model.objects.get(id=value)
except self.model.DoesNotExist:
raise ValidationError(self.error_messages['invalid_choice'], code='invalid_choice')
return value
def valid_value(self, value):
"Check to see if the provided value is a valid choice"
if any(value.id == int(choice[0]) for choice in self.choices):
return True
return False