Filtering Django queryset with custom function - django

I'm developing a REST API for an existing system that uses custom permission handling. I'm attempting to use the built-in generics from the Django REST Framework, but I'm running into trouble filtering the list views using my custom permissions. An example of my current view is:
class WidgetList(generics.ListCreateAPIView):
permission_classes = (permissions.IsAuthenticated,)
model = Widget
serializer_class = WidgetSerializer
filter_backends = (filters.DjangoFilterBackend,)
filter_fields = ('widget_type', 'widget_owner')
def get_queryset(self):
"""
Overwrite the query set to check permissions
"""
qs_list = [w.id for w in self.model.objects.all() if
canReadWidget(self.request.user, w)]
return self.model.objects.filter(id__in=qs_list)
This works, however I feel like the get_queryset function could be improved. Because my canReadWidget is custom, I have to evaluate self.model.objects.all() and check which widgets the user can read, but the function must return a query set so I use the id__in=qs_list part. The result being that I make two database calls for what is really just one list fetch.
Is there a standard way to handle this kind of per-object filtering for a generic list view?

At some point, it's better to drop the default generic views or function and roll your own.
You should have a look at the ListModelMixin and override the list to deal with the list instead of turning it into a queryset.
You should adapt the filtering and pagination but you won't hit the DB twice as you currently do.

first install django-filter package and register in settings.py
Write this code on filter.py file
import django_filters
from .models import CustomUser
class UserFilter(django_filters.FilterSet):
first_name = django_filters.CharFilter(label="First Name", lookup_expr='icontains')
last_name = django_filters.CharFilter(label="Last Name", lookup_expr='icontains')
email = django_filters.CharFilter(label="Email", lookup_expr='icontains')
mobile_number = django_filters.CharFilter(label="Mobile No.", lookup_expr='icontains')
##Change Your Fields What You Want To Filtering
class Meta:
model = Widget
fields = {'is_verify'}
On Your Views File write this code:
class WidgetViewSet(MyModelViewSet):
queryset = Widget.objects
serializer_class = "pass your serializer"
def get_filter_data(self):
_data = self.queryset.all()
data = UserFilter(self.request.GET, queryset=_data)
return data.qs.order_by('-id')
def get_queryset(self):
return self.get_filter_data()

Related

Django Rest FrameWork Reduce number of queries using group by

I am writing an api using Django Rest Frameworks. The api fetches a list of clients.A Clients has many projects. My api should returns the list of clients with number of projects completed, pending and total. My api works, but it has too many sql queries. The api is paginated
class ClientViewSet(ModelViewSet):
"""
A simple view for creating clients, updating and retrieving
"""
model = Client
queryset = Client.objects.all()
serializer_class = ClientSerializer
Now my client Serializer
class ClientSerializer(serializers.ModelSerializer):
total_projects_count = serializers.SerializerMethodField()
on_going_projects_count = serializers.SerializerMethodField()
completed_projects_count = serializers.SerializerMethodField()
class Meta:
model = Client
fields = __all__
def get_total_projects_count(self, obj):
return obj.total_projects_count()
def get_on_going_projects_count(self, obj):
return obj.on_going_project_count()
def get_completed_projects_count(self, obj):
return obj.completed_projects_count()
Project has a client foreign key. I tried to fetch all products like below and group by using annotate. But annotate worked only on a single field.
projects = Project.objects.filter(client__in=queryset).values('client', 'status')
How to do group by on multiple fields and pass that extra argument to serializer. Or is there any better approach. I also tried prefetch_related but the total_projects_count was still exuecting new sql queries
You need to annotate the calculated fields in the queryset and then, instead of calling the methods, use the annotated columns to return the relevant result. This will make sure that all data is retrieved using a single query, which will definitely be faster.
Update your queryset.
class ClientViewSet(ModelViewSet):
"""
A simple view for creating clients, updating and retrieving
"""
model = Client
queryset = Client.objects.annotate(total_projects_count_val=...)
serializer_class = ClientSerializer
Then, in your serializer, use the annotated column
class ClientSerializer(serializers.ModelSerializer):
total_projects_count = serializers.SerializerMethodField()
on_going_projects_count = serializers.SerializerMethodField()
completed_projects_count = serializers.SerializerMethodField()
class Meta:
model = Client
fields = __all__
def get_total_projects_count(self, obj):
return obj.total_projects_count_val
...
Looking at the method names, I think you will need Case-When annotation.
I reduced the query by using the below queries
from django.db.models import Count, Q
pending = Count('project', filter=Q(project__status="pending"))
finished = Count('project', filter=Q(project__status="finished"))
queryset = Client.objects.annotate(pending=pending).annotate(finished=finished)
Now was able to access queryset[0].finished etc . As I was using pagination provided drf the query generated was
SELECT "clients_client"."id",
"clients_client"."created_at",
"clients_client"."updated_at",
"clients_client"."client_name",
"clients_client"."phone_number",
"clients_client"."email",
"clients_client"."address_lane",
"clients_client"."state",
"clients_client"."country",
"clients_client"."zipCode",
"clients_client"."registration_number",
"clients_client"."gst",
COUNT("projects_project"."id") FILTER (WHERE "projects_project"."status" = 'pending') AS "pending",
COUNT("projects_project"."id") FILTER (WHERE "projects_project"."status" = 'finished') AS "finished"
FROM "clients_client"
LEFT OUTER JOIN "projects_project"
ON ("clients_client"."id" = "projects_project"."client_id")
GROUP BY "clients_client"."id"
ORDER BY "clients_client"."id" ASC
LIMIT 6

Django REST API Listview

Currently I'm trying to develop personal blog with Django/REST API, and I have trouble for that.
There are a number of posts in blog and I want to control those posts with Hyperlink. I made it by using ModelViewSet, however, whole data in detailView is also shown in ListView.
The thing is, I only want "url" and "title" of posts to be shown in ListView while DetailView contains full data.
Here is my code and current results given by REST framework.
Don't mind IndexView
# serializers
class PostSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = Post
fields = '__all__'
# views
class PostViewSet(viewsets.ModelViewSet):
queryset = Post.objects.all()
serializer_class = serializers.PostSerializer
permission_classes = (IsAdminUser, )
Post List in REST API:
Post instance in REST API:
As far as I'm aware, you need a separate serializer for the list view.
You could create a custom serializer that takes in a fields arg to select specific fields. But its probably simpler to just have a separate one for the ListView. Also, for the list view, if you are only showing a subset of the model fields, you can use the only() function on the queryset to only return the model data that you need. For example:
qs = MyModel.objects.all().only('field_a', 'field_b', 'field_c')
Here is the custom serializer if you decide to go that way:
class CustomSerializer(serializers.ModelSerializer):
def __init__(self, *args, **kwargs):
selected_fields = kwargs.pop('selected_fields', None)
# used pop function so selected_fields is not passed to superclass
super().__init__(*args, **kwargs)
if selected_fields:
# make sure only fields for the model are allowed
fields = set(selected_fields)
current_fields = set(self.fields.keys())
for field in current_fields - fields:
self.fields.pop(field)
class MyModelSerializer(CustomSerializer):
class Meta:
model = MyModel
fields = '__all__'
In the list view:
required_fields = ('field_a', 'field_b', 'field_c')
data_to_return = MyModelSerializer(model_queryset, many=True, fields=required_fields).data
return Response(data)

use django-filter with object url_params

I have successfully implemented django-filter on my Django-rest-framework server.
I have the following filter_class
filters.py
class EmploymentFilter(filters.FilterSet):
class Meta:
model = EmploymentCheck
fields = ['instructions',]
views.py
class EmploymentCheckViewSet(viewsets.ModelViewSet):
pagination_class = ContentRangeHeaderPagination
serializer_class = EmploymentCheckSerializer
queryset = EmploymentCheck.objects.all()
filter_class = EmploymentFilter
the filter works when i send a get request as
/employmentcheck/?instructions=2
However, I have implemented a front end with react-admin.
My front-end, sends a request with the url_params as objects
/employmentcheck/?filter={"instruction_id":"2"}&range=[0,24]&sort=["id","DESC"]/
Notice how the URL specifies a filter object, in which, it defines the parameters to filter against.
My question is, how and where can I filter my model without changing the URL pattern from my client?
Any other advise that spans the scope of my question is equally welcomed
Models.py
class EmploymentCheck(models.Model):
instructions = models.ForeignKey(Instruction, on_delete=models.CASCADE, null=True)
I simply omitted the filter_class, and instead overrode the get_queryset() method of the viewset.
Allow me to make some changes, the commented out code would fail if you sent a request with no 'filter' query params. e.g GET:/api/employee/ as opposed to GET:/api/employee/?filter{"instruction"=""}
So I opted to check if the query_params.get('key') was none, if so, i pass an empty string.
Any better pythonic way of handling this is encouraged
ViewSet
class EmploymentCheckViewSet(viewsets.ModelViewSet):
serializer_class = EmploymentCheckSerializer
def get_queryset(self):
queryset = EmploymentCheck.objects.all()
if self.request.query_params.get('filter') is None:
return queryset #return the queryset as is
else:
_instructions = self.request.query_params.get('filter') #returned as a string
_instructions = json.loads(_instructions)#convert to dictionary
if queryset and any(_instructions):
queryset = queryset.filter(instructions = _instructions['instruction_id'])
return queryset
Note: When you override the get_queryset, you must explicitly define the base_name argument of the router.register method in your app.urls.py. Refer to django-rest-framework routers.
urls.py
router = DefaultRouter()
router.register(r'employmentcheck', views.EmploymentCheckViewSet, 'EmploymentCheck')#explicitly set the base_name attribute when the viewset defines a custom get_queryset method

Django REST Framework: Setting up prefetching for nested serializers

My Django-powered app with a DRF API is working fine, but I've started to run into performance issues as the database gets populated with actual data. I've done some profiling with Django Debug Toolbar and found that many of my endpoints issue tens to hundreds of queries in the course of returning their data.
I expected this, since I hadn't previously optimized anything with regard to database queries. Now that I'm setting up prefetching, however, I'm having trouble making use of properly prefetched serializer data when that serializer is nested in a different serializer. I've been using this awesome post as a guide for how to think about the different ways to prefetch.
Currently, my ReadingGroup serializer does prefetch properly when I hit the /api/readinggroups/ endpoint. My issue is the /api/userbookstats/ endpoint, which returns all UserBookStats objects. The related serializer, UserBookStatsSerializer, has a nested ReadingGroupSerializer.
The models, serializers, and viewsets are as follows:
models.py
class ReadingGroup(models.model):
owner = models.ForeignKeyField(settings.AUTH_USER_MODEL)
users = models.ManyToManyField(settings.AUTH_USER_MODEL)
book_type = models.ForeignKeyField(BookType)
....
<other group related fields>
def __str__(self):
return '%s group: %s' % (self.name, self.book_type)
class UserBookStats(models.Model):
reading_group = models.ForeignKey(ReadingGroup)
user = models.ForeignKey(settings.AUTH_USER_MODEL)
alias = models.CharField()
total_books_read = models.IntegerField(default=0)
num_books_owned = models.IntegerField(default=0)
fastest_read_time = models.IntegerField(default=0)
average_read_time = models.IntegerField(default=0)
serializers.py
class ReadingGroupSerializer(serializers.ModelSerializer):
users = UserSerializer(many = True,read_only=True)
owner = UserSerializer(read_only=True)
class Meta:
model = ReadingGroup
fields = ('url', 'id','owner', 'users')
#staticmethod
def setup_eager_loading(queryset):
#select_related for 'to-one' relationships
queryset = queryset.select_related('owner')
#prefetch_related for 'to-many' relationships
queryset = queryset.prefetch_related('users')
return queryset
class UserBookStatsSerializer(serializers.HyperlinkedModelSerializer):
reading_group = ReadingGroupSerializer()
user = UserSerializer()
awards = AwardSerializer(source='award_set', many=True)
class Meta:
model = UserBookStats
fields = ('url', 'id', 'alias', 'total_books_read', 'num_books_owned',
'average_read_time', 'fastest_read_time', 'awards')
#staticmethod
def setup_eager_loading(queryset):
#select_related for 'to-one' relationships
queryset = queryset.select_related('user')
#prefetch_related for 'to-many' relationships
queryset = queryset.prefetch_related('awards_set')
#setup prefetching for nested serializers
groups = Prefetch('reading_group', queryset ReadingGroup.objects.prefetch_related('userbookstats_set'))
queryset = queryset.prefetch_related(groups)
return queryset
views.py
class ReadingGroupViewset(views.ModelViewset):
def get_queryset(self):
qs = ReadingGroup.objects.all()
qs = self.get_serializer_class().setup_eager_loading(qs)
return qs
class UserBookStatsViewset(views.ModelViewset):
def get_queryset(self):
qs = UserBookStats.objects.all()
qs = self.get_serializer_class().setup_eager_loading(qs)
return qs
I've optimized the prefetching for the ReadingGroup endpoint (I actually posted about eliminating duplicate queries for that endpoint here), and now I'm working on the UserBookStats endpoint.
The issue I'm having is that, with my current setup_eager_loading in the UserBookStatsSerializer, it doesn't appear to use the prefetching set up by the eager loading method in the ReadingGroupSerializer. I'm still a little hazy on the syntax for the Prefetch object - I was inspired by this excellent answer to try that approach.
Obviously the get_queryset method of UserBookStatsViewset doesn't call setup_eager_loading for the ReadingGroup objects, but I'm sure there's a way to accomplish the same prefetching.
prefetch_related() supports prefetching inner relations by using double underscore syntax:
queryset = queryset.prefetch_related('reading_group', 'reading_group__users', 'reading_group__owner')
I don't think Django REST provides any elegant solutions out of the box for fetching all necessary fields automatically.
An alternative to prefetching all nested relationships manually, there is also a package called django-auto-prefetching which will automatically traverse related fields on your model and serializer to find all the models which need to be mentioned in prefetch_related and select_related calls. All you need to do is add in the AutoPrefetchViewSetMixin to your ViewSets:
from django_auto_prefetching import AutoPrefetchViewSetMixin
class ReadingGroupViewset(AutoPrefetchViewSetMixin, views.ModelViewset):
def get_queryset(self):
qs = ReadingGroup.objects.all()
return qs
class UserBookStatsViewset(AutoPrefetchViewSetMixin, views.ModelViewset):
def get_queryset(self):
qs = UserBookStats.objects.all()
return qs
Any extra prefetches with more complex Prefetch objects can be added in the get_queryset method on the ViewSet.

Django Rest Framework Update or Create if not exists

I want to use the perform an update or create in django-rest-framework, by passing or not the id field. I've got this model
class Etiqueta(models.Model):
name_tag = models.CharField(max_length=200, blank=False, null=False)
description_tag = models.TextField(max_length=500, blank=False, null=False)
def __unicode__(self):
return self.name_tag
And in django-rest-framework I've got this serializer
from myapp.modulos.estado_1.models import Etiqueta
from rest_framework import serializers, viewsets
# Serializers define the API representation.
class TagSerializer(serializers.ModelSerializer):
class Meta:
model = Etiqueta
fields = (
'id',
'name_tag',
'description_tag'
)
# ViewSets define the view behavior.
class TagViewSet(viewsets.ModelViewSet):
queryset = Etiqueta.objects.all()
serializer_class = TagSerializer
Normally when I create an object, I perform a POST to the URL without the /:id, but if I've got an object with a local id, I want him to be created in the REST with the same id (remote id), django overwrite my local id and creates a new one. Does anybody know how achieve this? Also it is important to mention that I'm working with google-app-engine, google-cloud-datastore and django-dbindexer.
This code should work for your case -
class TagViewSet(viewsets.ModelViewSet):
queryset = Etiqueta.objects.all()
serializer_class = TagSerializer
def get_object(self):
if self.request.method == 'PUT':
obj, created = Etiquetta.objects.get_or_create(pk=self.kwargs.get('pk'))
return obj
else:
return super(TagViewSet, self).get_object()
You should have a look at how Django REST framework does currently and adapts your create method to update whenever you have an id field.
The original ViewSet.create is here and the ViewSet.update is here.
Please note that you will probably end up with two different serializers for /tag/ and /tag/:id since the later should not allow the id field to be writable while the former should.
I've write a drf views mixin for updating an object by id, if no corresponding object, just create it then update.