Global Pagination not working on DRF Project - django

I have written an API on DRF which returns a list of data based on certain conditions, but the data is very large and global pagination is not applying on it. As a result, speed slows down and therefore, data is not shown properly on a single page.
I have adding following code in settings.py file:
REST_FRAMEWORK = {
"DEFAULT_PAGINATION_CLASS": "rest_framework.pagination.PageNumberPagination",
"PAGE_SIZE": 10
}
This is my API:
class TeacherViewSet(ModelViewSet):
queryset = Teacher.objects.all()
serializer_class = serializers.TeacherSerializer
authentication_classes = [TokenAuthentication]
def list(self, request, *args, **kwargs):
response = []
for teacher in queryset:
name = Student.objects.filter(teacher=teacher).values("name")
res = {"name": name}
response.append(res)
return Response(response)
Any thing wrong I am doing?

Since you are overriding list method you are disabling pagination feature. Default list method looks like this:
def list(self, request, *args, **kwargs):
queryset = self.filter_queryset(self.get_queryset())
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True)
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(queryset, many=True)
return Response(serializer.data)
Note paginate_queryset and get_paginated_response methods which perform pagination. So if you need to override list you should include these methods as well:
def list(self, request, *args, **kwargs):
response = []
queryset = self.filter_queryset(self.get_queryset())
queryset = self.paginate_queryset(queryset)
for teacher in queryset:
name = Student.objects.filter(teacher=teacher).values("name")
res = {"name": name}
response.append(res)
return self.get_paginated_response(response)
Not related to original question but please note that performing DB query inside a loop is considered a bad practice and could affect performance of your view.
Instead of fetching student for each teacher inside for loop consider to use prefetch_related.

Related

Custom queries in django serializers

I have the following view that allows me to determine defined prices for the logged in user and captures the information of the provider model requested by the user:
class ProductStoDists(mixins.ListModelMixin, generics.GenericAPIView):
permission_classes = (permissions.IsAuthenticated,IsSavedRole)
queryset = SellProduct.objects.select_related("provider", "product", "coupon_percentage", "coupon_quality").filter(~Q(stock = None), Q(disabled_sale=False))
serializer_class = SellProductSerializer
parser_classes = (MultiPartParser,)
filter_backends = [DjangoFilterBackend, filters.SearchFilter]
filterset_fields = ["product", "provider", "provider__state_prin_venue", "product__sectors", "product__categories", "provider__type_provider"]
pagination_class = Pagination
def get(self, request, *args, **kwargs):
return self.list(request, *args, **kwargs)
##method_decorator(cache_page(60*60))
##method_decorator(vary_on_cookie)
def list(self, request, *args, **kwargs):
queryset = self.filter_queryset(self.get_queryset())
page = self.paginate_queryset(queryset)
entity = Provider.objects.filter(administrator=request.user)
data_price = []
if page is not None:
serializer = self.get_serializer(page, many=True)
object_dataset = DataSetIds()
if request.user.role != "lab":
prices = []
for sell_product_cost in serializer.data:
provider = Provider.objects.filter(id_provider=sell_product_cost["provider"]).select_related("country_prin_venue", "state_prin_venue", "city_prin_venue", "administrator")
price_object = CustomPrices(entity, request.user.id, provider)
if provider[0].type_provider == "dist":
if request.user.role == "dist" or request.user.role == "sto":
sell_product_cost["distributors"] = price_object.get_sell_distributors(sell_product_cost["id_sell"])
else:
sell_product_cost["distributors"] = None
else:
if request.user.role == "sto":
sell_product_cost["storage"] = price_object.get_sell_storage(sell_product_cost["id_sell"])
else:
sell_product_cost["storage"] = None
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(queryset, many=True)
return Response(serializer.data)
How could I make the "storage" or distributors field be done from the serializardor and not from the view?, since I consider that this can slow down the response of the customer service.
Without knowing your model definitions, the answer likely involves performing this work at the SQL builder level rather than after querying the data. If you can't, then this points to an architectural problem that limits the abilities of your application.
For example let's say your SellProduct model has a foreign key to Provider. If you need to conditionally add information to the queryset then you can do so using annotate and Case
queryset = self.filter_queryset(self.get_queryset())
queryset = queryset.select_related('provider') \
.annotate(
distributors=Case(
When(provider__type_provider='dist', then=Value('foo')),
default=Value('other foo'),
output_field=CharField()
)
But if you're relying on python methods to fill in bits of information for each instance returned from the queryset then you need to re-assess your model structure and determine if you can better relate information to avoid this situation.

Django passing multiple ids through URL

For POST/GET (etc) requests I have the following URL for one user:
v1/users/userid123
registered as so:
router.register(r"v1/users", accounts_views_v1.UserViewSet)
What modifications should I make so that I can pass multiple user IDs like:
v1/users/?ids=["userid123","userid456"]?
It worked without any modification for another model of mine, with the same criteria, but this one keeps giving a 403 error (before even going into the method!) when I try it
Code per request:
My viewset is insanely long, here's the beginning though
class UserViewSet(MultipleDBModelViewSet):
serializer_class = UserSerializer
queryset = User.objects.none()
#workspace_specific
def get_queryset(self):
group_ids = json.loads(self.request.query_params.get("group_id", "[]"))
queryset = None
if group_ids:
queryset = User.objects.filter(group_memberships__group__in=group_ids).distinct()
else:
queryset = User.objects.all()
return queryset.select_related("workspace_role").prefetch_related(
"group_memberships__group_role", "group_memberships__group"
)
and my URLS:
PREFIX = settings.REST_FRAMEWORK_ROUTER_PREFIX
if PREFIX:
PREFIX = r"^" + str(PREFIX) + r"/"
router = BulkRouter()
single_object_router = SingleObjectRouter()
lazy_single_object_create_or_update_router = LazySingleObjectCreateOrUpdateRouter()
...
router.register(r"v1/users", accounts_views_v1.UserViewSet)
...
urlpatterns = [...]
GET method:
def list(self, request, *args, **kwargs):
queryset = self.filter_queryset(self.get_queryset())
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True, exclude=None)
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(queryset, many=True, exclude=None)
return Response(serializer.data)
I found the kink, this function wraps all the others:
def permission_check_wrapper(self, request, *args, pk=None, **kwargs)
of course, when there are IDs instead of one pk, it doesn't work - how can I pass ids instead?

Filter the queryset based on the domain making the request

I'm building a SaaS that will manage several websites on the frontend with NextJS (Universal react framework) using Django Rest Framework.
I want to filter the data based on the domain making the request, on the frontend I'm sending through the headers the domain and on the backend I'm filtering the data based on the domain, the problem is that when I try to return the data with the code below I get:
AttributeError: 'Response' object has no attribute 'model'
Here is my code:
class ListProperties(generics.ListAPIView):
queryset = models.Property.objects.all()
serializer_class = frontend.PropertyCard
filter_class = filters.PropertyFilterset
pagination_class = pagination.PropertyPageNumberPagination
def get_queryset(self):
domain = self.request.META['HTTP_DOMAIN']
qs = self.filter_queryset(self.queryset.filter(company__domain=domain))
serialized = self.get_serializer(qs,many=True)
return Response(serialized.data)
The expected result should be the data that corresponds to the domain passed through the headers. Filtered (if filters are applied) and paginated.
The qet_queryset [drf-doc] is supposed to, like the name suggests, return a QuerySet, not the response of that queryset. You should simply return:
class ListProperties(generics.ListAPIView):
queryset = models.Property.objects.all()
serializer_class = frontend.PropertyCard
filter_class = filters.PropertyFilterset
pagination_class = pagination.PropertyPageNumberPagination
def get_queryset(self):
domain = self.request.META['HTTP_DOMAIN']
return self.queryset.filter(company__domain=domain)
The list(..) function will automatically apply the defined filters [GitHub]:
def list(self, request, *args, **kwargs):
queryset = self.filter_queryset(self.get_queryset())
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True)
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(queryset, many=True)
return Response(serializer.data)
The (optional) pagination, serialization, and construction of the response are all handled by the ListAPIView [drf-doc] itself. If you want to override how the response is constructed for the given queryset, you should override the list(..) method [classy].

Django Rest Framework: DRYer pagination for custom actions

Suppose I need to set up several GET endpoints that look like this objects/past, objects/future. Example:
#action(detail=False, methods=["GET"], name="Past Objects")
def past(self, request, *args, **kwargs):
startdate = datetime.datetime.now()
some_user = UserProfile.objects.get(user__username="someuser")
queryset = self.queryset.filter(
other__attribute__profile=some_user,
creation_date__lte=startdate
).order_by("-creation_date")
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True)
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(queryset, many=True)
return Response(serializer.data)
The above works just fine. But is there anyway to avoid the page = ... -> serializer= ... part?
I have specified this in my ModelViewSet:
pagination_class = CustomObjectPagination
But it seems the pagination is only auto-applied to default methods like get_queryset and not custom actions. Do I have to write this boilerplate every time I specify a custom action like past?
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True)
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(queryset, many=True)
return Response(serializer.data)
Edit: Should have made it clearer that I'm asking specifically whether there's a built-in way to do the above.
I don't think we have any built-in to apply pagination on actions. But, we can have a simple decorator to do this. Make sure that your action returns a list or QuerySet when using this decorator.
from functools import wraps
from django.db.models import QuerySet
def paginate(func):
#wraps(func)
def inner(self, *args, **kwargs):
queryset = func(self, *args, **kwargs)
assert isinstance(queryset, (list, QuerySet)), "apply_pagination expects a List or a QuerySet"
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True)
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(queryset, many=True)
return Response(serializer.data)
return inner
#paginate
#action(detail=False, methods=["GET"], name="Past Objects")
def past(self, request, *args, **kwargs):
startdate = datetime.datetime.now()
some_user = UserProfile.objects.get(user__username="someuser")
queryset = self.queryset.filter(
other__attribute__profile=some_user,
creation_date__lte=startdate
).order_by("-creation_date")
return queryset
I wrote a simple function for that:
def response_with_paginator(viewset, queryset):
page = viewset.paginate_queryset(queryset)
if page is not None:
serializer = viewset.get_serializer(page, many=True)
return viewset.get_paginated_response(serializer.data)
return Response(viewset.get_serializer(queryset, many=True).data)
Usage is like:
#action(...)
def comments(self, ...):
queryset = Comment.objects.filter(...)
return response_with_paginator(self, queryset)

Order data by non-database field

I'm using Django rest framework, Here is my serializers.py for social app:
class SocialPostSerializer(serializers.ModelSerializer):
likes = serializers.SerializerMethodField() # define field
class Meta:
model = SocialPost
def get_likes(self, obj):
post_id = obj.id
#I get post_like from django-redis
post_like = get_redis_connection("default")
likes = post_like.get("post"+":"+str(post_id))
if likes == None:
return 0
else:
likes = likes.decode('utf-8')
return likes
With the code above, I got what I need from the API.
Since 'likes' doesn't exist in my database(Mysql here), I can't using order_by('likes') to sort the data with django ORM
I follow the doc here ListCreateAPIView which lead me to override list(): (I had override create() and get_queryset() before)
from operator import itemgetter
class PostList(generics.ListCreateAPIView):
permission_classes = (permissions.IsAuthenticatedOrReadOnly,)
queryset = Post.objects.all()
serializer_class = PostAllSerializer
def list(self, request, *args, **kwargs):
queryset = self.filter_queryset(self.get_queryset())
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True)
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(queryset, many=True)
#problem here
serializer.data = sorted(serializer.data, key=itemgetter(serializer.data['likes']))
return Response(serializer.data)
def create(self, request, *args, **kwargs):
act_id = request.data.get('act')
act = Act.objects.get(pk=act_id)
if act.act_type == 0:
if request.user != act.user:
return Response(status=403)
return super().create(request, args, kwargs)
def perform_create(self, serializer):
serializer.save(user=self.request.user)
def get_queryset(self):
queryset = Post.objects.all().order_by('-post_create_time')
act_id = self.request.query_params.get('act_id', None)
post_author = self.request.query_params.get('post_author', None)
if act_id is not None:
queryset = queryset.filter(act=act_id)
if post_author is not None:
queryset = queryset.filter(user__user_name=post_author)
return queryset
Nothing happened, What is wired is even when I uncomment
return Response(serializer.data)
Still nothing happened, Which part is wrong?
Another question is when I wanna add some extra data like 'question' when I use django FBV:
def results(request, question_id):
question = get_object_or_404(Question, pk=question_id)
return render(request, 'polls/results.html', {'question': 'some data I wanna add'})
Is it possible to add the data in serializer.data? For example, I wanna display user_id from which user request this api:
def list(self, request, *args, **kwargs):
queryset = self.filter_queryset(self.get_queryset())
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True)
return self.get_paginated_response(serializer.data)
user_id = request.user.id #how can I add the user_id into response data?
serializer = self.get_serializer(queryset, many=True)
return Response(serializer.data)
I look around the docs here Including extra context.
serializer = AccountSerializer(account, context={'request': request})
I don't really understand how to add data in it.
"Since 'likes' doesn't exist in my database(Mysql here), I can't using order_by('likes') to sort the data with django ORM"
You wont be able to sort the results using django ORM as the likes field is not a part of your DB table. The only way to do it is to sort the serializer.data that you get in your view. In your case serializer.data will be a list of dictionary, you can use sort command for list and sort on likes using lambda.
One thing to take care here is as you will be doing the sort by loading the data in memory, make sure that you dont load a lot of data. Have a check on memory utilization.
"Another question is when I wanna add some extra data like 'question' when I use django FBV"
I did'nt understand what is needed here.