Django Rest Framework : Pagination with annotated Queryset - django

I am currently doing a project which requires pagination. A list of Images is provided to the user and whether they were Favorited or not by user.
Favorited Images are kept in a separate table.
In order to provide a list of images and ones that were Favorited by the user an annotate.
def _get_queryset(self, request):
user_favorited = DjImagingImagefavorites.objects.filter(ifv_img_recordid__exact=OuterRef('pk'), ifv_user_id__exact=request.user.profile)
queryset = DjImagingImage.objects.all().annotate(favorited=Exists(user_favorited))
return queryset
Then in the list function
def list(self, request):
images = self._get_queryset(request)
page = self.paginate_queryset(images) #Breaks here
The Query then throws an error.
]Cannot use an aggregate or a subquery in an expression used for the group by list of a GROUP BY clause.
Due to an oddities of how the paginate function performs the count and constructs an illegal sql statement.
Question is - Is their a better way about going to do this or should this work perfectly as I thought it should have?

A solution that I found works is coded below
def list(self, request):
images = self._get_queryset(request)
#page = self.paginate_queryset(images)
return self.paginate(images)
I create a wrapper called paginate
def paginate(self, queryset):
"""Calculate paginated QuerySet and wrap in a Response.
Args:
queryset - the queryset to be paginated
"""
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True)
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(queryset, many=True)
return Response(serializer.data)
For those who are wondering the queryset I am constructing
def get_queryset(self):
"""Return the modified/prepared master QuerySet.
Returns:
QuerySet<DjImagingImage>: The modified master QuerySet
"""
return DjImagingImage.objects.annotate(
mine=FilteredRelation('djimaginguserspecifics',condition=Q(djimaginguserspecifics__usp_emp_employeeid=self.request.user.profile)),
usp_favorite=F('mine__usp_favorite'),
usp_inhistory=F('mine__usp_inhistory'),
usp_emp_employeeid=F('mine__usp_emp_employeeid'),
usp_lastviewed=F('mine__usp_lastviewed'),
comment_count=Count('djimagingimagecomment'))

Related

How to filter nested data by property of parent object, using Django Rest Framework viewset

I have nested data; a List contains many Items. For security, I filter Lists by whether the current user created the list, and whether the list is public. I would like to do the same for items, so that items can only be updated by authenticated users, but can be viewed by anybody if the list is public.
Here's my viewset code, adapted from the List viewset code which works fine. This of course doesn't work for Items because the item doesn't have the properties "created_by" or "is_public" - those are properties of the parent list.
Is there a way I can replace "created_by" and "is_public" with the list properties? i.e. can I get hold of the parent list object in the item's get_queryset method, and check it's properties?
The alternative is that I assign "created_by" and "is_public" to the item as well, but I would prefer not to do that because it is duplicated data. The lists's properties should control the item's permissions.
class ItemViewSet(viewsets.ModelViewSet):
permission_classes = [permissions.AllowAny, ]
model = Item
serializer_class = ItemSerializer
def get_queryset(self):
# restrict any method that can alter a record
restricted_methods = ['POST', 'PUT', 'PATCH', 'DELETE']
if self.request.method in restricted_methods:
# if you are not logged in you cannot modify any list
if not self.request.user.is_authenticated:
return Item.objects.none()
# you can only modify your own lists
# only a logged-in user can create a list and view the returned data
return Item.objects.filter(created_by=self.request.user)
# GET method (view item) is available to owner and for items in public lists
if self.request.method == 'GET':
if not self.request.user.is_authenticated:
return Item.objects.filter(is_public__exact=True)
return Item.objects.filter(Q(created_by=self.request.user) | Q(is_public__exact=True))
# explicitly refuse any non-handled methods
return Item.objects.none()
Many thanks for any help!
Edit: between Lucas Weyne's answer and this post I think I have got this sorted now. Here's my working code in api.py:
from rest_framework import viewsets, permissions
from .models import List, Item
from .serializers import ListSerializer, ItemSerializer
from django.db.models import Q
class IsOwnerOrReadOnly(permissions.BasePermission):
def has_object_permission(self, request, view, obj):
# handle permissions based on method
# Read permissions are allowed to any request,
# so we'll always allow GET, HEAD or OPTIONS requests.
if request.method in permissions.SAFE_METHODS:
return True
if hasattr(obj, 'created_by'):
return obj.created_by == request.user
if hasattr(obj, 'list'):
if hasattr(obj.list, 'created_by'):
return obj.list.created_by == request.user
class ListViewSet(viewsets.ModelViewSet):
permission_classes = [IsOwnerOrReadOnly]
model = List
serializer_class = ListSerializer
def get_queryset(self):
# can view public lists and lists the user created
if self.request.user.is_authenticated:
return List.objects.filter(
Q(created_by=self.request.user) |
Q(is_public=True)
)
return List.objects.filter(is_public=True)
def pre_save(self, obj):
obj.created_by = self.request.user
class ItemViewSet(viewsets.ModelViewSet):
permission_classes = [IsOwnerOrReadOnly]
model = Item
serializer_class = ItemSerializer
def get_queryset(self):
# can view items belonging to public lists and lists the usesr created
if self.request.user.is_authenticated:
return Item.objects.filter(
Q(list__created_by=self.request.user) |
Q(list__is_public=True)
)
return Item.objects.filter(list__is_public=True)
Django allows lookups that span relationships. You can filter Item objects across List properties, just use the field name of related fields across models, separated by double underscores, until you get to the field you want.
class ItemViewSet(viewsets.ModelViewSet):
permission_classes = [IsOwnerOrReadyOnly]
serializer_class = ItemSerializer
def get_queryset(self):
if self.request.user.is_authenticated
return Item.objects.filter(
Q(list__created_by=self.request.user) |
Q(list__is_public__exact=True)
)
return Item.objects.filter(list__is_public=True)
To allow items to be updated only by its owners, write a custom object-level permission class.
class IsOwnerOrReadOnly(permissions.BasePermission):
def has_object_permission(self, request, view, obj):
# Read permissions are allowed to any request,
# so we'll always allow GET, HEAD or OPTIONS requests.
if request.method in permissions.SAFE_METHODS:
return True
# Instance must have an attribute named `created_by`.
return obj.list.created_by == request.user

Django Rest Framework return empty list in results randomly, but count still correct

When I'm trying to use custom filter it's working all the time in local development, but when we deploy to Production (Docker Swarm) We found an issue that sometime the API response is return empty results randomly, but the count is correct. Below is the example results from the API
API Response
{
'count': 1,
'next': 'http://localhost:8000/api/somethings/?email=test%40example.com&limit=0&offset=0',
'previous': None,
'results': []
}
Right now we need to restart a uwsgi service (By restarting docker swarm for this service) and the problem is fixed for a moment and randomly happen again.
Here is our DRF view
class SomeView(ListCreateAPIView):
queryset = SomeModel.objects.all()
serializer_class = SomeModelSerializer
filter_backends = (OrderingFilter, DjangoFilterBackend)
filter_class = CustomFilter
ordering = ('id',)
def list(self, request, *args, **kwargs):
if request.GET.get('all', None):
# Do something
serializer = self.get_serializer(queryset, many=True)
return Response(serializer.data)
else:
return super(SomeView, self).list(self, request, *args, **kwargs)
Here is our CustomFilter
from django_filters.rest_framework.filters import CharFilter
import rest_framework_filters as filters
class CustomFilter(filters.FilterSet):
json_field_separator = '___'
json_field_is = CharFilter(name='json_field', method='filter_json_field')
json_field_is_not = CharFilter(name='json_field', method='exclude_json_field')
def split_lookup_field(self, value):
return dict(field.split(':') for field in value.split(self.json_field_separator))
def filter_json_field(self, queryset, name, value):
try:
lookup_field = self.split_lookup_field(value)
return queryset.filter(**lookup_field)
except (ValueError, FieldError):
return queryset.none()
def exclude_json_field(self, queryset, name, value):
try:
lookup_field = self.split_lookup_field(value)
except (ValueError, FieldError):
return queryset.none()
for query_arg, query_value in lookup_field.items():
queryset = queryset.exclude(**{query_arg: query_value})
return queryset
class Meta:
model = SomeModel
exclude = ['image', 'json_field']
Here is a version of Package we use for this project
Django==1.10.8
djangorestframework==3.6.4
django-filter==1.0.4
djangorestframework-filters==0.10.2
In GenericAPIView you can find a method which is called: get_queryset(), which looks like this:
def get_queryset(self):
"""
Get the list of items for this view.
This must be an iterable, and may be a queryset.
Defaults to using `self.queryset`.
This method should always be used rather than accessing `self.queryset`
directly, as `self.queryset` gets evaluated only once, and those results
are cached for all subsequent requests.
You may want to override this if you need to provide different
querysets depending on the incoming request.
(Eg. return a list of items that is specific to the user)
"""
assert self.queryset is not None, (
"'%s' should either include a `queryset` attribute, "
"or override the `get_queryset()` method."
% self.__class__.__name__
)
queryset = self.queryset
if isinstance(queryset, QuerySet):
# Ensure queryset is re-evaluated on each request.
queryset = queryset.all()
return queryset
It is a quick call, but I think your queryset is not re-evaluated on each request.
Take a look at this comment: # Ensure queryset is re-evaluated on each request.

How to upgrade an API from pre 3.0 for Django Rest Framework and getting RetrieveAPIView to work with multiple PKs

I'm upgrading an API from pre 3.0 for Django Rest Framework. It's turning out to be a real struggle for RetrieveAPIView.
Here's my old code:
class ObjectLogLatest(generics.RetrieveAPIView):
serializer_class = ObjectLogSerializer
model = models.ObjectLog
def get_queryset(self):
model_one = self.kwargs['s_pk']
model_two = self.kwargs['p_pk']
user = self.request.user
if self.request.user.is_superuser:
try:
return models.ObjectLog.objects.filter(model_one=model_one).filter(model_two=model_two).latest('created')
except:
pass
else:
try:
return models.SolutionLog.objects.filter(user=self.request.user).filter(model_one=model_one).filter(model_two=model_two).latest('created')
except:
pass
Here's my new code:
class MultipleFieldLookupMixin(object):
"""
Apply this mixin to any view or viewset to get multiple field filtering
based on a `lookup_fields` attribute, instead of the default single field filtering.
"""
def get_object(self):
queryset = self.get_queryset() # Get the base queryset
queryset = self.filter_queryset(queryset) # Apply any filter backends
filter = {}
for field in self.lookup_fields:
filter[field] = self.kwargs[field]
return get_object_or_404(queryset, **filter) # Lookup the object
class ObjectLogLatest(MultipleFieldLookupMixin, generics.RetrieveAPIView):
serializer_class = ObjectLogSerializer
lookup_fields = ('model_one', 'model_two')
def get_queryset(self):
model_one = models.Model_one.objects.get(id=self.kwargs['s_pk'])
model_two = models.Model_two.objects.get(id=self.kwargs['p_pk'])
if self.request.user.is_superuser:
try:
return models.ObjectLog.objects.filter(model_one=model_one).filter(model_two=model_two).latest('created')
except:
pass
else:
try:
return models.ObjectLog.objects.filter(user=self.request.user).filter(model_one=model_one).filter(model_two=model_two).latest('created')
except:
pass
Here's the URL:
url(r'^api/sll/(?P<p_pk>\d+)/(?P<s_pk>\d+)/$', scen_views.SolutionLogLatest.as_view()),
I'm wondering how to do the same thing as the old code to find the specific ObjectLog while filtering for model_one and model_two.
The mixin comes from this article: http://www.django-rest-framework.org/api-guide/generic-views/
Any thoughts are welcome. Am I even headed in the right direction?

django MultiValueDictKeyError while requesting get object

I have made a form to filter ListView
class SingleNewsView(ListView):
model = News
form_class = SearchForm
template_name = "single_news.html"
def get(self, request, pk, **kwargs):
self.pk = pk
pub_from = request.GET['pub_date_from']
pub_to = request.GET['pub_date_to']
return super(SingleNewsView,self).get(request,pk, **kwargs)
My form fields are pub_date_from and pub_date_to. When I run the site it says:
MultiValueDictKeyError .
I don't know what's going on. When I remove the two line of getting pub_from and pub_to the site works fine. I want these two values to filter the queryset.
On first request there is no form data submitted so request.GET would not have any data. So doing request.GET['pub_date_from'] will fail. You shall use .get() method
pub_from = request.GET.get('pub_date_from')
pub_to = request.GET.get('pub_date_to')
If these keys are not in the dict, will return None. So handle the such cases appropriately in your code.
Also, if you want to filter objects for ListView add get_queryset() method to return filtered queryset as explained here Dynamic filtering

Django Rest, OrderingFilter using ForeignKey field?

I'm using Django Rest's OrderingFilter to order my API endpoint results (http://www.django-rest-framework.org/api-guide/filtering#orderingfilter)
like so:
/endpoint?ordering=-id
Is it possible to give it a foreign key field to order by?
like:
/endpoint?ordering=myfk__id
Trying the above doesn't seem to work for me
Just overwrite list method of ListModelMixin, then you can use all ORM ordering.
class ViewSet(mixins.ListModelMixin, viewsets.GenericViewSet):
def list(self, request, *args, **kwargs):
queryset = self.filter_queryset(self.get_queryset())
params = request.GET.get('ordering', '')
if params:
ordering = [param.strip() for param in params.split(',')]
queryset = queryset.order_by(*ordering)
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True)
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(queryset, many=True)
return Response(serializer.data)
Specify the ordering_fields in your view. So your field should look like:
ordering_fields = ('myfk__id',)
EDIT:
Looks like this is not currently allowed in DRF. See here. That being said, that site includes a snippet of some code you can use to implement it. Basically, use the filter code posted to subclass the OrderingFilter and use that as your filter instead.