Update Two
So unfortunately, #Reza Torkaman Ahmadi's idea didn't work out in the end. This is because our program has a filtering function that relies on get_queryset, and overriding the get_queryset method in our views was messing up that function.
So, my partner and I discussed it, and here's what he came up with.
class OrderbyFilter(filters.OrderingFilter):
def get_ordering(self, request, queryset, view):
"""
Ordering is set by a comma delimited ?$orderby=... query parameter.
Extends the OrderingFilter of the django rest framework to redefine
ordering parameters to "asc" and "desc".
"""
params = request.query_params.get(self.ordering_param)
if params:
field_queries = [param.strip() for param in params.split(',')]
fields = []
for field_query in field_queries:
field_query = field_query.split()
if len(field_query) <= 2:
while "asc" in field_query:
field_query.remove("asc")
for i, field in enumerate(field_query):
if field == "desc":
field_query[i-1] = "-" + field_query[i-1]
while "desc" in field_query:
field_query.remove("desc")
fields.append(field_query[0])
else:
fields.append([param.strip() for param in params.split(',')])
ordering = self.remove_invalid_fields(queryset, fields, view, request)
if ordering:
return ordering
return self.get_default_ordering(view)
Basically, this function over-rides Django REST's source code, specifically the get_ordering function in the OrderingFilter. What it does is, if 'asc' is in the query after the field, it removes it and treats it like normal (for normal ascension ordering)
Otherwise if 'desc' is there, it removed the 'desc', and applies a hyphen.
Update Answered. Applied #Reza Torkaman Ahmadi idea and it works great after modifying it to fit my needs. Thanks mate!
Currently, in Django API rest framework, if a user wants to see a list of something in ascending or descending order, they have to do the following:
'Datastream&order_by=-name' shows names in descending order
'Datastream&order_by=name' shows names in ascending order.
I want to make a custom query where typing in 'Datastream&order_by=asc name' will order the names by ascending order, and 'desc name' will do so in descending order.
I've looked at some of the source code for the REST framework, but I may be looking in the wrong area. Getting stumped on what I should do. Any ideas?
You can do it in your own way. like this:
class DatastreamViewSet(ModelViewSet):
def get_queryset(self):
queryset = super(DatastreamViewSet, self).get_queryset()
order_by = self.request.query_params.get('order_by', '')
if order_by:
order_by_name = order_by.split(' ')[1]
order_by_sign = order_by.split(' ')[0]
order_by_sign = '' if order_by_sign == 'asc' else '-'
queryset = queryset.order_by(order_by_sign + order_by_name)
return queryset
this will look for query parameter order_by if is supplied then will split it by space, the first one will indicate to use + or - sign on order_by filter, and the second will be the name of it. so put it all together and create a text, pass it to order_by and your good to go.
for example:
?order_by=asc name
will be like this in django =>
return queryset.order_by('name')
Django Rest Framework's standard way:
from rest_framework.filters import OrderingFilter
Then on your APIView or ViewSet
filter_backends = (OrderingFilter,)
ordering_fields = ['field_name']
query parameter is ordering and supports reverse ordering as well.
GET https://example.com/?ordering=-field_name
Related
While creating a front end for a Django module I faced the following problem inside Django core:
In order to display a link to the next/previous object from a model query, we can use the extra-instance-methods of a model instance: get_next_by_FIELD() or get_previous_by_FIELD(). Where FIELD is a model field of type DateField or DateTimeField.
Lets explain it with an example
from django.db import models
class Shoe(models.Model):
created = models.DateTimeField(auto_now_add=True, null=False)
size = models.IntegerField()
A view to display a list of shoes, excluding those where size equals 4:
def list_shoes(request):
shoes = Shoe.objects.exclude(size=4)
return render_to_response(request, {
'shoes': shoes
})
And let the following be a view to display one shoe and the corresponding
link to the previous and next shoe.
def show_shoe(request, shoe_id):
shoe = Shoe.objects.get(pk=shoe_id)
prev_shoe = shoe.get_previous_by_created()
next_shoe = shoe.get_next_by_created()
return render_to_response('show_shoe.html', {
'shoe': shoe,
'prev_shoe': prev_shoe,
'next_shoe': next_shoe
})
Now I have the situation that the show_shoe view displays the link to the previous/next regardless of the shoes size. But I actually wanted just shoes whose size is not 4.
Therefore I tried to use the **kwargs argument of the get_(previous|next)_by_created() methods to filter out the unwanted shoes, as stated by the documentation:
Both of these methods will perform their queries using the default manager for the model. If you need to emulate filtering used by a custom manager, or want to perform one-off custom filtering, both methods also accept
optional keyword arguments, which should be in the format described in Field lookups.
Edit: Keep an eye on the word "should", because then also (size_ne=4) should work, but it doesn't.
The actual problem
Filtering using the lookup size__ne ...
def show_shoe(request, shoe_id):
...
prev_shoe = shoe.get_previous_by_created(size__ne=4)
next_shoe = shoe.get_next_by_created(size__ne=4)
...
... didn't work, it throws FieldError: Cannot resolve keyword 'size_ne' into field.
Then I tried to use a negated complex lookup using Q objects:
from django.db.models import Q
def show_shoe(request, shoe_id):
...
prev_shoe = shoe.get_previous_by_created(~Q(size=4))
next_shoe = shoe.get_next_by_created(~Q(size=4))
...
... didn't work either, throws TypeError: _get_next_or_previous_by_FIELD() got multiple values for argument 'field'
Because the get_(previous|next)_by_created methods only accept **kwargs.
The actual solution
Since these instance methods use the _get_next_or_previous_by_FIELD(self, field, is_next, **kwargs) I changed it to accept positional arguments using *args and passed them to the filter, like the **kwargs.
def my_get_next_or_previous_by_FIELD(self, field, is_next, *args, **kwargs):
"""
Workaround to call get_next_or_previous_by_FIELD by using complext lookup queries using
Djangos Q Class. The only difference between this version and original version is that
positional arguments are also passed to the filter function.
"""
if not self.pk:
raise ValueError("get_next/get_previous cannot be used on unsaved objects.")
op = 'gt' if is_next else 'lt'
order = '' if is_next else '-'
param = force_text(getattr(self, field.attname))
q = Q(**{'%s__%s' % (field.name, op): param})
q = q | Q(**{field.name: param, 'pk__%s' % op: self.pk})
qs = self.__class__._default_manager.using(self._state.db).filter(*args, **kwargs).filter(q).order_by('%s%s' % (order, field.name), '%spk' % order)
try:
return qs[0]
except IndexError:
raise self.DoesNotExist("%s matching query does not exist." % self.__class__._meta.object_name)
And calling it like:
...
prev_shoe = shoe.my_get_next_or_previous_by_FIELD(Shoe._meta.get_field('created'), False, ~Q(state=4))
next_shoe = shoe.my_get_next_or_previous_by_FIELD(Shoe._meta.get_field('created'), True, ~Q(state=4))
...
finally did it.
Now the question to you
Is there an easier way to handle this? Should shoe.get_previous_by_created(size__ne=4) work as expected or should I report this issue to the Django guys, in the hope they'll accept my _get_next_or_previous_by_FIELD() fix?
Environment: Django 1.7, haven't tested it on 1.9 yet, but the code for _get_next_or_previous_by_FIELD() stayed the same.
Edit: It is true that complex lookups using Q object is not part of "field lookups", it's more part of the filter() and exclude() functions instead. And I am probably wrong when I suppose that get_next_by_FIELD should accept Q objects too. But since the changes involved are minimal and the advantage to use Q object is high, I think these changes should get upstream.
tags: django, complex-lookup, query, get_next_by_FIELD, get_previous_by_FIELD
(listing tags here, because I don't have enough reputations.)
You can create custom lookup ne and use it:
.get_next_by_created(size__ne=4)
I suspect the method you've tried first only takes lookup arg for the field you're basing the get_next on. Meaning you won't be able to access the size field from the get_next_by_created() method, for example.
Edit : your method is by far more efficient, but to answer your question on the Django issue, I think everything is working the way it is supposed to. You could offer an additional method such as yours but the existing get_next_by_FIELD is working as described in the docs.
You've managed to work around this with a working method, which is OK I guess, but if you wanted to reduce the overhead, you could try a simple loop :
def get_next_by_field_filtered(obj, field=None, **kwargs):
next_obj = getattr(obj, 'get_next_by_{}'.format(field))()
for key in kwargs:
if not getattr(next_obj, str(key)) == kwargs[str(key)]:
return get_next_by_field_filtered(next_obj, field=field, **kwargs)
return next_obj
This isn't very efficient but it's one way to do what you want.
Hope this helps !
Regards,
I have a Forum Topic model that I want to order on a computed SerializerMethodField, such as vote_count. Here are a very simplified Model, Serializer and ViewSet to show the issue:
# models.py
class Topic(models.Model):
"""
An individual discussion post in the forum
"""
title = models.CharField(max_length=60)
def vote_count(self):
"""
count the votes for the object
"""
return TopicVote.objects.filter(topic=self).count()
# serializers.py
class TopicSerializer(serializers.ModelSerializer):
vote_count = serializers.SerializerMethodField()
def get_vote_count(self, obj):
return obj.vote_count()
class Meta:
model = Topic
# views.py
class TopicViewSet(TopicMixin, viewsets.ModelViewSet):
queryset = Topic.objects.all()
serializer_class = TopicSerializer
Here is what works:
OrderingFilter is on by default and I can successfully order /topics?ordering=title
The vote_count function works perfectly
I'm trying to order by the MethodField on the TopicSerializer, vote_count like /topics?ordering=-vote_count but it seems that is not supported. Is there any way I can order by that field?
My simplified JSON response looks like this:
{
"id": 1,
"title": "first post",
"voteCount": 1
},
{
"id": 2,
"title": "second post",
"voteCount": 8
},
{
"id": 3,
"title": "third post",
"voteCount": 4
}
I'm using Ember to consume my API and the parser is turning it to camelCase. I've tried ordering=voteCount as well, but that doesn't work (and it shouldn't)
This is not possible using the default OrderingFilter, because the ordering is implemented on the database side. This is for efficiency reasons, as manually sorting the results can be incredibly slow and means breaking from a standard QuerySet. By keeping everything as a QuerySet, you benefit from the built-in filtering provided by Django REST framework (which generally expects a QuerySet) and the built-in pagination (which can be slow without one).
Now, you have two options in these cases: figure out how to retrieve your value on the database side, or try to minimize the performance hit you are going to have to take. Since the latter option is very implementation-specific, I'm going to skip it for now.
In this case, you can use the Count function provided by Django to do the count on the database side. This is provided as part of the aggregation API and works like the SQL COUNT function. You can do the equivalent Count call by modifying your queryset on the view to be
queryset = Topic.objects.annotate(vote_count=Count('topicvote_set'))
Replacing topicvote_set with your related_name for the field (you have one set, right?). This will allow you to order the results based on the number of votes, and even do filtering (if you want to) because it is available within the query itself.
This would require making a slight change to your serializer, so it pulls from the new vote_count property available on objects.
class TopicSerializer(serializers.ModelSerializer):
vote_count = serializers.IntegerField(read_only=True)
class Meta:
model = Topic
This will override your existing vote_count method, so you may want to rename the variable used when annotating (if you can't replace the old method).
Also, you can pass a method name as the source of a Django REST framework field and it will automatically call it. So technically your current serializer could just be
class TopicSerializer(serializers.ModelSerializer):
vote_count = serializers.IntegerField(read_only=True)
class Meta:
model = Topic
And it would work exactly like it currently does. Note that read_only is required in this case because a method is not the same as a property, so the value cannot be set.
Thanks #Kevin Brown for your great explanation and answer!
In my case I needed to sort a serializerMethodField called total_donation which is the sum of donations from the UserPayments table.
UserPayments has:
User as a foreignKey
sum which is an IntegerField
related_name='payments'
I needed to get the total donations per User but only donations that have a status of 'donated', not 'pending'. Also needed to filter out the payment_type coupon, which is related through two other foreign keys.
I was dumbfounded how to join and filter those donations and then be able to sort it via ordering_fields.
Thanks to your post I figured it out!
I realized it needed to be part of the original queryset in order to sort with ordering.
All I needed to do was annotate the queryset in my view, using Sum() with filters inside like so:
class DashboardUserListView(generics.ListAPIView):
donation_filter = Q(payments__status='donated') & ~Q(payments__payment_type__payment_type='coupon')
queryset = User.objects.annotate(total_donated=Sum('payments__sum', filter=donation_filter ))
serializer_class = DashboardUserListSerializer
pagination_class = DashboardUsersPagination
filter_backends = [filters.OrderingFilter]
ordering_fields = ['created', 'last_login', 'total_donated' ]
ordering = ['-created',]
I will put it here because the described case is not the only one.
The idea is to rewrite the list method of your Viewset to order by any of your SerializerMethodField(s) also without moving your logic from the Serializer to the ModelManager (especially when you work with several complex methods and/or related models)
def list(self, request, *args, **kwargs):
response = super().list(request, args, kwargs)
ordering = request.query_params.get('ordering')
if "-" in ordering:
response.data['results'] = sorted(response.data['results'], key=lambda k: (k[ordering.replace('-','')], ), reverse=True)
else:
response.data['results'] = sorted(response.data['results'], key=lambda k: (k[ordering], ))
return response
I am working with django-rest-framework and I have an API that returns me the info with a filter like this:
http://example.com/api/products?category=clothing&in_stock=True
--this returns me 10 items
But it also returns the whole Model data if I dont put the filters, this is the default way.
http://example.com/api/products/
--this returns me more than 100 (all the Model Table)
How can I disable this default operation, I mean, how can I make a filter to be necesary to make this api works? or even better! how can I make the last URL to return an empty json response?
UPDATE
Here is some code:
serializers.py
class OEntradaDetalleSerializer(serializers.HyperlinkedModelSerializer):
item = serializers.RelatedField(source='producto.item')
descripcion = serializers.RelatedField(source='producto.descripcion')
unidad = serializers.RelatedField(source='producto.unidad')
class Meta:
model = OEntradaDetalle
fields = ('url','item','descripcion','unidad','cantidad_ordenada','cantidad_recibida','epc')
views.py
class OEntradaDetalleViewSet(BulkUpdateModelMixin,viewsets.ModelViewSet):
filter_backends = (filters.DjangoFilterBackend,)
filter_fields = ('cantidad_ordenada','cantidad_recibida','oentrada__codigo_proveedor','oentrada__folio')
queryset = OEntradaDetalle.objects.all()
serializer_class = OEntradaDetalleSerializer
urls.py
router2 = BulkUpdateRouter()
router2.register(r'oentradadetalle', OEntradaDetalleViewSet)
urlpatterns = patterns('',
url(r'^api/',include(router2.urls)),
)
URL EXAMPLE
http://localhost:8000/api/oentradadetalle/?oentrada__folio=E01
THIS RETURNS ONLY SOME FILTERED VALUES
http://localhost:8000/api/oentradadetalle/
THIS RETURNS EVERYTHING IN THE MODEL (I need to remove this or make it return some empty data)
I would highly recommend using pagination, to prevent anyone from being able to return all of the results (which likely takes a while).
If you can spare the extra queries being made, you can always check if the filtered and unfiltered querysets match, and just return an empty queryset if that is the case. This would be done in the filter_queryset method on your view.
def filter_queryset(self, queryset):
filtered_queryset = super(ViewSet, self).filter_queryset(queryset)
if queryset.count() === len(filtered_queryset):
return queryset.model.objects.none()
return filtered_queryset
This will make one additional query for the count of the original queryset, and if it is the same as the filtered queryset, an empty queryset will be returned. If the queryset was actually filtered, it will be returned and the results will be what you are expecting.
Suppose I have a Person model that has a first name field and a last name field. There will be many people who have the same first name. I want to write a TastyPie resource that allows me to get a list of the unique first names (without duplicates).
Using the Django model directly, you can do this easily by saying something like Person.objects.values("first_name").distinct(). How do I achieve the same thing with TastyPie?
Update
I've adapted the apply_filters method linked below to use the values before making the distinct call.
def apply_filters(self, request, applicable_filters):
qs = self.get_object_list(request).filter(**applicable_filters)
values = request.GET.get('values', '').split(',')
if values:
qs = qs.values(*values)
distinct = request.GET.get('distinct', False) == 'True'
if distinct:
qs = qs.distinct()
return qs
values returns dictionaries instead of model objects, so I don't think you need to override alter_list_data_to_serialize.
Original response
There is a nice solution to the distinct part of the problem here involving a light override of apply_filters.
I'm surprised I'm not seeing a slick way to filter which fields are returned, but you could implement that by overriding alter_list_data_to_serialize and deleting unwanted fields off the objects just before serialization.
def alter_list_data_to_serialize(self, request, data):
data = super(PersonResource, self).alter_list_data_to_serialize(request, data)
fields = request.GET.get('fields', None)
if fields is not None:
fields = fields.split(',')
# Data might be a bundle here. If so, operate on data.objects instead.
data = [
dict((k,v) for k,v in d.items() if k in fields)
for d in data
]
return data
Combine those two to use something like /api/v1/person/?distinct=True&values=first_name to get what you're after. That would work generally and would still work with additional filtering (&last_name=Jones).
I'm trying to order by a count of a manyToMany field is there a way to do this with TastyPie?
For example
class Person(models.Model):
friends = models.ManyToMany(User, ..)
I want PersonResource to spit out json that is ordered by the number of friends a person has...
is that possible?
I know this is an old question, but I recently encountered this problem and came up with a solution.
Tastypie doesn't easily allow custom ordering, but it is easy to modify the queryset it uses.
I actually just modified the default queryset for the model using a custom manager.
for instance:
class PersonManager(models.Manager):
def get_query_set(self):
return super(PersonManager self).get_query_set().\
annotate(friend_count=models.Count('friends'))
class Person(models.Model):
objects = PersonManager()
friends = ...
You could also add the annotation in Tastypie, wither in the queryset=... in the Meta class, or overriding the get_object_list(self,request) method.
I wasn't able to get the results ordering as per coaxmetal's solution, so I solved this a different way, by overriding the get_object_list on the Resource object as per http://django-tastypie.readthedocs.org/en/latest/cookbook.html. Basically if the 'top' querystring parameter exists, then the ordered result is returned.
class MyResource(ModelResource):
class Meta:
queryset = MyObject.objects.all()
def get_object_list(self, request):
try:
most_popular = request.GET['top']
result = super(MyResource, self).get_object_list(request).annotate(num_something=Count('something')).order_by('num_something')
except:
result = super(MyResource, self).get_object_list(request)
return result
I have not used TastyPie, but your problem seems to be more general. You can't have custom ordering in a Django ORM query. You're better off storing tuples of the form (Person, friend_count). This is pretty easy:
p_list = []
for person in Person.objects.all():
friendcount = len(person.friends.all())
p_list.append((person, friendcount))
Then, you can use the built in sorted function like so:
sorted_list = [person for (person, fc) in sorted(p_list, key=lambda x: x[1])]
The last line basically extracts the Persons from a sorted list of Persons, sorted on the no of friends one has.
`