I have a DRF ModelSerializer class that serializes anOrder model. This serializer has a field:
num_modelA = serializers.SerializerMethodField()
`
def get_num_modelA(self, o):
r = ModelA.objects.filter(modelB__modelC__order=o).count()
return r
Where ModelA has a ForeignKey field modelB, ModelB has a ForeignKey field modelC, and ModelC has a ForeignKey field order.
The problem with this is obviously that for each order that gets serialized it makes an additional query to the DB which slows performance down.
I've implemented a static method setup_eager_loading as described here that fixed the N+1 query problem for other fields I was having.
#staticmethod
def setup_eager_loading(queryset):
# select_related for "to-one" relationships
queryset = queryset.select_related('modelD','modelE')
return queryset
My idea was I could use prefetch_related as well to reduce the number of queries. But I am unsure how to do this since Order and ModelA are separated by multiple foreign keys. Let me know if any other information would be useful
You can work with an annotation:
from django.db.models import Count
# …
#staticmethod
def setup_eager_loading(queryset):
# select_related for "to-one" relationships
return queryset.select_related('modelD','modelE').annotate(
num_modelA=Count('modelC__modelB__modelA')
)
in the serializer for your Order, you can then use num_modelA as an IntegerField:
from rest_framework import serializers
class OrderSerializer(serializers.ModelSerializer):
num_modelA = serializers.IntegerField()
class Meta:
model = Order
fields = ['num_modelA', 'and', 'other', 'fields']
Related
I have Serializer which defines a custom column, in the serializers.py which I am trying to filter a query on.
The itemsoutstanding column example below, I am trying to filter it from my view but it returns "is not defined"
class OverView(serializers.ModelSerializer):
itemsoutstanding = serializers.SerializerMethodField()
class Meta:
model = Order
fields = ['id','total_price', 'created_at','itemsoutstanding']
def get_itemsoutstanding(self, obj):
count= Items.objects.filter(order=obj.id).count()
return count
In my view I am trying to filter on the serializer column, but it says it's not defined
queryset = Order.objects.all()
serializer_class = OverView
queryset = Order.objects.filter(shop=shop)
queryset = queryset.filter(itemsoutstanding> 0)
Is there any way to filter based on the serializer columns?
You will need to annotate the queryset in the definition, you may need to check the django docs for annotations and probably adjust this code to your models but the idea is this:
from django.db.models import Count
queryset = Order.objects.annotate(itemsoutstanding=Count("items"))
Then you can use the itemsoutstanting in the filter and in the serializer as a field:
class OverView(serializers.ModelSerializer):
class Meta:
model = Order
fields = ['id','total_price', 'created_at','itemsoutstanding']
The queryset has nothing to do with the serializer, so it does not know of your defined field in the serializer. That custom field is only added to the serialized data and not the queryset. What you need to do is annotate the field in the view when you get the queryset.
Having query like SELECT *, 'hello' AS world FROM myApp_myModel I'd like to serialize it to json.
Doesn't seem like a big deal, and there are plenty of similar questions on SO but none seems to give straight answer.
So far I've tried:
data = myModel.objects.raw(query)
# gives: ModelState is not serializable
json.dumps([dict(r.__dict__) for r in data])
# doesn't serialize 'world' column, only model fields:
serializers.serialize('json', data)
#dear God:
for r in data:
for k in dict(r.__dict__):
print(getattr(r,k))
The issue:
Builtin django core serializers are not ready to include extra fields ( from raw neither from annotation expression) It just takes model fields from _meta.local_fields.
You can see it at django django/core/serializers/base.py source code:
concrete_model = obj._meta.concrete_model #obj is an object model
...
for field in concrete_model._meta.local_fields:
if field.serialize or field is pk_parent:
if field.remote_field is None:
if (self.selected_fields is None
or field.attname in self.selected_fields):
self.handle_field(obj, field)
else:
if (self.selected_fields is None
or field.attname[:-3] in self.selected_fields):
self.handle_fk_field(obj, field)
django rest framework at rescue:
To solve your issue you can use a non builtin functionality. You can include a REST package in your project. For example django rest framework can handle extra fields:
from django.db.models import F
from aula.apps.alumnes.models import MyModel
from rest_framework.renderers import JSONRenderer
data=MyModel.objects.annotate(dummy = F('some_field') )
class MyModelSerializer(serializers.ModelSerializer):
dummy = serializers.CharField()
class Meta:
model = MyModel
fields = ('some_other_field','dummy')
read_only_fields = (
'dummy',
)
m=MyModelSerializer(data, many=True)
JSONRenderer().render(m.data)
You can create a DRF searializer for the task:
http://www.django-rest-framework.org/api-guide/serializers/
i.e.
class MyModelSerializer(serializers.ModelSerializer):
world = serializers.ReadOnlyField()
class Meta:
model = MyModel
fields = (world, ...)
you can also use serializer inheritance etc - see the docs.
There is a clean way you can do this using Django Rest Framework
First off did you know You can also execute queries containing fields that aren’t defined on the model when doing a Raw query
for example ( REF )
>>> people = Person.objects.raw('SELECT *, age(birth_date) AS age FROM myapp_person')
>>> for p in people:
... print("%s is %s." % (p.first_name, p.age))
John is 37.
Jane is 42.
That means you can use a standard serializer. You just need to tell the serializer what to do with fields that were not originally on the model consider the below. Needed to join 3 tables to a user. The user, the company they belong to and the companies membership. If your table has thousands of users and you did the standard serialiser method field, it would result in thousands of queries to get the related companies membership each time. so instead here was the solution I used
# api.py
class UserSAMAExportListApiView(ListAPIView):
serializer_class = UserExportSerializer
model = User
def get_queryset(self):
q = User.objects.raw(
"""
SELECT
[users_user].[id],
[users_user].[email],
[companies_company].[title] AS company__title,
[companies_company].[registration_number] AS company__registration_number,
[memberships_membership].number AS company__membership__number
FROM [users_user]
LEFT OUTER JOIN [dbo].[companies_company]
ON ([users_user].[company_id] = [companies_company].[id])
LEFT OUTER JOIN [memberships_membership]
ON ([companies_company].[id] = [memberships_membership].[company_id])
WHERE ([memberships_membership].[expiry_date] >= %s)
"""
, [date.today(),]
)
return q
Then just tell your standard serialiser that there are some new fields you defined
# serializers.py
class UserExportSerializer(ModelSerializer):
class Meta:
model = User
fields = (
'id',
'email',
'company__title',
'company__registration_number',
'company__membership__number',
)
def build_unknown_field(self, field_name, model_class):
"""
Return a two tuple of (cls, kwargs) to build a serializer field with. For fields that werent originally on
The model
"""
return fields.CharField, {'read_only': True}
And that's it DRF will handle the rest in a standard way and do proper serialization for you
Note you have to override the build_unknown_fields method. This is simply saying convert all the non-standard model fields to Text, if you want you can check the field name and convert to other formats here.
I have a serializer that gives me everything fine.
ModelClassASerializer((serializers.ModelSerializer)):
.....
status = serializers.SerializerMethodField()
def get_status(self, obj):
....
status = ModelB.objects.get(id=obj.id).status
....
return status
class Meta:
model = ModelClassA
fields = (...)
But if I want to make a filtering based on that status, I can't. I am using django_filters.rest_framework.FilterSet for the filtering. There is no relation between models.
What is the best way to do that filtering?
It looks like objects in ModelA share the same IDs as those in ModelB. If that's the case, you can use a subquery to match on IDs. If the IDs do not correspond with each other, then this query will be nonsensical. You want to create the following queryset:
from django.db.models import Subquery
from myapp.models import ModelA, ModelB
pks = ModelB.objects.filter(status='foo').values('pk')
ModelA.objects.filter(pk__in=Subquery(pks))
To create the above will django-filter, you'll need to use the method argument on a filter.
from django_filters import rest_framework as filters
class ModelAFilter(filters.FilterSet):
status = filters.ChoiceFilter(choices=(('foo', 'Foo'), ...), method='filter_status')
class Meta:
model = ModelA
fields = []
def filter_status(self, queryset, name, value):
pks = ModelB.objects.filter(status=value).values('pk')
return queryset.filter(pk__in=Subquery(pks))
My Django-powered app with a DRF API is working fine, but I've started to run into performance issues as the database gets populated with actual data. I've done some profiling with Django Debug Toolbar and found that many of my endpoints issue tens to hundreds of queries in the course of returning their data.
I expected this, since I hadn't previously optimized anything with regard to database queries. Now that I'm setting up prefetching, however, I'm having trouble making use of properly prefetched serializer data when that serializer is nested in a different serializer. I've been using this awesome post as a guide for how to think about the different ways to prefetch.
Currently, my ReadingGroup serializer does prefetch properly when I hit the /api/readinggroups/ endpoint. My issue is the /api/userbookstats/ endpoint, which returns all UserBookStats objects. The related serializer, UserBookStatsSerializer, has a nested ReadingGroupSerializer.
The models, serializers, and viewsets are as follows:
models.py
class ReadingGroup(models.model):
owner = models.ForeignKeyField(settings.AUTH_USER_MODEL)
users = models.ManyToManyField(settings.AUTH_USER_MODEL)
book_type = models.ForeignKeyField(BookType)
....
<other group related fields>
def __str__(self):
return '%s group: %s' % (self.name, self.book_type)
class UserBookStats(models.Model):
reading_group = models.ForeignKey(ReadingGroup)
user = models.ForeignKey(settings.AUTH_USER_MODEL)
alias = models.CharField()
total_books_read = models.IntegerField(default=0)
num_books_owned = models.IntegerField(default=0)
fastest_read_time = models.IntegerField(default=0)
average_read_time = models.IntegerField(default=0)
serializers.py
class ReadingGroupSerializer(serializers.ModelSerializer):
users = UserSerializer(many = True,read_only=True)
owner = UserSerializer(read_only=True)
class Meta:
model = ReadingGroup
fields = ('url', 'id','owner', 'users')
#staticmethod
def setup_eager_loading(queryset):
#select_related for 'to-one' relationships
queryset = queryset.select_related('owner')
#prefetch_related for 'to-many' relationships
queryset = queryset.prefetch_related('users')
return queryset
class UserBookStatsSerializer(serializers.HyperlinkedModelSerializer):
reading_group = ReadingGroupSerializer()
user = UserSerializer()
awards = AwardSerializer(source='award_set', many=True)
class Meta:
model = UserBookStats
fields = ('url', 'id', 'alias', 'total_books_read', 'num_books_owned',
'average_read_time', 'fastest_read_time', 'awards')
#staticmethod
def setup_eager_loading(queryset):
#select_related for 'to-one' relationships
queryset = queryset.select_related('user')
#prefetch_related for 'to-many' relationships
queryset = queryset.prefetch_related('awards_set')
#setup prefetching for nested serializers
groups = Prefetch('reading_group', queryset ReadingGroup.objects.prefetch_related('userbookstats_set'))
queryset = queryset.prefetch_related(groups)
return queryset
views.py
class ReadingGroupViewset(views.ModelViewset):
def get_queryset(self):
qs = ReadingGroup.objects.all()
qs = self.get_serializer_class().setup_eager_loading(qs)
return qs
class UserBookStatsViewset(views.ModelViewset):
def get_queryset(self):
qs = UserBookStats.objects.all()
qs = self.get_serializer_class().setup_eager_loading(qs)
return qs
I've optimized the prefetching for the ReadingGroup endpoint (I actually posted about eliminating duplicate queries for that endpoint here), and now I'm working on the UserBookStats endpoint.
The issue I'm having is that, with my current setup_eager_loading in the UserBookStatsSerializer, it doesn't appear to use the prefetching set up by the eager loading method in the ReadingGroupSerializer. I'm still a little hazy on the syntax for the Prefetch object - I was inspired by this excellent answer to try that approach.
Obviously the get_queryset method of UserBookStatsViewset doesn't call setup_eager_loading for the ReadingGroup objects, but I'm sure there's a way to accomplish the same prefetching.
prefetch_related() supports prefetching inner relations by using double underscore syntax:
queryset = queryset.prefetch_related('reading_group', 'reading_group__users', 'reading_group__owner')
I don't think Django REST provides any elegant solutions out of the box for fetching all necessary fields automatically.
An alternative to prefetching all nested relationships manually, there is also a package called django-auto-prefetching which will automatically traverse related fields on your model and serializer to find all the models which need to be mentioned in prefetch_related and select_related calls. All you need to do is add in the AutoPrefetchViewSetMixin to your ViewSets:
from django_auto_prefetching import AutoPrefetchViewSetMixin
class ReadingGroupViewset(AutoPrefetchViewSetMixin, views.ModelViewset):
def get_queryset(self):
qs = ReadingGroup.objects.all()
return qs
class UserBookStatsViewset(AutoPrefetchViewSetMixin, views.ModelViewset):
def get_queryset(self):
qs = UserBookStats.objects.all()
return qs
Any extra prefetches with more complex Prefetch objects can be added in the get_queryset method on the ViewSet.
Basically, I want to filter out inactive users from a related field of a ModelSerializer. I tried Dynamically limiting queryset of related field as well as the following:
class MySerializer(serializers.ModelSerializer):
users = serializers.PrimaryKeyRelatedField(queryset=User.objects.filter(active=True), many=True)
class Meta:
model = MyModel
fields = ('users',)
Neither of these approaches worked for just filtering the queryset. I want to do this for a nested related Serializer class as a field (but couldn't even get it to work with a RelatedField).
How do I filter queryset for nested relation?
I'll be curious to see a better solution as well. I've used a custom method in my serializer to do that. It's a bit more verbose but at least it's explicit.
Some pseudo code where a GarageSerializer would filter the nested relation of cars:
class MyGarageSerializer(...):
users = serializers.SerializerMethodField('get_cars')
def get_cars(self, garage):
cars_queryset = Car.objects.all().filter(Q(garage=garage) | ...).select_related()
serializer = CarSerializer(instance=cars_queryset, many=True, context=self.context)
return serializer.data
Obviously replace the queryset with whatever you want. You don't always need the to give the context (I used it to retrieve some query parameters in the nested serializer) and you probably don't need the .select_related (that was an optimisation).
One way to do this is to create a method on the Model itself and reference it in the serializer:
#Models.py
class MyModel(models.Model):
#...
def my_filtered_field (self):
return self.othermodel_set.filter(field_a = 'value_a').order_by('field_b')[:10]
#Serialziers.py
class MyModelSerialzer(serializers.ModelSerializer):
my_filtered_field = OtherModelSerializer (many=True, read_only=True)
class Meta:
model = MyModel
fields = [
'my_filtered_field' ,
#Other fields ...
]
Another way to avoid the SerializerMethodField solution and therefore still allow writing to the serializer as well would be to subclass the RelatedField and do the filtering there.
To only allow active users as values for the field, the example would look like:
class ActiveUsersPrimaryKeyField(serializers.PrimaryKeyRelatedField):
def get_queryset(self):
return super().get_queryset().filter(active=True)
class MySerializer(serializers.ModelSerializer):
users = ActiveUsersPrimaryKeyField(many=True)
class Meta:
model = MyModel
fields = ('users',)
Also see this response.
Note that this only restricts the set of input values to active users, though, i.e. only when creating or updating model instances, inactive users will be disallowed.
If you also use your serializer for reading and MyModel already has a relation to a user that has become inactive in the meantime, it will still be serialized. To prevent this, one way is to filter the relation using django's Prefetch objects. Basically, you'll filter out inactive users before they even get into the serializer:
from django.db.models import Prefetch
# Fetch a model instance, eagerly prefetching only those users that are active
model_with_active_users = MyModel.objects.prefetch_related(
Prefetch("users", queryset=User.objects.filter(active=True))
).first()
# serialize the data with the serializer defined above and see that only active users are returned
data = MyModelSerializer(model_with_active_users).data