I need to list all my devices. To do this I use a prefetch related to reduce the amount of queries. But one of them is consuming to much time.. I wonder if it can't go better.
I will start with the model construction: I want a list of devices. This is the device model:
class Device(models.Model):
name = models.CharField(max_length=250, null=True, blank=True)
def get_active_gateway(self):
from backend.gateways.models import Gateway
all_gatewaydevices = self.gatewaydevices.all()
for gd in all_gatewaydevices:
if not gd.end_date:
return gd.gateway
return None
In the real code the model is larger, but that code is irrelevant. As you can see, a device has some gatewaydevices (which is a model between gateway and device)
The gatewaydevice model looks like:
class GatewayDevice(models.Model):
gateway = models.ForeignKey(
Gateway, on_delete=models.CASCADE, related_name="devices"
)
device = models.ForeignKey(
Device, on_delete=models.CASCADE, related_name="gatewaydevices"
)
So in my list of devices, I want for every device, the linked gateway.
This is my view:
class AdminDeviceView(GenericAPIView):
def get_permissions(self):
return IsAuthenticated()
# noinspection PyMethodMayBeStatic
def get_serializer_class(self):
return AdminDeviceInfoSerializer
#swagger_auto_schema(
responses={
200: openapi.Response(
_("Successfully fetched all data from devices."),
AdminDeviceInfoSerializer,
)
}
)
def get(self, request):
devices = (
Device.objects.prefetch_related(
"gatewaydevices__gateway",
)
.all()
)
serializer_class = self.get_serializer_class()
serializer = serializer_class(devices, many=True)
devices_data = serializer.data
return Response(
{"total": devices.count(), "items": devices_data}, status=status.HTTP_200_OK
)
This is the part of the serializer that is important:
#staticmethod
def get_gateway(device):
gateway = device.get_active_gateway()
return GatewaySimpleSerializer(gateway).data if gateway else None
Can this get any faster/more efficient?
get_active_gateway is very inefficient - it check each object separately. And when called in get_gateway - it does not utilize prefetch bonuses.
You can filter and get only valid Gateway objects directly in prefetch.
Also using to_attr to map result to same field name as in serializer.
(*not tested)
class AdminDeviceView(GenericAPIView):
# ...
def get(self, request):
# ...
devices = Device.objects.prefetch_related(
Prefetch(
"gatewaydevices",
queryset=Gateway.objects.filter(
devices__end_date__isnull=True
).order_by().distinct(), # distinct in case there are many
to_attr="gateway",
)
)
# ...
class AdminDeviceInfoSerializer(serializers.ModelSerializer):
gateway = GatewaySimpleSerializer(many=True, read_only=True)
# ...
Related
Please be gentle. I'm a Django newb and I find the level of abstraction just plain overwhelming.
My ultimate goal is to modify an image file on its way into the model. That part may or may not be relevant, but assistance came my way in this post which advised me that I should be making changes inside a validator:
REST Django - How to Modify a Serialized File Before it is Put Into Model
Anyway, at the moment I am simply trying to get the context of the request so I can be sure to do the things to the thing only when the request is a POST. However, inside my validator, the self.context is just an empty dictionary. Based on what I have found out there, there should be a value for self.context['request'].
Here is what I have:
Serializer with validator method:
class MediaSerializer(serializers.ModelSerializer):
class Meta:
model = Media
fields = '__all__'
def validate_media(self, data):
print(self.context)
#todo: why is self.context empty?
#if self.context['request'].method == 'POST':
# print('do a thing here')
return data
def to_representation(self, instance):
data = super(MediaSerializer, self).to_representation(instance)
return data
The view along with the post method
class MediaView(APIView):
queryset = Media.objects.all()
parser_classes = (MultiPartParser, FormParser)
permission_classes = [permissions.IsAuthenticated, ]
serializer_class = MediaSerializer
def post(self, request, *args, **kwargs):
user = self.request.user
print(user.username)
request.data.update({"username": user.username})
media_serializer = MediaSerializer(data=request.data)
# media_serializer.update('username', user.username)
if media_serializer .is_valid():
media_serializer.save()
return Response(media_serializer.data, status=status.HTTP_201_CREATED)
else:
print('error', media_serializer.errors)
return Response(media_serializer.errors, status=status.HTTP_400_BAD_REQUEST)
The Model:
class Media(models.Model):
objects = None
username = models.ForeignKey(User, to_field='username',
related_name="Upload_username",
on_delete=models.DO_NOTHING)
date = models.DateTimeField(auto_now_add=True)
#temp_media = models.FileField(upload_to='upload_temp', null=True)
media = models.FileField(upload_to='albumMedia', null=True)
#todo: potentially this will go to a temp folder, optimize will be called and then permananent home will be used -jjb
#MEDIA_ROOT path must be /src/upload
file_type = models.CharField(max_length=12)
MEDIA_TYPES = (
('I', "Image"),
('V', "Video")
)
media_type = models.CharField(max_length=1, choices=MEDIA_TYPES, default='I')
ACCESSIBILITY = (
('P', "Public"),
('T', "Tribe"),
('F', "Flagged")
)
user_access = models.CharField(max_length=1, choices=ACCESSIBILITY, default='P')
So I'm just trying to figure out how to fix this context problem. Plus if there are any other tips on how to get where I'm going, I'd be most appreciative.
PS I'm pretty new here. If I wrote this question in a way that is inappropriate for stack overflow, please be kind, and I will correct it. Thanks.
I don't think you need to worry about checking if the request is a POST inside the validate_media() method. Generally validation only occurs during POST, PATCH, and PUT requests. On top of that, validation only occurs when you call is_valid() on the serializer, often explicitly in a view, as you do in your post() function. As long as you never call is_valid() from anywhere other than post(), you know that it is a POST. Since you don't support patch() or put() in your view, then this shouldn't be a problem.
inside my validator, the self.context is just an empty dictionary
You must explicitly pass in context when creating a serializer for it to exist. There is no magic here. As you can see in the source code context defaults to {} when you don't pass it in.
To pass in context, you can do this:
context = {'request': request}
media_serializer = MediaSerializer(data=request.data, context=context)
Even better, just pass in the method:
context = {'method': request.method}
media_serializer = MediaSerializer(data=request.data, context=context)
You can make the context dictionary whatever you want.
I am implementing some APIs using Django Rest Framework, and using the generateschema command to generate the OpenApi 3.0 specs afterwards.
While working on getting the schema to generate correctly, I realized that my code seemed to be duplicating a fair bit of logic between the FilterBackend and Serializer I was using. Both of them were accessing and validating the query parameters from the request.
I like the way of specifying the fields in the Serializer (NotesViewSetGetRequestSerializer in my case), and I would like to use that in my FilterBackend (NoteFilterBackend in my case). It would be nice to have access to the validated_data within the filter, and also be able to use the serializer to implement the filtering schemas.
Are there good solutions out there for only needing to specify your request query params once, and re-using with the filter and serializer?
I've reproduced a simplified version of my code below. I'm happy to provide more info on ResourceURNRelatedField if it's needed (it extends RelatedField and uses URNs instead of primary keys), but I think this would apply to any kind of field.
class NotesViewSet(generics.ListCreateAPIView, mixins.UpdateModelMixin):
allowed_methods = ("GET")
queryset = Note.objects.all()
filter_backends = [NoteFilterBackend]
serializer_class = NotesViewSetResponseSerializer
def get(self, request, *args, **kwargs):
query_params_dict = request.query_params
request_serializer = NotesViewSetGetRequestSerializer(data=query_params_dict)
request_serializer.is_valid(raise_exception=True)
validated_data = request_serializer.validated_data
member = validated_data.get("member_urn")
team = validated_data.get("team_urn")
if not provider_can_view_member(request.user, member, team):
return custom404(
request,
HttpResponseNotFound(
"Member does not exist!. URN={}".format(member.urn())
),
)
return super(NotesViewSet, self).list(request, *args, **kwargs)
class NotesViewSetGetRequestSerializer(serializers.Serializer):
member_urn = ResourceURNRelatedField(queryset=User.objects.all(), required=True)
team_urn = ResourceURNRelatedField(queryset=Team.objects.all(), required=True)
privacy_scope = serializers.CharField(required=False)
def validate_privacy_scope(self, value):
choices = dict(Note.PRIVACY_SCOPE_CHOICES)
if value and value not in choices:
raise serializers.ValidationError(
"bad privacy scope {}. Supported values are: {}".format(value, choices)
)
else:
return value
class NoteFilterBackend(BaseFilterBackend):
def filter_queryset(self, request, queryset, view):
member_urn = request.query_params.get("member_urn")
customer_uuid = URN.from_urn(member_urn).id
privacy_scope = request.query_params.get("privacy_scope")
team_urn = request.query_params.get("team_urn")
team_uuid = URN.from_urn(team_urn).id
queryset = queryset.filter(customer__uuid=customer_uuid)
if privacy_scope == Note.PRIVACY_SCOPE_TEAM_PROVIDERS:
queryset = queryset.filter(team__uuid=team_uuid)
return queryset
I was building my api in DJango and rest framework . Please see my model file
class StaffUser(models.Model):
staff_id=models.CharField(max_length=100,null=True,)
name=models.CharField(max_length=100,null=True)
user=models.OneToOneField(User, on_delete=models.CASCADE,related_name='staffs')
roles=models.ManyToManyField(BranchRole,related_name='holding_staffs')
published_date = models.DateTimeField(blank=True, null=True)
class Meta:
db_table = 'staff_details'
def save(self,*args,**kwargs):
email =kwargs['email']
password=kwargs['password']
del kwargs['email']
del kwargs['password']
self.published_date = timezone.now()
self.user=User.objects.create_user(
email=email,
password=password,
is_staff=True,
is_active=1
)
super(StaffUser,self).save(**kwargs)
return self
def __str__(self):
return self.name
When I am trying to call this save function in viewset , I am getting following exception.
"Exception on crate :TypeError("save() got an unexpected keyword argument 'name'"
Please help me to resolve this error. Please see my code in viewset
class StaffUserViewSet(viewsets.ModelViewSet):
"""
This api deals all operations related with module management
You will have `list`, `create`, `retrieve`,
update` and `destroy` actions.
Additionally we also provide an action to update status.
"""
serializer_class = StaffUserSerializer
permission_classes = [permissions.AllowAny]
queryset = StaffUser.objects.all()
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.ARG={
'staff_id':str,
'phone_number':str,
'email':str,
'name':str,
'password':str,
'address':str,
'id':int,
'roles':dict,
}
self.REPLACING_VALUES={
'model_name':'content_type__model_name',
'content_type_id':'content_type__id',
'app_name':'content_type__app_label' ,
'app_label':'content_type__app_label'
}
self.DEFAULT_PARAMETERS={
'content_type__status__active':True
}
self.VIEW_PARAMETERS_LIST=[
]
self.detailed_view=False
# if any user argument having another label that model relation
# will replace the smame with replacement values"""
self.api_model=StaffUser()
def create(self, request):
#over ride create method in view set
status='Sucess'
# set up detailed view is False
# alter this value to True if you need to alter thsi
content=[]
message="Sucessfull Created"
try:
query_params = data_formatter.request_to_dic(request.POST) # custom method to get input as dictionary
print(query_params)#it works
if (query_params is not None):
input_params=data_formatter.convert_type(query_params,self.ARG) #it works-custom method for convert
print(input_params)
validated_value=validations.is_data_valid(input_params,{'arg':self.ARG})#it works-custom method for convert
print(validated_value)
if(validated_value==1):
#To create group based on name
staff_obj=StaffUser().save(
**input_params
)
else:
status='Fail'
message=validations.ERROR_MESSAGES[validated_value]
except Exception as e:
message='Exception on crate :'+repr(e)
status='Fail'
return Response({"status":status,"data":content,"message":message})
you're using the wrong signature of save()
instead of
staff_obj=StaffUser().save(**input_params)
you should use
staff_obj=StaffUser.objects.create(**input_params)
or
staff_obj=StaffUser(**input_params)
staff_obj.save()
Read carefully the documentation
One more thing, you can rewrite this
email =kwargs['email']
password=kwargs['password']
del kwargs['email']
del kwargs['password']
in this way
email = kwargs.pop('email')
password = kwargs.pop('password')
UPDATE
Honestly not sure it'a a good idea to do this job in the save() method; why don't create a model's classmethod ?
#classmethod
def create staff_user(cls, **kwargs):
email = kwargs.pop('email')
password = kwargs.pop('password')
kwargs['published_date'] = timezone.now()
kwargs['user'] = User.objects.create_user(
email=email,
password=password,
is_staff=True,
is_active=1
)
return cls.objects.create(**kwargs)
and then call it in your View
staff_obj = StaffUser.create_staff_user(**input_params)
?
In an application I'm building, I've created a series of custom model managers and querysets to have a higher level api.
The problem comes when I execute an admin action. The queryset passed to it seems to be a generic one, and I would like to have access to my custom queryset to be able to use the filtering functions I created in it.
This is the action:
def mark_payment_as_sent_action():
''' Admin action to mark payment as sent '''
def mark_payment_as_sent(modeladmin, request, queryset):
# #####################################################################
# This is what I currently do
payments = queryset.filter(status=models.Payment.S_PENDING)
# This is what I want to do
payments = queryset.pending()
# #####################################################################
# Do stuff with filtered payments
return HttpResponseRedirect("...")
mark_payment_as_sent.short_description = "Mark as sent"
return mark_payment_as_sent
These are the custom model manager an query set:
class PaymentQuerySet(models.query.QuerySet):
def pending(self):
return self.filter(status=self.model.S_PENDING)
class PaymentManager(models.Manager):
use_for_related_fields = True
def get_query_set(self):
return PaymentQuerySet(self.model)
def pending(self, *args, **kwargs):
return self.get_query_set().pending(*args, **kwargs)
And finally the model and admin classes:
class Payment(models.Model):
status = models.CharField(
max_length=25,
choices=((S_PENDING, 'Pending'), ...)
)
objects = managers.PaymentManager()
#admin.register(models.Payment)
class PaymentsAdmin(admin.ModelAdmin):
actions = (
admin_actions.mark_payment_as_sent_action(),
)
Any hint on how can I tell Django to use my queryset when calling an admin action?
Thanks a lot.
As noamk noted, the problem was the method name. Django renamed the get_query_set method to get_queryset.
Now it's working petectly.
class PaymentQuerySet(models.query.QuerySet):
def pending(self):
return self.filter(status=self.model.S_PENDING)
class PaymentManager(models.Manager):
use_for_related_fields = True
def get_queryset(self):
return PaymentQuerySet(self.model)
def pending(self):
return self.get_queryset().pending()
I am using rest_framework v3.1.3 in django 1.8. I am pretty new to django.
Here are the relevant model definitions
#python_2_unicode_compatible
class UserFitbit(models.Model):
user = models.OneToOneField(User, related_name='fituser')
fitbit_user = models.CharField(max_length=32)
auth_token = models.TextField()
auth_secret = models.TextField()
#this is a hack so that I can use this as a lookup field in the serializers
#property
def user__userid(self):
return self.user.id
def __str__(self):
return self.user.first_name + ' ' + self.user.last_name
def get_user_data(self):
return {
'user_key': self.auth_token,
'user_secret': self.auth_secret,
'user_id': self.fitbit_user,
'resource_owner_key': self.auth_token,
'resource_owner_secret': self.auth_secret,
'user_id': self.fitbit_user,
}
def to_JSON(self):
return json.dumps(self, default=lambda o: o.__dict__,
sort_keys=True, indent=4)
class Challenge(models.Model):
name=models.TextField()
status=models.TextField() #active, pending, ended, deleted
start_date=models.DateField()
end_date=models.DateField()
#members=models.ManyToManyField(UserFitbit)
members=models.ManyToManyField(User)
admin=models.ForeignKey(UserFitbit,related_name='admin')
#for each member get stats between the start and end dates
def memberstats(self):
stats = []
for member in self.members.all():
fbu = UserFitbit.objects.filter(user__id=member.id)
fu = UserData.objects.filter(userfitbit=fbu)
fu = fu.filter(activity_date__range=[self.start_date,self.end_date])
fu = fu.annotate(first_name=F('userfitbit__user__first_name'))
fu = fu.annotate(user_id=F('userfitbit__user__id'))
fu = fu.annotate(last_name=F('userfitbit__user__last_name'))
fu = fu.values('first_name','last_name','user_id')
fu = fu.annotate(total_distance=Sum('distance'),total_steps=Sum('steps'))
if fu:
stats.append(fu[0])
return stats
def __str__(self):
return 'Challenge:' + str(self.name)
class Meta:
ordering = ('-start_date','name')
And here is the serializer for the challenge
class ChallengeSerializer(serializers.ModelSerializer):
links = serializers.SerializerMethodField(read_only=True)
memberstats = MemberStatSerializer(read_only=True,many=True)
#these are user objects
#this should provide a hyperlink to each member
members = serializers.HyperlinkedRelatedField(
#queryset defines the valid selectable values
queryset=User.objects.all(),
view_name='user-detail',
lookup_field='pk',
many=True,
)
class Meta:
model=Challenge
fields = ('id','name','admin','status','start_date','end_date','members','links','memberstats',)
read_only_fields = ('memberstats','links',)
def get_links(self, obj) :
request = self.context['request']
return {
'self': reverse('challenge-detail',
kwargs={'pk':obj.pk},request=request),
}
As you can see the Challenge has a many to many relationship with User. This is the built in User model from django not UserFitBit defined here.
With these definitions when I go to the api browser for a challenge I need to be able to select the users based on their name, but the select only shows their User id property and the hyperlink url. I would like the members to be User objects, but I don't know how to change the text for the select options since I don't think I can change the built in User object. What is the best way to change the select box options to show the users name from the User object rather than the username field and hyperlink?
Here is an image:
I'm not sure if this is the best way but after reading DRF's source code, I would try this.
Subclass the HyperlinkedRelatedField and override the choices property.
import six
from collections import OrderedDict
class UserHyperLinkedRelatedField(serializers.HyperLinkedRelatedField):
#property
def choices(self):
queryset = self.get_queryset()
if queryset is None:
return {}
return OrderedDict([
(
six.text_type(self.to_representation(item)),
six.text_type(item.get_full_name())
)
for item in queryset
])
then would simply replace the field in the serializer.
members = UserHyperlinkedRelatedField(
queryset=User.objects.all(),
view_name='user-detail',
lookup_field='pk',
many=True,
)
The DRF docs also mentioned that there's a plan to add a public API to support customising HTML form generation in future releases.
Update
For DRF 3.2.2 or higher, there will be an available display_value method.
You can do
class UserHyperLinkedRelatedField(serializers.HyperLinkedRelatedField):
def display_value(self, instance):
return instance.get_full_name()
Because this is a many related field I also had to extend the ManyRelatedField and override the many_init method of the RelatedField to use that class. Can't say I understand all of this just yet, but it is working.
class UserManyRelatedField(serializers.ManyRelatedField):
#property
def choices(self):
queryset = self.child_relation.queryset
iterable = queryset.all() if (hasattr(queryset, 'all')) else queryset
items_and_representations = [
(item, self.child_relation.to_representation(item))
for item in iterable
]
return OrderedDict([
(
six.text_type(item_representation),
item.get_full_name() ,
)
for item, item_representation in items_and_representations
])
class UserHyperlinkedRelatedField(serializers.HyperlinkedRelatedField):
#classmethod
def many_init(cls, *args, **kwargs):
list_kwargs = {'child_relation': cls(*args, **kwargs)}
for key in kwargs.keys():
if key in MANY_RELATION_KWARGS:
list_kwargs[key] = kwargs[key]
return UserManyRelatedField(**list_kwargs)
members = UserHyperlinkedRelatedField(
queryset=User.objects.all(),
view_name='user-detail',
lookup_field='pk',
many=True,
)