Problem
I have a backend with Django (Wagtail CMS) and a frontend with react. I am using them as a news site. I am new in Django and my other teammate manages the front end. I am providing them with wagtail's default API with some customization(I have exposed all needed fields in BaseAPIView). In my API there were 25 blog posts for the first time and now there are 60 posts. The first time the API loading time was 1.02 sec and now the API load time is 1.69sec. I am afraid of what will happen when the posts hit 5000+! Because I am not limiting anything in the API.
Plan
I am planning to use radis cache for the API. But I can't understand what should I set in the API views! because my API views is this.
#api.py
from wagtail.api.v2.views import PagesAPIViewSet,BaseAPIViewSet
api_router = WagtailAPIRouter('wagtailapi')
class ProdPagesAPIViewSet(BaseAPIViewSet):
renderer_classes = [JSONRenderer]
filter_backends = [FieldsFilter,
ChildOfFilter,
AncestorOfFilter,
DescendantOfFilter,
OrderingFilter,
TranslationOfFilter,
LocaleFilter,
SearchFilter,]
meta_fields = ["type","seo_title","search_description","first_published_at"]
body_fields = ["id","type","seo_title","search_description","first_published_at","title"]
listing_default_fields = ["type","seo_title","search_description","first_published_at","id","title","alternative_title","news_slug","blog_image","video_thumbnail","categories","blog_authors","excerpt","content","content2","tags","story_type"]
nested_default_fields = []
def get_queryset(self):
return super().get_queryset().filter(story_type='Story').order_by('-first_published_at')
name = "stories"
model = AddStory
api_router.register_endpoint("stories", ProdPagesAPIViewSet)
Will adding radis cache in API solve the loading problem or should I add a cache in the front end?
I tried to add pagination in the API. Well, I was unsuccessful, but somehow if I could add pagination will it be better?
Sugestion Seeking
What will be better for me adding radis cache or adding pagination?
Related
I am using low level caching API
I want to cache the queryset result in the django rest API, the problems i am facing is:
Queryset result is dependent on input provided to the API.
{
"status": "Deactive",
"location" : ["India","us"],
"hashtagID": "1"
}
Suppose , I cache this result using some key, say Status_1
cache_key = "Status_1"
query_set = ***query to filter the model using the above input***
cache.set(cache_key, query_set)
Now, if again user gives the same input to the API, How would i come to know that query_set respected to that input is already cached??
Simply, i want to create such a caching system in Django rest framework, where we can cache queryset which vary according to input provided to rest API.
I. I'm new to using Django REST Swagger for API documentation. I'm trying to understand how to edit a) 'Implementation Notes' and b) parameter descriptions. Image below -
Here's my viewset for the 'Crash' model.
class CrashViewSet(viewsets.ModelViewSet):
"""
retrieve: Get a single Crash instance
list: Get a list of all Crashes
"""
queryset = Crash.objects.all()
serializer_class = CrashSerializer
filter_backends = (SearchFilter,DjangoFilterBackend,OrderingFilter,)
search_fields = ('crash_id','crash_hr_short_desc','urb_area_short_nm','fc_short_desc',
'hwy_compnt_short_desc','mlge_typ_short_desc', 'specl_jrsdct_short_desc',
'jrsdct_grp_long_desc','st_full_nm','isect_st_full_nm','rd_char_short_desc',
'isect_typ_short_desc','crash_typ_short_desc','collis_typ_short_desc',
'rd_cntl_med_desc','wthr_cond_short_desc','rd_surf_short_desc','lgt_cond_short_desc',
'traf_cntl_device_short_desc','invstg_agy_short_desc','crash_cause_1_short_desc',
'crash_cause_2_short_desc','crash_cause_3_short_desc','pop_rng_med_desc','rd_cntl_med_desc')
filter_fields = ('ser_no','cnty_id','alchl_invlv_flg','crash_day_no','crash_mo_no','crash_yr_no','crash_hr_no',
'schl_zone_ind','wrk_zone_ind','alchl_invlv_flg','drug_invlv_flg','crash_speed_invlv_flg',
'crash_hit_run_flg',)
ordering_fields = '__all__'
I have made some edits to the DocString but don't quite understand how to proceed so as to get a more thorough descriptions of the endpoints and their fields. I tried going through the Django REST Swagger tutorial but I'm not given any direction on how to format the DOCSTRING. What's the best way to do this?
II. Another minor issue is that I want to get rid of the 'Django Login' button since this is public API. How do I do this?
III. What are some best practices for API documentation?
I'm just start working with django and DRF, and occure a problem, that is looks like DRF cache responses. I mean - I can change object, create new, or delete it - and DRF keep response, thats nothing is changed. For example, I create an object, but modelViewSet still return data where this object does not presented. But if I directly request it object - it show that it's created. And so with any another actions. I can't find topic about caching in DRF, and look like I have not any django chaching middlewares, so I have no idea what is going on.
Only one thing that helps - restart server ( I'm using default dev-server).
One more thing - all data is ok when it's rendered by django views, not DRF views.
Here is one of the serializers/modelViewSets that I'm using. It simple as it possible. And also - I'm not using django cache backends. At least - I have not any in my settings.
class WorkOperationSerializer(serializers.ModelSerializer):
class Meta:
model = WorkOperation
class WorkOperationAPIView(viewsets.ModelViewSet):
serializer_class = WorkOperationSerializer
queryset = WorkOperation.objects.all()
def get_queryset(self):
return self.queryset
You can read here about django queryset caching. The best advice seems to be: re-run the .all() method to get fresh results. Using object.property may give you cached results.
I want to integrate haystack with django-cms making a search view. My CMS has pages with view restrictions (only a few authenticated users have access to some pages).
The problem is: when making a search, haystack gives me list with results from all pages, including pages to which the current user has not view permissions.
How can I integrate Haystack in a way that filters results, showing only the ones to which the current user has permissions for? If that's not possible, how to configure haystack letting it index only pages without view-restrictions? Any help is appreciated.
In my solution to this problem I use aldryn_search to do the integration of Haystack and django-cms. aldryn_search returns a listwith results from all pages, including the ones the current user hast not view-permissions for. To resolve this issue I'm inheriting from AldrynSearchView and override the get_queryset method like this:
def get_queryset(self):
queryset = super(IntranetSearchView, self).get_queryset()
for result in queryset.load_all():
page = result.object.page
# Begin: modified copy (queryset.exclude added) of cms.utils.decorators.cms_perms
if page:
if page.login_required and not self.request.user.is_authenticated():
queryset = queryset.exclude(id=result.id)
if not page.has_view_permission(self.request, user=self.request.user):
queryset = queryset.exclude(id=result.id)
# End: Copy
return queryset
using queryset.exclude() to exclude results the current user has not permissions for. After that I inherit from AldrynSearchApphook overriding the urls with my new View and than doing a apphoook_pool.register of the modified Apphook.
I'm using Django Rest Framework and DRF-extensions (http://chibisov.github.io/drf-extensions/docs/#how-key-constructor-works).
I have an API call /user/:id/products, which provides the full list of products that a user has access to. Because it's an expensive query, I'm caching it based on method id, format, language, user, and retrieval SQL query.
I have another API call /pay which handles all of our payments. The product ids are POST arguments to that API call.
After a user pays, immediately it directs them to the product page--but since the product query is cached, it still displays the old list of products, not the updated list with the new products they just purchased. I know about the cache.delete method, but getting the key from /pay is difficult since it's a completely different API call.
What I would like to do is add a parameter force_refresh to the /user/:id/products API call, that, if present, calculates the hash key as if the force_refresh parameter weren't present and then clears the cache for that key. However, I can't find the right place in KeyConstructor within drf-extension's caching framework to stick this in. What is the best way for me to let the server know to delete the cache for this query?
Example for using DRF-extensions caching with all GET params as key hash on ReadOnlyModelViewSet.
Define your key constructor (key hash):
from rest_framework_extensions.key_constructor.constructors import DefaultKeyConstructor
from rest_framework_extensions.key_constructor.bits import QueryParamsKeyBit
class QueryParamsKeyConstructor(DefaultKeyConstructor):
all_query_params = QueryParamsKeyBit('*') # Don't change, * is default on drf-extension dev version only
And your view:
class UserViewSet(ListCacheResponseMixin, viewsets.ReadOnlyModelViewSet):
serializer_class = UserSerializer
list_cache_key_func = QueryParamsKeyConstructor()
set DRF-extensions and django caching settings in settings.py, for example:
CACHES = {
"default": {
"BACKEND": "redis_cache.RedisCache",
"LOCATION": "redis://127.0.0.1:6379/0"
}
}
REST_FRAMEWORK_EXTENSIONS = {
'DEFAULT_CACHE_RESPONSE_TIMEOUT': 20
}
What I would like to do is add a parameter force_refresh to the /user/:id/products API call
I don't think it's a good idea. What if your client made /pay request and then used his own routing to /user/:id/products ? With force_refresh attribute you are adding state for request and it is not idempotent anymore.
If you have high read rate and low write rate I would suggest you to look at creation of custom key bit with global cached invalidation, based on post save signals.
If you want to make cache invalidation more specific I would suggest you to denormalize your User model. For example:
class User(UserBaseModel):
...
products_updated_date = models.DateTimeField()
You should update products_updated_date every time you make payments:
class UserViewSet(CacheResponseMixin, ReadOnlyModelViewSet):
...
#action('pay')
def pay(self):
...
self.request.user.products_updated_date = now()
self.request.user.save()
...
And finally you should create custom key bit with users's products_updated_date field usage:
from rest_framework_extensions.key_constructor.bits import (
KeyBitBase
)
class UserProductsUpdatedKeyBit(KeyBitBase):
def get_data(self, **kwargs):
request = kwargs['request']
if (hasattr(request, 'user') and
request.user and
request.user.is_authenticated()):
return request.user.products_updated_date
You can add UserProductsUpdatedKeyBit to your key constructor class for /user/:id/products routing.