I am using low level caching API
I want to cache the queryset result in the django rest API, the problems i am facing is:
Queryset result is dependent on input provided to the API.
{
"status": "Deactive",
"location" : ["India","us"],
"hashtagID": "1"
}
Suppose , I cache this result using some key, say Status_1
cache_key = "Status_1"
query_set = ***query to filter the model using the above input***
cache.set(cache_key, query_set)
Now, if again user gives the same input to the API, How would i come to know that query_set respected to that input is already cached??
Simply, i want to create such a caching system in Django rest framework, where we can cache queryset which vary according to input provided to rest API.
Related
Problem
I have a backend with Django (Wagtail CMS) and a frontend with react. I am using them as a news site. I am new in Django and my other teammate manages the front end. I am providing them with wagtail's default API with some customization(I have exposed all needed fields in BaseAPIView). In my API there were 25 blog posts for the first time and now there are 60 posts. The first time the API loading time was 1.02 sec and now the API load time is 1.69sec. I am afraid of what will happen when the posts hit 5000+! Because I am not limiting anything in the API.
Plan
I am planning to use radis cache for the API. But I can't understand what should I set in the API views! because my API views is this.
#api.py
from wagtail.api.v2.views import PagesAPIViewSet,BaseAPIViewSet
api_router = WagtailAPIRouter('wagtailapi')
class ProdPagesAPIViewSet(BaseAPIViewSet):
renderer_classes = [JSONRenderer]
filter_backends = [FieldsFilter,
ChildOfFilter,
AncestorOfFilter,
DescendantOfFilter,
OrderingFilter,
TranslationOfFilter,
LocaleFilter,
SearchFilter,]
meta_fields = ["type","seo_title","search_description","first_published_at"]
body_fields = ["id","type","seo_title","search_description","first_published_at","title"]
listing_default_fields = ["type","seo_title","search_description","first_published_at","id","title","alternative_title","news_slug","blog_image","video_thumbnail","categories","blog_authors","excerpt","content","content2","tags","story_type"]
nested_default_fields = []
def get_queryset(self):
return super().get_queryset().filter(story_type='Story').order_by('-first_published_at')
name = "stories"
model = AddStory
api_router.register_endpoint("stories", ProdPagesAPIViewSet)
Will adding radis cache in API solve the loading problem or should I add a cache in the front end?
I tried to add pagination in the API. Well, I was unsuccessful, but somehow if I could add pagination will it be better?
Sugestion Seeking
What will be better for me adding radis cache or adding pagination?
In any rest API,
Data comes in request body
We perform some Logic on
data and perform some queries
3.finally API responds serialized data.
My question is :-
Where to put data processing Logic? And what is the efficient way of designing any REST API?
Logic on data is mainly put on the views. You can use any views, be it functional, generic API views, and viewsets. There are three types of views in DRF.
You should be comfortable in using one of them, however should understand all of them. Go through this link.
https://micropyramid.com/blog/generic-functional-based-and-class-based-views-in-django-rest-framework/
I personally find comfort in using CBV (Class-Based Views). It is in the views where most of the SQL joins take place too.
For eg:
class GetReviewAPIView(ListAPIView):
permission_classes = [IsAuthenticated]
serializer_class = ReviewSerializer
def get_queryset(self):
user = self.request.user
return Review.objects.filter(user=user)
The above is a good example of CBV where the API call gets all the reviews of a particular user only. There is a SQL join happening between the User table and Review table.
Also, you can write the logic in the serializer class as well instead of view. This is done mainly when you have to write the logic for each serializer field differently and set characteristics for each field. I can give you an example for this as well if you want.
I'm storing user's data and articles using the Pocket API into a SQLite Database using Django Framework. How can I efficiently maintain consistency between the database and the data received from the API? I'm planning to update the database once every 24 hours.
Should be simple with chron jobs. If you already have a model with logic to fetch data from the Pocket API and parse it, it's as simple as:
class MyCronJob(CronJobBase):
RUN_EVERY_MINS = 120 # every 2 hours
schedule = Schedule(run_every_mins=RUN_EVERY_MINS)
code = 'my_app.my_cron_job' # a unique code
def do(self):
data = fetch_your_data()
save_data(data)
If you don't have the parsing logic yet, I'd suggest Django Rest Framework serializers. Using the serializers is simple and saving the data shrinks down to:
def save_data(json_data):
serializer = YourPocketDataSerializer(json_data)
serializer.save()
I have a table called 'users' and 'location'. Users table has a foreign key that relates to location table. I have a users serializer to get the JSON. What would I do to get the hyperlinks for the users table using its primary key?
In django rest framework documentation, I couldn't find a solution. I tried using hyperlinkrelatedfield. But still I couldn't achieve this. Can someone help me in finding the solution?
Using rest-framework HyperlinkedRelatedField does not work because it was never built to expose the URL of the object being requested. Mainly because since the client already has the url of the user, why send it back again? Nevertheless you can achieve this by doing something like this.
class UserSerializer(serializers.ModelSerializer):
user_url = serializers.SerializerMethodField()
class Meta:
model = User
def get_label_location(self, obj):
return HyperlinkedRelatedField(view_name='user-detail',
read_only=True) \
.get_url(obj, view_name='label-detail',
request=self.context['request'], format=None)
Take note on a few things,
view-name param to the HyperlinkedRelatedField should be based on your url configuration
read-only has to be true since otherwise you'll have to specify the queryset. But since we have the object needed to generate the url we can ignore that.
I've set format param to None but you might want to set it based on your settings.
You can read up about SerializerMethodField here.
I'm using Django Rest Framework and DRF-extensions (http://chibisov.github.io/drf-extensions/docs/#how-key-constructor-works).
I have an API call /user/:id/products, which provides the full list of products that a user has access to. Because it's an expensive query, I'm caching it based on method id, format, language, user, and retrieval SQL query.
I have another API call /pay which handles all of our payments. The product ids are POST arguments to that API call.
After a user pays, immediately it directs them to the product page--but since the product query is cached, it still displays the old list of products, not the updated list with the new products they just purchased. I know about the cache.delete method, but getting the key from /pay is difficult since it's a completely different API call.
What I would like to do is add a parameter force_refresh to the /user/:id/products API call, that, if present, calculates the hash key as if the force_refresh parameter weren't present and then clears the cache for that key. However, I can't find the right place in KeyConstructor within drf-extension's caching framework to stick this in. What is the best way for me to let the server know to delete the cache for this query?
Example for using DRF-extensions caching with all GET params as key hash on ReadOnlyModelViewSet.
Define your key constructor (key hash):
from rest_framework_extensions.key_constructor.constructors import DefaultKeyConstructor
from rest_framework_extensions.key_constructor.bits import QueryParamsKeyBit
class QueryParamsKeyConstructor(DefaultKeyConstructor):
all_query_params = QueryParamsKeyBit('*') # Don't change, * is default on drf-extension dev version only
And your view:
class UserViewSet(ListCacheResponseMixin, viewsets.ReadOnlyModelViewSet):
serializer_class = UserSerializer
list_cache_key_func = QueryParamsKeyConstructor()
set DRF-extensions and django caching settings in settings.py, for example:
CACHES = {
"default": {
"BACKEND": "redis_cache.RedisCache",
"LOCATION": "redis://127.0.0.1:6379/0"
}
}
REST_FRAMEWORK_EXTENSIONS = {
'DEFAULT_CACHE_RESPONSE_TIMEOUT': 20
}
What I would like to do is add a parameter force_refresh to the /user/:id/products API call
I don't think it's a good idea. What if your client made /pay request and then used his own routing to /user/:id/products ? With force_refresh attribute you are adding state for request and it is not idempotent anymore.
If you have high read rate and low write rate I would suggest you to look at creation of custom key bit with global cached invalidation, based on post save signals.
If you want to make cache invalidation more specific I would suggest you to denormalize your User model. For example:
class User(UserBaseModel):
...
products_updated_date = models.DateTimeField()
You should update products_updated_date every time you make payments:
class UserViewSet(CacheResponseMixin, ReadOnlyModelViewSet):
...
#action('pay')
def pay(self):
...
self.request.user.products_updated_date = now()
self.request.user.save()
...
And finally you should create custom key bit with users's products_updated_date field usage:
from rest_framework_extensions.key_constructor.bits import (
KeyBitBase
)
class UserProductsUpdatedKeyBit(KeyBitBase):
def get_data(self, **kwargs):
request = kwargs['request']
if (hasattr(request, 'user') and
request.user and
request.user.is_authenticated()):
return request.user.products_updated_date
You can add UserProductsUpdatedKeyBit to your key constructor class for /user/:id/products routing.