Django API sudden degrade in performance - django

I am working on a project where all REST API are implemented in django and UI is taken care by Vue.js. Since last week our dev and qa environments are facing long TTFB response time even for simple select queries to Postgres Database. To investigate this I added some logs to see how much time it takes to execute the query. Here is the code below
import datetime
from django.core.signals import request_started, request_finished
from django.conf import settings
class TenantListViewSet(APIView):
"""
API endpoint that returns all tenants present in member_master table.
"""
#ldap_login_required
#method_decorator(login_required)
def get(self, request):
settings.get_start_time = datetime.datetime.now()
print(f'Total time from started to get begin {settings.get_start_time-settings.request_start_time}')
errors = list()
messages = list()
try:
settings.db_query_start_time = datetime.datetime.now()
tenant_list_objects = list(MemberMaster.objects.all())
print(MemberMaster.objects.all().explain(verbose=True, analyze=True))
settings.db_query_end_time = datetime.datetime.now()
print(f'Total time taken to execute DB query {settings.db_query_end_time-settings.db_query_start_time}')
db_start = datetime.datetime.now()
serializer = MemberMasterSerializer(tenant_list_objects, many=True)
db_end = datetime.datetime.now()
print(f'Total time taken to serialize data {db_end-db_start}')
tenant_list = serializer.data
response = result_service("tenant_list", tenant_list, errors, messages)
settings.get_end_time = datetime.datetime.now()
print(f'Total time taken to finish get call {settings.get_end_time-settings.get_start_time}')
return JsonResponse(response.result["result_data"], status=response.result["status_code"])
except Exception as e:
if isinstance(e, dict) and 'messages' in e:
errors.extend(e.messages)
else:
errors.extend(e.args)
response = result_service("tenant_list", [], errors, messages)
return JsonResponse(response.result['result_data'], status=400)
def started(sender, **kwargs): # First method that gets invoked when GET API is called
settings.request_start_time = datetime.datetime.now()
def finished(sender, **kwargs): # Last method that gets invoked before sending the response back
return_from_get = datetime.datetime.now() - settings.get_end_time
total = datetime.datetime.now() - settings.request_start_time
print(f'Total time from get return to finished {return_from_get}')
print(f'API total time {total}')
request_started.connect(started) # Added hooks to django signal to check the time
request_finished.connect(finished) # Added hooks to django signal to check the time
These are the logs for the above API which has 126 rows in the table member_master
Total time taken to check user is authenticated 0:00:00.000374
Total time from started to get begin 0:00:02.060007
Seq Scan on member_master (cost=0.00..11.20 rows=120 width=634) (actual time=0.004..0.032 rows=126 loops=1)
Output: member_master_id, member_name, mbr_sk, mbr_uid, member_guid
Planning time: 0.028 ms
Execution time: 0.065 ms
Total time taken to execute DB query 0:00:02.026839
Total time taken to serialize data 0:00:00.000132
Total time taken to finish get call 0:00:02.267809
Total time from get return to finished 0:00:00.001454
API total time 0:00:04.329273
As you can see the execution time of the query itself is 0.065 ms but the time taken by django is 2 seconds. I tried to use raw query instead of list(MemberMaster.objects.all()) and the issue still exist. Also I am not able to reproduce this in my local environment but happens in both dev and qa environments. Hence I did not use django debug toolbar. No changes were made recently to any model and this issue exist for all API's in the project. Any idea what might be the issue here?
Model code
class MemberMaster(models.Model):
member_master_id = models.AutoField(primary_key=True)
member_name = models.CharField(max_length=256)
mbr_sk = models.ForeignKey('Member', on_delete=models.DO_NOTHING, db_column='mbr_sk')
mbr_uid = models.CharField(max_length=32, blank=True, null=True)
member_guid = models.UUIDField(blank=True, null=True)
class Meta:
managed = False
db_table = 'member_master'
app_label = 'queryEditor'

Related

How to optimize a server request that has to send back several items with dynamic attributes that need to be calculated every time a user requests it?

I have an Angular UI app connecting to a Django API that uses GraphQL (using Graphene) and Postgres for DB.
My application has many courses and each course can have several chapters. The users signing in can see access courses and not others because a course could have a prerequisite. So they will see a course listed but it will be "locked" for them and a message will say that they need to complete the particular prerequisite before it can be accessed. Like this, we need some other attributes to be sent along with the list of courses:-
'locked' - Boolean - whether a course is locked for the current logged-in user or not.
'status' - ENUM - PENDING/SUBMITTED/GRADED/RETURNED/FLAGGED
'completed' - Boolean - whether the course is completed or not
When a user requests the list of courses, these 3 attributes are calculated for each item in the list before it is compiled and sent back to the user.
And this is done for each of the chapters inside the course too. And the chapter might contain upto 30 chapters or so. So this really takes a LOT of time!
I've implemented caching as well, but because these values change often (eg. when the user completes a chapter) they are constantly invalidated and it doesn't make sense to keep these attributes server-side cached to begin with.
Here's the code for how the chapters are processed for the query for list of chapters:-
#login_required
#user_passes_test(lambda user: has_access(user, RESOURCES['CHAPTER'], ACTIONS['LIST']))
def resolve_chapters(root, info, course_id=None, searchField=None, limit=None, offset=None, **kwargs):
current_user = info.context.user
# Checking if this is cached
cache_entity = CHAPTER_CACHE[0]
cache_key = generate_chapters_cache_key(cache_entity, searchField, limit, offset, course_id, current_user)
cached_response = fetch_cache(cache_entity, cache_key)
if cached_response:
return cached_response
# If not cached...
qs = rows_accessible(current_user, RESOURCES['CHAPTER'], {'course_id': course_id})
if searchField is not None:
filter = (
Q(searchField__icontains=searchField.lower())
)
qs = qs.filter(filter)
if offset is not None:
qs = qs[offset:]
if limit is not None:
qs = qs[:limit]
set_cache(cache_entity, cache_key, qs)
return qs
And I'm using this code to dynamically insert the three attributes into each item in the list of chapters that the above code returns:-
class ChapterType(DjangoObjectType):
completed = graphene.Boolean()
completion_status = graphene.String()
locked = graphene.String()
def resolve_completed(self, info):
user = info.context.user
completed = CompletedChapters.objects.filter(participant_id=user.id, chapter_id=self.id).exists()
return completed
def resolve_completion_status(self, info):
user = info.context.user
status = ExerciseSubmission.StatusChoices.PENDING
try:
completed = CompletedChapters.objects.get(participant_id=user.id, chapter_id=self.id)
status = completed.status
except:
pass
return status
def resolve_locked(self, info):
user = info.context.user
locked = is_chapter_locked(user, self)
return locked
class Meta:
model = Chapter
And the method is_chapter_locked() is quite complex in itself:-
def is_chapter_locked(user, chapter):
locked = None
# Letting the user see it if they are a grader
user_role = user.role.name;
grader = user_role == USER_ROLES_NAMES['GRADER']
# Checking if the user is the author of the course or a grader
if chapter.course.instructor.id == user.id or grader:
# If yes, we mark it as unlocked
return locked
course_locked = is_course_locked(user, chapter.course) # Checking if this belongs to a course that is locked
if course_locked:
# If the course is locked, we immediately return locked is true
locked = 'This chapter is locked for you'
return locked
# If the course is unlocked we
completed_chapters = CompletedChapters.objects.all().filter(participant_id=user.id)
required_chapters = MandatoryChapters.objects.all().filter(chapter_id=chapter.id)
required_chapter_ids = required_chapters.values_list('requirement_id',flat=True)
completed_chapter_ids = completed_chapters.values_list('chapter_id',flat=True)
pending_chapter_ids = []
for id in required_chapter_ids:
if id not in completed_chapter_ids:
pending_chapter_ids.append(id)
if pending_chapter_ids:
locked = 'To view this chapter, you must have completed '
pending_chapters_list = ''
for id in pending_chapter_ids:
try:
chapter= Chapter.objects.get(pk=id, active=True)
if pending_chapters_list != '':
pending_chapters_list += ', '
pending_chapters_list += '"' + str(chapter.section.index) +'.'+str(chapter.index)+'. '+chapter.title +'"'
except:
pass
locked += pending_chapters_list
return locked
As can be seen, there is a lot of dynamic processing that is done for fetching the list of chapters. And this is taking a considerably long time, even with caching of the query from the database before the dynamic attributes are calculated.
I am looking for strategies to minimize the dynamic calculation. What kind of an approach works best for performance optimizations in situations like this?
Thank you.

How to use session timeout in django rest view?

I am implementing a view for a game using Django REST's APIView. I am very new to Django and have never done this before so I'm not sure how to implement this.
The main idea is that a game only lasts 5 minutes. I am sending a resource to the user and creating a session object. This view. should be unavailable after 5 minutes. Is there such a thing as a view timeout?
Will the session timeout then work for the post request as well or do I need to implement it there as well?
This is my view:
The out commented code at the end is what I was thinking of doing. Can I even do it in the view directly? How else can I do this and test it?
views.py
class GameView(APIView):
"""
API View that retrieves the game,
retrieves an game round as well as a random resource per round
allows users to post tags that are verified and saved accordingly to either the Tag or Tagging table
"""
def get(self, request, *args, **kwargs):
current_score = 0
if not isinstance(request.user, CustomUser):
current_user_id = 1
else:
current_user_id = request.user.pk
random_resource = Resource.objects.all().order_by('?').first()
resource_serializer = ResourceSerializer(random_resource)
gameround = Gameround.objects.create(user_id=current_user_id,
gamesession=gamesession,
created=datetime.now(),
score=current_score)
gameround_serializer = GameroundSerializer(gameround)
return Response({'resource': resource_serializer.data,
'gameround': gameround_serializer.data,
})
# TODO: handle timeout after 5 min!
# now = timezone.now()
# end_of_game = start_time + timezone.timedelta(minutes=5)
# if :
# return Response({'resource': resource_serializer.data, 'gameround': gameround_serializer.data,})
# else:
# return Response(status=status.HTTP_408_REQUEST_TIMEOUT)
*Testing the out commented code in Postman always leads to a 408_request_timeout.

RuntimeWarning: DateTimeField received a naive datetime while time zone support is active

Here I am writing a middleware to delete objects older than 3 months. The date_before_2_month and datetime_before_2_months is working fine. I haven't tested yet for filtering But in my console it is giving me a runtime warning saying activity_time received a naive datetime.
Is this warning a issue(needs to solve) or we can ignore it ?
Also does my filter parameters are good for querying the objects from model which are 2 months old ?
activity_time in model is DateTimeField(auto_now_add=True)
class UserActivityLogDeleteMiddleware(object): # middleware for deleting user activity logs which are older than 2 months
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
date_before_2_month = datetime.date.today() - relativedelta(months=2)
# converting into datetime
datetime_before_2_month = datetime.datetime.combine(date_before_2_month, datetime.time(9, 00, 00))
# deleting logs older than 2 months
UserActivityLog.objects.filter(activity_time__lt=datetime_before_2_month).delete()
response = self.get_response(request)
return response
Generally you want to keep your datetimes timezone aware if you're using timezones through your app.
There are a couple of improvements I can see to your code. When using relativedelta you can set the hour and minute to make life a bit easier.
from django.utils import timezone
...
def __call__(self, request):
# Getting a datetime for 2 months ago at 9am
dt_2_months_ago = timezone.now() - relativedelta(months=2, hour=9, minute=0)
# deleting logs older than 2 months
UserActivityLog.objects.filter(activity_time__lt=dt_2_months_ago).delete()
response = self.get_response(request)
return response
One note here, are you sure you want to be running this in middleware? It'll run on every single request, every time you get js, every ajax request etc. Just a thought.

query is getting UTC timestamp from the database but local time is stored in the database

For some reason my queryset is returning a UTC time, however in the database the time it is supposed to be getting is local time. Does anyone know why this is happening? Thanks for your help
The last_checkin_time method is what gets the user's last checkin timestamp from the database, and right now I just have it posting in my toggle method in the time_delta variable so I can see what value its getting. (once i get this timezone thing figured out the time_delta will be an actual time delta)
Heres my model manager
class UserActivityManager(models.Manager):
def current(self, user):
current_obj = self.get_queryset().filter(user=user).order_by('-timestamp').first()
return current_obj
def last_checkin_time(self, user):
last_activity_time = self.get_queryset().order_by('-timestamp').filter(user=user, activity="checkin").first()
return last_activity_time
def toggle(self, user):
last_item = self.current(user)
activity = "checkin"
time_delta = None
last_checkin = self.last_checkin_time(user)
if last_item is not None:
if last_item.timestamp <= tz.localize(datetime.datetime.now()):
pass
if last_item.activity == "checkin":
activity = "checkout"
time_delta = last_checkin.timestamp
obj = self.model(
user=user,
activity=activity,
time_delta = time_delta,
)
obj.save()
return obj
And heres what the table in my database looks like (focusing on the time_delta field in the last few rows)
EDIT:
Also I should mention the timestamp field in my model is set to auto_now_add=True ie.
timestamp = models.DateTimeField(auto_now_add=True)
Not sure if this is causing the problem
Instead of using tz.localize(last_checkin.timestamp) I needed to use tz.normalize(last_checkin.timestamp). It seems since the timestamp was already set to UTC I needed to change it with the normalize method, rather than the localize

Django: How to automatically reset a boolean field to it's default after some time (eg. 6 months) to make full access for a page expire

I am fairly new to django and I have the problem of creating full access for a site. The user has to give some additional information to get full access after signing up. I want the full access to automatically expire after 6 months. I defined a custom user model with the extra condition:
models.py
from django.db import models
from django.contrib.auth.models import AbstractUser
class CustomUser(AbstractUser):
full_name = models.CharField(blank=True, max_length=255)
has_full_access = models.BooleanField(default=False)
#some other stuff
After typing in some data for getting full access, the user gets redirected to this view which sets the boolean to true:
views.py
def data_gathered_done(request):
current_user = CustomUser.objects.get(id=request.user.id)
current_user.has_full_access = True
current_user.save()
#some other stuff
I want this boolean field to automatically reset to it's default (False) 6 months after the full access has been granted. How can I do that?
I'd do it with a property on the Model.
from datetime import datetime, timedelta
from django.db import models
from django.contrib.auth.models import AbstractUser
expire_after = timedelta(days=180)
class CustomUser(AbstractUser):
full_name = models.CharField(blank=True, max_length=255)
full_access_since = models.DatetimeField(auto_add_now=True)
#some other stuff
#property
def has_full_access(self):
return datetime.now() - expire_after < self.full_access_since
Then you can use the Boolean normally
from django.http import HttpResponseForbidden
user = CostumUser.objects.get(pk=123)
if not user.has_full_access:
return HttpResponseForbidden()
I'm a little late to this question, but I had a somewhat similar problem recently where I needed a boolean "lock" that "expired" after a 90 minutes. I didn't want to install any third-party dependencies or packages to do this.
The scenario: When a user accesses an "edit mode" view/template from a given model instance's detail view, I need to lock out all other users to prevent concurrency issues.
However after X minutes, I want others to be able to edit so I needed the UI menu options to revert back.
(Note: In my case I have to deal with concurrency at the database level as well, but this solution deals with the UI.) However, the logic could be extended to handle other time-based access issues within a site or webapp.
If I handled this only client side (say with AJAX), a user might lock a model and potentially their computer blows up, hence no AJAX fires to unlock. Has to be back-end. Like the answer above, a function that checks timestamps on the model seems like the way to go, but then again I have users all over the world - how do I deal with daylight savings and different timezones?
My solution was to use a non-DST timezone as a time constant so I didn't have to worry about that. Who cares what timezone I'm benchmarking - it's just a back-end method that checks durations.
models.py
class SomeProduct(models.Model):
name = models.CharField()
description = models.TextField()
lock = models.BooleanField(default=False)
timestamp = models.DateTimeField(null=True, blank=True, auto_now_add=False)
def __str__(self):
return str(self.name)
views.py
import datetime
import pytz
def update_product_view(request, slug): # This view shows forms and locks out other editors
qs = SomeProduct.objects.get(slug=slug):
if qs.lock == False:
qs.lock = True
now = datetime.datetime.now(pytz.timezone('US/Hawaii')) #Hawaii time is constant, no DST
qs.timestamp = now
qs.save()
elif qs.lock == True:
now = datetime.datetime.now(pytz.timezone('US/Hawaii'))
qs.timestamp = now
qs.save()
else:
pass
# Forms and other view logic here...
def product_view(request, slug): # This view unlocks the model if enough time has passed
qs = SomeProduct.objects.get(slug=slug):
if qs is not None:
try: # in case no timestamp has been set
now = datetime.datetime.now(pytz.timezone('US/Hawaii'))
then = qs.timestamp
delta = (now-then).total_seconds() # compare the difference
minutes = 60 #seconds
if delta > 90*minutes:
qs.lock = False # if 90 or more minutes have passed, unlock the model
qs.save()
else:
pass
except:
pass
else:
pass
# context and other view logic here...
template
{% if obj.lock == True %}
# adjust edit options or hide buttons accordingly
{% else obj.lock == False %}
# show button that leads to edit view url
{% endif %}
This is a pretty simplified version of my code, but the basics are there. I also have some JS on the front end that informs the user with a timeclock, exit edit mode URL that unlocks the model, etc. Your needs may vary. If anybody has some perspective on how I can make this better or any "gotchas" I'd love to learn something so please share. For now this works!