Multiple ajax request (Singleton Pattern) - django

I am using django restless for an ajax POST request, which took almost 10 to 20 seconds.
Here is my code.
class testEndPoint(Endpoint):
def post(self, request):
testForm = TestEmailForm(request.data)
if testForm.is_valid():
sometable = EmailTable.object.get(**condition)
if sometable.is_email_sent == false:
#Send Email
#Took around 15 seconds
sometable.is_email_sent = true
sometable.save()
else:
result = testForm.errors
return serialize(result)
i am calling it via $.ajax, but the problem is if two request hit this url with milliseconds time difference, both request passed through if sometable.is_email_sent = false: condition.
How can i prevent multiple submission. Right now i have moved sometable.is_email_sent = true;sometable.save(); before email send part, but i need more generic solution as there are dozen more places where this is happening. I am on django 1.5
django restless

You should disable the originating input element before you start your ajax call (that will prevent the majority of these issues).
The remaining problems can be solved by using select_for_update
class testEndPoint(Endpoint):
#transaction.commit_manually
def post(self, request):
testForm = TestEmailForm(request.data)
if testForm.is_valid():
condition['is_email_sent'] = False
try:
rows = EmailTable.objects.select_for_update().filter(**condition)
for row in rows:
row.is_email_sent = True
row.save()
#Send Email
except:
transaction.rollback()
raise
else:
transaction.commit()
else:
result = testForm.errors
return serialize(result)
select_for_update will lock the rows until the end of the transaction (i.e. it needs to be inside a transaction). By adding is_email_sent=False to the condition, we can remove the if. I've moved the changing of is_email_sent above the "Send Email", but it is not strictly necessary -- in any case it will be undone by the transaction rolling back if there is an exception.

Related

Django asyncio for saving (big amount of) objects - Nothing saved

I want to fetch categories from a Magento API and display them in a template. In the same time, I want to save them in DB for an ulterior use.
Categories are too many and the render of the template takes more than 30 sec.
I start to learn using asyncio but couldn't get my way with it. I surely missed something.
First, my URL leads to the function that retrieves the categories
#login_required
def get_categories(request):
Category.objects.all().delete()
try:
cats = fetch_categories()
tree = cats['children_data']
except:
print('erreur : impossible de récupérer les catégories (fetch_categories)')
asyncio.run(parse_categories(tree))
return render(request, 'categories/categories_list.html', {'tree': tree})
When I get the "categories tree", I send it to
async def parse_categories(tree):
for lvl1 in tree:
all_tasks = []
asyncio.create_task(save_cat(lvl1))
# main products categories (turbo, injectors ...)
for lvl2 in lvl1['children_data']:
asyncio.create_task(save_cat(lvl2))
# sub categories like RENAULT, DACIA
for lvl3 in lvl2['children_data']:
asyncio.create_task(save_cat(lvl3))
for lvl4 in lvl3['children_data']:
asyncio.create_task(save_cat(lvl4))
for lvl5 in lvl4['children_data']:
asyncio.create_task(save_cat(lvl5))
My save() function is async. I'm not sure it should be. Before I started using async, it was working.
async def save_cat(cat):
cat_id = cat['id']
new_cat = Category()
new_cat.id = cat_id
new_cat.name = cat.get('name', None)
new_cat.parent = cat.get('parent_id', None)
new_cat.url = cat.get('path', None)
new_cat.is_active = cat.get('is_active', None)
new_cat.position = cat.get('position', None)
new_cat.level = cat.get('level', None)
new_cat.save()
When I run, no error. The context is well sent to the template and displays well. But no category is saved.
I also tried to make a task list with asyncio.create_task in each level and execute the loop at the end of parse_categories() like said in this thread, without success.
all_tasks.append(asyncio.create_task(save_cat(lvl1)))
[...]
responses = asyncio.gather(*all_tasks, return_exceptions=True)
loop = asyncio.get_event_loop()
loop.run_until_complete(responses)
loop.close()
Any clue to solve my case will be welcome

How to optimize a server request that has to send back several items with dynamic attributes that need to be calculated every time a user requests it?

I have an Angular UI app connecting to a Django API that uses GraphQL (using Graphene) and Postgres for DB.
My application has many courses and each course can have several chapters. The users signing in can see access courses and not others because a course could have a prerequisite. So they will see a course listed but it will be "locked" for them and a message will say that they need to complete the particular prerequisite before it can be accessed. Like this, we need some other attributes to be sent along with the list of courses:-
'locked' - Boolean - whether a course is locked for the current logged-in user or not.
'status' - ENUM - PENDING/SUBMITTED/GRADED/RETURNED/FLAGGED
'completed' - Boolean - whether the course is completed or not
When a user requests the list of courses, these 3 attributes are calculated for each item in the list before it is compiled and sent back to the user.
And this is done for each of the chapters inside the course too. And the chapter might contain upto 30 chapters or so. So this really takes a LOT of time!
I've implemented caching as well, but because these values change often (eg. when the user completes a chapter) they are constantly invalidated and it doesn't make sense to keep these attributes server-side cached to begin with.
Here's the code for how the chapters are processed for the query for list of chapters:-
#login_required
#user_passes_test(lambda user: has_access(user, RESOURCES['CHAPTER'], ACTIONS['LIST']))
def resolve_chapters(root, info, course_id=None, searchField=None, limit=None, offset=None, **kwargs):
current_user = info.context.user
# Checking if this is cached
cache_entity = CHAPTER_CACHE[0]
cache_key = generate_chapters_cache_key(cache_entity, searchField, limit, offset, course_id, current_user)
cached_response = fetch_cache(cache_entity, cache_key)
if cached_response:
return cached_response
# If not cached...
qs = rows_accessible(current_user, RESOURCES['CHAPTER'], {'course_id': course_id})
if searchField is not None:
filter = (
Q(searchField__icontains=searchField.lower())
)
qs = qs.filter(filter)
if offset is not None:
qs = qs[offset:]
if limit is not None:
qs = qs[:limit]
set_cache(cache_entity, cache_key, qs)
return qs
And I'm using this code to dynamically insert the three attributes into each item in the list of chapters that the above code returns:-
class ChapterType(DjangoObjectType):
completed = graphene.Boolean()
completion_status = graphene.String()
locked = graphene.String()
def resolve_completed(self, info):
user = info.context.user
completed = CompletedChapters.objects.filter(participant_id=user.id, chapter_id=self.id).exists()
return completed
def resolve_completion_status(self, info):
user = info.context.user
status = ExerciseSubmission.StatusChoices.PENDING
try:
completed = CompletedChapters.objects.get(participant_id=user.id, chapter_id=self.id)
status = completed.status
except:
pass
return status
def resolve_locked(self, info):
user = info.context.user
locked = is_chapter_locked(user, self)
return locked
class Meta:
model = Chapter
And the method is_chapter_locked() is quite complex in itself:-
def is_chapter_locked(user, chapter):
locked = None
# Letting the user see it if they are a grader
user_role = user.role.name;
grader = user_role == USER_ROLES_NAMES['GRADER']
# Checking if the user is the author of the course or a grader
if chapter.course.instructor.id == user.id or grader:
# If yes, we mark it as unlocked
return locked
course_locked = is_course_locked(user, chapter.course) # Checking if this belongs to a course that is locked
if course_locked:
# If the course is locked, we immediately return locked is true
locked = 'This chapter is locked for you'
return locked
# If the course is unlocked we
completed_chapters = CompletedChapters.objects.all().filter(participant_id=user.id)
required_chapters = MandatoryChapters.objects.all().filter(chapter_id=chapter.id)
required_chapter_ids = required_chapters.values_list('requirement_id',flat=True)
completed_chapter_ids = completed_chapters.values_list('chapter_id',flat=True)
pending_chapter_ids = []
for id in required_chapter_ids:
if id not in completed_chapter_ids:
pending_chapter_ids.append(id)
if pending_chapter_ids:
locked = 'To view this chapter, you must have completed '
pending_chapters_list = ''
for id in pending_chapter_ids:
try:
chapter= Chapter.objects.get(pk=id, active=True)
if pending_chapters_list != '':
pending_chapters_list += ', '
pending_chapters_list += '"' + str(chapter.section.index) +'.'+str(chapter.index)+'. '+chapter.title +'"'
except:
pass
locked += pending_chapters_list
return locked
As can be seen, there is a lot of dynamic processing that is done for fetching the list of chapters. And this is taking a considerably long time, even with caching of the query from the database before the dynamic attributes are calculated.
I am looking for strategies to minimize the dynamic calculation. What kind of an approach works best for performance optimizations in situations like this?
Thank you.

How to use session timeout in django rest view?

I am implementing a view for a game using Django REST's APIView. I am very new to Django and have never done this before so I'm not sure how to implement this.
The main idea is that a game only lasts 5 minutes. I am sending a resource to the user and creating a session object. This view. should be unavailable after 5 minutes. Is there such a thing as a view timeout?
Will the session timeout then work for the post request as well or do I need to implement it there as well?
This is my view:
The out commented code at the end is what I was thinking of doing. Can I even do it in the view directly? How else can I do this and test it?
views.py
class GameView(APIView):
"""
API View that retrieves the game,
retrieves an game round as well as a random resource per round
allows users to post tags that are verified and saved accordingly to either the Tag or Tagging table
"""
def get(self, request, *args, **kwargs):
current_score = 0
if not isinstance(request.user, CustomUser):
current_user_id = 1
else:
current_user_id = request.user.pk
random_resource = Resource.objects.all().order_by('?').first()
resource_serializer = ResourceSerializer(random_resource)
gameround = Gameround.objects.create(user_id=current_user_id,
gamesession=gamesession,
created=datetime.now(),
score=current_score)
gameround_serializer = GameroundSerializer(gameround)
return Response({'resource': resource_serializer.data,
'gameround': gameround_serializer.data,
})
# TODO: handle timeout after 5 min!
# now = timezone.now()
# end_of_game = start_time + timezone.timedelta(minutes=5)
# if :
# return Response({'resource': resource_serializer.data, 'gameround': gameround_serializer.data,})
# else:
# return Response(status=status.HTTP_408_REQUEST_TIMEOUT)
*Testing the out commented code in Postman always leads to a 408_request_timeout.

Update request.session variable before function ends

At a Django server, I have a page with a single button that starts a function.
This function takes a while to complete, and I tried writing updates on the process to a request.session variable with the intent of checking on its contents from a separate page.
It seems however that request.session variables are not updated until the function they are included in are done. At least the variable does not update until then.
Am I right, and if so, is there a way to write to request.session variable before the function's completion?
The sessions are set up properly, I can write and read variables with other examples. For now I'll also just make a temporary db record to store the status update info and read it from there, but I'm curious as of this request.session thing - is my guess correct, and is there a way around?
update:
views.py
#login_required
def autolink(request):
result, time = access_check(request, 'super')
if not result:
return redirect('index')
result = f_autolink(request)
if result is None:
result = request.session.get('technical', '')
return render(request, 'autolink.html', {'result': result, })
functions.py
def f_autolink(request):
if request.method == 'GET':
result = None
elif request.method == 'POST':
request.session['technical'] = 'starting the job'
result = f_kzd_autolink(request)
else:
result = None
return result
def f_kzd_autolink(request):
homeless = Kzd.objects.filter(payment__isnull=True, returned=False).distinct()
result = []
count = homeless.count()
counter = 0
request.session['technical'] = 'Starting link job - {c} records to process'.format(c=count)
for h in homeless:
counter += 1
request.session['technical'] = 'Checking record {c1} of {c}'.format(c1=counter, c=count)
/* long code that makes the h in homeless cycle run for about 3 minutes, unrelated to the question */
so basically, the view shows request.session.get('technical', ''), but neither function writes to it until they are done (it then writes about processing the last record).
The session is saved on a per-request basis when it was modified or when the setting settings.SESSION_SAVE_EVERY_REQUEST is set to True in your settings.
So the simple answer yes, the session is saved by the session middleware when processing the response created by a view. But you could do it manually by calling request.session.save() inside your view.
If you have code, that runs very long it would be better to imediately create a response and use tools like celery to asynchronously process your task.
And you should consider storing your data into an own database table/ own model if it is not really related to the user session.

Django - Check users messages every request

I want to check if a user has any new messages each time they load the page. Up until now, I have been doing this inside of my views but it's getting fairly hard to maintain since I have a fair number of views now.
I assume this is the kind of thing middleware is good for, a check that will happen every single page load. What I need it to do is so:
Check if the user is logged in
If they are, check if they have any messages
Store the result so I can reference the information in my templates
Has anyone ever had to write any middleware like this? I've never used middleware before so any help would be greatly appreciated.
You could use middleware for this purpose, but perhaps context processors are more inline for what you want to do.
With middleware, you are attaching data to the request object. You could query the database and find a way to jam the messages into the request. But context processors allow you to make available extra entries into the context dictionary for use in your templates.
I think of middleware as providing extra information to your views, while context processors provide extra information to your templates. This is in no way a rule, but in the beginning it can help to think this way (I believe).
def messages_processor(request):
return { 'new_messages': Message.objects.filter(unread=True, user=request.user) }
Include that processor in your settings.py under context processors. Then simply reference new_messages in your templates.
I have written this middleware on my site for rendering messages. It checks a cookie, if it is not present it appends the message to request and saves a cookie, maybe you can do something similar:
class MyMiddleware:
def __init__(self):
#print 'Initialized my Middleware'
pass
def process_request(self, request):
user_id = False
if request.user.is_active:
user_id = str(request.user.id)
self.process_update_messages(request, user_id)
def process_response(self, request, response):
self.process_update_messages_response(request, response)
return response
def process_update_messages(self, request, user_id=False):
update_messages = UpdateMessage.objects.exclude(expired=True)
render_message = False
request.session['update_messages'] = []
for message in update_messages:
if message.expire_time < datetime.datetime.now():
message.expired = True
message.save()
else:
if request.COOKIES.get(message.cookie(), True) == True:
render_message = True
if render_message:
request.session['update_messages'].append({'cookie': message.cookie(), 'cookie_max_age': message.cookie_max_age})
messages.add_message(request, message.level, message)
break
def process_update_messages_response(self, request, response):
try:
update_messages = request.session['update_messages']
except:
update_messages = False
if update_messages:
for message in update_messages:
response.set_cookie(message['cookie'], value=False, max_age=message['cookie_max_age'], expires=None, path='/', domain=None, secure=None)
return response