I have a site which uses the standard Django comments (well, a subclass of that, but mostly the same). I want to cache the list of comments that's rendered on each page, as that's quite a big and slow query. But, while I know how to cache individual querysets, I can't see how best to do it for the comments app.
It looks like the querysets for these lists of comments are generated in the BaseCommentNode templatetag. So I can't see an easy way to see if there's a cached version of that QS and return that if so... what's my best way of nicely caching this query?
(I'm also caching every page for all logged-out users, with a 5 minute expiry, but think my site would benefit from caching queries like this for a longer period.)
Related
Often, I end up using several similar queries in my Django views, with one varying lookup field.
This accumulate easily, and lead to severe impacts on page load times. How can one go about optimising this?
{
"pg1": Product.objects.filter(groups__group="pg1")[:5],
"pg2": Product.objects.filter(groups__group="pg2")[:5],
"pg3": Product.objects.filter(groups__group="pg3")[:5],
"pg4": Product.objects.filter(groups__group="pg4")[:5],
}
If I understand properly you are writing query for each page? what happens when you have 100's of pages? No, That's not the way you do it!
Use django.core.paginator in django which helps you manage pagination for your views.
More information could be found here
https://docs.djangoproject.com/en/2.1/topics/pagination/
I have a fairly large django project consisting of several individual apps. I am farming out some of the front-end work (CSS, HTML tweaks) to people who aren't over-familiar with django. To that end I'd like to generate a list of templates for each URL pattern any given engineer is working on. This will save much time that would otherwise be spent manually tracking down the templates used during a view's render phase.
For example, if Bob is working on URLs beginning with /accounts/ then I'd like to generate a list of all the templates used by any view that handles requests to those URLs.
My initial thought is to use something in the test framework since that has access to the templates rendered during a request. However, I can't guarantee that all URLs or views will be exercised (sadly I don't have 100% test coverage), and a missed template is unlikely to be noticed. I don't mind writing a set of tests that simply exercise each view, but don't want to duplicate existing efforts. Also certain views require POSTed data or authentication to function correctly - although I suspect that's an issue I'll have to face no matter what approach is used.
Are there any utilities or snippets that will do what I need?
django-debug-toolbar is a must for developing with Django, It includes a panel detailing all templates used during a request.
I've found the SQL panel is the most helpful for improving page load times as it details slow and duplicate queries.
It can slow down requests when enabled, disabling all panels but those that you use helps.
How can I make Django Rest Frameworks browsable UI fast with RelatedField?
I'm aware this has already been asked here: Django REST Framework: slow browsable UI because of large related table but the answer is no longer valid for new versions of DRF
Including two PrimaryKeyRelatedFields gives me a 5s+ load time, removing them takes me back down to under .3
I've tried setting html_cutoff=100 or even html_cutoff=1but it seems to make no difference to load times.
Any ideas? currently on DRF '3.3.2'
Edit: tables involved have 12000 to 120 records - but it would be great to handle much larger amounts
Since DRF version 3.4.4, it's possible to limit number of relationships being displayed by using selected fields cutoffs.
From DRF documentations:
When rendered in the browsable API relational fields will default to only displaying a maximum of 1000 selectable items. If more items are present then a disabled option with "More than 1000 items…" will be displayed.
...
You can also control these globally using the settings HTML_SELECT_CUTOFF and HTML_SELECT_CUTOFF_TEXT.
This question is similar or duplicate of this one Django REST Framework: slow browsable UI because of large related table.
In essence it's N+1 Problem and in context of Django it can be fixed by eager loading of data by calling prefetch_related() or select_related() on QuerySet. Check this answare
Not quite the answer I am looking for, but currently it looks like there is activity around this already on github - https://github.com/tomchristie/django-rest-framework/issues/3329 with a little luck, one of those patches will get merged soon
I'm started using the Django REST framework in preparation for production, but unfortunately, it is performing quite slowly.
I am calling an array of 500 dictionaries, each with 5 key-value pairs each. In the shell, the call-time is not noticeable at all - you press enter, and it's done. Previously, when I was serving the same content directly without the REST framework, there was also no noticeable delay. However, with the REST framework, it takes about 1 - 2 seconds after the page has rendered for the content to display.
I do not think this is due to javascript as hitting the same details through the browseable API results in a similar delay.
Also, I am NOT cacheing at the moment.
There's no way anyone else is going to be able to debug this for you from the details given in the question.
Are you reusing an existing generic view or writing your own view?
Are you serializing the data, if so what does your serializer definition look like?
Does the issue manifest when rendering to JSON, or just when rendering to the Browsable API?
You mention serving the content without REST framework - what did the views looks like before, and what do they look like after?
The REST framework views are trivial, so use a profiling tool, or simply override them and add some timing. Likewise the renderers are trivial - subclass whatever renderer you're using at the moment, override .render() and add a couple of timing calls either side of calling the parent's .render() method.
If you think you've narrowed down a problem to a specific area then throw together a minimal test case and submit it as an issue.
The serialization itself is unlikely to be an issue, I've used the same serialization engine to replicate Django's fixture dumping and there was no significant degradation.
It's feasible that if you're doing lookups across model relationships you might need to add .select_related() or .prefetch_related() calls when constructing the queryset, exactly as you would with any other Django view.
Edit: Note that following on from this post there were significant serializer speed improvements made, as noted in this ticket.
Say, I have a page with a photo gallery. Each thumbnail has e.g. a photo, country, author and so on. I render these items/widgets using template tags (which load specified templates) - it goes that way because of DRY (I use these items/widgets separately in different places on the page).
And it is very slow.
I have performed some profiling using django-debug-toolbar:
SQL Queries: default 84.81 ms (147 queries)
But:
Total CPU time: 5768.360 msec
Which is too long to wait.
After some analysis it turned out that the main culprit is templating enginge.
When I want to display e.g. 150 photos, 600 associated items/widgets are being rendered via templates. It means 600 I/O operations or even more. Moving these widgets to main template solves the problem, but does not keep DRY.
So my question is how one can avoid such a behaviour? Be DRY and slow or no-DRY and fast?
I'd rather be DRY and fast...
After several hours of profiling and searching...
Thanks for your help, but in this case it seems to me that the best solution so far is to use Template fragment caching:
I tried it and gained 70-80% speed performance!
{% load cache %}
{% cache 3600 mywidget_id %}
.. rendered mywidget is cached ..
{% endcache %}
You might want to try the caching template loader, django.template.loaders.cached.Loader - it should certainly reduce the amount of IO needed.
Edit to add You need to be careful of assuming that just because the majority of time is spent in the template rendering phase, that the query count is not to blame. Don't forget that querysets are lazy, and unless you're specifically slicing or iterating them in the view, they will only be evaluated when the template is loaded. I would say that reducing your query count through good use of select_related and other techniques should be significant help.
I'm assuming since you're using the debug toolbar that you're getting these numbers in development. However, because of this, these are not "real" numbers.
The built-in Django server is good for development, but it has a number of shortcomings that make it much slower that a real webserver would be. First, it's single threaded, so that means no parallel requests. This also means that IO ops are discrete. Second, it's tasked with not just serving requests to Django, but also static resources.
Long and short, if you want to truly profile your site for page load times, you'll need to install a true webserver locally. Basically set it up like you would in your production environment. I'd be willing to wager the request times will be far better, then.
I had the same issue, and maybe for the same reason. I optimized the performance of a custom template tag. The number of db requests fell from 640 to 2, and the resulting db time was under 20ms. My page however, had become slower! 7s -> 10s. Sigh. I tried a cached template loader, without effect.
I had given up, disabled the django debug toolbar, after which the response time fell to 1.2s, unbelievable!! In my case, the huge response time was only caused by the debug toolbar! A related issue can be found here. I'm not diving deeper in this toolbar issue, as I was planning to cache the template tag using template fragment caching anyways. During development I have 10s response time every 15 minutes, which I can live with. Anyways, hope this helps.
This might not apply to this particular problem but in some cases it helped me to use select_related on the queries. Might not be the case but it might lower your query count.
Time spent on queries is relatively little.
But ORM takes much more time to generate these queries and parse the results into the model instances.
So, despite huge number of queries your app is CPU bound because of the slow ORM (in your case it may take a second). So you have to reduce number of queries anyway.
Most probably the queries are made inside of your template tag. So you have to get desired data in a few queries, and set it on the photo instances.
{% for photo in photos|annotate_comment_count %}
...
{% endfor %}
def annotate_comment_count(photo_list):
counts = dict(Comment.objects.filter(photo__in=photo_list).values('photo') \
.annotate(count=models.Count('id')))
for photo in photo_list:
photo.comments_count = counts[photo.pk]
return photo_list
So, inside your templatetag you don't have to query comments count for a single photo, but you already have this information in comments_count attribute. And you achieved this in one query.