I'm writing a blog application. All the pages (lists of posts, detail of the post) are really static, I can predict when the must be update (for example when I write a new post or a comment is added). I could use #cache_page to cache entire views.
The only problem is that in every page I have some data collected from Twitter that I want to update every 5 minutes.
Django offers template caching, per-view caching and the low level cache framework. With the low level framework I can avoid calculating most of what must be displayed on the page (like caching Post queries, comments, tags...).
What is the best approach to my problem? How to aggressively cache almost everything for a view / template but a few parts?
I want to avoid using iframes.
Thanks
You can not exclude certain parts of a Django template for the cache not should this work in any other template engine I know of.
My advice would be to use JavaScript to asynchronously load you're ever changing content. It should be particularly easy with Twitter as the already offer a great API.
It that doesn't suit you, you can always use Django template caching, to cache only parts of your template.
One option might be to set up Varnish on the server. I'm not familiar with Varnish myself, but as I understand it you can use Edge Side Includes to cache only certain fragments of a page.
Obviously it may not suit your use case, but it sounds like a possibility.
Yes it does! I just tested within an Nginx/gunicorn setup. Sub-questions:
How does it notice the change in the template(s)?
Does performance suffer in any way because of this "feature"?
Can this "feature" be disabled/re-enabled?
Django does not cache the loading of templates by default. Because they are not cached, they get loaded from disk every time they are rendered and that is why you'll see template changes without reloading the application.
If you're interested in caching the template loading process to improve performance (it will help a lot if you are rendering a number of different templates per request), take a look at this post.
On a roll with Django questions today.
The caching framework looks pretty awesome and I'd like to use it sitewide. Rather than set an explicit expiry time for my views, I'd prefer to cache them indefinitely and only invalidate/delete the cache when the content changes. Dream scenario, right?
Is there some way to hook into Django's automatic admin so that when a CRUD operation happens, the relevant cache gets deleted? I expect I'd have to somehow tell the admin panel which model should invalidate which class, but in principle, is this possible? Some kind of callback I can add? Any alternatives?
thanks!
Matt
Two part answer:
Clear cache on a CRUD event? Easy as pie — use Django signals.
Clear only the relevant parts of the cache? This is a genuinely hard problem. On the surface it may look straightforward, but the dependencies can be very difficult to discern for all but the most trivial cases.
We sort of solved part 2 by extending the django caching code to embed object class/id info into the name, and then caching at a sub-page level. On a CRUD event we could do a simple regexp through the cached item names and prune as needed.
All in all, I think it was yet another case of Premature Optimization and it's not at all clear that it made any difference. Next time I'll wait until there is a proven, measurable performance problem before doing something like this.
This is a newbie question from someone who doesn't know much about HTTP caching :)
I'm using Django with the #never_cache decorator.
Is there a way I can instruct the browser to cache the page unless the content has changed, in which case the browser should reload the page?
Thanks.
I disagree with Dominic and there is a very good reason to generate the page, see if it's changed and throw it away if it hasn't - and that's to avoid the need to transfer the entire page over the internet. This only makes sense if your page is quite cheap to generate and is fairly large, but it can be a quick win.
The mechanism for doing this is the ETag header. Django has good support for this, just set USE_ETAGS in settings.py and you'll get the benefit of returning 304 Not Modified responses where appropriate on all your pages.
I think reading this would be a good starting point:
https://web.archive.org/web/20180101014856/http://eflorenzano.com/blog/2008/11/29/drop-dead-simple-django-caching/
An excerpt:
Caching is easy to screw up. Usually it's a manual process which is error-prone and tedious. It's actually quite easy to cache, but knowing when to invalidate which caches becomes a lot harder. [...] The underlying idea is that every Django model has a primary key, which makes for an excellent key to a cache. Using this basic idea, we can cover a fairly large use case for caching, automatically, in a much more deterministic way.
I'm playing around with Django on my website hosting service.
I found out that a simple Django page, which has only some static text, and is rendered from a very simple template I created takes a significant time to render. When compared to a static HTML page, I am getting ~2 seconds difference in the load times. Keep in mind this is a simple test of mine with nothing complicated. Also note that my web hosting is on a shared server (not dedicated), so I might be hitting some CPU limitations.
Seems to me that either:
I have some basic CGI/Apache/Django configuration wrong
Django takes significant overhead, at least in this specific scenario.
I find #1 not probable since I followed my web hosting service wiki on how to set up Django. So we are left with the overhead problem.
My question is which web framework do you find the best to use in scenarios where the website is hosted on a shared server, and CPU/memory overhead must be kept to minimum?
Edit: seems that my configuration is something I might want to look at, and perhaps later on I'll be opening a question on how to best configure Django.
For now, I would appreciate answers focusing on your experience, in general, with web frameworks, and which of those you found to be the best in terms of performance in the aforementioned scenario.
"I have some basic CGI/Apache/Django configuration wrong"
Correct.
First. The very first time Django returns a page, it takes forever. A lot of initialization happens for the first request.
Second. What specific configuration are you using. We just switched from mod_python to mod_wsgi in daemon mode and are very happy with the performance changes.
Third. What database are you using?
Fourth. What test configuration are you using?
Fifth. What caching parameters and reverse proxy are you using?
Odds are good that you have a lot of degrees of freedom in your configuration.
Edit
The question "which of those you found to be the best in terms of performance" is largely impossible to answer.
See http://wiki.python.org/moin/WebFrameworks
There are dozens of frameworks. Few people can examine more than a few to do head-to-head comparison.
The best possible performance is achieved through static content. A Python app that makes static pages (for instance a collection of Jinja templates) is fastest.
After that, it's largely impossible to say. Even http://werkzeug.pocoo.org/ involves some processing overheads that may be unacceptable in the above scenario. Python can be slow.
Django, with a modicum of effort, is often fast enough. Serving static content separately from dynamic content, for example, can be a huge speedup.
Since Django does so much automatically, there's a huge victory in not having to write every little administrative page.
I'd say there has to be something funky with your setup there to get such a large performance difference. Try mod_wsgi (if you're not already) and follow the excellent suggestions by the posters above. If Django genuinely was this slow in all cases, there's just no way companies would be able to use it for production applications. It's more than likely not to be Django that is holding the request up. Once you have the .pyc files all sorted (automatically generated bytecode), then the execution should be fairly zippy.
However, if you don't actually need all Django has to offer, then why use it? I'm using it in quite a large production application, and we're not using all of its features… if you're doing something fairly simple, you may want to consider using something like web.py or Werkzeug (or something non-Python-based if you'd rather).
Frameworks like Django or Ruby on Rails grew out of real world needs. As different as these needs were, as different they turned out.
Here is my Experience:
As a former PHP programmer, I prefered CakePHP for simple stuff and Symfony for more advanced applications. I had a look into Ruby, but the documentation sucked back then. Now I'm using Django. Django works very well for me. In contrast to Symfony I feel like Django brings less flexibility out of the Box, but its easier to extend.
Another approach would be to use 'no framework' CherryPy
I think the host may be an issue. I do Django development on my localhost (Mac) and it's way better. I like WebFaction for cheap hosting and Amazon ec2 for premium hosting.
The framework is strong and it can handle heavy sites - don't obsess about that stuff. The important thing is to create a clean product, Django can handle it. There are about a thousand steps to take when you see how the application handles in the wild, but for now, just trust us that you don't need to worry about the inherent speed of the framework before exhausting a whole slew of parameters including a dedicated VPS/instance when you need it.
Also, following on your edit - I personally don't think performance is a major issue in programming. Here are the issues in terms of concern:
UI/UX efficiency
UI/UX speed (application caching)
Well designed models/views
Optimization of the system (n-tier architecture, etc...)
Optimization of the process (good QA to reduce failures/bottlenecks from deployment)
Optimization of the subsystems (database, etc...)
Hardware
Framework internal optimization
Don't waste time with comparing framework speeds. Their advantage is in extensible code, smart architectures, etc...
On a side note, DO NOT NOT USE A FRAMEWORK FOR A NEW WEB APPLICATION. I'm sorry I can't say it loud enough, but it's an absolute requirement nowadays. It's not even a debate about not using one, just which one to use.
I personally chose Django, which is great. But I can't definitively knock the others out there.
It's possibly both. Django does have stuff for caching built-in, which would be worth trying. Regardless, any non-cached page will nearly always take longer than a static file. A file has to be read in both cases, and in the case of a dynamic page, it also must be executed. And then, in both cases, sent over to the client.
Definetely shared hosting is not the best choice to run heavy frameworks such as Django or CakePHP. If you can afford it, buy VPS.
As for performance, probably your host uses Python with mod-python, which is not recommended now. WSGI is preferred standard for Python powered webapps.