I'm writing a blog application. All the pages (lists of posts, detail of the post) are really static, I can predict when the must be update (for example when I write a new post or a comment is added). I could use #cache_page to cache entire views.
The only problem is that in every page I have some data collected from Twitter that I want to update every 5 minutes.
Django offers template caching, per-view caching and the low level cache framework. With the low level framework I can avoid calculating most of what must be displayed on the page (like caching Post queries, comments, tags...).
What is the best approach to my problem? How to aggressively cache almost everything for a view / template but a few parts?
I want to avoid using iframes.
Thanks
You can not exclude certain parts of a Django template for the cache not should this work in any other template engine I know of.
My advice would be to use JavaScript to asynchronously load you're ever changing content. It should be particularly easy with Twitter as the already offer a great API.
It that doesn't suit you, you can always use Django template caching, to cache only parts of your template.
One option might be to set up Varnish on the server. I'm not familiar with Varnish myself, but as I understand it you can use Edge Side Includes to cache only certain fragments of a page.
Obviously it may not suit your use case, but it sounds like a possibility.
Related
this question could be a duplicate but I have a specific use case.
The app is a single page Django app. The user can choose from several options in a drop-down box. Depending on the choice, a few input boxes will need to be rendered. As an example, if the user chooses to order a pizza, the options in another drop-down menu should be related to the toppings whereas an order for drinks should provide options related to the type or brands. The resulting number of input boxes could be 5 with one option or 10 with a different option.
I can think of rendering the page using JS or by using Python in the back-end. Because this app will be a commercial one, I do not need SEO as users need to log into the app first. I also want to minimize the amount of code that a competitor can 're-use' from my work.
Will client-side rendering open up security problems? If not, is client-side the better way to go?
This question is more of a theoretical/opinion-based nature than technical, but let me provide some answers.
Will client-side rendering open up security problems?
Generally, web application security is a server-side concern, not client-side. You can do things like input validation on the client-side, but the minimum practice for security is to sanitize, validate, and authenticate all request data anyway, so the client-side checks are more of for convenience and improved user experience than security. I'm not saying that there are no such things as client-side security concerns, but it's something I don't think is generally a cause of worry. Client-side rendering specifically and especially doesn't sound like something to be careful about: regardless of what your client-side code does, whatever <form> and <input> markup it generates, your server-side code should always handle the submitted data as if it could be malicious.
Is client-side the better way to go?
There are so many more factors to consider in order to answer this, so it's largely a matter of opinion. But since you're asking about Django, then you might want to reduce overall development friction by maximizing Django's features and design—and Django, in my view, is largely a static markup-first framework, meaning minimal use (at first, at least) of client-side JavaScript. Django Forms and Class-Based Views (CBV), for example, work well together to allow rapid development of non-single-page applications.
Your specific use case of an initial drop-down choice determining the main form to be presented could be developed very rapidly in the traditional Django way by giving up your single-page-application requirement, and just providing some initial menu page that will lead to the different views and forms (pizza vs. drinks, etc.), the latter of which you could build rapidly with the help of CBVs. (By the way, your specific use case doesn't seem too unique, actually. It's just the fundamental issue of complexity for which we have programming concepts such as polymorphism and inheritance in object-oriented programming—hence the appropriateness of CBVs.)
I know that single-page applications are nice, and is the fashionable thing nowadays, but I think people underestimate the speed of old-fashioned HTML applications. And by speed I mean not just the user's client-side experience (HTML pages load rather rapidly with HTTP2 and CDNs and all the other modern Web infrastructure tech these days), but also development time.
Besides, you can always just add single-page-like experiences in a progressive manner. Django is particularly suited to an agile-style development strategy where you'd build initial functionality rapidly without much client-side JS, and then just add rich client-side experiences (using React or Vue or something similar) where it will add the most value for users.
I also want to minimize the amount of code that a competitor can 're-use' from my work.
I don't know the full context, but generally I wouldn't worry about this. If you won't do much client-side rendering, then there won't be much client-side code to ‘steal’. But even if you do, unless you specifically write your client-side code in a way that maximizes reusability (either for yourself or for others), I think coders tend anyway to write highly-coupled code, which is to say, your client-side code will tend to be highly dependent on your server-side code's specifics, which means poor reusability. Your competitors could copy your client-side code all they want, but the cost of making it work with their own back-end will be so high that it wouldn't be worth it, they'll just want to write their own.
On a roll with Django questions today.
The caching framework looks pretty awesome and I'd like to use it sitewide. Rather than set an explicit expiry time for my views, I'd prefer to cache them indefinitely and only invalidate/delete the cache when the content changes. Dream scenario, right?
Is there some way to hook into Django's automatic admin so that when a CRUD operation happens, the relevant cache gets deleted? I expect I'd have to somehow tell the admin panel which model should invalidate which class, but in principle, is this possible? Some kind of callback I can add? Any alternatives?
thanks!
Matt
Two part answer:
Clear cache on a CRUD event? Easy as pie — use Django signals.
Clear only the relevant parts of the cache? This is a genuinely hard problem. On the surface it may look straightforward, but the dependencies can be very difficult to discern for all but the most trivial cases.
We sort of solved part 2 by extending the django caching code to embed object class/id info into the name, and then caching at a sub-page level. On a CRUD event we could do a simple regexp through the cached item names and prune as needed.
All in all, I think it was yet another case of Premature Optimization and it's not at all clear that it made any difference. Next time I'll wait until there is a proven, measurable performance problem before doing something like this.
This is a newbie question from someone who doesn't know much about HTTP caching :)
I'm using Django with the #never_cache decorator.
Is there a way I can instruct the browser to cache the page unless the content has changed, in which case the browser should reload the page?
Thanks.
I disagree with Dominic and there is a very good reason to generate the page, see if it's changed and throw it away if it hasn't - and that's to avoid the need to transfer the entire page over the internet. This only makes sense if your page is quite cheap to generate and is fairly large, but it can be a quick win.
The mechanism for doing this is the ETag header. Django has good support for this, just set USE_ETAGS in settings.py and you'll get the benefit of returning 304 Not Modified responses where appropriate on all your pages.
I think reading this would be a good starting point:
https://web.archive.org/web/20180101014856/http://eflorenzano.com/blog/2008/11/29/drop-dead-simple-django-caching/
An excerpt:
Caching is easy to screw up. Usually it's a manual process which is error-prone and tedious. It's actually quite easy to cache, but knowing when to invalidate which caches becomes a lot harder. [...] The underlying idea is that every Django model has a primary key, which makes for an excellent key to a cache. Using this basic idea, we can cover a fairly large use case for caching, automatically, in a much more deterministic way.
Suppose you are running a Django site, and have a legacy PHP forum to support and integrate into your site, since current Django forum solutions are not mature enough.
What is the best way to do this?
Currently I have a simple view, which renders a very simple template which extends my site's base template, and the content area has nothing but an <IFRAME> which holds the forum as its src. A small jQuery function is used to maximize the <IFRAME>'s height (once it finishes loading) so as to contain 100% of the forum content.
But all of this sounds pretty awkward. How would you go about this?
There are a few options. None are ideal (but mixing two platforms never is!)
Use iframes as you've suggested (bad as the address in the address bar is always that of the django page and if somebody copes a link off the forum, it will be the PHP forum, not the django holder)
Use iframes but instead of using the same src all the time, parse the URL and append the relative bit onto the src of the iframe. ie if django sees /forum/this-url, set the src to http://forum-address/this-url and make sure all your links target parent. This has the advantage of showing the correct link in the address bar at all times (rather than it always being /forum/). You'll need to hack your forum for this to work.
Proxy the content and inject it into the page properly. You'll need to pass cookies and it might get really messy but in most terms, this is a great way to integrate things because your links will always be correct. You'll need to butcher your forum theme to strip out everything outside and including the <body> tags.
Theme your forum in the same way as the Django site. This would give best performance but you might have issues if you use dynamic stuff in your django template. An option to get around this is by having the django template cache things to memcache and using php-memcache to pull them out into your forum template.
I have done both 3 and 4 in the past. I used 3 for a very simple form (so didn't have to deal with cookies and sessions as you will). I used 4 for integrating a FluxBB forum into a Wordpress install. Both PHP but it would be uber bloat to load FluxBB inside Wordpress. I cached the dynamic template things into memcache and pulled them out in the forum template.
For this, I would probably suggest going with #4. It's a pain in the arse having to maintain two themes but it's by far the fastest performing solution.
When I read the question summary I immediately thought that you would need some kind of script, which could be linked to a signal via the Dispatcher in Django, to syncronize the user database from your Django site to the forum. This would keep the authentication side of things in check - but you still need to do one of the things that Oli has suggested, to make them look the same.
Themeing will probably be the least hassle-free route, but that's not to say it will be easy!
I'm trying to put a workable plan together for a charity that could really make good use of a forum and a wiki, but a crucial part of its operations happen in parts of the world where dial-up connection dominates and probably will continue to do so for the foreseeable future.
This site was recommended as one that behaves well even on a dial-up connection, so I thought I'd ask for some help here!
The site I want to hook this on to is using Drupal. Anyone out there with experiences like this who could maybe help?
Behaving well on dial-up involves sitting down and optimizing your HTML, CSS, and images to be as small as possible, and then ensuring that your server is sending sane HTTP headers for caching. Make sure your CSS stylesheet is external, and shared across all pages. If dial-up is a major issue, you'll want to stick to a single stylesheet if possible. Avoid JavaScript, because those computers usually don't have the processing power for it either. If you must use JavaScript, jQuery is extremely small and very fast and highly recommended, but I suspect that for most content-oriented websites, it won't be necessary.
To be honest, if you produce valid XHTML/HTML5, valid CSS, and you follow all of the usual best practices for standards-based web design (no table layouts, semantic markup, etc), dial-up really won't be an issue. It'll just work.
To tweak the maximum performance out of your site you might want to install this and use it on your site when you are done with the initial development- ySlow - this will analyse your pages and highlight all the areas you can improve. It's really a great tool for optimising site download speeds.
You should be able to accomplish this, but to be honest you are going to loose a lot in the way of user experience by creating a dial-up friendly site. It basically means you have to do the following to optimize for the experience:
Keep JS to a minimum
Make sure the JS is minified.
Reduce large image requirements w/ CSS and some optimal planning of layout
Make sure caching is enabled in the headers so that new files only get downloaded when nessisary.
If you do all this, you should have a site that is acceptable on dialup.
There are already some hints on how to keep page sizes and load times down.
To complement this, you could use a software that simulates limited bandwith. This helps you test the speed of your site on dialup.
There are several available (just google "simulate dialup").
Sloppy e.g. seems quite usable.
You could also do what Google does for Gmail, i.e. provide 2 versions of your view, one for slow connections that uses plain old HTML, and one for faster connections. You could make the default one the slow one, but provide a link to enable the faster one.
Gmail also has a built-in mechanism that detects when you load the page whether it's going fast or not and will automatically revert to the plain HTML view if it's too slow, which is another fancier alternative.
Your main goal should be minimum page size (keep only HTML in pages, all styling information should be externalized in css files for caching, same for JavaScript in js files) and minimum round trips to server (full requests and post backs). Contrary to popular belief a JS heavy site could work like a charm if you perform a lot of heavy duty client side and keep the server roundtrips clean with the minimum amount of data needed (think JQuery and AJAX here with small partial renderings).
P.S. If u'r using .NET throw ViewState away.