I wonder is it a bad idea to get data from request session or is it better to parse the data into dict context and render it (Need to do it for each view)?
you can add this to your TEMPLATE_CONTEXT_PROCESSORS if you are accessing the request object in templates frequently (like I do for URL get parameter processing).
"django.core.context_processors.request",
It's common practice to send what you need through the context in the view.
I feel like it gives you a little more security/certainty in what you're doing because you can keep your logic in the view where it should be rather than doing any checks in the template for things being in the request.
edit
The above is only true if you're looking to do something rarely. If you're regularly adding an element of the request to your templates you should indeed, as everybody else suggests, be writing context processors to make what you require available to all views.
Take a look at the docs; TEMPLATE_CONTEXT_PROCESSORS
Also give this chapter of the django book a read as it'll be very helpful; Chapter 9: Advanced Templates
Specifically this section;
Guidelines for Writing Your Own Context Processors
Here are a few tips for rolling your own:
Make each context processor responsible for the smallest subset of functionality possible. It’s easy to use multiple processors, so you might as well split functionality into logical pieces for future reuse.
Keep in mind that any context processor in TEMPLATE_CONTEXT_PROCESSORS will be available in every template powered by that settings file, so try to pick variable names that are unlikely to conflict with variable names your templates might be using independently. As variable names are case-sensitive, it’s not a bad idea to use all caps for variables that a processor provides.
It doesn’t matter where on the filesystem they live, as long as they’re on your Python path so you can point to them from the TEMPLATE_CONTEXT_PROCESSORS setting. With that said, the convention is to save them in a file called context_processors.py within your app or project.
Django gives you a way to put data into every template, it is called context processors.
http://www.b-list.org/weblog/2006/jun/14/django-tips-template-context-processors/
https://docs.djangoproject.com/en/1.7/ref/templates/api/
Related
I have an html file which is the base,where other html documents extends.Its a static page but i want to have variable in the menu.I don't think it's wise to create a view for it,since i don't intend to let users visit the base alone.So where in my project can I store site-wide dynamic variables that can be called on any page without explicitly stating them in their views.
Thank you in advance.
For user specific variables, use session.
For global constants (not variables!), use settings.py.
For global variables, consider to store it in database so it can be multithreading & multiprocess safe.
I looked around and saw different approaches,but one that doesn't compromise the DRY philosophy the most for me is registering a tag in your project then input it in the base template.Its neater See here https://stackoverflow.com/a/21062774/6629594 for an example
Storage can take any number of places, I put mine in a stats model in the db so you get all the goodness of that (and make it easy to access in views).
I then have a context processor written as so:
#context_processors.py:
def my_custom_context_processor(request):
return {'custom_context_variable1':'foo','custom_context_variable2':'bar'}
Add this to your context processors in settings.py:
TEMPLATE_CONTEXT_PROCESSORS = (
...
"my_app.context_processors.ny_custom_context_processor",
)
Provided you use render() to render your templates you can then you can just use:
{{ custom_context_variable1 }}
to return 'foo' in your template. Obviously returning strings is for example only, you can use anything you like so long as your context processor returns a dict.
you can also try using php pages.
Then acces the variable on each page with an include 'file containing the var.php' on every page.
None of this will be visible in the source html as it is only processed on the server side.
If you you would like to try this, mail me and I will send you some sample code.
I have a considerable number of custom template tags that perform a variety of functions, including:
simple string transformation
display of complex ui elements
timestamp manipulation and formatting
handling and display of user avatars
etc...
All of these functions reside in a single file: app/templatetags/custom_tags.py. When I want to use one of these tags in a template, I import all of them using {% load custom_tags %}.
However, only a small subset of the available tags are actually used in any given template. In other words, all these functions are being 'loaded' into the template, yet only a few of them are called in a specific web request.
Is this inefficient, in terms of performance? Should I be loading code more conservatively -- i.e., splitting up my custom tags into separate files and only loading the subset that I need?
Or does this not matter, because all tags are loaded in memory -- i.e., subsequent calls to {% load custom_tags %} elsewhere in the application won't result in any additional overhead?
I apologize if there are incorrect assumptions or premises in this question. I'd love to have a better understanding of the implications of importing python code in general, or in a Django environment specifically.
For Django <= 1.8:
The load tag is defined here and actually does the loading here and here. Both places call to get_library, defined here. According to the docstrings there, yes, it caches template tag/filter libraries within the same process in the dictionary initalized here.
For Django 1.9:
The modules for template tags are now being loaded even earlier, when the parser is instantiated, and the libraries are being stored directly on the parser. Load tags call out to find_library here and here, which just gets the already-loaded tag directly from the parser.
Aside from the actual loading activity
As #spectras points out below, regardless of the Django version, the tag's loading behavior is, strictly speaking, a side effect, and the tag returns (<=1.8/1.9) a no-op node(<=1.8/1.9), which renders no content--so there's not really a performance consideration as far as that goes.
I know that in django integration, it is easy to test if a page would load successfully by making sure status code is 200. However, the project I am working on have pages that might partially load (certain sections of the page will silently fail to load). What is the best way to catch this situation? Is there a way to insert such error into the http response?
I know I can potentially do regex on the text on the page to check for things that might not load or I can probably check that name of certain css class exist. But that does not seem to be too robust an approach.
This will greatly depend on your implementation details, but there are two suitable approaches to testing templates that may help you:
If the partial page loading can be tested/triggered by using nothing more than template syntax, create test templates that conditionally print some text you can match against in the response, such as WORKED or FOO.
If it's something that largely depends upon the context the template receives, then one-off test views, which you define alongside your test case and call directly by passing in a mocked request, work as well. In this case, you'll likely rely on the test view to raise exceptions if the page rendering won't proceed as expected, otherwise everything went well.
Alternatively, you can even mix the two. In this case, you'll rely on the view to generate a HTTP response which you'll then check for some test text.
If that doesn't work, you can resort to overriding the templates. The general problem is that you can't rely on matching against text because it's global. The template can change and potentially cause your tests to misfire. What you can then do is have specific test settings that add additional directories for template discovery where you can provide different template implementations which contain text that does not change which would be suitable and safe for matching against in the test. The difficulty with this approach is that it does not neatly document itself, as opposed to the previous two approaches.
In our company we make news portals for a pretty big number of local newspapers (currently 13, going to 30 next month and more in the future), each with 2k to 100k page views/day. Since we are evolving from a situation where each site was heavily customized to one where each difference is a matter of configuration or custom template, our software is already pretty much the same for all sites. Right now our deployment strategy is one gunicorn instance for each site (with 1-17 workers each, depending on the site traffic), on a 16-core server and 12GB RAM. The problem with this setup is that each worker (regular pre-forked gunicorn) takes 110MB, whether its being used or not. Now with the new sites we would need to add more RAM to serve not that much many requests, so basically it doesn't scale. Also, since we are moving from this model where each site is independent, each site has its own database and I quite like it that way, especially since we are using relational databases (mysql, but migrating to pgsql), so its much easier to shard this way.
I'm doing some research and experimenting with running all sites on one gunicorn instance, so I could use the servers fully and add more servers behind a load balancer when it came to it. The problem is that django assumes in a lot of places that only one site is running per process, so for what I've thought of so far I'd have to implement:
A middleware that takes the HTTP_HOST from the request and places an identifier on a threadlocal variable.
A template loader that uses that variable to load custom templates accordingly.
Monkey patch django.db.model.Model, probably adding a metaclass (not even sure that's possible, but I think I would need it because of the custom managers we sometimes need to use) that would overwrite the managers for one that would first call db_manager(identifier) on the original manager and then call the intended method. I would also need to overwrite the save and delete methods to always include the using=identifier parameter.
I guess I would need to stop using inclusion_tag decorators, not a big problem, but I need to think of other cases like this.
Heavy and ugly patching of urlresolvers if I need custom or extra urls for each site. I don't need them now, but probably will at some point.
And this is just is what I came up with without even implementing it and seeing where it breaks, I'm sure I'd need many more changes for it to work. So I really don't want to do it, especially with the extra maintenance effort I'll need, but I don't see any alternatives and would love to learn that someone already solved this in a better way. Of course I could also stop using django altogether (I already have many reasons to do so) but that would mean a major rewrite and having two maintain two incompatible branches of the software until the new one reached feature parity with the django version, so to me it seems even worse than all the ugly hacks.
I've recently developed an e-commerce system with similar requirements -- many instances running from the same project sharing almost everything. The previous version of the system was a bunch of independent installations (~30) so it was pretty unmaintainable. I'm sure the requirements still differ from yours (for example, all instances shared the same models in my case), but it still might be useful to share my experience.
You are right that Django doesn't help with scenarios like this out of the box, but it's actually surprisingly easy to work it around. Here is a brief description of what I did.
I could see a synergy between what I wanted to achieve and django.contrib.sites. Also because many third-party Django apps out there know how to work with it and use it, for example, to generate absolute URLs to the current site. The major problem with sites is that it wants you to specify the current site id in settings.SITE_ID, which a very naive approach to the multi host problem. What one naturally wants, and what you also mention, is to determine the current site from the Host request header. To fix this problem, I borrowed the hook idea from django-multisite: https://github.com/shestera/django-multisite/blob/master/multisite/threadlocals.py#L19
Next I created an app encapsulating all the functionality related to the multi host aspect of my project. In my case the app was called stores and among other things it featured two important classes: stores.middleware.StoreMiddleware and stores.models.Store.
The model class is a subclass of django.contrib.sites.models.Site. The good thing about subclassing Site is that you can pass a Store to any function where a Site is expected. So you are effectively still just using the old, well documented and tested sites framework. To the Store class I added all the fields needed to configure all the different stores. So it's got fields like urlconf, theme, robots_txt and whatnot.
The middleware class' function was to match the Host header with the corresponding Store instance in the database. Once the matching Store was retrieved, It would patch the SITE_ID in a way similar to https://github.com/shestera/django-multisite/blob/master/multisite/middleware.py. Also, it looked at the store's urlconf and if it was not None, it would set request.urlconf to apply its special URL requirements. After that, the current Store instance was stored in request.store. This has proven to be incredibly useful, because I was able to do things like this in my views:
def homepage(request):
featured = Product.objects.filter(featured=True, store=request.store)
...
request.store became a natural additional dimension of the request object throughout the project for me.
Another thing that was defined on the Store class was a function get_absolute_url whose implementation looked roughly like this:
def get_absolute_url(self, to='/'):
"""
Return an absolute url to this `Store` or to `to` on this store.
The URL includes http:// and the domain name of the store.
`to` can be an object with `get_absolute_url()` or an absolute path as string.
"""
if isinstance(to, basestring):
path = to
elif hasattr(to, 'get_absolute_url'):
path = to.get_absolute_url()
else:
raise ValueError(
'Invalid argument (need a string or an object with get_absolute_url): %s' % to
)
url = 'http://%s%s%s' % (
self.domain,
# This setting allowed for a sane development environment
# where I just set it to ".dev:8000" and configured `dnsmasq`.
# The same value was also removed from the `Host` value in the middleware
# before looking up the `Store` in database.
settings.DOMAIN_SUFFIX,
path
)
return url
So I could easily generate URLs to objects on other than the current store, e.g.:
# Redirect to `product` on `store`.
redirect(store.get_absolute_url(product))
This was basically all I needed to be able to implement a system allowing users to create a new e-shop living on its own domain via the Django admin.
I am deleting a single path from the Django cache like this:
from models import Graph
from django.http import HttpRequest
from django.utils.cache import get_cache_key
from django.db.models.signals import post_save
from django.core.cache import cache
def expire_page(path):
request = HttpRequest()
request.path = path
key = get_cache_key(request)
if cache.has_key(key):
cache.delete(key)
def invalidate_cache(sender, instance, **kwargs):
expire_page(instance.get_absolute_url())
post_save.connect(invalidate_cache, sender = Graph)
This works - but is there a way to delete recursively? My paths look like this:
/graph/123
/graph/123/2009-08-01/2009-10-21
Whenever the graph with id "123" is saved, the cache for both paths needs to be invalidated. Can this be done?
You might want to consider employing a generational caching strategy, it seems like it would fit what you are trying to accomplish. In the code that you have provided, you would store a "generation" number for each absolute url. So for example you would initialize the "/graph/123" to have a generation of one, then its cache key would become something like "/GENERATION/1/graph/123". When you want to expire the cache for that absolute url you increment its generation value (to two in this case). That way, the next time someone goes to look up "/graph/123" the cache key becomes "/GENERATION/2/graph/123". This also solves the issue of expiring all the sub pages since they should be referring to the same cache key as "/graph/123".
Its a bit tricky to understand at first but it is a really elegant caching strategy which if done correctly means you never have to actually delete anything from cache. For more information here is a presentation on generational caching, its for Rails but the concept is the same, regardless of language.
Another option is to use a cache that supports tagging keys and evicting keys by tag. Django's built-in cache API does not have support for this approach. But at least one cache backend (not part of Django proper) does have support.
DiskCache* is an Apache2 licensed disk and file backed cache library, written in pure-Python, and compatible with Django. To use DiskCache in your project simply install it and configure your CACHES setting.
Installation is easy with pip:
$ pip install diskcache
Then configure your CACHES setting:
CACHES = {
'default': {
'BACKEND': 'diskcache.DjangoCache',
'LOCATION': '/tmp/path/to/directory/',
}
}
The cache set method is extended by an optional tag keyword argument like so:
from django.core.cache import cache
cache.set('/graph/123', value, tag='/graph/123')
cache.set('/graph/123/2009-08-01/2009-10-21', other_value, tag='/graph/123')
diskcache.DjangoCache uses a diskcache.FanoutCache internally. The corresponding FanoutCache is accessible through the _cache attribute and exposes an evict method. To evict all keys tagged with /graph/123 simply:
cache._cache.evict('/graph/123')
Though it may feel awkward to access an underscore-prefixed attribute, the DiskCache project is stable and unlikely to make significant changes to the DjangoCache implementation.
The Django cache benchmarks page has a discussion of alternative cache backends.
Disclaimer: I am the original author of the DiskCache project.
Checkout shutils.rmtree() or os.removedirs(). I think the first is probably what you want.
Update based on several comments: Actually, the Django caching mechanism is more general and finer-grained than just using the path for the key (although you can use it at that level). We have some pages that have 7 or 8 separately cached subcomponents that expire based on a range of criteria. Our component cache names reflect the key objects (or object classes) and are used to identify what needs to be invalidated on certain updates.
All of our pages have an overall cache-key based on member/non-member status, but that is only about 95% of the page. The other 5% can change on a per-member basis and so is not cached at all.
How you iterate through your cache to find invalid items is a function of how it's actually stored. If it's files you can use simply globs and/or recursive directory deletes, if it's some other mechanism then you'll have to use something else.
What my answer, and some of the comments by others, are trying to say is that how you accomplish cache invalidation is intimately tied to how you are using/storing the cache.
Second Update: #andybak: So I guess your comment means that all of my commercial Django sites are going to explode in flames? Thanks for the heads up on that. I notice you did not attempt an answer to the problem.
Knipknap's problem is that he has a group of cache items that appear to be related and in a hierarchy because of their names, but the key-generation logic of the cache mechanism obliterates that name by creating an MD5 hash of the path + vary_on. Since there is no trace of the original path/params you will have to exhaustively guess all possible path/params combinations, hoping you can find the right group. I have other hobbies that are more interesting.
If you wish to be able to find groups of cached items based on some combination of path and/or parameter values you must either use cache keys that can be pattern matched directly or some system that retains this information for use at search time.
Because we had needs not-unrelated to the OP's problem, we took control of template fragment caching -- and specifically key generation -- over 2 years ago. It allows us to use regexps in a number of ways to efficiently invalidate groups of related cached items. We also added a default timeout and vary_on variable names (resolved at run time) configurable in settings.py, changed the ordering of name & timeout because it made no sense to always have to override the default timeout in order to name the fragment, made the fragment_name resolvable (ie. it can be a variable) to work better with a multi-level template inheritance scheme, and a few other things.
The only reason for my initial answer, which was indeed wrong for current Django, was because I have been using saner cache keys for so long I literally forgot the simple mechanism we walked away from.