I am trying to apply cache in my Django project using memcache.
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': '127.0.0.1:11211',
}
}
I am applying per view cache as below:
from django.views.decorators.cache import cache_page
url(r'^(?P<slug>[\w-]+)/default/$', cache_page(60 * 15)(default_view), name='default_view')
I am trying to dedug the code in my IDE.
I found the following function at lib/python3.6/site-packages/django/views/decorators/cache.py
def cache_page(timeout, *, cache=None, key_prefix=None):
return decorator_from_middleware_with_args(CacheMiddleware)(
cache_timeout=timeout, cache_alias=cache, key_prefix=key_prefix
)
I tried to create a stop at return line. But I found when i refresh the url the code does not stop at this line.
As already #Alasdair mentioned in the comment if you set a breakpoint in cache_page body your it will be executed only once.
If you want to follow the request you should set your breakpoint in CacheMiddleware methods either in process_response or process_request
Related
Problem:
I want to figure out a way to disable throttling when running my tests with pytest -vv
Details:
I have this default throttling policy in my settings.py file:
'DEFAULT_THROTTLE_RATES': {
'anon': '100/day',
'user': '1000/day'
}
I also have this fixture in my confest.py which returns an error whenever I exceed the limit of requests:
def get_token(user, client):
response = client.post(
"/email-login",
{"email":user.email, "password": "B9vX95phJDi3C4"},
)
return {
"HTTP_AUTHORIZATION": f"Bearer {response.json()['token']['access']}"
}
What I have tried:
I have attempted to use the solution in this GitHub Issue: https://github.com/encode/django-rest-framework/issues/1336, but it doesn't work in my case.
First you need to create a way to differentiate between test env and otherwise. Like we do for PROD and DEV using settings.DEBUG config.
My recommendation is to create an env variable test=Trueand then in your settings.py write -
if os.environ.get("test", False):
REST_FRAMEWORK = {
'DEFAULT_THROTTLE_CLASSES': [
'rest_framework.throttling.AnonRateThrottle',
'rest_framework.throttling.UserRateThrottle'
],
'DEFAULT_THROTTLE_RATES': {
'anon': '100/day',
'user': '1000/day'
}
}
else it does nothing, and drf will not throttle.
I was able to resolve this problem with the following steps:
I created a new settings file which inherited from the base settings file. i.e from settings import *
Then I deleted the DEFAULT_THROTTLE_RATES key i.e del REST_FRAMEWORK["DEFAULT_THROTTLE_RATES"]
Next thing I did was to point to the new settings file in pytest.ini i.e DJANGO_SETTINGS_MODULE="new_settings.py"
Now the tests will use the new settings file
#ra123 has the right idea in general. As another approach, with all Django projects I add something like this to my settings/__init__.py (or just settings.py if you do a one file thing). It looks at argv to see if its in test mode
IS_TESTING = bool(set(sys.argv[:2]) & {"pytest", "test", "jenkins"})
REST_FRAMEWORK = { "YOUR_CONFIG": "..." }
# at the very very end, AFTER your settings are loaded:
if IS_TESTING:
# override your rest framework settings in test mode
REST_FRAMEWORK["DEFAULT_THROTTLE_CLASSES"] = []
# some other handy things, for making tests faster/easier
PASSWORD_HASHERS = ("django.contrib.auth.hashers.MD5PasswordHasher",)
EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"
DEFAULT_FILE_STORAGE = "inmemorystorage.InMemoryStorage"
I ended up with it this way so we don't have to worry about it ever getting the wrong settings. It also helps keep things centralized, so (for example) you don't call sentry.init in testing mode, even if there is a sentry_url in the environment.
Django cannot persist my cached data even if I set timeout to none.
My settings.py contains this:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'django_cache', #'/var/tmp/django_cache',
'TIMEOUT': None,
}
}
I save data with this line of code:
cache.set('array', array, timeout=None)
I fetch data like this:
array = cache.get('array')
try:
iterator = iter(array)
except TypeError:
# Array not iterable: my app gets here when cache data is lost
else:
# I go through the array and get needed info
I don't think it's a MAX_ENTRIES issue because I only have one array with 39 elements.
When the data is lost, the array becomes not iterable (because empty).
I also tried using file cache cause I suspected that restarting the Django app may clear the RAM cache but had the same issue.
I've tried file caching with the following configuration in settings.py:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
'LOCATION': os.path.join(BASE_DIR, 'filecache'),
'TIMEOUT': None,
}
}
The "filecache" directory is created and contains a non-empty cache file but I still loose my cached array somehow.
Background
I have a website running Django 2.0 on AWS ElasticBeanstalk. I have a couple views on my website that take some time to calculate, so I thought I'd look into some simple caching. I decided on LocMemCache because it looked like the quickest to set up that would meet my needs. (I'm using AWS, so using Memcached apparently requires ElastiCache, which adds cost and is additional setup overhead that I wanted to avoid.)
The views do not change often, and the site is not high-traffic, so I put long timeouts on the caches. There are three views where I have enabled caching:
A report generated inside a template – uses Template Fragment caching
A list of locations requested by AJAX and used in a JS library – uses per-view caching
A dynamically-generated binary file download – uses per-view caching
The caching is set up and works great.
The data that goes into these views is added and edited by other staff at my company, that are used to their changes appearing immediately. So in order to address questions such as, "I updated this data, why has the webpage not updated?" I wanted to create a "Clear Server Cache" button, accessible by staff, to force a cache reset.
The button is set up and functioning. It requests a view that calls cache.clear() from django.core.cache. I used the sledgehammer cache.clear() approach because the way to specify an individual per-view cache in code seems to be a bit clunky and convoluted, so the "clear it all" approach seemed adequate. And at the very least it should always "work" in the sense that all the data will get re-loaded again.
The Problem
When I use the button to call cache.clear(), it only clears the Template Fragment cache. It does not seem to clear the per-view caches. Why?
According to Django Documentation,
Be careful with this; clear() will remove everything from the cache, not just the keys set by your application.
So why is it not touching the per-view caches? Doesn't the warning seem to indicate that clear() is dangerous specifically because it's a sledgehammer and nothing at all is spared? What am I missing?
Does AWS use some kind of special memory that's immune to this sort of culling? (If this is the case, then why are the Template Fragments successfully cleared?) I did notice (and find it interesting) that the cache remains even after deploying a new image to the same environment.
I could switch to using Database caching, but I'd like to understand why this isn't working so I don't need to abandon LocMemCache as an option to ever use in the future.
I could also move the others to use Template Fragment caching, but if I ever expand the caching to fit other needs, I will want to be able to use per-view caching. Also, this solution would be less than ideal for a binary-file-download view.
settings.py
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
## 'LOCATION': '',
},
}
portfolio.html (Cache #1 – Template Fragment)
{% load static cache compress %}
...
<div id="total-portfolio-content">
{% cache 7200 portfolio %}{% include 'reports/total_portfolio/report_include.html' %}{% endcache %}
</div>
map/urls.py (Cache #2 – Per-view)
from django.conf.urls import include, url
app_name = 'map'
urlpatterns = [
## Yes, I know this uses the old-style url(). I have plans to upgrade the entire project.
url(r'^(?P<tg>[\w-]+)/data.geojson$',
cache_page(60 * 60 * 12)(
views.NamedGeoJSONLayerView.as_view(model=FacilityCoord)),
name='tg-data'),
]
resources/urls/__init__.py (Cache #3 – Per-view)
from django.conf.urls import include, url
app_name = 'resources'
urlpatterns = [
url(r'^download/$',
cache_page(60 * 60 * 12)(
views.DownloadMetricXLSX.as_view()),
name='download'),
]
myadmin/views.py (Cache Clear button)
from django.core.cache import cache
#staff_member_required(login_url=login_url)
def clear_cache(request):
cache.clear()
## And because that doesn't seem to work as advertised, I also tried....
## taken from <https://djangosnippets.org/snippets/1080/>
try:
cache._cache.clear() # in-memory caching
cache._expire_info.clear()
except AttributeError:
# I think this only applies to filesystem caching? Just grasping at straws.
old_freq = cache._cull_frequency
old_max = cache._max_entries
cache._max_entries = 0
cache._cull_frequency = 1
cache._cull()
cache._cull_frequency = old_freq
cache._max_entries = old_max
return JsonResponse({'success': True})
The problem is in the response headers. The cache_page decorator automatically adds a max-age option to the Cache-Control header in the response. So the cache clear was working properly, clearing the local memory on the server, but the user's browser was instructed not to ask the server for updated data for the duration of the timeout. And my browser was happily complying (even after Ctrl-F5).
Fortunately, there are other decorators you can use to deal with this without much difficulty, now that it's clear what's happening. Django provides a number of other decorators, such as cache_control or never_cache.
I ended up using never_cache, which turned the urls files into...
from django.conf.urls import include, url
from django.views.decorators.cache import never_cache, cache_page
app_name = 'map'
urlpatterns = [
url(r'^(?P<tg>[\w-]+)/data.geojson$',
never_cache(cache_page(60 * 60 * 12)(
views.NamedGeoJSONLayerView.as_view(model=FacilityCoord))),
name='tg-data'),
]
and
from django.conf.urls import include, url
from django.views.decorators.cache import never_cache, cache_page
app_name = 'resources'
urlpatterns = [
url(r'^download/$',
never_cache(cache_page(60 * 60 * 12)(
views.DownloadMetricXLSX.as_view())),
name='download'),
]
So Im running into this CACHE error when I try to runserver or syncdb.
Here is the traceback: https://gist.github.com/1538051
I tried inserting this into the settings.py file:
CACHE_BACKEND = {
'default': {
'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
}
}
But that gave another error which makes no sense to me.
if backend_uri.find(':') == -1:
AttributeError: 'dict' object has no attribute 'find'
Can someone help me figure what the problem is and how I can go about fixing it.
NB: I am working on the dev server
If you're using Django 1.2 or lower, CACHE_BACKEND (docs) accepts a string:
CACHE_BACKEND = 'dummy://'
If you're using Django 1.3+, CACHE_BACKEND has been deprecated in favour of CACHES (docs):
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
}
}
I'm not sure why your installation isn't creating the appropriate default - it could be that there's something else going on with your installation, but I don't really have enough information without knowing more about your settings.py etc.
First off, What version of django are you using? The dictionary- style backend config is new to django 1.3, and your traceback suggests that you are on something like 1.2.4.
If that's the case, you will need to use the older uri-style cache settings. Something like
CACHE_BACKEND = 'dummy://'
would match what you are trying to set in your question.
The full documentation on caching, relevant to Django 1.2, is available here: http://docs.djangoproject.com/en/1.2/topics/cache/
How can I ask Django to tell me when it encounters, for example, an undefined variable error while it's rendering templates?
I've tried the obvious DEBUG = True and TEMPLATE_DEBUG = True, but they don't help.
Put this in your debug settings:
class InvalidString(str):
def __mod__(self, other):
from django.template.base import TemplateSyntaxError
raise TemplateSyntaxError(
"Undefined variable or unknown value for: \"%s\"" % other)
TEMPLATE_STRING_IF_INVALID = InvalidString("%s")
This should raise an error when the template engine sees or finds an undefined value.
According to the django documentation,
undefined variables are treated as ''(empty string) by default. While in if for regroup, it's None.
If you are going to identify the variable undefined, change TEMPLATE_STRING_IF_INVALID in settings.
'%s' makes the invalid variable to be rendered as its variable name, in this way, u can identify easily.
how-invalid-variables-are-handled
Finding template variables that didn't exist in the context was important to me as several times bugs made it into production because views had changed but templates had not.
I used this technique, implemented in manage.py, to achieve the effect of breaking tests when template variables not found in the context were used. Note that this technique works with for loops and if statements and not just {{ variables }}.
import sys
# sometimes it's OK if a variable is undefined:
allowed_undefined_variables = [
'variable_1',
'variable_2',
]
if 'test' in sys.argv:
import django.template.base as template_base
old_resolve = template_base.Variable.resolve
def new_resolve(self, context):
try:
value = old_resolve(self, context)
except template_base.VariableDoesNotExist as e:
# if it's not a variable that's allowed to not exist then raise a
# base Exception so Nodes can't catch it (which will make the test
# fail)
if self.var not in allowed_undefined_variables:
raise Exception(e)
# re-raise the original and let the individual Nodes deal with it
# however they'd like
raise e
return value
template_base.Variable.resolve = new_resolve
How to log a warning on undefined variable in a template
It seems that Django relies on undefined variables being a simple empty string. So instead of changing this behaviour or making it throw an exception, let's keep it the same but have it log a warning instead!
In your settings.py file:
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
# ...
'OPTIONS': {
# ...
'string_if_invalid': InvalidStringShowWarning("%s"),
},
}
]
(string_if_invalid replaces TEMPLATE_STRING_IF_INVALID in newer Django versions.)
And further up, you'll need to define the InvalidStringShowWarning class, making it behave while logging a warning:
class InvalidStringShowWarning(str):
def __mod__(self, other):
import logging
logger = logging.getLogger(__name__)
logger.warning("In template, undefined variable or unknown value for: '%s'" % (other,))
return ""
def __bool__(self): # if using Python 2, use __nonzero__ instead
# make the template tag `default` use its fallback value
return False
You should be able to see the warning in the output of python manage.py runserver.
I believe that's a major oversight on Django's part and the primary reason I prefer not to use their default template engine. The sad truth is that, at least for now (Django 1.9), you can't achieve this effect reliably.
You can make Django raise an exception when {{ undefined_variable }} is encountered - by using "the hack" described in slacy's answer.
You can't make Django raise the same exception on {% if undefined_variable %} or {% for x in undefined_variable %} etc. "The hack" doesn't work in such cases.
Even in cases in which you can, it is strongly discouraged by Django authors to use this technique in production environment. Unless you're sure you don't use Django's built-in templates in your app, you should use "the hack" only in DEBUG mode.
However, if you're stuck with Django's templates for now, I would recommend to use slacy's answer, just make sure you're in DEBUG mode.
Read up on how invalid variable are handled in templates. Basically, just set TEMPLATE_STRING_IF_INVALID to something in your settings.py.
TEMPLATE_STRING_IF_INVALID = "He's dead Jim! [%s]"
I am use next:
import logging
from django.utils.html import format_html
from django.utils.safestring import mark_safe
class InvalidTemplateVariable(str):
"""
Class for override output that the Django template system
determinated as invalid (e.g. misspelled) variables.
"""
# styles for display message in HTML`s pages
styles = mark_safe('style="color: red; font-weight: bold;"')
def __mod__(self, variable):
"""Overide a standart output here."""
# access to current settings
from django.conf import settings
# display the message on page in make log it only on stage development
if settings.DEBUG is True:
# format message with captured variable
msg = 'Attention! A variable "{}" does not exists.'.format(variable)
# get logger and make
logger = self.get_logger()
logger.warning(msg)
# mark text as non-escaped in HTML
return format_html('<i {}>{}</i>', self.styles, msg)
# on production it will be not displayed
return ''
def get_logger(self):
"""Create own logger with advanced error`s details."""
logger = logging.getLogger(self.__class__.__name__)
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
return logger
Usage in settings file (by default it settings.py):
TEMPLATES = [
{
......
'OPTIONS': {
.....................
'string_if_invalid': InvalidTemplateVariable('%s'),
.....................
},
},
]
or directly
TEMPLATES[0]['OPTIONS']['string_if_invalid'] = InvalidTemplateVariable('%s')
A result if DEBUG = True:
On page
In console
> System check identified 1 issue (0 silenced). October 03, 2016 -
> 12:21:40 Django version 1.10.1, using settings 'settings.development'
> Starting development server at http://127.0.0.1:8000/ Quit the server
> with CONTROL-C. 2016-10-03 12:21:44,472 - InvalidTemplateVariable -
> WARNING - Attention! A variable "form.media" does not exists.
You can use the pytest-django setting FAIL_INVALID_TEMPLATE_VARS
The invaild vars get checked if pytest executes the code.
[pytest]
DJANGO_SETTINGS_MODULE = mysite.settings
FAIL_INVALID_TEMPLATE_VARS = True
If there is a undefined variable in templates, django won't tell you.
You can print this variable in view.