This is mostly curiosity, but is the DB penalty for wrapping an entire view with #transaction.atomic a negligible one?
I'm thinking of views where the GET of a form or its re-display after a validation fail involves processing querysets. (ModelChoiceFields, for example, or fetching an object that the template displays.)
It seems to me to be far more natural to use with transaction.atomic() around the block of code which actually alters a bunch of related DB objects only after the user's inputs have validated.
Am I missing something?
From the source code:
def atomic(using=None, savepoint=True, durable=False):
# Bare decorator: #atomic -- although the first argument is called
# `using`, it's actually the function being decorated.
if callable(using):
return Atomic(DEFAULT_DB_ALIAS, savepoint, durable)(using)
# Decorator: #atomic(...) or context manager: with atomic(...): ...
else:
return Atomic(using, savepoint, durable)
It's the same. In both cases the function is returning an Atomic object which handles whether the transaction should commit or not.
Related
I'm trying to utilize django's row-level-locking by using the select_for_update utility. As per the documentation, this can only be used when inside of a transaction.atomic block. The side-effect of using a transaction.atomic block is that if my code throws an exception, all the database changes get rolled-back. My use case is such that I'd actually like to keep the database changes, and allow the exception to propagate. This leaves me with code looking like this:
with transaction.atomic():
user = User.objects.select_for_update.get(id=1234)
try:
user.do_something()
except Exception as e:
exception = e
else:
exception = None
if exception is not None:
raise exception
This feels like a total anti-pattern and I'm sure I must be missing something. I'm aware I could probably roll-my-own solution by manually using transaction.set_autocommit to manage the transaction, but I'd have thought that there would be a simpler way to get this functionality. Is there a built in way to achieve what I want?
I ended up going with something that looks like this:
from django.db import transaction
class ErrorTolerantTransaction(transaction.Atomic):
def __exit__(self, exc_type, exc_value, traceback):
return super().__exit__(None, None, None)
def error_tolerant_transaction(using=None, savepoint=True):
"""
Wraps a code block in an 'error tolerant' transaction block to allow the use of
select_for_update but without the effect of automatic rollback on exception.
Can be invoked as either a decorator or context manager.
"""
if callable(using):
return ErrorTolerantTransaction('default', savepoint)(using)
return ErrorTolerantTransaction(using, savepoint)
I can now put an error_tolerant_transaction in place of transaction.atomic and exceptions can be raised without a forced rollback. Of course database-related exceptions (i.e. IntegrityError) will still cause a rollback, but that's expected behavior given that we're using a transaction. As a bonus, this solution is compatible with transaction.atomic, meaning it can be nested inside an atomic block and vice-versa.
When I want to select objects with a get() function like
personalProfile = World.objects.get(ID=personID)
If get function doesn't return find a value, a "matching query does not exist." error occurs.
If I don't need this error, I'll use try and except function
try:
personalProfile = World.objects.get(ID=personID)
except:
pass
But I think this is not the best way since I use
except:
pass
Please recommend some idea or code sample to fight with this issue
That depends on what you want to do if it doesn't exist..
Theres get_object_or_404:
Calls get() on a given model manager, but it raises Http404 instead of the model’s DoesNotExist exception.
get_object_or_404(World, ID=personID)
Which is very close to the try except code you currently do.
Otherwise theres get_or_create:
personalProfile, created = World.objects.get_or_create(ID=personID)
Although, If you choose to continue with your current approach, at least make sure the except is localised to the correct error and then do something with that as necessary
try:
personalProfile = World.objects.get(ID=personID)
except MyModel.DoesNotExist:
raise Http404("No MyModel matches the given query.")
The above try/except handle is similar to what is found in the docs for get_object_or_404...
A get_or_none() function has been proposed, multiple times now. The rejection notice is feature creep, which you might or might not agree with. The functionality is present --with slightly different semantics-- in the first() queryset method.
But first things first:
The manager throws World.DoesNotExist, a specialized subclass of ObjectDoesNotExist when a World object was not found:
try:
personalProfile = World.objects.get(ID=personID)
except World.DoesNotExist:
pass
There's also get_object_or_404() which raises a Http404 exception when the object was not found.
You can also roll your own get_or_none(). A possible implementation could be:
def get_or_none(queryset, *args, **kwargs):
try:
return queryset.get(*args, **kwargs)
except ObjectDoesNotExist:
return None
Note that this still raises MultipleObjectsReturned when more than one matching object is found. If you always want the first object regardless of any others, you can simplify using first(), which returns None when the queryset is empty:
def get_or_none(queryset, *args, **kwargs):
return queryset.filter(*args, **kwargs).first()
Note however, for this to work reliably, you need a proper order for objects, because in the presence of multiple objects first() might be non-deterministic (it probably returns the first object from the database index used to filter the query and neither indexes not the underlying tables need be sorted or even have a repeatable order).
Use both, however, only when the use of the object to retrieve is strictly optional for the further program flow. When failure to retrieve an object is an error, use get_object_or_404(). When an object should be created when it does not exist, use get_or_create(). In those cases, both are better suited to simplify program flow.
As alasdair mentioned you could use the built in first() method.
It returns the object if it exists or None if it's not
personalProfile = World.objects.filter(ID=personID).first()
I am seeing some really surprising and frustrating behavior with Django testing. Model objects are being "found" by a related lookup, but no model objects exist. (I apologize for the weird description here...the behavior is bizarre enough that I don't know quite how to describe it. Do the objects exist? Do I exist? Do you??)
I need them to exist, so I have a method in place that creates them if they don't exist. The problem is that on one line, Django finds that they do exist, and therefore they are not created...and then on the next line we can confirm that no such objects exist.
My tests are giving Errors in test_something() related to the absence of the necessary TaskMetadata object.
#the model
class TaskMetadata(models.Model):
task = models.OneToOneField(ContentType)
...
#the test
class SimpleTest(TestCase):
def setUp(self):
some_utility_function()
def test_something(self):
...something that requires TaskMetadata...
def some_utility_function():
task = ...whatever...
ctype = ContentType.objects.get_for_model(task)
try:
ctype.taskmetadata
except TaskMetadata.DoesNotExist:
...create TaskMetadata...
print "Created TaskMetadata object for %s" % task.__name__
else:
print "TaskMetadata object already exists for %s" % task.__name__
print ctype.taskmetadata
print "ALL OF THEM!! %s" % TaskMetadata.objects.all()
and the printed result of some_utility_function():
TaskMetadata object already exists for SomeTask
some task
ALL OF THEM!! [] # <-- NOTE EMPTY QUERYSET
In summary: "Yes, TaskMetadata object exists. Yes, TaskMetadata object exists. No, there are no TaskMetadata objects at all!!"
So, seriously, what on earth is going on here? Is this a cache problem? I tried clearing the cache (wild guess; I don't have CACHES configured in settings.py)
def setUp(self):
cache.clear()
some_utility_function()
Does not help. Transactions maybe? I'm stumped. How do I even debug this?
UPDATE:
See a minimal django project that replicates the issue here.
When the first testcase runs, TaskMetadata.objects.all() is NOT an empty queryset (it is in fact populated with objects, as I would expect); when the second testcase (exactly the same as the first) runs, it is empty.
I suspect this has something to do with database flushing between testcases that is clearing out the TaskMetadata objects, but the related lookup is cached, and so the next time some_utility_function() is called for the next testcase, it doesn't create any TaskMetadata objects. 1) Is that plausible? 2) How to work around it? 3) This is a Django bug, right?
Django bug ticket
In your tearDown method you need to call ContentType.objects.clear_cache(). This is because Django caches calls to ContentType.objects.get_for_model. Having a one-to-one to content type is a bit weird, so I don't think django needs to make any changes for this, especially as it should be a one line fix for you.
The problem here is the "finally" clause.
A finally clause is always executed before leaving the try statement, whether an exception has occurred or not.
http://docs.python.org/2/tutorial/errors.html
So, the finally clause containing the print statements will always be executed.
In each view of my application I need to have navigation menu prepared. So right now in every view I execute complicated query and store the menu in a dictionary which is passed to a template. In templates the variable in which I have the data is surrounded with "cache", so even though the queries are quite costly, it doesn't bother me.
But I don't want to repeat myself in every view. I guessed that the best place to prepare the menu is in my own context processor. And so I did write one, but I noticed that even when I don't use the data from the context processor, the queries used to prepare the menu are executed. Is there a way to "lazy load" such data from CP or do I have to use "low level" cache in CP? Or maybe there's a better solution to my problem?
Django has a SimpleLazyObject. In Django 1.3, this is used by the auth context processor (source code). This makes user available in the template context for every query, but the user is only accessed when the template contains {{ user }}.
You should be able to do something similar in your context processor.
from django.utils.functional import SimpleLazyObject
def my_context_processor(request):
def complicated_query():
do_stuff()
return result
return {
'result': SimpleLazyObject(complicated_query)
If you pass a callable object into the template context, Django will evaluate it when it is used in the template. This provides one simple way to do laziness — just pass in callables:
def my_context_processor(request):
def complicated_query():
do_stuff()
return result
return {'my_info': complicated_query}
The problem with this is it does not memoize the call — if you use it multiple times in a template, complicated_query gets called multiple times.
The fix is to use something like SimpleLazyObject as in the other answer, or to use something like functools.lru_cache:
from functools import lru_cache:
def my_context_processor(request):
#lru_cache()
def complicated_query():
result = do_stuff()
return result
return {'my_info': complicated_query}
You can now use my_info in your template, and it will be evaluated lazily, just once.
Or, if the function already exists, you would do it like this:
from somewhere import complicated_query
def my_context_processor(request):
return {'my_info': lru_cache()(complicated_query)}
I would prefer this method over SimpleLazyObject because the latter can produce some strange bugs sometimes.
(I was the one who originally implemented LazyObject and SimpleLazyObject, and discovered for myself that there is curse on any code artefact labelled simple.)
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
We just now started doing the A/B testing for our Django based project. Can I get some information on best practices or useful insights about this A/B testing.
Ideally each new testing page will be differentiated with a single parameter(just like Gmail). mysite.com/?ui=2 should give a different page. So for every view I need to write a decorator to load different templates based on the 'ui' parameter value. And I dont want to hard code any template names in decorators. So how would urls.py url pattern will be?
It's useful to take a step back and abstract what A/B testing is trying to do before diving into the code. What exactly will we need to conduct a test?
A Goal that has a Condition
At least two distinct Paths to meet the Goal's Condition
A system for sending viewers down one of the Paths
A system for recording the Results of the test
With this in mind let's think about implementation.
The Goal
When we think about a Goal on the web usually we mean that a user reaches a certain page or that they complete a specific action, for example successfully registering as a user or getting to the checkout page.
In Django we could model that in a couple of ways - perhaps naively inside a view, calling a function whenever a Goal has been reached:
def checkout(request):
a_b_goal_complete(request)
...
But that doesn't help because we'll have to add that code everywhere we need it - plus if we're using any pluggable apps we'd prefer not to edit their code to add our A/B test.
How can we introduce A/B Goals without directly editing view code? What about a Middleware?
class ABMiddleware:
def process_request(self, request):
if a_b_goal_conditions_met(request):
a_b_goal_complete(request)
That would allow us to track A/B Goals anywhere on the site.
How do we know that a Goal's conditions has been met? For ease of implementation I'll suggest that we know a Goal has had it's conditions met when a user reaches a specific URL path. As a bonus we can measure this without getting our hands dirty inside a view. To go back to our example of registering a user we could say that this goal has been met when the user reaches the URL path:
/registration/complete
So we define a_b_goal_conditions_met:
a_b_goal_conditions_met(request):
return request.path == "/registration/complete":
Paths
When thinking about Paths in Django it's natural to jump to the idea of using different templates. Whether there is another way remains to be explored. In A/B testing you make small differences between two pages and measure the results. Therefore it should be a best practice to define a single base Path template from which all Paths to the Goal should extend.
How should render these templates? A decorator is probably a good start - it's a best practice in Django to include a parameter template_name to your views a decorator could alter this parameter at runtime.
#a_b
def registration(request, extra_context=None, template_name="reg/reg.html"):
...
You could see this decorator either introspecting the wrapped function and modifying the template_name argument or looking up the correct templates from somewhere (like a Model). If we didn't want to add the decorator to every function we could implement this as part of our ABMiddleware:
class ABMiddleware:
...
def process_view(self, request, view_func, view_args, view_kwargs):
if should_do_a_b_test(...) and "template_name" in view_kwargs:
# Modify the template name to one of our Path templates
view_kwargs["template_name"] = get_a_b_path_for_view(view_func)
response = view_func(view_args, view_kwargs)
return response
We'd need also need to add some way to keep track of which views have A/B tests running etc.
A system for sending viewers down a Path
In theory this is easy but there are lot of different implementations so it's not clear which one is best. We know a good system should divide users evenly down the path - Some hash method must be used - Maybe you could use the modulus of memcache counter divided by the number of Paths - maybe there is a better way.
A system for recording the Results of the Test
We need to record how many users went down what Path - we'll also need access to this information when the user reaches the goal (we need to be able to say what Path they came down to met the Condition of the Goal) - we'll use some kind of Model(s) to record the data and either Django Sessions or Cookies to persist the Path information until the user meets the Goal condition.
Closing Thoughts
I've given a lot of pseudo code for implementing A/B testing in Django - the above is by no means a complete solution but a good start towards creating a reusable framework for A/B testing in Django.
For reference you may want to look at Paul Mar's Seven Minute A/Bs on GitHub - it's the ROR version of the above!
http://github.com/paulmars/seven_minute_abs/tree/master
Update
On further reflection and investigation of Google Website Optimizer it's apparent that there are gaping holes in the above logic. By using different templates to represent Paths you break all caching on the view (or if the view is cached it will always serve the same path!). Instead, of using Paths, I would instead steal GWO terminology and use the idea of Combinations - that is one specific part of a template changing - for instance, changing the <h1> tag of a site.
The solution would involve template tags which would render down to JavaScript. When the page is loaded in the browser the JavaScript makes a request to your server which fetches one of the possible Combinations.
This way you can test multiple combinations per page while preserving caching!
Update
There still is room for template switching - say for example you introduce an entirely new homepage and want to test it's performance against the old homepage - you'd still want to use the template switching technique. The thing to keep in mind is your going to have to figure out some way to switch between X number of cached versions of the page. To do this you'd need to override the standard cached middleware to see if their is a A/B test running on the requested URL. Then it could choose the correct cached version to show!!!
Update
Using the ideas described above I've implemented a pluggable app for basic A/B testing Django. You can get it off Github:
http://github.com/johnboxall/django-ab/tree/master
If you use the GET parameters like you suggsted (?ui=2), then you shouldn't have to touch urls.py at all. Your decorator can inspect request.GET['ui'] and find what it needs.
To avoid hardcoding template names, maybe you could wrap the return value from the view function? Instead of returning the output of render_to_response, you could return a tuple of (template_name, context) and let the decorator mangle the template name. How about something like this? WARNING: I haven't tested this code
def ab_test(view):
def wrapped_view(request, *args, **kwargs):
template_name, context = view(request, *args, **kwargs)
if 'ui' in request.GET:
template_name = '%s_%s' % (template_name, request.GET['ui'])
# ie, 'folder/template.html' becomes 'folder/template.html_2'
return render_to_response(template_name, context)
return wrapped_view
This is a really basic example, but I hope it gets the idea across. You could modify several other things about the response, such as adding information to the template context. You could use those context variables to integrate with your site analytics, like Google Analytics, for example.
As a bonus, you could refactor this decorator in the future if you decide to stop using GET parameters and move to something based on cookies, etc.
Update If you already have a lot of views written, and you don't want to modify them all, you could write your own version of render_to_response.
def render_to_response(template_list, dictionary, context_instance, mimetype):
return (template_list, dictionary, context_instance, mimetype)
def ab_test(view):
from django.shortcuts import render_to_response as old_render_to_response
def wrapped_view(request, *args, **kwargs):
template_name, context, context_instance, mimetype = view(request, *args, **kwargs)
if 'ui' in request.GET:
template_name = '%s_%s' % (template_name, request.GET['ui'])
# ie, 'folder/template.html' becomes 'folder/template.html_2'
return old_render_to_response(template_name, context, context_instance=context_instance, mimetype=mimetype)
return wrapped_view
#ab_test
def my_legacy_view(request, param):
return render_to_response('mytemplate.html', {'param': param})
Justin's response is right... I recommend you vote for that one, as he was first. His approach is particularly useful if you have multiple views that need this A/B adjustment.
Note, however, that you don't need a decorator, or alterations to urls.py, if you have just a handful of views. If you left your urls.py file as is...
(r'^foo/', my.view.here),
... you can use request.GET to determine the view variant requested:
def here(request):
variant = request.GET.get('ui', some_default)
If you want to avoid hardcoding template names for the individual A/B/C/etc views, just make them a convention in your template naming scheme (as Justin's approach also recommends):
def here(request):
variant = request.GET.get('ui', some_default)
template_name = 'heretemplates/page%s.html' % variant
try:
return render_to_response(template_name)
except TemplateDoesNotExist:
return render_to_response('oops.html')
A code based on the one by Justin Voss:
def ab_test(force = None):
def _ab_test(view):
def wrapped_view(request, *args, **kwargs):
request, template_name, cont = view(request, *args, **kwargs)
if 'ui' in request.GET:
request.session['ui'] = request.GET['ui']
if 'ui' in request.session:
cont['ui'] = request.session['ui']
else:
if force is None:
cont['ui'] = '0'
else:
return redirect_to(request, force)
return direct_to_template(request, template_name, extra_context = cont)
return wrapped_view
return _ab_test
example function using the code:
#ab_test()
def index1(request):
return (request,'website/index.html', locals())
#ab_test('?ui=33')
def index2(request):
return (request,'website/index.html', locals())
What happens here:
1. The passed UI parameter is stored in the session variable
2. The same template loads every time, but a context variable {{ui}} stores the UI id (you can use it to modify the template)
3. If user enters the page without ?ui=xx then in case of index2 he's redirected to '?ui=33', in case of index1 the UI variable is set to 0.
I use 3 to redirect from the main page to Google Website Optimizer which in turn redirects back to the main page with a proper ?ui parameter.
You can also A/B test using Google Optimize. To do so you'll have to add Google Analytics to your site and then when you create a Google Optimize experiment each user will get a cookie with a different experiment variant (according to the weight for each variant). You can then extract the variant from the cookie and display various versions of your application. You can use the following snippet to extract the variant:
ga_exp = self.request.COOKIES.get("_gaexp")
parts = ga_exp.split(".")
experiments_part = ".".join(parts[2:])
experiments = experiments_part.split("!")
for experiment_str in experiments:
experiment_parts = experiment_str.split(".")
experiment_id = experiment_parts[0]
variation_id = int(experiment_parts[2])
experiment_variations[experiment_id] = variation_id
However there is a django package that integrates well with Google Optimize: https://github.com/adinhodovic/django-google-optimize/.
And here is a blog post on how to use the package and how Google Optimize works: https://hodovi.cc/blog/django-b-testing-google-optimize/.