I'm trying to utilize django's row-level-locking by using the select_for_update utility. As per the documentation, this can only be used when inside of a transaction.atomic block. The side-effect of using a transaction.atomic block is that if my code throws an exception, all the database changes get rolled-back. My use case is such that I'd actually like to keep the database changes, and allow the exception to propagate. This leaves me with code looking like this:
with transaction.atomic():
user = User.objects.select_for_update.get(id=1234)
try:
user.do_something()
except Exception as e:
exception = e
else:
exception = None
if exception is not None:
raise exception
This feels like a total anti-pattern and I'm sure I must be missing something. I'm aware I could probably roll-my-own solution by manually using transaction.set_autocommit to manage the transaction, but I'd have thought that there would be a simpler way to get this functionality. Is there a built in way to achieve what I want?
I ended up going with something that looks like this:
from django.db import transaction
class ErrorTolerantTransaction(transaction.Atomic):
def __exit__(self, exc_type, exc_value, traceback):
return super().__exit__(None, None, None)
def error_tolerant_transaction(using=None, savepoint=True):
"""
Wraps a code block in an 'error tolerant' transaction block to allow the use of
select_for_update but without the effect of automatic rollback on exception.
Can be invoked as either a decorator or context manager.
"""
if callable(using):
return ErrorTolerantTransaction('default', savepoint)(using)
return ErrorTolerantTransaction(using, savepoint)
I can now put an error_tolerant_transaction in place of transaction.atomic and exceptions can be raised without a forced rollback. Of course database-related exceptions (i.e. IntegrityError) will still cause a rollback, but that's expected behavior given that we're using a transaction. As a bonus, this solution is compatible with transaction.atomic, meaning it can be nested inside an atomic block and vice-versa.
Related
This is mostly curiosity, but is the DB penalty for wrapping an entire view with #transaction.atomic a negligible one?
I'm thinking of views where the GET of a form or its re-display after a validation fail involves processing querysets. (ModelChoiceFields, for example, or fetching an object that the template displays.)
It seems to me to be far more natural to use with transaction.atomic() around the block of code which actually alters a bunch of related DB objects only after the user's inputs have validated.
Am I missing something?
From the source code:
def atomic(using=None, savepoint=True, durable=False):
# Bare decorator: #atomic -- although the first argument is called
# `using`, it's actually the function being decorated.
if callable(using):
return Atomic(DEFAULT_DB_ALIAS, savepoint, durable)(using)
# Decorator: #atomic(...) or context manager: with atomic(...): ...
else:
return Atomic(using, savepoint, durable)
It's the same. In both cases the function is returning an Atomic object which handles whether the transaction should commit or not.
I'm attempting to update my code that uses commit_manually to atomic to go to Django 1.7 then 1.8 but I'm having issues not being able to rollback while in an atomic block. The issue stems from a flag we added to out import routines, dry_run. In the prior Django versions #commit_manually allowed us to do the following:
if self.dry_run:
transaction.rollback()
else:
transaction.commit()
If I attempt to preform a roll back in an atomic block it throws an error:
TransactionManagementError : "This is forbidden when an 'atomic' block is active."
To get this to work I tried to make use of set_autocommit
Example:
def do_some_import(self)
transaction.set_autocommit(False)
#import routine
if self.dry_run:
transaction.rollback()
transaction.set_autocommit(True)
else:
transaction.commit()
transaction.set_autocommit(True)
But this feels wrong any suggestions or insights?
Have you tried set_rollback()? I think that will solve your problem.
#transaction.atomic
def do_some_import(self):
#import routine
if self.dry_run:
transaction.set_rollback(True)
I have written a test which checks whether IntegrityError was raised in case of duplicate records in the database. To create that scenario I am issue a REST API twice. The code looks like this:
class TestPost(APITestCase):
#classmethod
def setUpClass(cls):
super().setUpClass()
common.add_users()
def tearDown(self):
super().tearDown()
self.client.logout()
def test_duplicate_record(self):
# first time
response = self.client.post('/api/v1/trees/', dict(alias="some name", path="some path"))
# same request second time
response = self.client.post('/api/v1/trees/', dict(alias="some name", path="some path"))
self.assertEqual(response.status_code, status.HTTP_400_BAD_RREQUEST)
But I get an error stack like this
"An error occurred in the current transaction. You can't "
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
How can I avoid this error this is certainly undesirable.
I've come across this issue today, and it took me a while to see the bigger picture, and fix it properly. It's true that removing self.client.logout() fixes the issue, and is probably not needed there, but the problem lies in your view.
Tests which are subclasses of TestCase wrap your test cases and tests in db transactions (atomic blocks), and your view somehow breaks that transaction. For me it was swallowing an IntegrityError exception. At that point, the transaction was already broken, but Django didn't know about it, so it couldn't correctly perform a rollback. Any query that would then be executed would cause a TransactionManagementError.
The fix for me was to correctly wrap the view code in yet another atomic block:
try:
with transaction.atomic(): # savepoint which will be rolled back
counter.save() # the operation which is expected to throw an IntegrityError
except IntegrityError:
pass # swallow the exception without breaking the transaction
This may not be a problem for you in production, if you're not using ATOMIC_REQUESTS, but I still think it's the correct solution.
Try removing self.client.logout() from the tearDown method. Django rolls back the transaction at the end of each test. You shouldn't have to log out manually.
Is it necessary to catch errors stemming from HTTP inputs? Is it ever a good idea to let it fail naturally (allow the exception to bubble up)?
I have a Django view for a server side interface for an AJAX call that looks something like this:
def some_view(request):
try:
some_int = int(request.POST.get('some_int')) # May raise ValueError or TypeError
except (ValueError, TypeError):
return HttpResponseBadRequest('some_int must be an int')
# ... Code that assumes some_int is an int
return HttpResponse('The normal response')
Is it ever acceptable in production code to have something like this?
def some_view(request):
some_int = int(request.POST.get('some_int')) # Ignore ValueError or TypeError raised
# ... Code that assumes some_int is an int
return HttpResponse('normal_response')
As I accept more parameters, I find that it is frustrating to maintain so many try/except blocks which are mostly the same and I end up with a ton of boiler plate code.
Of course I tried to refactor this into a separate function but since Django requires an HttpResponse to be returned, not raised as an exception, I can't plug it into a view without a try/except block. Also, conversions to int aren't the only thing I check... there are a lot of business logic sanity checks performed depending on the input as well. For example, I would validate that the JSON passed is of a specific format (i.e. array of objects of int array, etc.).
My views end up being 70+ lines of code just for sanity checks and a few lines of code that actually generate the response. Somehow I feel like there should be a more elegant way but I haven't found one so I'm considering forgoing all checks and just letting Django take care of it. Is this a bad idea?
I'm aware of the following potential problems if I don't catch the exceptions:
The same HTTP 500 is returned for all errors
If logging is enabled in production, it would probably log an error every time an invalid input occurs
Are there other problems I should be aware of? It just feels wrong not to catch exceptions from user inputs even though there's not much I can do about it in terms of recovery logic.
I think the best way to handle this is by writing your own middleware that catches the exceptions and turns them into your desired response.
That might look something like this:
# views.py
def some_view(request):
some_int = int(request.POST.get('some_int'))
# ... Code that assumes some_int is an int
return HttpResponse('normal_response')
# middleware.py
class MyValidationMiddleware(object):
def process_exception(self, request, e):
if isinstance(e, ValueError):
return HttpResponseBadRequest('Input did not validate')
else:
return None # let it bubble up
(Since middleware is site-wide, you might want to explicitly define your own input-validation exceptions to distinguish them from other sorts of exceptions that might occur.)
Alternatively, you could do the same sort of thing with per-view decorators.
I would like to know if I have something like this:
def functionA():
with transaction.atomic():
#save something
functionB()
def functionB():
with transaction.atomic():
#save another thing
Someone knows what will happen? If functionB fails, functionA will rollback too?
Thank you!
Yes, it will. Regardless of nesting, if an atomic block is exited by an exception it will roll back:
If the block of code is successfully completed, the changes are committed to the database. If there is an exception, the changes are rolled back.
Note also that an exception in an outer block will cause the inner block to roll back, and that an exception in an inner block can be caught to prevent the outer block from rolling back. The documentation addresses these issues. (Or see here for a more comprehensive follow-up question on nested transactions).
with transaction.atomic(): # Outer atomic, start a new
transaction.on_commit(foo)
try:
with transaction.atomic(): # Inner atomic block, create a savepoint
transaction.on_commit(bar)
raise SomeError() # Raising an exception - abort the savepoint
except SomeError:
pass
# foo() will be called, but not bar()