Intermittent Deadlock with Django LiveServerTestCase, Selenium, and Postgres - django

Question

In testing a Django/Postgres app using LiveServerTestCase and Selenium I'm seeing intermittent deadlock problems. LiveServerTestCase inherits from TransactionTestCase, so all DB tables are truncated after each test runs. But sometimes that truncation causes deadlock because one of the tables is locked by an unresolved Postgres transaction. I can see that because this query returns a row:
select * from pg_stat_activity
where datname='test' and current_query='<IDLE> in transaction';
So some activity in my application must be creating an unresolved transaction. I've combed the tests to make sure they wait for any updates to complete before exiting and am convinced that's not it.
Looking at the Postgres logs I see these two lines frequently, without a corresponding COMMIT or ROLLBACK:
SHOW default_transaction_isolation
BEGIN
I suspect these are causing the deadlock. Any idea what might be issuing this SQL or how to disable it? This is Django 1.5.

Answer 1

The root cause of this deadlock is Django 1.5's autocommit behavior. By default Django 1.5 runs with an open transaction, which is only closed by a COMMIT if you do an UPDATE or INSERT. "Read" operations (SELECT) cause the unmatched BEGIN statements I mentioned above. It appears that deadlock happens if a SELECT occurs just before the end-of-test TRUNCATE. To avoid deadlock the test must exit only after all requests have completed, even if the requests cause no DB writes. That can be tricky if Ajax calls are updating parts of the page asynchronously after an update.
A better solution is to use Django 1.6, where atomic() is the only (non-deprecated) transaction-creating primitive. It doesn't open transactions for read operations, and doesn't leave dangling BEGIN statements. Tests can follow the common-sense approach of not exiting while "write" requests are pending.

Answer 2

For any future travelers:
We experienced the same issue on Django 3.2 + Postgres 12. When the build server was at a high load with multiple parallel builds, the live server kept receiving AJAX calls from the Selenium container, which interrupted the post-test TRUNCATE, causing a deadlock.
Our solution was to just add a 1-second sleep at the end of each test case:
class CustomLiveTestCase(StaticLiveServerTestCase):
def tearDown(self):
time.sleep(1)
This gave the live server enough time to process any lingering AJAX calls after the test finished, removing the deadlocks.