django.db.utils.InterfaceError: connection already closed - django

Question

Stack:
Ubuntu (20.04 LTS)
Nginx
Postgresql (v13.3)
An AWS load balancer sends traffic to the Ubuntu instance(k8s cluster), which is handled by Nginx, which forwards on to Django (4.0.3) running in gunicorn (19.9.0). Django connects to the database using psycopg2 (2.8.6).
The issue I have is that the database connection seems to shut down randomly.
Django reports errors like this:
`InterfaceError: connection already closed
File "django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
File "django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "contextlib.py", line 79, in inner
return func(*args, **kwds)
File "django/views/generic/base.py", line 84, in view
return self.dispatch(request, *args, **kwargs)
File "django/utils/decorators.py", line 46, in _wrapper
return bound_method(*args, **kwargs)
File "django/views/decorators/cache.py", line 62, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "django/views/generic/base.py", line 119, in dispatch
return handler(request, *args, **kwargs)
File "django/views/generic/detail.py", line 108, in get
self.object = self.get_object()
File "django/views/generic/detail.py", line 53, in get_object
obj = queryset.get()
File "django/db/models/query.py", line 492, in get
num = len(clone)
File "django/db/models/query.py", line 302, in __len__
self._fetch_all()
File "django/db/models/query.py", line 1507, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "django/db/models/query.py", line 57, in __iter__
results = compiler.execute_sql(
File "django/db/models/sql/compiler.py", line 1359, in execute_sql
cursor = self.connection.cursor()
File "django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "django/db/backends/base/base.py", line 284, in cursor
return self._cursor()
File "django/db/backends/base/base.py", line 262, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "django/db/backends/base/base.py", line 262, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "django/db/backends/postgresql/base.py", line 256, in create_cursor
cursor = self.connection.cursor()`
Postgresql does not report any major errors: I can only assume that the connection was closed correctly and Django was not expecting it.
This happens quite rarely, but enough to be a concern: once every 3 days.
Any suggestions of how to investigate this further would be gratefully accepted. Thanks in advance!

Answer 1

I'm using django-service-objects and Django-Q for a distributed data mining code. I ran into a error "django.db.utils.InterfaceError: connection already closed" when I was running Django-Q cluster async_task(..., sync=True) to debug and attempted to query the Django PostgreSQL 13 DB using Model.object.get(pk=...).
I resolved the issue by setting db_transaction = False within the Service class.
class dataMining(Service):
db_transaction = False
https://django-service-objects.readthedocs.io/en/stable/pages/usage.html#database-transactions
The effect of db_transaction=True (enabled by default) is to "...process method on services ... inside a transaction". Meaning changes to the Django PostgreSQL DB are not actually saved until the def process(self) method of the Service class completes successfully. https://django-service-objects.readthedocs.io/en/stable/pages/usage.html#service
I suspect the self.object = self.get_object() call in your code sample is experiencing the same problem of being 'locked out' during a process method within a DB transaction.