Django+Celery+postgresql_psycopg2 connection already closed error - django

Question

After forcefully killing (kill -9) the idle db connections. Celery log showing the below error
[2016-03-08 19:28:04,765: CRITICAL/MainProcess] Task users.tasks.refresh_trend_buzz_messages[85a2430a-9a39-4979-b99d-fcaae86b3ba1] INTERNAL ERROR: InterfaceError('connection already closed',)
Traceback (most recent call last):
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/celery/app/trace.py", line 253, in trace_task
I, R, state, retval = on_error(task_request, exc, uuid)
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/celery/app/trace.py", line 201, in on_error
R = I.handle_error_state(task, eager=eager)
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/celery/app/trace.py", line 85, in handle_error_state
}[self.state](task, store_errors=store_errors)
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/celery/app/trace.py", line 118, in handle_failure
req.id, exc, einfo.traceback, request=req,
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/celery/backends/base.py", line 120, in mark_as_failure
traceback=traceback, request=request)
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/celery/backends/base.py", line 251, in store_result
request=request, **kwargs)
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/djcelery/backends/database.py", line 29, in _store_result
traceback=traceback, children=self.current_task_children(request),
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/djcelery/managers.py", line 42, in _inner
return fun(*args, **kwargs)
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/djcelery/managers.py", line 181, in store_result
'meta': {'children': children}})
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/djcelery/managers.py", line 87, in update_or_create
return get_queryset(self).update_or_create(**kwargs)
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/djcelery/managers.py", line 70, in update_or_create
obj, created = self.get_or_create(**kwargs)
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/django/db/models/query.py", line 373, in get_or_create
return self.get(**lookup), False
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/django/db/models/query.py", line 301, in get
num = len(clone)
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/django/db/models/query.py", line 77, in __len__
self._fetch_all()
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/django/db/models/query.py", line 854, in _fetch_all
self._result_cache = list(self.iterator())
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 710, in results_iter
for rows in self.execute_sql(MULTI):
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 780, in execute_sql
cursor = self.connection.cursor()
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 159, in cursor
cursor = util.CursorWrapper(self._cursor(), self)
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 131, in _cursor
return self.create_cursor()
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 131, in _cursor
return self.create_cursor()
File "/home/.virtualenvs/backend/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 139, in create_cursor
cursor = self.connection.cursor()
InterfaceError: connection already closed
I'm using Django==1.6.1, celery==3.1.10, django-celery==3.1.10 and psycopg2==2.4.1
How can I fix the error ?

Answer 1

This is the expected behaviour. Each celery worker will open a connection to the database and try to reuse it to avoid the overhead of negotiation every time. By killing the database processes you close the connection, hence the error.
What are you trying to achieve by killing the database processes?

Answer 2

You need to adjust max_connections in your PostgreSQL configuration.
You will need at least 1 connection for each celery worker plus enough for any other client code.