Django celery worker __init__() error - django

Question

The tech stack looks like this:
Django frontend and backend
Celery worker queue for asynchronously processing time consuming tasks
Within the past day or two I've noticed a lot of these kinds of stack traces both locally and in production:
[2012-07-05 20:31:01,583: CRITICAL/MainProcess] Task site_endpoint.tasks.async_inbound_message[a950736c-ff93-420c-9fbf-6deb2b88ff4d] INTERNAL ERROR: TypeError('__init__() takes at least 3 arguments (1 given)',)
Traceback (most recent call last):
File "/projects/site/venv/lib/python2.7/site-packages/celery/execute/trace.py", line 192, in trace_task
R = I.handle_error_state(task, eager=eager)
File "/projects/site/venv/lib/python2.7/site-packages/celery/execute/trace.py", line 91, in handle_error_state
}[self.state](task, store_errors=store_errors)
File "/projects/site/venv/lib/python2.7/site-packages/celery/execute/trace.py", line 114, in handle_failure
task.backend.mark_as_failure(req.id, exc, self.strtb)
File "/projects/site/venv/lib/python2.7/site-packages/celery/backends/base.py", line 96, in mark_as_failure
traceback=traceback)
File "/projects/site/venv/lib/python2.7/site-packages/celery/backends/base.py", line 229, in store_result
return self._store_result(task_id, result, status, traceback, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/backends/database.py", line 26, in _store_result
traceback=traceback)
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/managers.py", line 40, in _inner
return fun(*args, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/managers.py", line 164, in store_result
"traceback": traceback})
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/managers.py", line 82, in update_or_create
return self.get_query_set().update_or_create(**kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/managers.py", line 66, in update_or_create
obj, created = self.get_or_create(**kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/query.py", line 385, in get_or_create
obj.save(force_insert=True, using=self.db)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/base.py", line 460, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/base.py", line 543, in save_base
for f in meta.local_fields if not isinstance(f, AutoField)]
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/fields/subclassing.py", line 28, in inner
return func(*args, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/fields/subclassing.py", line 28, in inner
return func(*args, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 276, in get_db_prep_save
return self.get_db_prep_value(value, connection=connection, prepared=False)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/fields/subclassing.py", line 53, in inner
return func(*args, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/picklefield/fields.py", line 154, in get_db_prep_value
value = force_unicode(dbsafe_encode(value, self.compress, self.protocol))
File "/projects/site/venv/lib/python2.7/site-packages/picklefield/fields.py", line 57, in dbsafe_encode
value = b64encode(dumps(deepcopy(value), pickle_protocol))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 328, in _reconstruct
args = deepcopy(args, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
y.append(deepcopy(a, memo))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 329, in _reconstruct
y = callable(*args)
TypeError: __init__() takes at least 3 arguments (1 given)
Looking at the database calls when this happens, I can see the following queries being executed:
LOG: statement: SELECT "celery_taskmeta"."id", "celery_taskmeta"."task_id", "celery_taskmeta"."status", "celery_taskmeta"."result", "celery_taskmeta"."date_done", "celery_taskmeta"."traceback" FROM "celery_taskmeta" WHERE "celery_taskmeta"."task_id" = 'a950736c-ff93-420c-9fbf-6deb2b88ff4d'
LOG: statement: SAVEPOINT s140735259576672_x4
LOG: statement: ROLLBACK
LOG: statement: BEGIN
LOG: statement: SELECT "celery_taskmeta"."id", "celery_taskmeta"."task_id", "celery_taskmeta"."status", "celery_taskmeta"."result", "celery_taskmeta"."date_done", "celery_taskmeta"."traceback" FROM "celery_taskmeta" WHERE "celery_taskmeta"."task_id" = 'a950736c-ff93-420c-9fbf-6deb2b88ff4d'
LOG: statement: SAVEPOINT s140735259576672_x5
LOG: statement: ROLLBACK
LOG: statement: BEGIN
LOG: statement: SELECT "celery_taskmeta"."id", "celery_taskmeta"."task_id", "celery_taskmeta"."status", "celery_taskmeta"."result", "celery_taskmeta"."date_done", "celery_taskmeta"."traceback" FROM "celery_taskmeta" WHERE "celery_taskmeta"."task_id" = 'a950736c-ff93-420c-9fbf-6deb2b88ff4d'
LOG: statement: SAVEPOINT s140735259576672_x6
I am having a tough time understanding what the source of this invalid object initialization is. Anyone have ideas?

Answer 1

It looks like you are using new-relic remote performance monitoring/forensic analysis. I have seen this kind of problem before when old versions of the library are being used. I recommend checking to make sure you are using the latest version of their client library.

Answer 2

This happened to me and I solved it by reading Python multiprocessing pool hangs at join? and http://bugs.python.org/issue9400.
The problem:
Celery somehow pickles the exceptions it gets and sends them somewhere (to the database/broker, I guess). So, if, at any time, a Celery task raises an exception that isn't pickleable, this bug will happen.
The solution:
You must ensure you're handling all the bizarre exceptions your Celery tasks code might be raising. The Traceback can give you hints, but it may not be very exact about everything.
If you really don't know where are your exceptions coming from, try putting your tasks code inside try...except: pass blocks.