Django Celery with Johnny Cache tasks failing with weird errors - django

Question

I am running Django 1.4.5, Celery 3.0.15, Django Celery 3.0.11, Johnny Cache 1.4.
The calls to ORM in celery tasks sometimes fail with weird errors like invalid literal for int() with base 10: 'a'" or <MaybeEncodingError: Error sending result: ''<ExceptionInfo: ObjectDoesNotExist()>''. Reason: ''PicklingError("Can\'t pickle <class \'scsite.models.DoesNotExist\'>: attribute lookup scsite.models.DoesNotExist failed",)''.>:
Here is a sample stack trace:
Task scsite.tasks.send_signup_email[aecf0561-65af-4d11-a3a0-63d88b7e1a70] raised exception: ValueError("invalid literal for int() with base 10: 'd'",)
Task scsite.tasks.send_signup_email[aecf0561-65af-4d11-a3a0-63d88b7e1a70] raised exception: ValueError("invalid literal for int() with base 10: 'd'",)
Stacktrace (most recent call last):
File "celery/execute/trace.py", line 192, in trace_task
R = I.handle_error_state(task, eager=eager)
File "scsite/tasks.py", line 537, in send_signup_email
email_html = email_template.render(Context(data))
File "django/template/base.py", line 142, in render
context.render_context.pop()
File "django/template/base.py", line 134, in _render
return self.nodelist.render(context)
File "django/template/base.py", line 823, in render
bit = self.render_node(node, context)
File "django/template/base.py", line 837, in render_node
return node.render(context)
File "django/template/loader_tags.py", line 155, in render
return self.render_template(self.template, context)
File "django/template/loader_tags.py", line 137, in render_template
output = template.render(context)
File "django/template/base.py", line 142, in render
context.render_context.pop()
File "django/template/base.py", line 134, in _render
return self.nodelist.render(context)
File "django/template/base.py", line 823, in render
bit = self.render_node(node, context)
File "django/template/base.py", line 837, in render_node
return node.render(context)
File "django/template/defaulttags.py", line 192, in render
nodelist.append(node.render(context))
File "django/template/defaulttags.py", line 474, in render
self.extra_context.iteritems()])
File "django/template/base.py", line 584, in resolve
obj = settings.TEMPLATE_STRING_IF_INVALID
File "django/template/base.py", line 721, in resolve
value = self._resolve_lookup(context)
File "django/template/base.py", line 781, in _resolve_lookup
raise
File "django/db/models/manager.py", line 119, in count
return self.get_query_set().count()
File "django/db/models/fields/related.py", line 461, in get_query_set
return super(RelatedManager, self).get_query_set().using(db).filter(**self.core_filters)
File "django/db/models/query.py", line 624, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "django/db/models/query.py", line 642, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "django/db/models/sql/query.py", line 1250, in add_q
can_reuse=used_aliases, force_having=force_having)
File "django/db/models/sql/query.py", line 1185, in add_filter
connector)
File "django/db/models/sql/where.py", line 69, in add
value = obj.prepare(lookup_type, value)
File "django/db/models/sql/where.py", line 320, in prepare
return self.field.get_prep_lookup(lookup_type, value)
File "django/db/models/fields/__init__.py", line 310, in get_prep_lookup
return self.get_prep_value(value)
File "django/db/models/fields/__init__.py", line 537, in get_prep_value
return int(value)
Task scsite.tasks.update_various_denorm_fields[edddb260-041d-4195-83d7-f2e731e5d1a5] raised exception: ValueError("invalid literal for int() with base 10: 'a'",)
Stacktrace (most recent call last):
File "celery/task/trace.py", line 242, in trace_task
R = I.handle_error_state(task, eager=eager)
File "celery/task/trace.py", line 415, in __protected_call__
return self.run(*args, **kwargs)
File "scsite/tasks.py", line 470, in update_various_denorm_fields
Person.objects.update_question_counts()
File "scsite/managers.py", line 715, in update_question_counts
person.update_question_count_denorm()
File "scsite/models.py", line 2712, in update_question_count_denorm
self.question_count_denorm = self.questions.count()
File "django/db/models/manager.py", line 119, in count
return self.get_query_set().count()
File "django/db/models/fields/related.py", line 567, in get_query_set
return super(ManyRelatedManager, self).get_query_set().using(db)._next_is_sticky().filter(**self.core_filters)
File "django/db/models/query.py", line 624, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "django/db/models/query.py", line 642, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "django/db/models/sql/query.py", line 1250, in add_q
can_reuse=used_aliases, force_having=force_having)
File "django/db/models/sql/query.py", line 1185, in add_filter
connector)
File "django/db/models/sql/where.py", line 69, in add
value = obj.prepare(lookup_type, value)
File "django/db/models/sql/where.py", line 320, in prepare
return self.field.get_prep_lookup(lookup_type, value)
File "django/db/models/fields/__init__.py", line 310, in get_prep_lookup
return self.get_prep_value(value)
File "django/db/models/fields/__init__.py", line 537, in get_prep_value
return int(value)
I believe the error is not in my code, it looks like cache becomes corrupt for some reason, and something like the following happens:
CustomModel with pk=123 is retrieved.
It is passed on to Celery to process.
For some reason, the pk of the Custom model becomes a character like 'a', 'd' instead of 123. Another call is made with MyModel.objects.get(pk='a') and it obviously fails.
Actually, revisiting the code for this, it seems I am not passing an object to Celery as I mentioned above, just running the following manager method from a simple celery task.
#task
def mytask():
Person.objects.update_question_count()
class PersonManager(models.Manager):
def update_question_counts(self):
persons = self.all() # 1
for person in persons:
person.update_question_count_denorm() #3
My guess is the following: when Line 1 runs, persons is obtained from cache, but while processing line 3, the cache obtained in Line 1 becomes corrupt.