Why are integration tests failing on updates to model objects when the function is run on Django q-cluster? - django

Question

I am running some django integration tests of code that passes some functions to Django-Q for processing. This code essentially updates some fields on a model object.
The app uses signals.py to listen for a post_save change of the status field of the Foo object. Assuming it's not a newly created object and it has the correct status, the Foo.prepare_foo() function is called. This is handled by services.py which hands it off to q-cluster. Assuming this executes without error hooks.py changes the status field of Foo to published, or keeps it at prepare if it fails. If it passes then it also sets prepared to True. (I know this sounds convoluted and with overlapping variables - part of the desire to get tests running is to be able to refactor).
The code runs correctly in all environments, but when I try to run tests to assert that the fields have been updated, they fail.
(If I tweak the code to bypass the cluster and have the functions run in-memory then the tests pass.)
Given this, I'm assuming the issue lies in how I've written the tests, but I cannot diagnose the issue.
tests_prepare_foo.py
import time
from django.test import TestCase, override_settings, TransactionTestCase
from app.models import Foo
class CreateCourseTest(TransactionTestCase):
reset_sequences = True
#classmethod
def setUp(cls):
cls.foo = Foo(status='draft',
prepared=False,
)
cls.foo.save()
def test_foo_prepared(self):
self.foo.status = 'prepare'
self.foo.save()
time.sleep(15) # to allow the cluster to finish processing the request
self.assertEquals(self.foo.prepared, True)
models.py
import uuid
from django.db import models
from model_utils.fields import StatusField
from model_utils import Choices
class Foo(models.Model):
ref = models.UUIDField(default=uuid.uuid4,
editable=False,
)
STATUS = Choices('draft', 'prepare', 'published', 'archived')
status = StatusField(null=True,
blank=True,
)
prepared = models.BooleanField(default=False)
def prepare_foo(self):
""""
...
do stuff
"""
signals.py
from django.dispatch import receiver
from django.db.models.signals import post_save
from django_q.tasks import async_task
from app.models import Foo
#receiver(post_save, sender=Foo)
def make_foo(sender, instance, **kwargs):
if not kwargs.get('created', False) and instance.status == 'prepare' and not instance.prepared:
async_task('app.services.prepare_foo',
instance,
hook='app.hooks.check_foo_prepared',
)
services.py
def prepare_foo(foo):
foo.prepare_foo()
hooks.py
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def check_foo_prepared(task):
foo = task.args[0]
if task.success:
logger.info("Foo Prepared: Successful")
foo.status = 'published'
foo.prepared = True
foo.save()
logger.info("Foo Status: %s", foo.status)
logger.info("Foo Prepared: %s", foo.prepared)
else:
logger.info("Foo Prepared: Unsuccessful")
foo.status = 'draft'
foo.save()
Finally - logs when the test is run with the cluster on:
q-cluster server
INFO:app.hooks:Foo Prepared: Successful
INFO:app.hooks: Foo Status: published
INFO:app.hooks: Foo Prepared: True
django server
h:m:s [Q] INFO Enqueued 1
F
======
FAIL
self.assertEquals(self.foo.prepared, True)
AssertionError: False != True
I think I'm either missing something obvious in my tests, or something really subtle, but I can't work out which. I've tried setting the cluster to run synchronously (sync=True), and in my test reloading Foo just before the assertion:
self.foo.save()
time.sleep(15)
test_foo = Foo.objects.get(pk=1)
self.assertEquals(test_foo.prepared, True)
But this also fails with
self.assertEquals(test_foo.prepared, True)
AssertionError: False != True
Which leads me to believe that the cluster is not updating the object under test (unlikely), or the assertion is being checked before the cluster has updated the object (more likely).
This is the first time I've written tests that require hand-offs to a cluster, so any pointers, suggestions gratefully received!