I'm having trouble moving away from django_nose.FastFixtureTestCase to django.test.TestCase (or even the more conservative django.test.TransactionTestCase). I'm using Django 1.7.11 and I'm testing against Postgres 9.2.
I have a Testcase class that loads three fixtures files. The class contains two tests. If I run each test individually as a single run (manage test test_file:TestClass.test_name), they each work. If I run them together, (manage test test_file:TestClass), I get
IntegrityError: Problem installing fixture '<path>/data.json': Could not load <app>.<Model>(pk=1): duplicate key value violates unique constraint "<app_model_field>_49810fc21046d2e2_uniq"
To me it looks like the db isn't actually getting flushed or rolled back between tests since it only happens when I run the tests in a single run.
I've stepped through the Django code and it looks like they are getting flushed or rolled back -- depending on whether I'm trying TestCase or TransactionTestCase.
(I'm moving away from FastFixtureTestCase because of https://github.com/django-nose/django-nose/issues/220)
What else should I be looking at? This seems like it should be a simple matter and is right within what django.test.TestCase and Django.test.TransactionTestCase are designed for.
Edit:
The test class more or less looks like this:
class MyTest(django.test.TransactionTestCase): # or django.test.TestCase
fixtures = ['data1.json', 'data2.json', 'data3.json']
def test1(self):
return # I simplified it to just this for now.
def test2(self):
return # I simplified it to just this for now.
Update:
I've managed to reproduce this a couple of times with a single test, so I suspect something in the fixture loading code.
One of my basic assumptions was that my db was clean for every TestCase. Tracing into the django core code I found instances where an object (in one case django.contrib.auth.User) already existed.
I temporarily overrode _fixture_setup() to assert the db was clean prior to loading fixtures. The assertion failed.
I was able to narrow the problem down to code that was in a TestCase.setUpClass() instead of TestCase.setUp(), and so the object was leaking out of the test and conflicting with other TestCase fixtures.
What I don't understand completely is I thought that the db was dropped and recreated between TestCases -- but perhaps that is not correct.
Update: Recent version of Django includes setUpTestData() that should be used instead of setUpClass()
Related
Recently I've changed db engine from SQLite to PostgreSQL. I have successfully migrated whole db design to PostgreSQL (just simple makemigaretions, migrate). Once I ran tests some have failed for some unknown reason (errors suggest that some objects were not created). Failure doesn't apply to all tests, just selected few. Everything has been working before.
I'd start investigating what's going on on test-by-test basis, but some bizarre behavior has appeared. Let's say my test is in class MyTestClass and test is called test_do_something and in MyTestClass there are other tests as well.
When I'm running python manage.py test MyTestClass I'm getting info that test_do_something has failed.
When I'm running python manage.py test MyTestClass.test_do_something everything passes.
On SQLite both ways pass.
I'm assuming that setUpTestData() and setUp() methods work same way in SQLite and PostgreSQL. Or they don't?
Any clue why such discrepancy might be happening?
EDIT
I think I've noticed what might be wrong, but I don't understand why. The problem is because my function I've used to call to create object which is later used only once. Which differs from SQLite execution.
What I mean in my test I have something like this:
def create_object(self):
self.client.post(reverse('myurl'), kwargs={'myargs':arg})
def test_mytest1(self):
# Do something
self.create_object()
# Do something
def test_mytest2(self):
# Do something
self.create_object()
# Do something
def test_mytest3(self):
# Do something
self.create_object()
# Do something
Only for one test create_object() will be executed.
I believe I've fond cause of those failures. As a matter of fact it hasn't been an issue with one-time execution of support function as I expected. The problem has been with hardcoded ids I've used for various reasons. It appears that object I've been hoping to see didn't exist.
Let me explain a bit more what I've experienced. E.g. I had test where I've been referring to particular object passing this object id in URL kwargs. Before this operation I've created object and passed id=1 as kwargs, because I assumed that if this is only place within this test and setUp() it will be 1. It appears that with PostgreSQL it's not so clear. It seems that ids are incremented despite DB flush. Which is completely different behavior then SQLite has been providing.
I'd very much appreciate if someone could provide some more detailed answer why is this happening. Is ID counter not zeroed in PostgreSQL on flush? It would look so.
I am attempting to write test cases in Django using Selenium. I want to use existent fixtures so my test database (SQLite3) has test data for every test.
I have some model test cases (just using the TestCase class) as follows;
from django.test import TestCase
from django.test import LiveServerTestCases
from missions.models import Mission, MissionDataRecord
class MissionModelTests(TestCase):
fixtures = ['myproject_common/fixtures/auth_initial_load.json', 'worklog/fixtures/worklogs_initial_load',
'missions_initial_load.json']
def test_object_name_is_mission_title(self):
mission = Mission.objects.get(id=1)
self.assertEqual(mission.title, str(mission))
def test_object_name_is_mission_title_again(self):
mission = Mission.objects.get(id=1)
self.assertEqual(mission.title, str(mission))
This works as expected when run like this (I get two test passes). However, for Selenium testing I need to use LiveServerTestCase instead of TestCase.
The simple example above is a model test, but for illustrative purposes of the issue I'm facing with Selenium, if I simply replace "TestCase" with "LiveServerTestCase" the first test passes, but the second test fails, with the error
django.db.utils.IntegrityError: Problem installing fixture
'[...]/fixtures/auth_initial_load.json': Could not load
auth.User(pk=1): UNIQUE constraint failed: auth_user.username
This error occurs in the _fixture_setup of /django/test/testcases.py. This seems to suggests that my fixtures (specifically the auth_initial_load fixture) is attempting to load again ON TOP of existing data. However, from django docs reading this should not be taking place, because each test should run in its own transaction (which I believe means the fixtures are loaded for each transaction).
What is going on here, and more importantly, how can I use LiveServerTestCase with my existing fixtures (similar to how I am using TestCase currently)? In reality I need to use StaticLiveServerTestCase, but Im guessing the code will be the same.
It turns out the way I was loading fixtures correctly after all. The issue was with my fixtures themselves, using hard coded primary (and foreign) keys. In my case two users were being created before the fixtures were loaded, so when fixtures tried to load with the same primary key, a UNIQUE constraint violation occurred. The solution was to re-generate my fixtures using the --natural-primary and --natural-foreign flags as suggested in this SO answer.
Say I have this model definition:
class Foo(Model):
...
class Bar(Model):
some_m2m_field = ManyToManyField(Foo)
and this code:
bar = Bar.objects.create()
bar.some_m2m_field.set(an_id_array_with_some_invalid_pks)
When I run that normally, the last line will, as it should, throw an IntegrityError. However, if I run the same code from a django.test.TestCase, the last line will NOT throw an error. It instead will wait until the _post_teardown() phase of the test to throw the IntegrityError.
Here's a small project that demonstrates the issue: https://github.com/t-evans/m2mtest
How do I fix that? I suppose that's configurable, but I haven't been able to find it...
Follow-up question:
Ultimately, I need to handle the case when there are bad IDs being passed to the m2m_field.set() method (and I need unit tests that verify that bad IDs are being handled correctly, which is why the delayed IntegrityError in the unit test won't work).
I know I can find the bad IDs by looping over the array and hitting the DB one for each ID. Is there a more efficient way to find the bad IDs or (better) simply tell the set() method to ignore/drop the bad IDs?
TestCase wraps tests in additional atomic() blocks, compared to TransactionTestCase, so to test specific database transaction behaviour, you should use TransactionTestCase.
I believe an IntegrityError is thrown only when the transaction is committed, as that's the moment the db would find out about missing ids.
In general if you want to test for db exceptions raised during a test, you should use a TransactionTestCase and test your code using:
with self.assertRaises(IntegrityError):
# do something that gets committed to db
See the answer from #dirkgroten for how to fix the unit test issue.
As for the followup question on how to more-efficiently eliminate the bad IDs, one way is as follows:
good_ids = Foo.objects.filter(id__in=an_id_array_with_some_invalid_ids).values_list('id', flat=True)
I have a series of unit tests (all subclasses of TransactionTestCase) spread out through multiple apps in a single Django project. When I run all of them in one go using ./manage.py test an error occurs in one of the tests. But when I run each app's tests individually, one at a time, using ./manage.py test my_project.app_name I get no errors.
The specific error I get is a FieldError in modelform_factory, but my question isn't so much about the specific solution to this error. I'm just curious what possible data/processes/whatever could bleed over between the supposedly-self-contained test cases in Django. Any thoughts?
(For the curious, if I make all my tests subclasses of TestCase (rather than TransactionTestCase) I get a bunch of different errors, but I've chalked those up to some separate issue relating to problems with rolling back the transactions within which Django encapsulates each test case. But who knows, maybe there's a connection?)
Found the answer:
Regardless of how many different test cases are run during one ./manage.py test call, and regardless of how much the database is rolled back/truncated (for TestCase and TransactionTestCase, respectively) between tests, all tests are run under the same python thread and with the same instantiations of all model base classes. Thus any variables in the thread or any modifications to class definitions persist across test instances.
In my case, it was both. A view got a user instance out of the currently running thread, but that user had been deleted when a previous test had been rolled back. Later on a view modified a list that was declared (as a blank list) in its parent class, thus altering the parent classes list to not be a blank list, and causing problems in a later test.
Is there a way to see errors when django unit tests loads fixtures?
It's annoying that something like missing file, duplicate keys or badly formatted fixtures files do not show up as error when running django unit test.
If duplicate primary keys generated an error when a fixture was loaded, many existing tests would be broken. Sometimes a fixture needs to contain that error, so it can be tested against.
You can write generic tests to check for the problems you've mentioned:
If your fixture fails to load, any query relying on that fixture will fail. Write a test that attempts to retrieve an object in that fixture (eg, YourObject.objects.get(pk=1) or YourObject.objects.get(name='My Test Entry').
To test if you have duplicate primary keys, compare YourObject.objects.all().aggregate(Count('id', distinct=True)) to YourObject.objects.all().aggregate(Count('id')). If those are not equal, you have duplicate primary keys.
To avoid badly formatted fixture files, generate them from validated data. Use manage.py dumpdata, and avoid manually creating or editing fixtures.
The first test mentioned will indicate if you have an invalid path. The last recommendation removes any issues with bad formatting.
The silent failure is a result of how loaddata works. It's looking for the filenames given in the fixtures = [] list in several locations, and may fail to find the files in any one of them. In order to report an error, loaddata must track if a file has been found yet, and that requires a patch. There's a trac ticket for the silent fail, but it has been triaged at 'Design Decision Needed' for 2 years now.
You have the option to make some noise about this on the Django mailing list, and use the unapproved patch in your development copy of Django.
You can try to define your own validation method in setUp(). Testing if your fixtures are loaded would be quite easy, however, validating your fixtures would require quite a bit work depending on what format is your fixture.