I have added a new database in my django project. Now I have run into issues with my test cases. I am keep getting this error message for every single of my test cases:
Database queries to 'new-database' are not allowed in this test
I have searched for this issue and the common solution comes down to adding databases = '__all__' or databases = {'default', 'new_database'} to the TestCase class
But the problem is that now we have a lot of these test cases in my django application and a lot of corresponding TestCase based classes. So it does not fill right (specifically from the scale point of view) to add this databases = '__all__' declaration or whatever to every single class.
Do we have any other and more proper solution for this issue?
(After all why django needs to make transaction to new_database in all other test cases every single time that does not seem needed at all?)
Related
We need to use real data for one of our many external data sources during Django tests:
data is externally managed and read-only
data is accessed through manage.py inspectdb generated ORM classes
data is highly sensitive and we are not allowed to store fixtures of the actual data
tables are legacy design and will be phased out, hundreds of tables with complex relations, even getting a single record is complex
there is too much to do and I am unwilling to spend the time it would take to generate the fixtures, guarantee they're obfuscated, get approval of the obfuscation and justify keeping it around just to bridge us for a few months
We understand the downsides: This violates test purity and introduces a potential safety risk. We are willing to compromise on both to get us past the next few months when we will phase out this problem data source.
In this case, I need Django to understand that I don't want it to stand up a test database for this source, and just use the actual source so we can run some quick checks and walk away.
What is the simplest way to achieve this, with full understanding and acceptance of the risks and recommendations against?
For us, the solution was a custom test runner.
With help from Django's Advanced Testing Topics documentation, we overrode the default DiscoverRunner like this:
from django.test.runner import DiscoverRunner
def should_create_db(db_name):
# analyse db_name, a key from DATABASES, to determine whether a test
# database should be created
return db_name != 'messy_legacy_database'
class CustomTestRunner(DiscoverRunner):
# override method from superclass to selectively skip database setup
def setup_databases(self, **kwargs):
# 'aliases' is a set of unique keys from settings DATABASES dictionary
aliases = kwargs.get('aliases')
filtered = set([i for i in aliases if should_create_db(i)])
kwargs['aliases'] = filtered
# 'aliases' now contains only keys which trigger test database creation
return super().setup_databases(**kwargs)
# there was no need to override teardown_databases()
Next we update settings.py to use our override instead of the default runner:
TEST_RUNNER = 'path.to.CustomTestRunner'
Finally we tell our test class which databases it can use:
from django.test import TestCase
class OurTest(TestCase):
databases = [
'default',
'messy_legacy_database',
]
def test_messy_legacy_database(self):
# go nuts on your messy legacy database testing calls
pass
In this way our tests now skip test database creation for our messy legacy databases, and the logic we test pulls data from the actual data sources, allowing us to implement quick checks to ensure these code paths work.
Say I have this model definition:
class Foo(Model):
...
class Bar(Model):
some_m2m_field = ManyToManyField(Foo)
and this code:
bar = Bar.objects.create()
bar.some_m2m_field.set(an_id_array_with_some_invalid_pks)
When I run that normally, the last line will, as it should, throw an IntegrityError. However, if I run the same code from a django.test.TestCase, the last line will NOT throw an error. It instead will wait until the _post_teardown() phase of the test to throw the IntegrityError.
Here's a small project that demonstrates the issue: https://github.com/t-evans/m2mtest
How do I fix that? I suppose that's configurable, but I haven't been able to find it...
Follow-up question:
Ultimately, I need to handle the case when there are bad IDs being passed to the m2m_field.set() method (and I need unit tests that verify that bad IDs are being handled correctly, which is why the delayed IntegrityError in the unit test won't work).
I know I can find the bad IDs by looping over the array and hitting the DB one for each ID. Is there a more efficient way to find the bad IDs or (better) simply tell the set() method to ignore/drop the bad IDs?
TestCase wraps tests in additional atomic() blocks, compared to TransactionTestCase, so to test specific database transaction behaviour, you should use TransactionTestCase.
I believe an IntegrityError is thrown only when the transaction is committed, as that's the moment the db would find out about missing ids.
In general if you want to test for db exceptions raised during a test, you should use a TransactionTestCase and test your code using:
with self.assertRaises(IntegrityError):
# do something that gets committed to db
See the answer from #dirkgroten for how to fix the unit test issue.
As for the followup question on how to more-efficiently eliminate the bad IDs, one way is as follows:
good_ids = Foo.objects.filter(id__in=an_id_array_with_some_invalid_ids).values_list('id', flat=True)
I'm having trouble moving away from django_nose.FastFixtureTestCase to django.test.TestCase (or even the more conservative django.test.TransactionTestCase). I'm using Django 1.7.11 and I'm testing against Postgres 9.2.
I have a Testcase class that loads three fixtures files. The class contains two tests. If I run each test individually as a single run (manage test test_file:TestClass.test_name), they each work. If I run them together, (manage test test_file:TestClass), I get
IntegrityError: Problem installing fixture '<path>/data.json': Could not load <app>.<Model>(pk=1): duplicate key value violates unique constraint "<app_model_field>_49810fc21046d2e2_uniq"
To me it looks like the db isn't actually getting flushed or rolled back between tests since it only happens when I run the tests in a single run.
I've stepped through the Django code and it looks like they are getting flushed or rolled back -- depending on whether I'm trying TestCase or TransactionTestCase.
(I'm moving away from FastFixtureTestCase because of https://github.com/django-nose/django-nose/issues/220)
What else should I be looking at? This seems like it should be a simple matter and is right within what django.test.TestCase and Django.test.TransactionTestCase are designed for.
Edit:
The test class more or less looks like this:
class MyTest(django.test.TransactionTestCase): # or django.test.TestCase
fixtures = ['data1.json', 'data2.json', 'data3.json']
def test1(self):
return # I simplified it to just this for now.
def test2(self):
return # I simplified it to just this for now.
Update:
I've managed to reproduce this a couple of times with a single test, so I suspect something in the fixture loading code.
One of my basic assumptions was that my db was clean for every TestCase. Tracing into the django core code I found instances where an object (in one case django.contrib.auth.User) already existed.
I temporarily overrode _fixture_setup() to assert the db was clean prior to loading fixtures. The assertion failed.
I was able to narrow the problem down to code that was in a TestCase.setUpClass() instead of TestCase.setUp(), and so the object was leaking out of the test and conflicting with other TestCase fixtures.
What I don't understand completely is I thought that the db was dropped and recreated between TestCases -- but perhaps that is not correct.
Update: Recent version of Django includes setUpTestData() that should be used instead of setUpClass()
I have a series of unit tests (all subclasses of TransactionTestCase) spread out through multiple apps in a single Django project. When I run all of them in one go using ./manage.py test an error occurs in one of the tests. But when I run each app's tests individually, one at a time, using ./manage.py test my_project.app_name I get no errors.
The specific error I get is a FieldError in modelform_factory, but my question isn't so much about the specific solution to this error. I'm just curious what possible data/processes/whatever could bleed over between the supposedly-self-contained test cases in Django. Any thoughts?
(For the curious, if I make all my tests subclasses of TestCase (rather than TransactionTestCase) I get a bunch of different errors, but I've chalked those up to some separate issue relating to problems with rolling back the transactions within which Django encapsulates each test case. But who knows, maybe there's a connection?)
Found the answer:
Regardless of how many different test cases are run during one ./manage.py test call, and regardless of how much the database is rolled back/truncated (for TestCase and TransactionTestCase, respectively) between tests, all tests are run under the same python thread and with the same instantiations of all model base classes. Thus any variables in the thread or any modifications to class definitions persist across test instances.
In my case, it was both. A view got a user instance out of the currently running thread, but that user had been deleted when a previous test had been rolled back. Later on a view modified a list that was declared (as a blank list) in its parent class, thus altering the parent classes list to not be a blank list, and causing problems in a later test.
Is there a way to see errors when django unit tests loads fixtures?
It's annoying that something like missing file, duplicate keys or badly formatted fixtures files do not show up as error when running django unit test.
If duplicate primary keys generated an error when a fixture was loaded, many existing tests would be broken. Sometimes a fixture needs to contain that error, so it can be tested against.
You can write generic tests to check for the problems you've mentioned:
If your fixture fails to load, any query relying on that fixture will fail. Write a test that attempts to retrieve an object in that fixture (eg, YourObject.objects.get(pk=1) or YourObject.objects.get(name='My Test Entry').
To test if you have duplicate primary keys, compare YourObject.objects.all().aggregate(Count('id', distinct=True)) to YourObject.objects.all().aggregate(Count('id')). If those are not equal, you have duplicate primary keys.
To avoid badly formatted fixture files, generate them from validated data. Use manage.py dumpdata, and avoid manually creating or editing fixtures.
The first test mentioned will indicate if you have an invalid path. The last recommendation removes any issues with bad formatting.
The silent failure is a result of how loaddata works. It's looking for the filenames given in the fixtures = [] list in several locations, and may fail to find the files in any one of them. In order to report an error, loaddata must track if a file has been found yet, and that requires a patch. There's a trac ticket for the silent fail, but it has been triaged at 'Design Decision Needed' for 2 years now.
You have the option to make some noise about this on the Django mailing list, and use the unapproved patch in your development copy of Django.
You can try to define your own validation method in setUp(). Testing if your fixtures are loaded would be quite easy, however, validating your fixtures would require quite a bit work depending on what format is your fixture.