Django 1.7 introduced database migrations.
When running the unit tests in Django 1.7, it forces a migrate, that takes a long time. So I would like to skip the django migrations, and create the database in the final state.
I know that ignoring the migrations can be a bad practice, as that part of the code would not be tested. But that's not the case: I'm running the full migrations in the CI test server (jenkins). I only want to skip the migrations in my local tests, where the speed matters.
Some context:
Until Django 1.6, when using South, I used the SOUTH_TESTS_MIGRATE setting:
By default, South’s syncdb command will also apply migrations if it’s run in non-interactive mode, which includes when you’re running tests - it will run every migration every time you run your tests.
If you want the test runner to use syncdb instead of migrate - for example, if your migrations are taking way too long to apply - simply set SOUTH_TESTS_MIGRATE = False in settings.py.
However, syncdb does not exist anymore, now it's migrate.
And from Django 1.8 I'll use the --keepdb parameter:
The --keepdb option can be used to preserve the test database between test runs. This has the advantage of skipping both the create and destroy actions which greatly decreases the time to run tests, especially those in a large test suite. If the test database does not exist, it will be created on the first run and then preserved for each subsequent run. Any unapplied migrations will also be applied to the test database before running the test suite.
So this question is limited to Django 1.7.
Look at this workaround, posted by Bernie Sumption to the Django developers mailing list:
If makemigrations has not yet been run, the "migrate" command treats
an app as unmigrated, and creates tables directly from the models just
like syncdb did in 1.6. I defined a new settings module just for unit
tests called "settings_test.py", which imports * from the main
settings module and adds this line:
MIGRATION_MODULES = {"myapp": "myapp.migrations_not_used_in_tests"}
Then I run tests like this:
DJANGO_SETTINGS_MODULE="myapp.settings_test" python manage.py test
This fools migrate into thinking that the app is unmigrated, and so
every time a test database is created it reflects the current
structure of models.py.
In Django 1.9, this situation is improved somewhat, and you can set the value to None:
MIGRATION_MODULES = {"myapp": None}
Here is the end of my settings file :
class DisableMigrations(object):
def __contains__(self, item):
return True
def __getitem__(self, item):
return None
TESTS_IN_PROGRESS = False
if 'test' in sys.argv[1:] or 'jenkins' in sys.argv[1:]:
logging.disable(logging.CRITICAL)
PASSWORD_HASHERS = (
'django.contrib.auth.hashers.MD5PasswordHasher',
)
DEBUG = False
TEMPLATE_DEBUG = False
TESTS_IN_PROGRESS = True
MIGRATION_MODULES = DisableMigrations()
based on this snippet
I disabled migrations only when tests are running
django-test-without-migrations adds a --nomigrations flag to manage.py test. Works like a charm.
From Django version 3.1 and up the correct way to do this is to use the MIGRATE setting in the database settings dict. See also the documentation.
#settings.py
DATABASES = {
'TEST': {
'NAME': 'Foo',
'MIGRATE': False
}
}
Update: Never mind, this change was reverted before 1.10 final was released. Hopefully it will return in a future version.
Note that as of Django 1.10 this can be controlled by a test database setting.
MIGRATE
Default: True
If set to False, Django won’t use migrations to create the test database.
I just figure out how to disable migrations after django 1.10,may be it could help for somebody. Here is link at git
class DisableMigrations(dict):
def __contains__(self, item):
return True
def __getitem__(self, item):
return None
DATABASES = DisableMigrations()
MIGRATION_MODULES = DisableMigrations()
Migrations for django 1.10 has two part,please look at load_disk and recorder
The part of load_disk for migrations model of app that be added at INSTALL_APP
And the part of recorder for database connection
For the version before 1.9 we need set MIGRATION_MODULES={'do.not.migrate':'notmigrations'} when you are running test
Now we need set it None like MIGRATION_MODULES={'do.not.migrate':None}
So if we do not want make migrations for any app, just extend a dict and return None for getitem function , and do the same at DATABASES, that is the right thing you need to do
PS: For command, you need to specify --setting=module.path.settings_test_snippet after test
PPS If you are working with pycharm ,do not set --settings options at Run/Debug configurations, just add path of settings_test_snippet.py at Custom setting. That just be fine!!
enjoy
https://gist.github.com/apollovy/22826f493ad2d06d9a9a22464730ce0b
MIGRATION_MODULES = {
app[app.rfind('.') + 1:]: 'my_app.migrations_not_used_in_tests'
for app in INSTALLED_APPS
}
For django 1.9 and up the answer of Guillaume Vincent does not work anymore, so here's a new solution:
I'm using this snippet in my settings file, after the definition of the INSTALLED_APPS
if os.environ.get('TESTS_WITHOUT_MIGRATIONS', False):
MIGRATION_MODULES = {
app.split('.')[-1]: None for app in INSTALLED_APPS
}
It iterates over all installed apps and marks each as having no migration module. See the django docs for more information.
Using this snippet you can run your tests, setting the environment variable TESTS_WITHOUT_MIGRATIONS, e.g.:
TESTS_WITHOUT_MIGRATIONS=1 ./manage.py test
Related
Under my Django project there are a few apps and all of them have unit tests. One of them that I'm working right now is supposed to be included only in dev/stage environments, so I'm enabling it using a environment variable.
When this variable is present it is added to INSTALLED_APPS and it is working just fine, the problem is that Django is executing the tests for this app even when it is not in INSTALLED_APPS, and it fails with the following message:
ImportError: Failed to import test module: debug.tests.unit.test_services`
...(traceback information)...
RuntimeError: Model class debug.models.Email doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.
When I define the app_label in the class Meta of models in this app the error is different, it says it can't find a table, I assume that this is because the app is not in INSTALLED_APPS, so it's migrations are not executed.
OperationalError: no such table: debug_email
I'm not sure why Django executes the tests for all apps, but not it's migrations.
Am I missing something from Django configuration for tests?
https://docs.python.org/3/library/unittest.html#unittest.TestLoader.discover says:
If load_tests exists then discovery does not recurse into the package, load_tests is responsible for loading all tests in the package.
So in the lowest __init__.py in your app which you don't always want run:
from django.apps import apps
def load_tests(loader, tests, pattern):
from django.conf import settings
if apps.is_installed("your_dev_app"):
# Actually load the tests - thanks to #barney-szabolcs
return loader.discover(start_dir=dirname(abspath(__file__)), pattern=pattern)
You need to return the discovered tests in load_tests.
So, adding to #DaveLawrence's answer, the complete code is:
# your_dev_app/__init__.py
from django.apps import apps
from os.path import dirname, abspath
def load_tests(loader, tests, pattern):
"""
loads tests for your_dev_app if it is installed.
"""
from django.conf import settings
if apps.is_installed("your_dev_app"):
return loader.discover(start_dir=dirname(abspath(__file__)), pattern=pattern)
When you run:
python manage.py test
the command will look per default recursive for all files with the pattern test*.py in the working directory. It isn't affected by INSTALLED_APPS in settings.py.
You can specify a certain app to test it:
python manage.py test app_label
or specify a path:
python manage.py test myapp/tests
If you want to exclude some tests you can tag them and use the option --exclude-tag.
Run python manage.py test --help to get information on all options.
The official documentation gives a lot of information on the different possibilities how to run the tests.
EDIT:
If you have apps that are required only in the development environment, but not in the production, you could split your settings.py. One possible solution would be to outsource all development settings into a file local_settings.py and exclude it from versioning or from the production branch, i.e. don't push it in the production environment.
local_settings.py
DEBUG = True
INSTALLED_APPS += (
# Django Debug Toolbar would be for example
# used only in development
'debug_toolbar',
'your dev app',
)
settings.py
try:
from .local_settings import *
except ImportError:
pass
Is there a way to exclude database from django migrations?
I have a sphinxsearch database in my django project:
DATABASES['sphinxsearch'] = {
'ENGINE': 'sphinxsearch.backend.sphinx',
...
}
And when I try to run manage.py makemigrations command, Django tries to run
SHOW FULL TABLES query against it
which leads to an error, because this is wrong syntax for sphinxql
File "C:\Anaconda\lib\site-packages\django\db\backends\mysql\introspection.py", line 56, in get_table_list
cursor.execute("SHOW FULL TABLES")
...
django.db.utils.ProgrammingError: (1064, "sphinxql: syntax error, unexpected IDENT, expecting VARIABLES near 'FULL TABLES'")
An exception to this rule is the makemigrations command.
It validates the migration history in the databases to catch problems with the existing migration files (which could be caused by editing them) before creating new migrations.
By default, it checks only the default database, but it consults the allow_migrate() method of routers if any are installed.
makemigrations always creates migrations for model changes, but if allow_migrate() returns False, any migration operations for the model_name will be silently skipped when running migrate on the db.
Changing the behavior of allow_migrate() for models that already have migrations may result in broken foreign keys, extra tables, or missing tables. When makemigrations verifies the migration history, it skips databases where no app is allowed to migrate.
class DBMigrateRouter(object):
"""
A router to control all database operations on models in the
auth application.
"""
def allow_migrate(self, db, app_label, model_name=None, **hints):
"""
Allows migration for default DB
"""
return db == 'default'
Add this class to settings file
DATABASE_ROUTERS = ['path.to.DBMigrateRouter']
This will skip migrations on databases other than "default".
I've got some Postgres stored procedures that my selenium tests will depend on. In development, I load them with a line in a script:
cat stored_procedures.sql | python manage.py dbshell
This doesn't work when unit testing, since a fresh database is created from scratch. How can I load stored procedures saved in a file into the test database before unit tests are run?
I think you have few ways to make this. In my opinion, the best solution - to add migration with your custom SQL. In future, you'll need that migration not only at development, but also at production stage. So, It would be not clear deploy procedure, if you'll store change to DB in few places.
Other way - just to add execution of your SQL to setUp method of testCase.
Additional migration
You should create a new empty migration ./manage.py makemigrations --empty myApp
Add your SQL code to operations list
operations = [
migrations.RunSQL('RAW SQL CODE')
]
Another solution to this is to create a management command that executes the required SQL Query and then just execute this command at the very beginning of your tests.
Below is my case:
Management command:
import os
from django.core.management import BaseCommand
from django.db import connection
from applications.cardo.utils import perform_query
from backend.settings import BASE_DIR
class Command(BaseCommand):
help = 'Loads all database scripts'
def handle(self, **options):
db_scripts_path = os.path.join(BASE_DIR, 'scripts', 'database_scripts')
utils_path = os.path.join(db_scripts_path, 'utils.sql')
with open(utils_path, mode='r') as f:
sql_query = f.read()
with connection.cursor() as cursor:
cursor.execute(sql_query)
Now you can just go to the terminal and type
python manage.py load_database_scripts
and the scripts will be loaded.
With a helper function like this one
def load():
with django_db_blocker.unblock():
call_command('load_database_scripts')
You just call this load function before the test suite runs.
I have been trying to synchronize changes to a model in a Django app using migrations in 1.7 (postgres 9.1 - let me know if you need more details of my environment), but manage.py migrate doesn't seem to do anything, and sqlmigrate doesn't emit any SQL.
I thought Django 1.7 - "No migrations to apply" when run migrate after makemigrations might be applicable to my situation, and I did find some history in the django_migrations table in my database. I deleted the records for the app I am trying to migrate.
Recently I gave up on getting the alter table statements to generate/run and dropped the original version of the table. And while manage.py migrate states it is applying the migration, nothing happens to the database.
Here are the steps I've been trying:
Delete the history.
rm -r myapp/migrations
../manage.py dbshell
myapp_db=> delete from django_migrations where app='myapp'
Create an initial migration.
cp myapp/models.py.orig myapp/models.py
../manage.py makemigrations myapp
../manage.py migrate
manage.py migrate returns the following:
....
Running migrations:
Applying myapp.0001_initial... FAKED
Then I swap in the new models and generate a new migration.
cp myapp/models.py.new myapp/models.py
../manage.py makemigrations myapp
The result of makemigrations is in myapp/migrations/0002_notificationlog.py:
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import models, migrations
class Migration(migrations.Migration):
dependencies = [
('myapp', '0001_initial'),
]
operations = [
migrations.CreateModel(
name='NotificationLog',
fields=[
('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
('tstamp', models.DateTimeField(help_text=b'Log time', auto_now_add=True)),
('recipient', models.CharField(max_length=100)),
('subject', models.TextField()),
],
options={
},
bases=(models.Model,),
),
]
Run this migration:
../manage.py migrate
manage.py migrate acts like everything is OK:
....
Running migrations:
Applying myapp.0002_notificationlog... OK
I can see the log entries appear in django_migrations, but the table is not created.
I'm lost. Any idea what to try next?
Update
When running migrate -v 3 as requested, I see
Running pre-migrate handlers for application auth
followed by a similar line for each installed app.
Then
Loading 'initial_data' fixtures...
Checking '/var/www/environment/default/myproj/myproj' for fixtures...
No fixture 'initial_data' in '/var/www/environment/default/myproj/myproj'.
repeated a total of 13 times, the number of unmanaged apps.
Then
Running migrations:
Applying myapp.0001_initial... FAKED
followed by
Running post-migrate handlers for application auth
with a similar line for each installed app.
For migration 0002, the output is the same, except for
Running migrations:
Applying myapp.0002_notificationlog... OK
Note also that sqlmigrate doesn't output anything either:
../manage.py sqlmigrate myapp 0002 -v 3
Produces nothing at all.
Update 2
I copied myapp into a new project and was able to run migrations on it, but migrations stopped working when I imported my main project settings. Are there settings I should be aware of that could affect migration execution, particularly if I've been using South with previous versions of Django?
The problem disappeared with generic project settings and reappeared with my old, complex project settings. I tracked the problem down to a database Router class that was missing an allow_migrate method.
DATABASE_ROUTERS = [ 'myproj.routers.DatabaseAppsRouter', ]
I use this router to handle queries for a separate app in the project (readonly/MySQL).
Sadly I can't blame anyone but myself, since the Django documentation states clearly:
Note that migrations will just silently not perform any operations on a model for which [allow_migrate] returns False. (link)
I had created this router some time ago and didn't add the allow_migrate method to my router class when I upgraded to Django 1.7. When I added the method and made sure it returned True when needed, migrations run and the problem is solved.
When i run my app test django don't create my app tables and throw an error.
My test file:
from django.test import TestCase
class MyTest(TestCase):
fixtures = ['initial_data.json']
def test_my_stuff(self):
[...]
When i run test:
DatabaseError: Problem installing fixture
'/home/.../djStock/stock/fixtures/initial_data.json': Could
not load stock.Provider(pk=1): (1146, "Table
'test_djstock.stock_provider' doesn't exist")
My app is correctly added in INSTALLED_APPS. What i miss ?
Must be have south migration files. Check if myapp/migration/ contain files for migration.
You have to comment out 'south' in INSTALLED_APPS before running any tests, otherwise if models are updated, you Django won't have the update reflected in the table creation, thus you get the error that table does not exist.
Assuming you are using Django 1.6.