I have an existing project which extensively uses South migrations to load data into its tables.
Since upgrading to Django 1.3 our unit tests no longer run because they cannot find the data they rely on.
Is this behaviour is due to one of the backwards incompatible changes in 1.3
Is there an easy way for me to convert all these migrations into fixtures?
Yes, this behavior is due to this change.
There seems to be a workaround in South trunk (see https://bitbucket.org/andrewgodwin/south/changeset/21a635231327 ) so you can try South development version (it is quite stable in my experience).
You may try to change the DB name in settings (in order to get clean environment), run ./manage.py syncdb and ./manage.py migrate and then do ./manage.py dumpdata
I hit this issue today. Eventually I ended up refactoring my migrations so that they use helper functions to actually insert the data, and then calling the same functions from the setUp() of my tests.
Some hints;
Make your helper functions take the model class as an argument, so you can call them with orm['yourapp.YourModel'] from the migration and with models.YourModel from the test. That also shows the main limitation: South works for models whose schema has changed since then, the test code can't do that. I was lucky in that this particular model hasn't changed.
If you want to keep the helper methods inside the migrations, you'll find that you can't directly import yourapp.migrations.0001_some_migration because identifiers can't start with numbers. Use something like migration_0001 = importlib.import_module('yourapp.migrations.0001_some_migration') instead of an import statement.
Related
I'm trying to create a new Django application whose migrations depend on other apps migrations. I want migrations on the other apps to have a dependency on migrations on the new application, even when the new application doesn't reference at all the models at the other apps.
For example, let's say I have application A (an existing app) and application B (my new application):
A has a migration called A.0001, and B has a migration called B.0001 which depends on A.0001. I now make a change at A.MyModel, so I need to run python manage.py makemigrations to generate a new migration A.0002.
What I want is A.0002 to automatically depend on B.0001.
How can I specify this dependency on new migrations in A without having to do it by hand each time I modify a model in A?
I tried to add empty modifications of models in A at the migration B.0001, but I haven't got it to work and it looks very hacky to me.
I found a solution that doesn't exactly solve what I asked, but can be a workaround for some people, so I will post it here.
There is an undocumented field of the django.Migration class, called run_before, that works just like an inverse dependencies.
A.0002 will depend on B.0001 if I simply define run_before = [ A.0002 ] at B.0001.
This question already has an answer here:
Migrating problems when porting Django project to Python 3 and Django 2
(1 answer)
Closed 3 years ago.
I'm working on upgrading my legacy application from Django 1.11.13 to 2.2.8. I've dutifully addressed every compatibility issue, but I've hit one I can't figure out how to resolve. When I try to start the webserver in my local environment, I get this error (only showing the end of the full error trace that appears):
File "/Users/me/my_app/my_model/migrations/0001_initial.py", line 37, in Migration
('entry', models.ForeignKey(to='my_model.Entry')),
TypeError: __init__() missing 1 required positional argument: 'on_delete'
I understand why on_delete is now required -- I just spent a while updating my models everywhere to accommodate this change -- but I have no idea how to fix this particular issue without going through dozens of old migrations files to make them conform?!
I tried squashmigrations to at least collapse the number of places I have to clean up, but I got the same exact TypeError.
I tried to use the old version of Django for squashmigrations. I was successful in avoiding the TypeError, but ended up with a huge mess of circular import errors.
Since I don't actually need the migration history to roll back, I tried to follow these instructions (scenario 2) to clear the migration history while keeping the existing database, but I couldn't run makemigrations to catch up on the changes I made to make my models Django 2.2 compliant, and when I decided I'd skip ahead and deal with that later, showmigrations failed with the same TypeError. (Is there some other way to get a fresh set of initial migrations based on the current database? It can't be based off the models since the models have upgrade-related changes not yet reflected in the database.)
I moved the migrations to a non-standard location, which got the server to start, but that makes it impossible to actually do anything migration related ever again, and of course once I move back, everything breaks again...
I've considered just deleting my entire database and all migration history, building the tables from scratch with a fresh set of initial migrations, and then resetting the data from a backup, but there are a few huge tables which would make this take quite a while... and this rather seems like the nuclear approach. Am I stuck with editing a large number of very old migrations to be compliant with Django 2.2 for no actual reason since I'm never going to roll my project that far back? How can that be right?
As Iain Shelvington mentions in a comment under the question,
First delete all of your migration files and folder, then run makemigrations with the "on_delete" - this should create some "initial" migration files. Then you'll have to log in to your DB and delete all entries for your apps and then you need to run manage.py migrate --fake - this will enter into the DB entries for the newly created migrations but will not apply them
We've changed our database, using django migrations (django v1.7+).
The data that exists in the database is no longer valid.
Basically I want to test a migration by, inside a unittest, constructing the pre-migration database, adding some data, applying the migration, then confirming everything went smoothly.
How does one:
hold back the new migration when loading the unittest
I found some stuff about overriding settings.MIGRATION_MODULES but couldn't work out how to use it. When I inspect executor.loader.applied_migrations it still lists everything. The only way I could prevent the new migration was to actually remove the file; not a solution I can use.
create a record in the unittest database (using the old model)
If we can prevent the migration then this should be pretty straightforward. myModel.object.create(...)
apply the migration
I think I can probably work this out now that I've found the test_executor: set a plan pointing to the migration file and execute it? Um, right? Got any code for that :-D
confirm the old data in the database now matches the new model
Again, I expect this should be pretty easy: just fetch the instance created before the migration and confirm it has changed in all the right ways.
So the challenge is really just working out how to prevent the unittest from applying the latest migration script and then applying it when we're ready?
Perhaps I have the wrong approach? Should I create fixtures, and just confirm that they're all good at the end? Do fixtures get loaded before the migrations are applied, or after they're all done?
By using the MigrationExecutor and picking out specific migrations with .migrate I've been able to, maybe?, roll it back to a specific state, then roll forward one-by-one. But that is popping up doubts; currently chasing down sqlite fudging around due to the lack of an actual ALTER TABLE instruction. Jury still out.
I wasn't able to prevent the unittest from starting with the current database schema, but I did find it is quite easy to revert to earlier points in the migration history:
Where "0014_nulls_permitted" is a file in the migrations directory...
from django.db.migrations.executor import MigrationExecutor
executor.migrate([("workflow_engine", "0014_nulls_permitted")])
executor.loader.build_graph()
NB: running the executor.loader.build_graph between invocations of executor.migrate seems to be a very important part of completing the migration and making things behave as one might expect
The migrations which are currently applicable to the database can be checked with something like:
print [x[1] for x in sorted(executor.loader.applied_migrations)]
[u'0001_initial', u'0002_fix_foreignkeys', ... u'0014_nulls_permitted']
I created a model instance via the ORM then ensured the database was in the old state by running some SQL directly:
job = Job.objects.create(....)
from django.db import connection
cursor = connection.cursor()
cursor.execute('UPDATE workflow_engine_job SET next_job_state=NULL')
Great. Now I know I have a database in the old state, and can test the forwards migration. So where 0016_nulls_banished is a migration file:
executor.migrate([("workflow_engine", "0016_nulls_banished")])
executor.loader.build_graph()
Migration 0015 goes through the database converting all the NULL fields to a default value. Migration 0016 alters the schema. You can scatter some print statements around to confirm things are happening as you think they should be.
And now the test can confirm that the migration has worked. In this case by ensuring there are no nulls left in the database.
jobs = Job.objects.all()
self.assertTrue(all([j.next_job_state is not None for j in jobs]))
We have used the following code in settings_test.py to ignore the migration for the tests:
MIGRATION_MODULES = dict(
(app.split('.')[-1], '.'.join([app, 'nonexistent_django_migrations_module']))
for app in INSTALLED_APPS
)
The idea here being that none of the apps have a nonexistent_django_migrations_module folder, and thus django will simply find no migrations.
I'm at a new job. One of my predecessors used the class_prepared signal to apply a prefix to all the table names, i.e. django_content_type is oursite_django_content_type. I think this was unnecessary and ill advised (I looked at doing this at a previous job and did not), as the documentation says that class_prepared is an implementation detail. Well -- now it's a problem.
The site seems to work okay (I didn't thoroughly test it yet) but I can't run our unit tests. This is because the migrations are a core feature now and the contenttypes migration defines the db_table_name.
I dropped some debugging statements and this is the sequence of events.
class_prepared is run for contenttypes -- the db_table_name for the class is altered to oursite_django_content_type.
The migration is run and it creates the django_content_type table.
Tests are run and they can't find oursite_django_content_type.
I think I'm going to have to migrate the tables to their default values, but -- Does anyone have a suggestion so that I might put this off so that my successor will have to deal with it?
I'm trying to use fixtures as a DB-agnostic way to get the data into my database, but this is much harder than it should be. I'm wondering what I'm doing wrong...
Specifically, when I do a syncdb followed by a migrate followed by a loaddata I run into trouble, since syncdb already creates data that loaddata tries to read from the dump. This leads to double entries and hence a crashing script.
This seems to be the same problem as described here: https://code.djangoproject.com/ticket/15926
But it's weird to me that this seems to be an ignored issue. Are fixtures not meant to actually put real (live) data in?
If so: is there any Django-format that is meant for this? Or is everyone just dumping data as SQL? And, if so, how would one migrate development data in SQLite to a production database?
syncdb will also load data from fixtures if you have the fixtures named correctly and in the correct location. See this link for more info.
https://docs.djangoproject.com/en/1.3/howto/initial-data/#automatically-loading-initial-data-fixtures
If you do not want the data to load on every syncdb then you will need to change the name of the fixture.
fixtures are an OK way to load your data, I have used it on a number of projects. On some projects when I have a ton of data I sometimes write a special load script that will take the data from my data source and load up my new django models, the custom script is a little more work, but gives you more flexibility.
I tend to stay away from using sql to load if I can, since SQL is usually DB specific, if you have to worry about loading on different database versions, stay away if you can.
"In general, using a fixture is a cleaner method since it’s database-agnostic, but initial SQL is also quite a bit more flexible."
OP here; this is what I came up with so far:
# some_app/management/commands/delete_all_objects.py
from django.core.management.base import BaseCommand, CommandError
from django.db.models import get_models
class Command(BaseCommand):
help = 'Deletes all objects'
def handle(self, *args, **options):
for model in get_models():
model.objects.all().delete()
And then just run delete_all_objects between after syncdb & migrate and before loaddata. I'm not sure I like it, I'm very surprised it's necessary, but it works.