I am changing from a profile model setup, to extending the User model. I am trying to get my South migrations set up according to this guide. I have a few other models that reference the User model, and so when I auto-generate my schema migration I get lines along the lines of:
db.alter_column(u'app_model', 'user',
self.gf('django.db.models.fields.related.ForeignKey')(to=orm['app.user']))
The problem is these migrations cause South to break when I try and migrate:
FATAL ERROR - The following SQL query failed: INSERT INTO "_south_new_app_model" () SELECT FROM "app_model";
The error was: near ")": syntax error
! Error found during real run of migration! Aborting.
! Since you have a database that does not support running
! schema-altering statements in transactions, we have had
! to leave it in an interim state between migrations.
! You *might* be able to recover with:
! The South developers regret this has happened, and would
! like to gently persuade you to consider a slightly
! easier-to-deal-with DBMS (one that supports DDL transactions)
! NOTE: The error which caused the migration to fail is further up.
Error in migration: app:0018_auto__chg_field_model_user.py
DatabaseError: near ")": syntax error
Note: this is a migration created with ./manage.py schemamigration app --auto. I haven't edited it.
The thing is, do I care? The type of the reference stored in the database is the same, and I've done the step of renaming the table. Everything seems to continue working if I remove the alter_table lines from the migration. Is this a terrible, terrible idea, or is it ok?
Sqlite3 is the problem. As you may see here, sqlite3 support just a limited subset of alter table operations so that is causing South to fail the migration. If you need to do this you'll have to port the test database to another engine which supports alter table. Which I recommend, sometimes migrations can be tricky and you may think it's working but is always better to test it.
If you think is working ok, so go on and just ignore the error, but caution, it may be working now but maybe the migrations didn't succeed and later you'll find strange database related bugs in your development environment.
Related
I'm working on a Django Project with Wagtail which uses two databases. The first one is the standard sql lite database for all django models (called db_tool.sqlite3), the other one is also sql lite but for a wagtail integration (called db.sqlite3).
I wanted to migrate to the db_tool.sqlite3 with the following command
python manage.py make migrations
python manage.py migrate --database db_tool
but now I get the following error message regarding wagtail, which I never got before.
django.db.utils.IntegrityError: NOT NULL constraint failed: wagtailcore_page.draft_title
First of all: I don't understand this, because I named the db_tool in particular and I wonder, why the wagtail integration raises an error when I try to migrate to db_tool.
Second: I see no particular field at my wagtail-pages called draft_title and I don't have any draft page at the moment.
Third: the error message also relates to a migration file of wagtail that can be found in the side-packages (see below). So maybe this is the root of the error, but I don't understand the correlation to the other error message, because since now it worked fine and I changed nothing exept of some content of my wagtail pages.
File "C:\Users\pubr\.conda\envs\iqps_web\lib\site-packages\wagtail\core\migrations\0001_squashed_0016_change_page_url_path_to_text_field.py", line 23, in initial_data
root = Page.objects.create(
The wagtail version I use here is wagtail 2.15.2 and I haven't updated it since I started the project...
Due to the fact, that my wagtail-database has the name of the default django-database, could it be possible, that I accidentally tried a migration which was ment for the tool_db.sqlite3 without naming it in the migrate-command and caused this error by doing that?
So I would be very grateful if anyone knows, where the error comes from, or at least, what I could try out to fix it...
Kind regards and thank you!
It isn't clear to me if your database is currently broken or not. Hopefully not, but if it is, please take a back up of each before doing anything else.
This does sound like you might have been trying to operate in the wrong database. Do you have DATABASE_ROUTERS configured? I think that might help you prevent code from one app from getting introduced into the wrong database. The example in the Django docs is mostly focused on read replicas but should be adaptable to your situation: https://docs.djangoproject.com/en/4.1/topics/db/multi-db/#an-example
If your databases are in an incorrect state, start by looking at the django_migrations file in each and then carefully pruning the messed up one until you get back to the separation you have been enforcing.
We've changed our database, using django migrations (django v1.7+).
The data that exists in the database is no longer valid.
Basically I want to test a migration by, inside a unittest, constructing the pre-migration database, adding some data, applying the migration, then confirming everything went smoothly.
How does one:
hold back the new migration when loading the unittest
I found some stuff about overriding settings.MIGRATION_MODULES but couldn't work out how to use it. When I inspect executor.loader.applied_migrations it still lists everything. The only way I could prevent the new migration was to actually remove the file; not a solution I can use.
create a record in the unittest database (using the old model)
If we can prevent the migration then this should be pretty straightforward. myModel.object.create(...)
apply the migration
I think I can probably work this out now that I've found the test_executor: set a plan pointing to the migration file and execute it? Um, right? Got any code for that :-D
confirm the old data in the database now matches the new model
Again, I expect this should be pretty easy: just fetch the instance created before the migration and confirm it has changed in all the right ways.
So the challenge is really just working out how to prevent the unittest from applying the latest migration script and then applying it when we're ready?
Perhaps I have the wrong approach? Should I create fixtures, and just confirm that they're all good at the end? Do fixtures get loaded before the migrations are applied, or after they're all done?
By using the MigrationExecutor and picking out specific migrations with .migrate I've been able to, maybe?, roll it back to a specific state, then roll forward one-by-one. But that is popping up doubts; currently chasing down sqlite fudging around due to the lack of an actual ALTER TABLE instruction. Jury still out.
I wasn't able to prevent the unittest from starting with the current database schema, but I did find it is quite easy to revert to earlier points in the migration history:
Where "0014_nulls_permitted" is a file in the migrations directory...
from django.db.migrations.executor import MigrationExecutor
executor.migrate([("workflow_engine", "0014_nulls_permitted")])
executor.loader.build_graph()
NB: running the executor.loader.build_graph between invocations of executor.migrate seems to be a very important part of completing the migration and making things behave as one might expect
The migrations which are currently applicable to the database can be checked with something like:
print [x[1] for x in sorted(executor.loader.applied_migrations)]
[u'0001_initial', u'0002_fix_foreignkeys', ... u'0014_nulls_permitted']
I created a model instance via the ORM then ensured the database was in the old state by running some SQL directly:
job = Job.objects.create(....)
from django.db import connection
cursor = connection.cursor()
cursor.execute('UPDATE workflow_engine_job SET next_job_state=NULL')
Great. Now I know I have a database in the old state, and can test the forwards migration. So where 0016_nulls_banished is a migration file:
executor.migrate([("workflow_engine", "0016_nulls_banished")])
executor.loader.build_graph()
Migration 0015 goes through the database converting all the NULL fields to a default value. Migration 0016 alters the schema. You can scatter some print statements around to confirm things are happening as you think they should be.
And now the test can confirm that the migration has worked. In this case by ensuring there are no nulls left in the database.
jobs = Job.objects.all()
self.assertTrue(all([j.next_job_state is not None for j in jobs]))
We have used the following code in settings_test.py to ignore the migration for the tests:
MIGRATION_MODULES = dict(
(app.split('.')[-1], '.'.join([app, 'nonexistent_django_migrations_module']))
for app in INSTALLED_APPS
)
The idea here being that none of the apps have a nonexistent_django_migrations_module folder, and thus django will simply find no migrations.
So, in a bit more detail I have a model with a field like: permalink = models.IntegerField(default=0)
I've not actually been using this field - but would now like to.
However, it seems all models on this table, permalink is now 57295730 - on all 2000 models!
In an attempt to debug, I tried completly wiping the DB, running migrate (~100 migrations) - but then creating a instance of the model, I am told permalink violates the not-null constraint though I am definitely passing it a value! I also get a list of the values I am passing it, but am not sure how to know which value/column relates to which field?
I've even tried removing DB, removing migrations, running a new makemigrations - and still get the null violation...
even stranger, it looks like this field has not been touched since the initial migration!
migrations$ egrep permalink *
0001_initial.py: ('permalink', models.IntegerField(default=0)),
migrations$
I'm running (k)ubuntu 14.04, postgres 9.3, python 3.4, django 1.9.4
Though I'd love to know how to fix this - my question is really "What can I do to debug this kind of situation?"
Well Not the answer I want - but a working answer:
do automated testing
use CI! Prevent this problem in the first place
And if you are not doing the above...
use git bisect (or if you cant, manually use git reset --hard HEAD~1 to find the problem!)
in mycase, I was over-riding the save function of the model in a... stupid way!
edit:
in a little more detail, I was setting permalink to be 1 greater than the current biggest value - but in an earlier commit, had removed the + 1
However, I did not notice this error quickly, as it did not happen with data in the DB.
So! the error was actually quite informative - had I been running my tests more often (or using CI) I would have been informed of the error instantly, and saved myself quite a headache!
so, in short: **write tests, run them - automatically **
I made a change, adding a unique constraint to a model, within the abc application and did a
./manage.py schemamigration abc --auto
That created a migration file but as well as the expected change the new migration file also contained a number of add_column statements which are adding columns which were previously added in an earlier migration (and which have been the subject of a migrate)
I'm really puzzled as to why this has happened and what to do about it.
Will the add_column statements just be ignored if I do another migrate ?
OK thanks to the #django-south irc channel I've figured this out.
This type of problem can arise when activity has taken place in different source control branches and, as a result of a merge, the dictionary of frozen models, which appears at the bottom of a south migration file, is missing some stuff which has already taken place. The result of this is that the next schemamigration tries to produce the "missing" changes.
The fix is to manually edit the migration file which was created by the schemamigration before doing migrate. This will get things back into synch.
There's some information about issues in the later part of this section : http://south.readthedocs.org/en/latest/tutorial/part5.html#team-workflow .
Thanks to carljm and maney on #django-south for helping me with this.
I'm trying to do a simple migrate and I'm getting the error
django.db.utils.DatabaseError: (1050, "Table 'introspect_test' already exists").
When looking at the actual MySQL database, I see no table called introspect_test nor do I have such a table defined in any Django model. A little bit of Google-Fu tells me that this introspect_test has something to do with foreign key constraints and South's error checking... or something.
I think this whole problem originated form my own error - while in the middle of executing a python manage.py migrate app_name I accidently hit CTRL-C and, thus, stopped the process.
I guess the question is: how do I get rid of this mysterious introspect_test so that I can migrate normally?
Thanks
Ugh, silly me... what I didn't realise is that the table introspect_test is alls caps and, thus, isn't sorted alphabetically. It turns out that, due to a small laptop screen, doing show tables; listed all the tables but cut off the top few. Since the table in question is labelled as INTROSPECT_TEST it is at the top.
I did a quick drop table INTROSPECT_TEST and then migrated normally.
Fixed.