Say I have these migrations:
0001_initial
0002_add_some_column
0003_some_data_migration
All is fine at that point, but if I add one more schema migration:
0004_add_bar_column
and then try to run the migrations against a new DB, or any DB that doesn't have 0003 yet, 0003 will bork out because "column bar does not exist".
What is the correct way to handle that scenario? Do data migrations always have to be re-done, when a schema migration is added, such that the data migrations always come last? Is there a way to make the data migration not care that "bar" doesn't exist yet? The data migration doesn't make use of "bar", but for some reason Django still thinks it needs it to exist at that point...
I'm using the build-in Django migrations, not South.
How are you accessing the models in the data migration?
Make sure you access the ORM through the apps / schema_editor rather than importing models directly.
The first argument passed to your migration worker function is an app registry that has the historical versions of all your models loaded into it to match where in your history the migration sits.
i.e. in your data migration you should not have a line this:
from my_app import MyModel
but rather, something more like this
MyModel = apps.get_model("my_app", "MyModel")
Related
I have created a model and migrated in Django, with a unique key constraint for one of the field. Now am trying to remove the unique constraint and generate another migration file with the new change, but it says "Nothing seems to have changed".
I tried with the command
python manage.py schemamigration --auto
PS: I am using OnetoOne relationship for the field.
Good question. A one to one relationship implies that one record is associated with another record uniquely. Even though the unique constraint is removed(for one to one field) in the code explicitly, it won't be reflected in your DB. So it won't create any migration file.
If you try the same thing for foreign constraint, it will work.
I find the django automigration file generation problematic and incomplete.
Actually I experienced another similar problem with django migration just yesterday.
How I solved it:
delete all migration files from the /migrations folder
do a fresh makemigrations
run python manage.py migrate --fake so django does not try to rebuild..
Hey presto! Working and models updated :D
I have 4 models. I want to redo 1 one of them that I've been working on. The other 3 models have user data that I don't want to lose.
I want to entirely drop one table, and sync it up with what's in models.py.
How do I do this?
You could remove the model from models.py, and create a migration which will drop the table.
./manage.py makemigrations
Then add the model back to your models.py, and create a new migration which will recreate the model.
./manage.py makemigrations
Finally, run your migrations and you should be done.
./manage.py migrate
It's not very clear what you want to do here.
You can write new model class and delete old model class that you want to remove. After that run migrations the normal way.
It will delete the table related to deleted model class and make whatever othe changes you have defined in the models.
I've had some migration issues over time and occasionally have run into a case where a field will not have been correctly migrated (almost certainly because I tried some fake migration to get my dev db in a working state).
Doing an automatic schema migration will check the migration chain against the model, but not check either of those against the actual db.
Is there a way to easily compare the database against the current models or migration chain and verify that the db, the models, and migration chain are consistent?
As a straw man imagine you delete your migrations, create a new initial migration, and fake migrate to that initial while deleting the ghost migrations.
Is it trivially possible to verify that the database is in sync with that initial migration?
The django-extensions application provides sqldiff management command, which shows difference between current database and your model. So if there is difference between your database and model (migrations should be same after running makemigrations command), you will see.
On a Django app with some self-made (but close to available plugin methods) multi-tenant implementation, I would like to run a migration (a simple add_column this time) with South that could apply on all schemas. I have a configuration very close to this one.
I would like to skip any pure SQL queries if possible. I can get a list of the schemas name from the ORM properly, but then I wonder if I have the possibility to access the tables from the various schemas in a somehow propre way.
I have a hook to be able to change the DB_HOST and DB_SCHEMA via parameters at some level, but I think not to be able to loop cleanly this way inside the forwards migration method of South.
This question is quite high-level, but I mainly wonder if somebody had to face the same kind of question and I am curious to know if there is some clever way to handle it !
Regards,
Matt
This is an outline of a solution, as posted on the South mailing list. The question as phrased is a little different from the one that was posted on the list: There, it was also mentioned that there are "common" tables, shared between all tenants, in a separate schema. Rmatt's own answer refers to this as the public schema.
The basic idea of my solution: Save the migration history for each database (schema) in the schema. To do this, we need to employ some database and Django tricks.
This implies that history records for migrations of apps on the public schema are saved in the public schema, while history for migrations of tenant apps is saved in the tenant schema -- effectively sharding the migration history table. Django does not really support this kind of sharding; it is easy enough to set up the writing by instance content, but there's no way to set up the reading.
So I suggested to create, per tenant, a "tenant-helper" schema, containing one view, named south_migrationhistory, which is a union of the south_migrationhistory tables from the public and tenant schemata. Then, set up a database router for the South MigrationHistory model, instructing it to:
syncdb to both public and tenant schemata
read always from tenant-helper schema
write to public or tenant schema, according to the app the migration belongs to
The result allows proper treatment of dependencies from the tenant app migrations to the public app migrations; and it means all you need to do, to migrate forwards, is loop on the migrate --all (or syncdb --migrate) command -- no need to fake backward migrations. The migrations for the public schema will be run with the migrations for the first tenant in the loop, and all other tenants will "see" them.
As an afterthought, it is probably possible to do this also without a helper schema - by renaming the south_migrationhistory table in the tenant schema, and installing a view with that name in the schema that returns the above-mentioned union when queried, and has an "instead-of insert" trigger to write to the renamed table.
Fine, not so many people seem to have experience or to be concerned with this quite specific problem. I have tried some things here and there and I also got some support from the South mailing-list that helped me to understand some points.
Basically, the solution I implemented is the following:
I have a quite normal migration file autogenerated via South's schemamigration. But I have changed the table name of the add_column and delete_column to schema.table_name. The schema is provided by importing the multi-tenant middleware.
The migration is then applied only if the schema is not run against the public schema. It is actually not meant to be run standalone, or only with database and schema kwargs, but rather from a migration runner that is a new django command.
The runner has unfortunately to call the migration externally, in order to go through the middleware each time again. One other trick is that we have to get the previous state of migration, in order to fake it back to the previous state for south after each tenant migration.
Here is my snippet :
from subprocess import call
import os
from django.core.management.base import BaseCommand
from south.models import MigrationHistory
from myapp.models import MyModel
class Command(BaseCommand):
def handle(self, *args, **options):
#the only allowed arg is the prefix version and it should have a length of 4 (i.e. 0002)
applied = MigrationHistory.objects.filter(app_name='myapp').latest('applied')
current_version = applied.migration[:4]
call_args = ['python', os.path.join('bin', 'manage.py'), 'migrate', 'myorderbird.app.backups']
if len(args) == 1 and len(args[0]) == 4:
call_args.append(args[0])
obje_call_args = None
for obje in MyModel.objects.all():
if obje.schema_exists:
# fake the migration of the previous venue back to the current version
if obje_call_args:
obje_call_args = obje_call_args[:4] + [current_version, '--fake'] + obje_call_args[len(obje_call_args)-3:]
call(obje_call_args)
# migrate the venue in the loop
obje_call_args = list(call_args)
obje_call_args.extend(['--database={}'.format(obje.db), '--schema={}'.format(obje.schema)])
call(venue_call_args)
I rewrote a lot of my models, and since I am just running a test server, I do ./manage.py reset myapp to reset the db tables and everything has been working fine.
But I tried to do it this time, and I get an error,
"The full error: contraint owner_id_refs_id_9036cedd" of relation "myapp_tagger" does not exist"
So I figured I would just nuke the whole site and start fresh. So i did ./manage.py flush then did a syncdb this did not raise an error and deleted all my data, however it did not update the database since when I try to access any of my_app's objects, i get a column not found error. I thought that flush was supposed to drop all tables. The syncdb said that no fixtures were added.
I assume the error is related to the fact that I changed the tagger model to having a foreignkey with a name owner tied to another object.
I have tried adding related_name to the foreignkey arguments and nothing seems to be working.
I thought that flush was supposed to drop all tables.
No. According to the documentation, manage.py flush doesn't drop the tables. Instead it does the following:
Returns the database to the state it was in immediately after syncdb was executed. This means that all data will be removed from the database, any post-synchronization handlers will be re-executed, and the initial_data fixture will be re-installed.
As stated in chapter 10 of The Django Book in the "Making Changes to a Database Schema" section,
syncdb merely creates tables that don't yet exist in your database — it does not sync changes in models or perform deletions of models. If you add or change a model's field, or if you delete a model, you’ll need to make the change in your database manually.
Therefore, to solve your problem you will need to either:
Delete the database and reissue manage.py syncdb. This is the process that I use when I'm still developing the database schema. I use an initial_data fixture to install some test data, which also needs to be updated when the database schema changes.
Manually issue the SQL commands to modify your database schema.
Use South.