After making some changes in my models (eg. new field in a model and a new model) what is the best way of reflecting these changes to my populated database?
PS: I wanted to see many solutions in one place rated. Apparently more solutions are already listed here.
Another technique is to use the dumpdata and loaddata arguments to manage.py, killing your database in-between:
python manage.py dumpdata > dump.json
With an external tool, drop any affected tables, or kill the whole db
python manage.py loaddata dump.json
See manage.py docs for more.
I've asked a similar question here and got quite a few answers.
There are quite a lot of ways of doing it, like manually doing the dumping and reloading with SQL, using fixtures or using one of the "emerging" schema-evolution packages for Django:
Django Evolution
South
dmigrations
(there's a DjangoCon video of a panel on schema-evolution in Django where these 3 solutions are discussed)
Depends on the scope of the changes. If it's beyond an ALTER, you're doing major surgery. Make backups of model as well as database so you can go back.
My preference is to put your new (revised, corrected, expanded) model in as a NEW application. It won't have URL's or anything, just a model.
Creating the new model as a new application. Create tests, just to be sure it works.
syncdb to build this interim implementation of the new model.
Write a little one-time utility to query your old model, and load your new model. You might want to try this in pure SQL. I prefer to write a simple query, build and save loop.
After the new model is loaded, you can dump this to a JSON file.
Once you've pulled the data out of your old model, you can rebuild your DB in the preferred new format.
Move the new model into your existing application.
Drop the old versions of the application's tables.
syncdb to build the new tables.
Load the JSON file with the data.
Django now has its own built-in migrations, documented at:
https://docs.djangoproject.com/en/dev/topics/migrations/
Look with manage.py sqlall what the parameters are for the new columns and manually add them in your database with Alter table statements. This way you don't have to redo your database; It requires some SQL knowledge though...
Take a look here (Scroll down to "Making Changes to a Database Schema")
Perform these steps in order may help you:
For more details,
clickhere: http://south.readthedocs.org/en/latest/
1) python manage.py schemamigration apps.appname --initial
Above step creates migration folder as default.
2) python manage.py migrate apps.appname --fake
generates a fake migration.
3) python manage.py schemamigration apps.appname --auto
Then you can add fields as you wish and perform the above command.
4) python manage.py migrate apps.appname
Then migrate the files to the database.
Related
I have a django 1.8 app working with a db.
I'm trying to change the schema of a table using the built-in migration.
Here are the steps I did:
In my dev invironment, I grabbed the app source and ran
python manage.py sycdb
then I ran
python manage.py loaddata ~/my_data.json
then I modified modes.py. Added a field and renamed a field...all from the same table 'TABLE1' which had no data.
then
python manage.py makemigrations myapp
python manage.py migrate
Error: django.db.utils.OperationalError: table "myapp_someother_table" already exists
then ran
python manage.py migrate --fake-initial
worked!
but when I browsed to the admin page for TABLE1, I get this error:
OperationalError: no such column: myapp_table1.my_new_field_id
I checked the db and yes, there is no such column.
How can I procceed from here? I prefer to fix this via django.
If I fix it straight in the db, then the migration goes out of sync.
Migrations do not automagically see that you have made changes. Migrations detect changes by comparing the current model with the historical model saved in the migration files.
In this case, you didn't have any historical models, since you didn't have any migrations. Django was not able to detect any changes in your models, even though they were different from your database.
The correct way to make changes to your model is to first run manage.py makemigration <my_app>, and then make the changes to your model, followed by another manage.py makemigrations.
You might not be able to do it via pure django and keep your data. I don't have personal experience with south but there are a lot of mentions if this tool. Just in case if nothing else works for you...
Here is what I did to make things work, but there must be a better way so please add more answers/comments...
I deleted the sqlite db and the migration folder
I made the desired changes to model.py
ran syncdb
ran loaddata to load the json data dump that I had saved previously.
just started the dev server
We all love South migration in django, normally you first use convert_to_south and then make model class changes, then run schemamigration and migrate
I am not sure if the following scenario has happened to anyone, the project is not tracked under version control (git stash would be awesome), you changed the model class so much and lost the undo/redo history, and you realised the app has not yet converted to south. Is there a command to remedy this?
Here's one idea.. read all the disclaimers here first, like which databases are supported for detecting FKs.
https://docs.djangoproject.com/en/dev/ref/django-admin/#django-admin-inspectdb
Update: the real answer is to use a mix of the bottom two ideas. Basically, inspectdb is a nice human readable way to figure out the difference between your model and the database. Then, I'd manually build alter table statements to match. Luckily, sqlall outputs the create statements, so most of it will be cut and paste for new fields.
Make a backup of everything.
Copy your models.py somewhere.
Type in python manage.py inspectdb
Find the model you're looking for and paste it into the original models.py
Set up south.
Paste the new / updated models.py back into your models.py
Run schemamigration then migrate.
You can also just do the migrations manually, then convert to south.
python manage.py dbshell
Describe table (depends on your db)
Find missing rows... use alter table statements as necessary. (SQL hints in python manage.py sqlall <myapp>
I'm using sqlite3 and pycharm to learn more about django, and googled to find that south is recommended to make it easier to modify models after they have been created.
I'm trying to follow the advice on http://south.aeracode.org/docs/tutorial/part1.html#starting-off.
The most success I've had so far is to create a simple model and run syncdb before adding south to installed_apps. That way the intial tables are created and I get a chance to create a super user. (Django admin seems to fret if there are no users).
Then I add south to installed_apps, and run django_manage.py schemamigration bookmarks --initial
It seems to work fine. A new directory is created called migrations with a couple of files in it in my app folder and an encouraging message.
"Created 0001_initial.py. You can now apply this migration with: ./manage.py migrate bookmarks"
The next step - django_manage.py" migrate bookmarks generates the following error message
django.db.utils.DatabaseError: no such table: south_migrationhistory.
I thought that table would be created in the first schememigration step. What am I missing? Can anyone help?
Marg
South uses a table if its own to keep track of which migrations have been applied. Before you can apply any migrations, this must have been created, using python ./manage.py syncdb.
As well as for setting up south, you will find syncdb sometimes necessary for non-south apps in your project, such as the very common django.contrib.auth.
Note that as a convenience, you can run both in one go like this
python ./manage.py syncdb --migrate
My latest (unsuccessful) effort was the following
Create application – synch db – superuser created
Test run –admin screen shows basic tables
Add south, and syncdb from command line with manage.py syncdb – south_migrationhistory table created. Add basic vanilla model
Tried various combinations of manage.py syncdb –manage, and
schemamigration from Pycharm (if run from within pycharm a
migrations directory is created within the app
– if run from the command line the directory does not seem to be
created.)
Django admin screen shows table – but if I try to edit
the table it says that it doesn’t exist
Check database structure
using SQLite browser - table for newly created model doesn’t exist
I’m starting to think that the whole thing is not worth the time wasting hassle – maybe I’m better off just modifying the tables in SQLite browser
Answer in the similar question:
Run syncdb to add the Django and South tables to the database.
What is the best solution if I want to upgrade (alter) my database schema (add new fields to tables by adding them just to Django models) without losing data in these tables? "syncdb" not adding them of course, so I need your advices how to alter tables without deleting them and recreating again with syncdb.
When south isn't an option I just manually write scripts for small changes. and big ones i use
./manage.py dumpdata appname
http://docs.djangoproject.com/en/dev/ref/django-admin/#dumpdata-appname-appname-appname-model
Throw that into a file. Run a regex replace to update any added /removed fields and then a reset of that app is possible. I have to admit i haven't done this in a while but i can get some specific code to do this for you if needed.
it loads back up with loaddata
edit
Django dump data for a single model? This Question is similar and might have the info i was talking about.
Still let me know if you need and i'll dig up my old script (or write out a nice simple one) for you.
UPDATE
./manage.py dumpdata appname --indent=4 > appname.json
#open your fav text editor and do a find/replace
./manage.py reset appname
./manage.py loaddata appname.json
That should do it. When you do a find replace you only need to remove fields that you don't have any more and add fields that aren't nullable. (as a minimum).
Notes: the --indent=4 nicely formats everything for you. It means 4 spaces.
the ./manage.py reset only works in pre django 1.3 (gah!) in django 1.3 you will have to do a ./manage dbshell and drop table. The sql for that is found from command ./manage.py sqlreset appname.
Learning curve...
http://south.aeracode.org/
South was merged into django core at version 1.7.
There is now a native django feature for data migration on schema changes.
Django 1.7 has built-in migrations support.
See https://docs.djangoproject.com/en/dev/releases/1.7/#schema-migrations
I'm using Django ( < v1.7), with SQLite3 as the database engine.
I am trying to add a new field to an already existing model's class. This class already has data associated to it.
manage.py syncdb is not adding the new fields to the db.
Is there was a way to:
Add the new Field to the already existing class, and make sure a db entry is allocated to it?
Populate this new field with an empty string for all the already existing db entries?
This answer is still getting visibility but is outdated. Since 1.7 Django ships with a built-in migration system, written by the same author as South, and has deprecated syncdb though it will still work.
You will simply need to run a few commands to automatically add new columns:
python manage.py makemigrations
python manage.py migrate
It will be useful to understand what's happening under the hood, but those are the basics. Please ask new questions to get answers on 1.7 and migrations if you are still reading this old post.
For django < 1.7
syncdb will not add any new columns. See
http://docs.djangoproject.com/en/dev/ref/django-admin/#syncdb
You will have to add them manually. For example,. replace <> with relevant info:
python manage.py dbshell
ALTER TABLE <appname_modelname> ADD COLUMN <column_type> DEFAULT '';
You can see what Django might have done to generate columns on a fresh syncdb by using:
python manage.py sqlall app_name
and copying and pasting ALTER TABLE statements from there.
Otherwise, you can look into third-party apps like Django-South which are database migration tools.
Install south in your django and you can easily handle the existing tables. check this
If you really want to use Django-South, install it on your django, after adding your new fields to your existing model run
python manage.py schemamigration --initial
It will create a file in your project app. then,
python manage.py migrate
thats it your table is altered.
There are other options of migration apps (although South is the most used).
I have used django-evolution for my projects and it was very easy to install and start using.
South seems to be more complete, but for simpler tasks, django-evolution may be suitable.