How would I rerun a data migration on Django 1.8+? If relevant, my migration is numbered 0011_my_data_migration.py and is the latest migration.
Fake back to the migration before the one you want to rerun.
./manage.py migrate --fake yourapp 0010_my_previous_data_migration
Then rerun the migration.
./manage.py migrate yourapp 0011_my_data_migration
Then you can fake back to the most recent migration that you have run. In your case, you said that 0011 was the latest, so you can skip this stage.
./manage.py migrate --fake yourapp 0014_my_latest_data_migration
Note that depending on the state of your database and the contents of the migrations, rerunning a migration like this might cause errors. Note the warning in the docs about the --fake option:
This is intended for advanced users to manipulate the current migration state directly if they’re manually applying changes; be warned that using --fake runs the risk of putting the migration state table into a state where manual recovery will be needed to make migrations run correctly.
Alasdair's answer gives a disclaimer about this, but faking a migration back to the previous one is only safe if your migration is idempotent, which means you can run it multiple times without side effects like duplicate data. Most people don't write their migrations this way, but it's good practice.
You have two options to make this process safe:
Make your data migrations idempotent. This means that any created data is either reused (like with the Model.objects.get_or_create() method) or deleted and recreated. Reused is the better option, as deleting and recreating will change database indexes and sequences.
Make reverse data migrations. You can do this by passing 2 functions to migrations.RunPython(). For example, if you have migrations.RunPython(add_countries), you would change that to migrations.RunPython(add_countries, remove_countries) and delete any relevant countries in the second function.
If you choose option #2 then you would run:
./manage.py migrate yourapp 0010_my_previous_data_migration
./manage.py migrate yourapp 0011_my_data_migration
If you wanted to make that a one liner so that you can use it over and over:
./manage.py migrate yourapp 0010_my_previous_data_migration && ./manage.py migrate yourapp 0011_my_data_migration
Based on the accepted answer, here's a script for reapplying a given migration.
#! /bin/bash
# This script re-applies a given migration.
app_name="$1"
migration_index="$2"
prev_migration_index="$(echo "$migration_index" | sed 's/^0*//' | awk '{ print $1 - 1 }' | xargs printf "%04d")"
last_migration_index="$(django-admin showmigrations --plan | grep -oP "\.\K\d{4}" | sort | tail -n 1)"
# fake-migrate to the migration prior to the one we want to reapply
django-admin migrate --fake "$app_name" "$prev_migration_index"
# reapply the migration
django-admin migrate "$app_name" "$migration_index"
# fake-migrate to the last migration
django-admin migrate --fake "$app_name" "$last_migration_index"
Usage:
$ bash reapply_migration.sh <app_name> <migration_to_reapply_index>
Related
I'm using django 1.9.6. I recently deleted my migrations and ran migrate --run-syncdb and makemigrations my_app. Today I added a new field to one of my models:
models.py:
value = models.PositiveSmallIntegerField(null=True)
I tried to migrate the changes, but makemigrations doesn't detect the change. It's just the development version, so I can resync (I don't have to preserve the data), but running --run-syncdb again doesn't detect it either.
Why isn't this migrating?
Delete every past migration files and __pycache__ files except __init__
then:
python manage.py makemigrations yourApp
After that make sure that the db is the same as the code in model.py (remove new changes) and run next line:
python manage.py migrate --fake-initial
Now add all your changes in the model.py and run next lines:
python manage.py makemigrations
python manage.py migrate
Best Regards,
Kristian
I had the same issue. I realised I had a property defined on the model with the same name as a field I was trying to add on the model. Ensure the model doesn't have a model property/method with the same name as the field you are trying to add.
You should not delete migrations, you should squash them. If you simply deleted the files you likely messed things up, the easiest way to recover is re-sync your code to get the files back. A more complex route is to delete all the records from the django_migrations table and re-init the migrations from scratch but there is more steps/issues than I can really get into and I don't recommend it.
The reason makemigrations is not detecting the change is likely because there is no migrations folder in that app. If you run python manage.py makemigrations your_app --initial it might detect and generate the migrations or it might freak out because of the difference in your files and the django_migrations table.
The --run-syncdb command is great when you don't care about the data, usually before you actually deploy but once you start using migrations you should not use the --run-syncdb command anymore. For instance, during initial development here is the code I run every model change instead of dealing with migrations:
dropdb mydb && createdb mydb && python manage.py migrate --run-syncdb && python manage.py loaddata initial
I store all the initial data in a fixtures file and the command wipes out the entire database, the --run-syncdb rebuilds the schema, and the initial data is loaded while skipping actual migration files.
So, if you don't care about any of your data or can easily move it to a fixture than you can drop and re-create the DB. You are then free to delete all migration folders and you can use the command above until you go live and need to move to using migrations.
*** UPDATE FOR DJANGO 1.11 *** STILL USING ON 3.2.5 ***
I started using Django 1.11 and noticed the test framework can fail if you have dependencies on framework models and don't actually have migrations. This is the new command i'm using to wipe everything out and start over while still developing.
set -e
dropdb yourdb
createdb yourdb
find . -name "migrations" -type d -prune -exec rm -rf {} \;
python manage.py makemigrations name every app you use seperated by space
python manage.py migrate
python manage.py loaddata initial
I put this in a builddb.sh at the root of my project (next to manage.py) so I can just run ./builddb.sh. Be sure to delete it on deploy so there is no accidents!
Try creating a migrations folder that contains an empty __init__.py file inside your app folder if there is no migrations folder already. And make migrations.
In my case, I created a new project and created an app. The python manage.py makemigrations returned no change detected although I had added a handful of models.
Found out that you need to manually add the name of your app in the INSTALLED_APPS list inside your settings.py file. Unless you do that, no change in the migration of that app can be detected.
If you're deleting a model, and expecting the changes to be picked up in migrations, ensure that the model doesn't have a *.pyc still lying around.
Another case is abstract classes. Make sure that your model’s metaclass does not have the abstract = True property.
Using django 1.7.7 I want to use django's migration to add or remove a field.
so I modified model.py and ran
python manage.py makemigrations myproj
Migrations for 'myproj':
0001_initial.py:
- Create model Interp
- Create model InterpVersion
python manage.py migrate myproj
Operations to perform:
Apply all migrations: myproj
Running migrations:
Applying myproj.0001_initial... FAKED
python manage.py runserver
Then checked the admin page, it is not updated.
Then I tried removing the migration folder and tried again; the migrate command says there are no migrations to apply.
How can I do the migration?
Note: I want to use the new technique using django's migration not the old south approach.
Make sure that the migrations/ folder contains a __init__.py file
Lost half an hour over that.
Deleting the migration directory is never a good idea, because Django then loses track of which migration has been applied and which not (and once an app is deployed somewhere it can become quite difficult to get things back in sync).
Disclaimer: whenever things like that occur it is best to backup the database if it contains anything valuable. If in early development it is not necessary, but once things on the backend get out of sync there is a chance of things getting worse. :-)
To recover, you could try resetting your models to match exactly what they were before you have added/removed the fields. Then you can run
$ python manage.py makemigrations myproj
which will lead to an initial migration (0001_initial...). Then you can tell Django to fake that migration, which means to tell it to set its internal counter to this 0001_initial:
With Django 1.7:
$ python manage.py migrate myproj
With Django >= 1.8:
$ python manage.py migrate myproj --fake-initial
Now, try to change your model and run makemigrations again. It should now create a 0002_foobar migration that you could run as expected.
In my case, the migrations were not being reflected in mysql database. I manually removed the row of 'myapp'(in your case 'myproj') from the table 'django_migrations' in mysql database and ran the same commands again for migration.
Most of the above solutions would help in the issue, however, I wanted to point out another possible (albeit rare) possibility that the allow_migrate method of database router may be returning False when it should have returned None.
Django has a setting DATABASE_ROUTERS which will be used to determine which database to use when performing a database query.
From the docs:
if you want to implement more interesting database allocation behaviors, you can define and install your own database routers.
A database router class implements up to four methods:
db_for_read(model, **hints)
db_for_write(model, **hints)
allow_relation(obj1, obj2, **hints)
allow_migrate(db, app_label, model_name=None, **hints)
From the documentation:
allow_migrate(db, app_label, model_name=None, **hints)
Determine if the migration operation is allowed to run on the database with alias db. Return True if the operation should run, False if it shouldn’t run, or None if the router has no opinion.
It is possible that one of the database routers in sequence is returning False for the migration that you're trying to run, in which case the particular operation will not be run.
I find Django migrations a bit of a mystery, and tend to prefer external tools (liquibase, for example).
However, I just ran into this "No migrations to apply" problem as well. I also tried removing the migrations folder, which doesn't help.
If you've already removed the migrations folder, here's an approach that worked for me.
First, generate the new "clean" migrations:
$ python manage.py makemigrations foo
Migrations for 'foo':
dashboard/foo/migrations/0001_initial.py
- Create model Foo
- Create model Bar
Then look at the SQL and see if it looks reasonable:
$ python manage.py sqlmigrate foo 0001
BEGIN;
--
-- Create model Foo
--
CREATE TABLE "foo" ("id" serial NOT NULL PRIMARY KEY, ... "created_at" timestamp with time zone NOT NULL, "updated_at" timestamp with time zone NOT NULL);
CREATE INDEX "..." ON "foo" (...);
COMMIT;
Then apply execute that same SQL on your database.
I'm using Postgres but it will be similar for other engines.
One way is to write the content to a file:
$ python manage.py sqlmigrate foo 0001 > foo.sql
$ psql dbname username < foo.sql
BEGIN
CREATE TABLE
CREATE INDEX
COMMIT
Another is pipe the SQL directly:
$ python manage.py sqlmigrate foo 0001 | psql dbname username
Or copy and paste it, etc.
pip install django-extensions
and add this in the install app of settings.py
INSTALLED_APPS = [
'django_extensions'
]
Then run
python ./manage.py reset_db
Then run migrations again
python manage.py makemigrations
python manage.py migrate
Now, run migrations for your installed apps
python manage.py makemigrations your_app_name
python manage.py migrtate your_app_name
Done! See Your Database...
In addition to the other answers, make sure that in models.py, you have managed = True in each table's meta
You can remove your db
python manage.py makemigrations
python manage.py migrate
python manage.py migrate --run-syncdb
and see your data base this is working :)
Similar to Andrew E above but with a few changes especially where you haven't deleted the migrations folder in your quest to resolve the issue
1 - In your intact migration folder just examine the 000*.py files counting from the highest down to initial.py till you find the one where your Model is defined, say 0002_entry.py
2 - python manage.py sqlmigrate app-name 0002 > 0002_sql.txt to capture the SQL commands
3 - Edit this file to ensure there are no hard CR/LFs and the ALTER, CREATE INDEX commands are each on own single line
4 - Log into your DB (I have Postgres) and run these commands
In Database delete row myproj from the table django_migrations.
Delete all migration files in the migrations folder.
Then run python manage.py makemigrations and python manage.py migrate commands.
Are there django commands that
A. Delete all tables
B. delete all data in all tables
C. Create all tables as defined in the model?
I cannot find these right now!
And by commands i mean those little things that are like
runserver
etc
A. Delete all tables
manage.py sqlclear will print the sql statement to drop all tables
B. delete all data in all tables
manage.py flush returns the database to the state it was in immediately after syncdb was executed
C. Create all tables as defined in the model?
manage.py syncdb Creates the database tables for all apps in INSTALLED_APPS whose tables have not already been created.
See this page for a reference of all commands: https://docs.djangoproject.com/en/dev/ref/django-admin/
But you should definitely look into using south as someone already mentioned. It's the best way to manage your database.
N.B: syncdb is deprecated in favour of migrate, since Django 1.7.
If you have the client libraries installed for your database, you can run this:
python manage.py sqlflush | python manage.py dbshell
This doesn't drop the tables, but truncates them.
There isn't a command that does the it all in one go, but this "one liner" will drop all the tables and then re-create them. It would only work if you were running on a system that provides these utilities at the shell.
echo 'from django.conf import settings; print settings.INSTALLED_APPS; quit();' | python manage.py shell --plain 2>&1 | tail -n1 | sed -r "s|^.*\((.*)\).*$|\1|; s|[',]| |g; s|django\.contrib\.||g" | xargs python manage.py sqlclear | python manage.py dbshell && python manage.py syncdb
Neither manage.py sqlclear nor manage.py reset is capable of dropping all tables at once, both require an appname parameter.
You should take a look at Django Extensions, it gives you access to manage.py reset_db as well as many other useful management commands.
I recommend using django-south. It allows you to sync your models to your database not just when you add a field, but also when you delete a field/model. I really find it to be an essential component of building a Django site. Once installed, you could run a command like:
./manage.py migrate app zero
You can learn more here: http://south.aeracode.org/docs/about.html
And a simpler oneliner to drop all the tables for django 1.5+:
python2 manage.py sqlflush | sed 's/TRUNCATE/DROP TABLE/g'| python2 manage.py dbshell
I was able todrop my tables. Run
python manage.py sqlclear appname
Take note of the commands given. Then run
python manage.py dbshell
Add the commands given in the first step. line by line. When you are done, type '.exit' (for SQlite3). Resync your DB and you should be good to go. To be sure, check the tables with:
python manage.py shell
>>> from yourapp import yourclasses
>>> yourviews.objects.all()
it should return a []. I hope this helps.
A implies B, right?
For A, see How to drop all tables from the database with manage.py CLI in Django?
For C,
python manage.py syncdb
Ofcourse for smart data migration I go with what #bento mentioned: django-south
I've accumulated quite a few migrations using South (0.7) and Django (1.1.2) which are starting to consume quite a bit of time in my unit tests. I would like to reset the baseline and start a fresh set of migrations. I've reviewed the South documentation, done the usual Google/Stackoverflow searching (e.g. "django south (reset OR delete OR remove) migration history") and haven't found anything obvious.
One approach I've contemplated would involve "starting over" by "removing" South or "clearing" the history manually (e.g. clear the db table, remove migration files from the migrations director) and just re-run,
./manage.py schemamigration southtut --initial
So, if anyone has done this before and has some tips/suggestions they would be greatly appreciated.
If you need to selectively (for just one app) reset migrations that are taking too long, this worked for me.
rm <app-dir>/migrations/*
python manage.py schemamigration <app-name> --initial
python manage.py migrate <app-name> 0001 --fake --delete-ghost-migrations
Don't forget to manually restore any dependencies on other apps by adding lines like depends_on = (("<other_app_name>", "0001_initial"),("<yet_another_app_name>", "0001_initial")) to your <app-dir>/migrations/0001_initial.py file, as the first attribute in your migration class just below class Migration(SchemaMigration):.
You can then ./manage.py migrate <app-name> --fake --delete-ghost-migrations on other environments, per this SO answer. Of course if you fake the delete or fake the migrate zero you'll need to manually delete any left-over db tables with a migration like this.
A more nuclear option is to ./manage.py migrate --fake --delete-ghost-migrations on the live deployment server followed by a [my]sqldump. Then pipe that dump into [my]sql on the environments where you need the migrated, fully-populated db. South sacrilege, I know, but worked for me.
EDIT - I'm putting a comment below at the top of this as it's important to read it before the > accepted answer that follows #andybak
#Dominique: Your advice regarding manage.py reset south is dangerous
and may destroy the database if there are any third party apps using
south in the project, as pointed out by #thnee below. Since your
answer has so many upvotes I'd really appreciate it if you could edit
it and add at least a warning about this, or (even better) change it
to reflect #hobs approach (which is just as convenient, but doesn't
affect other apps) - thanks! – chrisv Mar 26 '13 at 9:09
Accepted answer follows below:
First, an answer by the South author:
As long as you take care to do it on all deployments simultaneously, there shouldn't be any problem with this. Personally, I'd do:
rm -r appname/migrations/
./manage.py reset south
./manage.py convert_to_south appname
(Notice that the “reset south” part clears migration records for ALL apps, so make sure you either run the other two lines for all apps or delete selectively).
The convert_to_south call at the end makes a new migration and fake-applies it (since your database already has the corresponding tables). There's no need to drop all the app tables during the process.
Here's what I'm doing on my dev + production server when I need to get rid of all these unneeded dev migrations:
Make sure we have the same DB schema on both sides
delete every migrations folder on both sides
run ./manage.py reset south (as the post says) on both sides = clears the south table *
run ./manage.py convert_to_south on both sides (faking 0001 migration)
then I can re-start to make migrations and push the migrations folders on my server
* except if you want to clean only one app among others, if so you'll need to edit your south_history table and delete only the entries about your app.
Thanks to the answers by Dominique Guardiola and hobs, it helped me solve a hard problem.
However there are a couple of issues with the solution, here is my take on it.
Using manage.py reset south is not a good idea if you have any third party apps that uses South, for example django-cms (basically everything uses South).
reset south will delete all migration history for all apps that you have installed.
Now consider that you upgrade to the latest version of django-cms, it will contain new migrations like 0009_do_something.py. South will surely be confused when you try to run that migration without having 0001 through 0008 in the migration history.
It is much better/safer to selectively reset only the apps that you are maintaining.
First of all, make sure that your apps don't have any desync between migrations on disk, and migrations that have been executed on the database. Otherwise there will be headache.
1. Delete migration history for my apps
sql> delete from south_migrationhistory where app_name = 'my_app';
2. Delete migrations for my apps
$ rm -rf my_app/migrations/
3. Create new initial migrations for my apps
$ ./manage.py schemamigration --initial my_app
4. Fake execute the initial migrations for my apps
This inserts the migrations into south_migrationhistory without touching actual tables:
$ ./manage.py migrate --fake my_app
Step 3 and 4 is actually just a longer variant of manage.py convert_to_south my_app, but I prefer that extra control, in such delicate situation as modifying the production database.
Like thnee (see her answer), we're using a gentler approach to the South author's (Andrew Godwin) suggestion quoted elsewhere here, and we're separating what we do with the code base from what we do to the database, during deployment, because we need the deployments to be repeatable:
What we do in the code:
# Remove all the migrations from the app
$ rm -fR appname/migrations
# Make the first migration (don't touch the database)
$ ./manage.py schemamigration appname --initial
What we do to the database once that code is deployed
# Fake the migration history, wiping out the rest
$ ./manage.py migrate appname --fake --delete-ghost-migrations
If you are just working on the dev machine, I wrote a management command that does pretty much what Dominique suggested.
http://balzerg.blogspot.co.il/2012/09/django-app-reset-with-south.html
In contrast of the south author suggestion, this will NOT HARM other installed apps using south.
Following is only if you want to reset all apps. Please backup your all databases prior to that work. Also note down your depends_on in initial files if there are any.
For once:
(1) find . -type d -name migrations -exec git rm -rf '{}' \;
(2) find . -type d -name migrations -exec rm -rf '{}' \;
(3) ./manage.py schemamigration <APP_NAME> --initial
(4) [GIT COMMIT]
Test bootstrapping your project before push. Then, for each local/remote machine, apply following:
(5) [GIT PULL]
(6) ./manage.py reset south
(7) ./manage.py migrate --fake
Do initial (3) for each app you want to re-involve. Note that, reset (6) will delete only migration history, therefore not harmful to libraries. Fake migrations (7) will put back migration history of any 3rd party apps installed.
delete necessary file from app folder
instance path
cd /usr/local/lib/python2.7/dist-packages/wiki/south_migrations
wiki -is my app
I am trying to get started with South. I had an existing database and I added South (syncdb, schemamigration --initial).
Then, I updated models.py to add a field and ran ./manage.py schemamigration myapp --auto. It seemed to find the field and said I could apply this with ./manage.py migrate myapp. But, doing that gave the error:
django.db.utils.DatabaseError: table "myapp_tablename" already exists
tablename is the first table listed in models.py.
I am running Django 1.2, South 0.7
since you already have the tables created in the database, you just need to run the initial migration as fake
./manage.py migrate myapp --fake
make sure that the schema of models is same as schema of tables in database.
Although the table "myapp_tablename" already exists error stop raising
after I did ./manage.py migrate myapp --fake, the DatabaseError shows
no such column: myapp_mymodel.added_field.
Got exactly the same problem!
1.Firstly check the migration number which is causing this. Lets assume it is: 0010.
2.You need to:
./manage.py schemamigration myapp --add-field MyModel.added_field
./manage.py migrate myapp
if there is more than one field missing you have to repeat it for each field.
3.Now you land with a bunch of new migrations so remove their files from myapp/migrations (0011 and further if you needed to add multiple fields).
4.Run this:
./manage.py migrate myapp 0010
Now try ./manage.py migrate myapp
If it doesn't fail you're ready. Just doublecheck if any field's aren't missing.
EDIT:
This problem can also occur when you have a production database for which you install South and the first, initial migration created in other enviorment duplicates what you already have in your db. The solution is much easier here:
Fake the first migration:
./manage migrate myapp 0001 --fake
Roll with the rest of migrations:
./manage migrate myapp
When I ran into this error, it had a different cause.
In my case South had somehow left in my DB a temporary empty table, which is used in _remake_table(). Probably I had aborted a migration in a way I shouldn't have. In any case, each subsequent new migration, when it called _remake_table(), was throwing the error sqlite3.pypysqlite2.dbapi2.OperationalError: table "_south_new_myapp_mymodel" already exists, because it did already exist and wasn't supposed to be there.
The _south_new bit looked odd to me, so I browsed my DB, saw the table _south_new_myapp_mymodel, scratched my head, looked at South's source, decided it was junk, dropped the table, and all was well.
If you have problems with your models not matching your database, like #pielgrzym, and you want to automatically migrate the database to match the latest models.py file (and erase any data that won't be recreated by fixtures during migrate):
manage.py schemamigration myapp --initial
manage.py migrate myapp --fake
manage.py migrate myapp zero
manage.py migrate myapp
This will only delete and recreate database tables that exist in your latest models.py file, so you may have garbage tables in your database from previous syncdbs or migrates. To get rid of those, precede all these migrations with:
manage.py sqlclear myapp | manage.py sqlshell
And if that still leaves some CRUFT lying around in your database then you'll have to do an inspectdb and create the models.py file from that (for the tables and app that you want to clear) before doing the sqlclear and then restore your original models.py before creating the --initial migration and migrating to it. All this to avoid messing around with the particular flavor of SQL that your database needs.
Perform these steps in order may help you:
1) python manage.py schemamigration apps.appname --initial
Above step creates migration folder as default.
2) python manage.py migrate apps.appname --fake
generates a fake migration.
3) python manage.py schemamigration apps.appname --auto
Then you can add fields as you wish and perform the above command.
4) python manage.py migrate apps.appname
If you have an existing database and app you can use the south conversion command
./manage.py convert_to_south myapp
This has to be applied before you do any changes to what is already in the database.
The convert_to_south command only works entirely on the first machine you run it on. Once you’ve committed the initial migrations it made into your VCS, you’ll have to run ./manage.py migrate myapp 0001 --fake on every machine that has a copy of the codebase (make sure they were up-to-date with models and schema first).
ref: http://south.readthedocs.org/en/latest/convertinganapp.html
As temporary solution, you can comment the Table creation in the migration script.
class Migration(migrations.Migration):
dependencies = [
(...)
]
operations = [
#migrations.CreateModel(
# name='TABLE',
# fields=[
# ....
# ....
# ],
#),
....
....
Or
If the existing table contains no rows (empty), then consider deleting the table like below. (This fix is recommended only if the table contains no rows). Also make sure this operation before the createModel operation.
class Migration(migrations.Migration):
dependencies = [
(...),
]
operations = [
migrations.RunSQL("DROP TABLE myapp_tablename;")
]
One more solution(maybe a temporary solution).
$ python manage.py sqlmigrate APP_NAME MIGRATION_NAME
eg.,.
$ python manage.py sqlmigrate users 0029_auto_20170310_1117
This will list all the migrations in raw sql queries. You can pick the queries which you want to run avoiding the part which creates the existing table