Django migrations has excellent behavior in terms of single migrations, where, assuming you leave atomic=True, a migration will be all-or-nothing: it will either run to completion or undo everything.
Is there a way to get this all-or-nothing behavior for multiple migrations? That is to say, is there a way to either run multiple migrations inside an enclosing transaction (which admittedly could cause other problems) or to rollback all succeeded migrations on failure?
For context, I'm looking for a single command or setting to do this so that I can include it in a deploy script. Currently, the only part of my deploy that isn't rolled back in the event of a failure are database changes. I know that this can be manually done by running python manage.py migrate APP_NAME MIGRATION_NUMBER in the event of a failure, but this requires knowledge of the last run migration on each app.
This isn't a feature in Django (at least as of 4.1). To do this yourself, your code needs to do the following:
Get a list of current apps that have migrations. This can be done by getting the intersection of apps from settings.INSTALLED_APPS and the apps in the MigrationRecorder.
Get the latest migration for each app.
Run migrations.
If migrations fail, run the rollback command for the latest migration for each app.
Here's an implementation of this as a custom management command: https://github.com/zagaran/django-migrate-or-rollback. This is done as a child-class of the migrate management command so that it exposes all of the command line options of migrate. To use this library, pip install django-migrate-or-rollback and add "django_migrate_or_rollback" to your INSTALLED_APPS in settings.py. Then you can use python manage.py migrate_or_rollback in place of the traditional migrate command.
Disclosure: I am the author of the library referenced (though I made the library based on this StackOverflow answer).
Related
On our staging server we observed a runtime error stating a field is missing from a database,
column our_table.our_field does not exist
LINE 1: ...d"."type", "our_table"...
The field was added during a recent update with a complicated migration squashing process. It's possible that some errors were made during this process, but "manage.py showmigrations" command shows that the migration has been applied and "manage.py makemigrations" does not create any new migration. As we do not run tests on our staging or production databases, we are trying to figure out the most effective method for identifying such errors.
In short, how can we identify mismatches between the database and Django models caused by an incorrect migration like the following?
python manage.py migrate our_app --fake
I suppose I am looking for something like
python manage.py check_database
Edit: Many thanks for the suggestions. However, this is more of a deployment than a development question because the problem likely occurred when our devops tried to apply the squashed migrations while retaining data on the staging server (which will be case on production). It was scary to learn that such inconsistency can occur when makemigrations and showmigrations do not show any problem and can therefore also happen on production.
The bottom line is that we need some way to ensure our database matches our models after deployment.
I have a django REST project and a PostgreSQL database deployed to DigitalOcean. When I develop locally, I have a separate dockerized REST server and a separate PostgreSQL database to test backend features without touching production data.
My question arises when I'm adding/modifying model fields that require me to make migrations using python [manage.py](https://manage.py) makemigrations and python [manage.py](https://manage.py) migrate command. Here is my current situation so far:
What I was supposed to do
IN LOCAL ENV, to create the migration files,
python manage.py makemigrations
python manage.py migrate
Now commit these newly created files, something like below.
git add app/migrations/...
git commit -m 'add migration files' app/migrations/...
IN PRODUCTION ENV, run only the below command.
python manage.py migrate
What I did so far
IN LOCAL ENV, created the migration files,
python manage.py makemigrations
python manage.py migrate
I committed & pushed the changes to production WITHOUT the created migration file
IN PRODUCTION ENV, ran BOTH commands.
python manage.py makemigrations
python manage.py migrate
The production server successfully added the isActive field to the database and is working fine, but I still have a 0011_user_isActive.py migration file in my local changes that hasn't been staged/committed/pushed to github repo.
And because I ran makemigrations command in production env, it probably created the same migration file that I haven't pushed from local env.
My questions are:
What happens if I push the local migration file to production? Wouldn't it create a conflict when I run migration command on digitalocean console in the future?
How should I fix this situation?
I am just scared I'm going to corrupt/conflict my production database as I'm very inexperienced in databases and have too much to risk at the moment. Would appreciate any tips on best practices when dealing with such situations!
As docs says:
The migration files for each app live in a “migrations” directory inside of that app, and are designed to be committed to, and distributed as part of, its codebase. You should be making them once on your development machine and then running the same migrations on your colleagues’ machines, your staging machines, and eventually your production machines.
So it's best practice is to push your migration files and make your local and production migration files sync.
And if you got conflict when pushing migraions files and pulling them, the makemigrations --merge command is for solving that.
Also docs says:
Because migrations are stored in version control, you’ll occasionally come across situations where you and another developer have both committed a migration to the same app at the same time, resulting in two migrations with the same number.
Don’t worry - the numbers are just there for developers’ reference, Django just cares that each migration has a different name. Migrations specify which other migrations they depend on - including earlier migrations in the same app - in the file, so it’s possible to detect when there’s two new migrations for the same app that aren’t ordered.
When this happens, Django will prompt you and give you some options. If it thinks it’s safe enough, it will offer to automatically linearize the two migrations for you. If not, you’ll have to go in and modify the migrations yourself - don’t worry, this isn’t difficult, and is explained more in Migration files below.
Also be aware that in case of updating existed data in production, you can use RunPython in migration file. Read about it here.
I am considering to add the command release: python manage.py migrate --no-input to my deployment, so Heroku is automatically migrating, once I push my repository. I now wonder if --no-input is a 'good' idea?
One more specific question: If migrate normally asks me 'Did you rename this field'. With --no-input, will it answer automatically yes? I couldn't find much detailed information in the official Django documentation.
We don't use the interactive flag (--no-input) in our pipeline and to the best of my knowledge it doesn't make much difference. If you inspected Django source code, you would find out that the interactive flag only falls through to the pre_migrate and post_migrate signals that are emitted prior to and post migration.
As far as I know, no internal Django app uses the interactive argument inside these signals. Some external packages perhaps could use that, but I personally have never encountered that.
As for your question:
One more specific question: If migrate normally asks me 'Did you rename this field'. With --no-input, will it answer automatically yes? I couldn't find much detailed information in the official Django documentation.
This happens in the makemigrations management command, not in migrate. You usually do (and should do) the former locally, so no need to include that in your deployment pipeline.
Background :-
I am using Django 1.3. We are using South as the module for DB migration and Git SCM.
Problem:-
What is the correct way to deal with the migrations Folder that is formed?
The main problem is I make changes in the DB schema in the development machine, when I upload it to the production server I have to migrate the existing schema. While doing that there is always some issue with the migration files.
Should I just add the migrations folder to the gitignore ? or is there a better way to go about it ?
You should add the migrations folder to your version control system and use the same files for production and development. You may run into some problems on your production system if you introduced your migrations not from the beginning and you have already existing tables.
Therefore you have to fake the first migration, which normally does the same thing as syncdb did when you created your database for the first time.
So when trying to apply migrations for your app for the first time on the production machine, execute manage.py migrate app_name 0001 --fake. This lets South know, that the first migration has already been applied (which already happend with syncdb) and when you run migrate again, it would continue with the following migrations.
how can i migrate entiry db at one step? south`s startmigration command can work only with single application
If you want to create a migration, manage.py startmigration is deprecated (see manage.py help startmigration), you should be using manage.py schemamigration (as described in the docs), and you should definitely be doing this individually on each app.
If you want to run migrations (in other words do the actual altering of the database, which I'm guessing is the case), The command for that is manage.py migrate which, if run without any arguments, will migrate all of your apps to the latest migration available.
My deployment script just has manage.py migrate, and it works fine without manual intervention, no matter how many apps have new migrations that need to be run.
Even with raw SQL, you wont be able to migrate an entire database in a single step as you need a query per table. You can however, create migrations for all apps, and then run them all at once. That's the closest thing you'll come to a one step migration.