On our staging server we observed a runtime error stating a field is missing from a database,
column our_table.our_field does not exist
LINE 1: ...d"."type", "our_table"...
The field was added during a recent update with a complicated migration squashing process. It's possible that some errors were made during this process, but "manage.py showmigrations" command shows that the migration has been applied and "manage.py makemigrations" does not create any new migration. As we do not run tests on our staging or production databases, we are trying to figure out the most effective method for identifying such errors.
In short, how can we identify mismatches between the database and Django models caused by an incorrect migration like the following?
python manage.py migrate our_app --fake
I suppose I am looking for something like
python manage.py check_database
Edit: Many thanks for the suggestions. However, this is more of a deployment than a development question because the problem likely occurred when our devops tried to apply the squashed migrations while retaining data on the staging server (which will be case on production). It was scary to learn that such inconsistency can occur when makemigrations and showmigrations do not show any problem and can therefore also happen on production.
The bottom line is that we need some way to ensure our database matches our models after deployment.
Related
This is a Heroku-specific issue with a Django project 1.11.24 running Python 3.6.5 and a Heroku postgres database.
During testing of two different branches during development, different conflicting migration files were deployed at different times to the Heroku server. We recognized this, and have now merged the migrations, but the order the Heroku psql db schema was migrated is out of order with the current migration files.
As a result, specific tables already exist, so on deploy applying the updated merged migration files errs with:
psycopg2.errors.DuplicateTable: relation "table_foo" already exists
In heroku run python manage.py showmigrations -a appname all of the migrations are shown as having run.
We've followed Heroku's docs and done the following:
Rolled back the app itself to before when the conflicting migrations took place and were run (https://blog.heroku.com/releases-and-rollbacks)
Rolled back the postgres db itself to a datetime before when the conflicting migrations took place and were run (https://devcenter.heroku.com/articles/heroku-postgres-rollback)
However, despite both app and db rollbacks, when we check the db tables themselves in the rollback in pql shell with \dt, the table causing the DuplicateTable err still exists, so the db rollback doesn't actually seem to effect the django_migrations table.
It's Heroku, so we can't fake the migrations.
We could attempt to drop the specific db tables that already exist (or drop the entire db, since it's a test server), but that seems like bad practice. Is there any other way to address this in Heroku? thanks
I eventually fixed this by manually modifying migration files to align with the schema dependency order that was established. Very unsatisfying fix, wish Heroku offered a better solution for this (or a longer postgres database rollback window)
Django migrations has excellent behavior in terms of single migrations, where, assuming you leave atomic=True, a migration will be all-or-nothing: it will either run to completion or undo everything.
Is there a way to get this all-or-nothing behavior for multiple migrations? That is to say, is there a way to either run multiple migrations inside an enclosing transaction (which admittedly could cause other problems) or to rollback all succeeded migrations on failure?
For context, I'm looking for a single command or setting to do this so that I can include it in a deploy script. Currently, the only part of my deploy that isn't rolled back in the event of a failure are database changes. I know that this can be manually done by running python manage.py migrate APP_NAME MIGRATION_NUMBER in the event of a failure, but this requires knowledge of the last run migration on each app.
This isn't a feature in Django (at least as of 4.1). To do this yourself, your code needs to do the following:
Get a list of current apps that have migrations. This can be done by getting the intersection of apps from settings.INSTALLED_APPS and the apps in the MigrationRecorder.
Get the latest migration for each app.
Run migrations.
If migrations fail, run the rollback command for the latest migration for each app.
Here's an implementation of this as a custom management command: https://github.com/zagaran/django-migrate-or-rollback. This is done as a child-class of the migrate management command so that it exposes all of the command line options of migrate. To use this library, pip install django-migrate-or-rollback and add "django_migrate_or_rollback" to your INSTALLED_APPS in settings.py. Then you can use python manage.py migrate_or_rollback in place of the traditional migrate command.
Disclosure: I am the author of the library referenced (though I made the library based on this StackOverflow answer).
I have a Postgres database full of data. And I made several changes to my Django app models.
mange.py makemigrations worked fine and created the migration files. But manage.py migrate execute only one file. And when I launch it again it doesn’t execute the rest as if they are already applied.
I deleted the migration files that were not applied and did another makemigration but it says no changes detected.
Any ideas how to reflect the models changes on the database without losing the data ?
Thanks
Django keeps track of which migrations it has applied already, so when you run the migrate command it will execute only the migrations that Django thinks that are missing.
I deleted the migration files that were not applied and did another makemigration but it says no changes detected.
This was a bad idea, it will make your migrations inconsistent.
If you want to go back in time, instead of deleting migrations, the proper way to do this is by reverting migrations. You can use the same migrate command and specify to which migration point you want your database model to be.
Check this answer for further information about reverting migrations; django revert last migration
As I was changing my models.py and migrating, I got an error message saying:
python manage.py makemigrations project_profile
CommandError: Conflicting migrations detected; multiple leaf nodes in the migration graph: (0033_auto_20180217_0912, 0036_auto_20180217_0927 in project_profile).
To fix them run 'python manage.py makemigrations --merge'
So, when I tried to follow the instructions, I got another error that one of my tables that the merged migration is now depending on do not exist anymore (I renamed it). Interestingly enough, this renaming took place during the merge operation. So, really Django should have known about it in the first place.
To resolve the situation, I deleted prior migrations up to and including the migrations that was not applied, the one that caused all the headache. I tried to makemigrations and migrate again. But, Django now throws another error saying some of the models it wants to create in the database already exist. Obviously, I do not want to delete those tables and loose all that information to appease Django. So, I had to resort to some hacking solutions and actually change those tables manually and do a fake migration in order to stop Django from complaining.
Having said all of that, I feel like there should be a more logical way about this. How do I resolve migrations during the merging?
I had the same issue, Then I was able to solve this by deleting the migrations file that django pointed out and starts with name auto. It occurred 2-3 times before it finally gave up and finally worked.
Alternatively you can django-dbbackup or django-import-export packages to backup the tables then clean your database and migrations. Then you can restore them back to the same state once migrations are stable.
Sources
dbbackup : https://django-dbbackup.readthedocs.io/en/stable/
import-export : https://django-import-export.readthedocs.io/en/latest/index.html
After refactoring some models in a bloated app (everything in appname/models.py) into a subfolder application (some of the models in appname/subapp/models.py) and running makemigrations, I'm getting the following error when running manage.py migrate:
ValueError: Found wrong number (2) of constraints for appname_modelname1(modelname2_id)
Getting rid of all migrations and starting over would be one option, but then I'd have to manually edit all existing production databases. Are there any alternatives to make the migrations apply smoothly?
Turns out that ./manage.py makemigrations makes overly complex migrations that will likely fail on a real constraint-enforcing database. If the objective is just to get rid of models, first make the migrations, then edit the migration file so that only the RemoveModel directives remain, and finally apply the migrations.