I have Django webapplication running on Heroku with Heroku Postgres. A few days ago I made a few changes to an app's models, adding and then removing unique=True to some model fields (among other things, just messing around a bit) and making some migrations. This surely broke something, and unfortunately I didn't discover this until a few days later, after I had made a lot of additional migrations in my local db. The only way I was able to fix it was with a:
python manage.py migrate --fake
This worked out OK on my local/dev postgres, but in production it is still broken when I try to make migrations and migrate. When running migrate:
django.db.utils.ProgrammingError: table "publicinteraction_smoothmessage_extra_software" does not
exist
Since it has been a few days, I don't know exactly which migration I faked or not, and the section in Django docs about "risk of manually having to correct --fake" was also seen too late...
Will it help to run python manage.py migrate publicinteraction zero or will this result in my data getting lost?
This is my first project in production with real users and encountering such issues is all new to a self-taught developer in Python and Django. Poking around and messing up a production database feels a bit itchy and way too nerve-wracking for my skill level at the moment.
How should I proceed without losing all my data? Best practices to keep in mind for the future when it comes to making migrations localy vs production is also very much appreciated :)
Related
On our staging server we observed a runtime error stating a field is missing from a database,
column our_table.our_field does not exist
LINE 1: ...d"."type", "our_table"...
The field was added during a recent update with a complicated migration squashing process. It's possible that some errors were made during this process, but "manage.py showmigrations" command shows that the migration has been applied and "manage.py makemigrations" does not create any new migration. As we do not run tests on our staging or production databases, we are trying to figure out the most effective method for identifying such errors.
In short, how can we identify mismatches between the database and Django models caused by an incorrect migration like the following?
python manage.py migrate our_app --fake
I suppose I am looking for something like
python manage.py check_database
Edit: Many thanks for the suggestions. However, this is more of a deployment than a development question because the problem likely occurred when our devops tried to apply the squashed migrations while retaining data on the staging server (which will be case on production). It was scary to learn that such inconsistency can occur when makemigrations and showmigrations do not show any problem and can therefore also happen on production.
The bottom line is that we need some way to ensure our database matches our models after deployment.
So I started working on a project that is right now on production but the preovious dev kinda made a mess with migrations (or maybe I am wrong), the migrations in developement and in production are not the same (he gitignored them) and I can not start working because there are missing migrations in development so the database is no fully migrated.
Im thinking I can just delete all migrations in development, reset them but I don't think that is a good idea in production. What should I do?
Migrations have two purposes: 1. Creating schema of database, 2. Migrating existing data of database (For example when you change a field from IntegerField to CharField, you need to write some migrations to convert integers saved in database to their equal char.) This one is only needed for production. If you are missing these migrations, it is not a problem, Because you only need schema of database to develop. But how can you make sure you have the correct schema? Run python manage.py makemigrations and then python manage.py migrate. You are good to go and there is no need to delete any prior migrations.
I have accidently deleted one of migrations folders and and have no backup for it.
What are my options?
DB is postgres. Right now everything is OK.(I have moved instead migration folder I have on my DEV server with SQL lite) So I am just getting red message on server that not all migrations have been applied.
But next time if i run migration i will be in trouble.
What is my way out?
Migrations are mainly for backward compatibility, and tracking/versioning of the changes to models/database. If you do not really care about historical changes, etc.. then you can just delete the migrations directory and do:
python manage.py makemigrations <app_name>
This creates the initial migrations (like starting from a clean slate) - and moving forward you can track the historical migrations moving forward. More on this can be read here
Now, when you run the migrations, you can do
python manage.py migrate <app_name> --fake-initial
to fake the initial migration.
Now might be a good time to have a version control on your application
Use version control.
You are not the first developer to delete important files, but often recovery takes less than a second - thanks to version control systems (also called revision control systems). Please stop everything else and install and use one of Git, Mercury or Subversion.
Don't use FTP
It's totally. I mean totally insecure. Always use SFTP
Don't use sqlite3 for local with other db for production
sqlite doesn't enforce strict type checking. Postgresql on the other hand is very particular about it. Additionally sqlite only has a subset of the functionality that's found on postgresql. Last but not least different RDBMS have different intricacies. If you use one locally and another in production, there is always a chance that your code will break when you deploy to live
Managing without the migration files
This is not a big loss as long as your database is in sync with your models.
If you database is not in sync with your models, you can use
./manage.py inspectdb
to recreate local models that represent the actual structure in the db. Then you do makemigrations and migrate (as explained by karthik) on the generated models.
Then replace that with your live models and do the step again.
The application I'm developing is starting to need migration for the database schemas. I've thought about django-south, but since I have no experience with that kind of stuff I'm a bit lost, this is my situation:
Development code: latest models, I didn't keep track of what changes I've made to the models.
Production code: running code, has old models. We have the server configured so we can make deployments just with a git pull :)
How can I update the code in production (and the DB) without breaking anything? I saw about the --initial statement but I don't think it works for this case, and also for convert_to_south to fake a migrationhistory, but I still don't get what should I do. Any help please?
Imo it would be better to create versioning for your project and deploy it with something like Fabric. This will contain your production environment nicely.
There's no magic with south, just add south to the installed apps setting and run an initial schemamigration then run a fake migration (migrate <app_name> --fake) so south 'knows' the current state of your models. In future releases (that contain schema changes) you can run schemamigration <app_name> --auto and migrate <app_name> to update the models accordingly.
To keep your migrations in one place use the migrations setting in your settings file:
SOUTH_MIGRATION_MODULES = {
'app_name_1': 'project_name.migrations.page',
'app_name_2': 'project_name.migrations.medialibrary',
}
You'll have to checkout the production version (to get the models back to production state), create an initial migration, copy that migration to your current development branch then create a schemamigration.
I'm using Django South for managing my database schema updates. As I'm currently developing locally, my models are changing a lot, and it's really annoying to change things with South:
$ bin/django schemamigration --auto core
(Please provide a default value for new field...)
...
$ bin/django migrate core
It often takes forever to do simple things like add and remove columns from the database, as South prompts me to provide default values even for columns I'm deleting.
Is there a setting that will cause South to work much more like Hibernates hibernate.hbm2ddl.auto setting and automatically, promptlessly, awesomely update my database schema without any fuss? Is there an alternative library for doing this?
South is really important to me for deployment migrations, but I need something to help me change things fast as I'm rapidly prototyping things.
I use fabric to help with local and production changes. This is a function in my fabfile.py. It helps with any changes I want to make.
def run_local():
"""
Installs requirements, syncs the database, migrates with south, and runs the server.
"""
local('pip install -r conf/requirements.txt')
local('python manage.py syncdb')
local('python manage.py migrate')
local('python manage.py runserver')