I am facing a challenge here. So I inhertied the models from previous developers and the tables were not properly built. I added some constraints and new tables in order to normalize those tables. Before pushing the application to the heroku I tested it on my local machine and it actually broke my database.
Now the heroku website is already in production, so there are user information. How should i approach this, do I need to destroy the existing database and create a new one and run the migrations
Be very, very careful. Applying migrations on production servers can cause irreversible damage if you are not careful, and so you should be prepared for every possible situation.
My best recommendation would be to create an entire duplicate copy of your live DB (using Heroku this is as simple as a PG dump/backup). You can then create a new staging site using the same code, upload the backup into a new Database instance, and then test against that. Live environments are not always the same as local ones. You can then run your migrations on the staging site, and see if there are any unexpected effects (the best way to do this would be by utilizing django test cases). If there are any issues, be sure to understand how the rollback process works with django migrations.
A good tutorial that is fairly recent can be found here: https://realpython.com/django-migrations-a-primer/
Related
I'm using Django and Postgresql to develop a web service.
Suppose we've 3~4 branch which for the different features or old-version bugfix purpose.
Then, I met a problem, when I was in branch A and change django model, and run migrate to change database in my local test desktop.
When I switch to another branch which has no migration file, database will inconsistent and cannot work when I try to run django, I've to delete the database and recreate it.
In general, what's the best/common way to deal with this kind demands for developer environment?
I understand your situation well and have been in same shoe several times.
Here is what I prefer(/do):
I am in branch bug-fix/surname_degrade
I changed the user data model [which generated user_migration_005] and then migrated the DB.
Then my boss came and pointed out that the user is not able to login due to login degrade.
So I have to switch branch and fix that first.
I can rollback the migration[user_migration_005] which I have done few moments back. With something like this python manage.py migrate user_migration_004
Switched branch and started working on hot-fix/login_degrade
When I switch back to my previous task , I can just do migration and proceed.
With this procedure I don't need to delete my all tables or restore old database or anything like that.
I am a newbie, will be extremely happy to hear your thoughts.
The major issue here is that, you database will change everytime You migrate,so either you mantain you database consistency among different branches, or You can do One thing, while using/testing (after declaring all the models)
1) Delete all database tables ( If you have a backup or dummy data )
2) Delete all existing migration files in you branch
3) Create new migrations
4) Migrate to new migrations
The above steps can also be done if the models are re modified, after modification just repeat the steps.
Run a different test database in each branch.
When you fork the design, fork the database
Make a clone of the database and migrate that.
Make sure when you push to git, you include your migrations, that wait when someone else pulls the branch and does a migrate django knows what changes were made to the database.
I'm building an e-commerce website with Django 1.8 and PostgreSQL 9.4. I'm interested in learning what techniques I can use when I change my database to avoid having problems with Django migrations, particularly in the event that I can't get migrations to run and I have to delete my migrations, rebuild my database, and restore from backups.
In development, I've found that when I change my database schema and re-run migrations, they only run successfully about 50% of the time. Clearly I'm doing some things wrong. What's worse is that when migrations don't work, it's not always easy to understand exactly why they failed and how to modify my migration files so that they will run. In those situations, I always have to delete the database and start over. This is acceptable in development but it's not a good strategy when I go into production.
What are some "best practices" or "do's and don'ts" you follow when you modify your model classes/database schema so as to increase the probability that your Django migrations will run? And are there any steps you take to ensure that you can restore your database in the event that your migrations won't run and you have to rebuild the database from scratch? I should add that I'm a one-person startup so I don't have the conflict issues that a team working from the same code base would have.
These Techniques are what I'm using
Work locally in the same environment what I'm working in it on server. Same version of the Django and database server then push the migrations itself, don't ignore it, and migrate on the server using there migrations.
This one I used once that I migrate manually, I created the tables, indices, relations using sql commands manually and it worked properly too.
I prefer the first one more
I have a Django site running django-cms, and three environments: local dev (currently a sqlite DB that's committed to the repo), staging (mysql), and prod (mysql). There are other django apps in the project that have their own tables in the DB(s), and schema changes are managed through South migrations.
We do development using a "git flow" process, meaning that features are developed in branches and merged into a "develop" branch when complete. From a deployment standpoint, the develop branch maps to the staging version of the website.
I'd like a way to manage data in these environments that doesn't involve manually crafting data migrations for django-cms, or blowing away the staging/prod databases to loaddata in changes.
What's a good working strategy for this? Is there a quasi-automated way to generate South data migrations? Or a way to have django-cms publish pages to different environments?
I run the exact same setup on multiple projects but almost never look to migrate data between dev, stage or production.
Development environments get messy with test data, stage environments get messy with development code and data which doesn't make it to production. Meaning that hopefully production stays clean and tidy.
Following on from this moving data between them should be done carefully and I'd almost never look to automate this in case of erroneous data making it to the production database.
If there is important information which you put in to your staging environment to demo to a client or perform 'final' testing on before deploying to production then I'd suggest performing a data migration with south on that specific app and deploying with that data migration.
For any other types of data migration, like CMS page setup etc, I'd recommend setting things up in CMS draft mode as you need them, them publishing.
I read somewhere that you would never run syncdb on a database, post its initial run.
Is this true?
I don't see what the problem could be. Do you?
running syncdb will not make changes to tables for any models already in the database (even if you have changed them).
for managing changes to models, consider south
Syncdb will create tables that don't exist, but not modify existing tables. So it's fairly safe to run in production. But it's not a reliable way to maintain a database schema. Look at the South package for a way to reliably maintain changes to your database schema between development and production. Should be part of django standard, IMHO.
On Heroku, as soon as you push new code, the web-serving instances restart... even if the underlying database schema additions/changes (via syncdb or south migrate) haven't yet been applied.
In many cases, this might just cause harmless errors undtil the syncdb/migrate is run soon afterward. But I'm concerned that in some cases, new code might half-work making unexpected changes in the pre-migration database.
What's the right way to be safe against this risk?
One technique might be to add the syncdb/migrate to the Procfile so it's run before web restart. But, in the case of multiple instances, or maybe even a case where the one old-code-instance is left running until the moment the one new-code-instance is known-up, there's still a variant of the issue where code is talking to a DB with a mismatched schema.
Is there a 'hold all web instances' feature (or common best practice) for letting the migrate complete without web traffic?
Or am I being overly concerned about a risk that is negligible in practice?
The safest way to handle migrations of this nature, Heroku or no, is to strictly adopt a compatibility approach with your schema and code:
Every additive or transformative schema change must be backwards-compatible;
Every destructive schema change must be performed after the code that depends on it has been removed;
Every code change must either be:
durable against the possibility that associated schema changes have not yet been made (for instance, removing a model or a field on a model) or
made only after the associated schema change has been performed (adding a model or a field on a model)
If you need to make a significant transformation of a model, this approach might require the following steps:
Create a new database table to hold your new model structure, and deploy that migration
Create a new model with the new structure, and code to copy changes from the old model to the new model when the old model changes, and deploy that code
Execute a migration or code action to copy all old model data to the new model
Update your codebase to use the new model rather than the old model, deleting the old model, and deploy that code
Execute a migration to delete the old model structure from the database
With some thought and planning, it can be used for more drastic changes as well:
Deploy code that completely removes dependence on a section of the database, presumably replacing those sections of the site with maintenance pages
Deploy a migration that makes drastic changes that would not for whatever reason work with the above dual-model workflow
Deploy code that brings the affected sections back with the new model structure supported
This can be hard to organize and requires strict discipline and firm understanding of your code's interaction with your database, but in practice, it does allow for most changes to be made with no more downtime than the server restart itself imposes.
Looks like fast-database changeovers are the way to go, but it requires a dedicated database.
http://devcenter.heroku.com/articles/fast-database-changeovers
Alternatively, here's a tutorial for copying the data from one database (e.g., production) to another database (e.g., staging), doing the schema/data migration (e.g., using django/south), then switching the app to use the newly-updated database instance.
http://devcenter.heroku.com/articles/migrating-data-between-plans
Seems reasonable, but potentially slow if there's a large amount of data.
The recommended method is this:
Add database changes for your new features to your existing code
Make the existing code compatible with the new schema
Deploy
Add the new features to your codebase
Deploy
This means that your database changes are already in place when the code starts to require them.
However....
There's a couple of issues with this. First that I know of no development shop that is organised enough to be able to handle this, as features just get built ad-hoc, and secondly that you're not really saving anything.
Generally speaking, unless your making big changes to a massive database your changes won't take long to apply and are usually over in a couple of seconds which a developer can work around quite happily issuing restarts etc when needed. The risk being that a user might get an error page. If the changes are larger, you have some alternatives. One is using maintenance mode to turn the site off for a few seconds.
To be honest, there is no clear cut way for how to handle this nicely as by definition your code needs to be in place for your database changes to start. The best way I've found to approach the problem is to look at each change individually and work out the smoothest path for each on a case by case basis.
Rehearsing deployments on a staging environment will mitigate the risk of a deploy going bad, and give you an idea of the impact.
Heroku recently released "buildpacks" which are the scripts they use to set up an environment for your application, from managing dependencies to restarting the instances. Essentially it's a more comprehensive Procfile which you can customize.
You can fork the Python buildpack and modify the script to run in the sequence you want. Append the command you run to syncdb to the end of bin/steps/django. Commit and put this repo on Github.
Unfortunately as of now it's not possible to modify the buildpack of an existing Heroku app, so you'll have to delete it and recreate one that points to your buildpack repo:
heroku create --stack cedar --buildpack git#github.com:...
This is the best solution because it
Doesn't cost anything at all
Doesn't require you to adapt your code to Heroku
Only syncs the db once per deployment
Hope this helps.