In Django, when working on multiple new features simultaneously, what is the best way to handle database migrations with git?
(as in python manage.py makemigrations , and python manage.py migrate which are used to update the database after the models.py file has been changed)
I’m in middle of working on feature1, on its own branch.
Now I need to create feature2 and push it to production.
Should I:
fork new feature2 branch off of production, create feature2, merge back to prod, migrate.
fork new feature2 branch off of production, create feature2, migrate, merge back to prod.
Or some other way?
And when I go back to feature1, what do i do to insure everything will be up to date?
If you use a database migration tool by command line in conjunction with git you can take advantage of up and downs SQL scripts.
Let's say you want to add some changes to your DB, through the migration tool you can define a namescript1-up.sql and namescript1-down.sql and you can checkout to a specific database version from the command line (after a git checkout).
For example, I use golang-migrate for my go apps (I think it can be used for any language when used by command line).
Anyway, I would say this is an improper use of a migration tool, which is more suitable to apply the scripts regarding only the schema variations and not regarding ordinary INSERT or UPDATE sql statements.
I would suggest working on a containerized version of your DB, so you can destroy and recreate anything on flight.
Related
I have accidently deleted one of migrations folders and and have no backup for it.
What are my options?
DB is postgres. Right now everything is OK.(I have moved instead migration folder I have on my DEV server with SQL lite) So I am just getting red message on server that not all migrations have been applied.
But next time if i run migration i will be in trouble.
What is my way out?
Migrations are mainly for backward compatibility, and tracking/versioning of the changes to models/database. If you do not really care about historical changes, etc.. then you can just delete the migrations directory and do:
python manage.py makemigrations <app_name>
This creates the initial migrations (like starting from a clean slate) - and moving forward you can track the historical migrations moving forward. More on this can be read here
Now, when you run the migrations, you can do
python manage.py migrate <app_name> --fake-initial
to fake the initial migration.
Now might be a good time to have a version control on your application
Use version control.
You are not the first developer to delete important files, but often recovery takes less than a second - thanks to version control systems (also called revision control systems). Please stop everything else and install and use one of Git, Mercury or Subversion.
Don't use FTP
It's totally. I mean totally insecure. Always use SFTP
Don't use sqlite3 for local with other db for production
sqlite doesn't enforce strict type checking. Postgresql on the other hand is very particular about it. Additionally sqlite only has a subset of the functionality that's found on postgresql. Last but not least different RDBMS have different intricacies. If you use one locally and another in production, there is always a chance that your code will break when you deploy to live
Managing without the migration files
This is not a big loss as long as your database is in sync with your models.
If you database is not in sync with your models, you can use
./manage.py inspectdb
to recreate local models that represent the actual structure in the db. Then you do makemigrations and migrate (as explained by karthik) on the generated models.
Then replace that with your live models and do the step again.
Migrations allow transforming from one database schema to another while maintaining current data in the database. Django allows creating migrations using the command python manage.py makemigrations
Each time makemigrations is run a new migration file 000n.. is added based on the changes detected in the models.py file.
Sometimes after making small changes to models.py, I want to run makemigrations but do not want a new migration to be created because the previous migrations haven't been used yet which allows them to be merged together, primarily because running each migration in production can take a lot of time when there is a lot of data in the database so merging migrations before hand may be preferable.
Is there a way to allow a new migration 000n.. to be merged with an existing migration 000(n-1).. ?
The command you are looking for is squashmigrations. It will merge all the unapplied migrations of a given app into a single file.
I want to run makemigrations but do not want a new migration to be
created because the previous migrations haven't been used yet
This is not a problem, Django runs migrations from top to bottom, meaning: your latest migration file will wait until other previous migration files are ran.
because running each migration in production can take a lot of time
when there is a lot of data in the database
How much data do you have in the database? If really much, then you must already have replications and redundant db servers. In this case, switch the reads and writes to one, say slave server, run the migrations in the master. and then switch the traffic back to master, and before that make sure that the lag between them is 0 and new schema is replicated properly among them
Actually we are group of 3 people working on the same project, and each one individually make changes in django database. After running migrations in individual machine, it creates migration file for each migration. When someone pushes updated code in remote git repository, it creates conflict with others' migrations of same name.
Because of this reason, I lost my whole data once. Kindly give suggestions what should I do with this migration thing?
here's what i do: whenever I want to fetch from remote i check if a duplicate migration will be fetched. (we have a script that checks for all migration directories if there are filenames which have the same starting number.) if that is the case, I 'merge' the migrations, usually like this:
Find the last migration before the duplicate, let's say it's migration 000X
Make sure you are on your local source version, before the duplicate is added.
migrate back to after migration n:
python manage migrate app 000X
pull the new version including the duplicates.
remove your duplicate migrations
run schemamigration
python manage schemamigration --auto
Now you should get a new migration adding your model changes on top of the changes that were made in the migration you pulled.
how can i migrate entiry db at one step? south`s startmigration command can work only with single application
If you want to create a migration, manage.py startmigration is deprecated (see manage.py help startmigration), you should be using manage.py schemamigration (as described in the docs), and you should definitely be doing this individually on each app.
If you want to run migrations (in other words do the actual altering of the database, which I'm guessing is the case), The command for that is manage.py migrate which, if run without any arguments, will migrate all of your apps to the latest migration available.
My deployment script just has manage.py migrate, and it works fine without manual intervention, no matter how many apps have new migrations that need to be run.
Even with raw SQL, you wont be able to migrate an entire database in a single step as you need a query per table. You can however, create migrations for all apps, and then run them all at once. That's the closest thing you'll come to a one step migration.
After making some changes in my models (eg. new field in a model and a new model) what is the best way of reflecting these changes to my populated database?
PS: I wanted to see many solutions in one place rated. Apparently more solutions are already listed here.
Another technique is to use the dumpdata and loaddata arguments to manage.py, killing your database in-between:
python manage.py dumpdata > dump.json
With an external tool, drop any affected tables, or kill the whole db
python manage.py loaddata dump.json
See manage.py docs for more.
I've asked a similar question here and got quite a few answers.
There are quite a lot of ways of doing it, like manually doing the dumping and reloading with SQL, using fixtures or using one of the "emerging" schema-evolution packages for Django:
Django Evolution
South
dmigrations
(there's a DjangoCon video of a panel on schema-evolution in Django where these 3 solutions are discussed)
Depends on the scope of the changes. If it's beyond an ALTER, you're doing major surgery. Make backups of model as well as database so you can go back.
My preference is to put your new (revised, corrected, expanded) model in as a NEW application. It won't have URL's or anything, just a model.
Creating the new model as a new application. Create tests, just to be sure it works.
syncdb to build this interim implementation of the new model.
Write a little one-time utility to query your old model, and load your new model. You might want to try this in pure SQL. I prefer to write a simple query, build and save loop.
After the new model is loaded, you can dump this to a JSON file.
Once you've pulled the data out of your old model, you can rebuild your DB in the preferred new format.
Move the new model into your existing application.
Drop the old versions of the application's tables.
syncdb to build the new tables.
Load the JSON file with the data.
Django now has its own built-in migrations, documented at:
https://docs.djangoproject.com/en/dev/topics/migrations/
Look with manage.py sqlall what the parameters are for the new columns and manually add them in your database with Alter table statements. This way you don't have to redo your database; It requires some SQL knowledge though...
Take a look here (Scroll down to "Making Changes to a Database Schema")
Perform these steps in order may help you:
For more details,
clickhere: http://south.readthedocs.org/en/latest/
1) python manage.py schemamigration apps.appname --initial
Above step creates migration folder as default.
2) python manage.py migrate apps.appname --fake
generates a fake migration.
3) python manage.py schemamigration apps.appname --auto
Then you can add fields as you wish and perform the above command.
4) python manage.py migrate apps.appname
Then migrate the files to the database.