I have a django REST project and a PostgreSQL database deployed to DigitalOcean. When I develop locally, I have a separate dockerized REST server and a separate PostgreSQL database to test backend features without touching production data.
My question arises when I'm adding/modifying model fields that require me to make migrations using python [manage.py](https://manage.py) makemigrations and python [manage.py](https://manage.py) migrate command. Here is my current situation so far:
What I was supposed to do
IN LOCAL ENV, to create the migration files,
python manage.py makemigrations
python manage.py migrate
Now commit these newly created files, something like below.
git add app/migrations/...
git commit -m 'add migration files' app/migrations/...
IN PRODUCTION ENV, run only the below command.
python manage.py migrate
What I did so far
IN LOCAL ENV, created the migration files,
python manage.py makemigrations
python manage.py migrate
I committed & pushed the changes to production WITHOUT the created migration file
IN PRODUCTION ENV, ran BOTH commands.
python manage.py makemigrations
python manage.py migrate
The production server successfully added the isActive field to the database and is working fine, but I still have a 0011_user_isActive.py migration file in my local changes that hasn't been staged/committed/pushed to github repo.
And because I ran makemigrations command in production env, it probably created the same migration file that I haven't pushed from local env.
My questions are:
What happens if I push the local migration file to production? Wouldn't it create a conflict when I run migration command on digitalocean console in the future?
How should I fix this situation?
I am just scared I'm going to corrupt/conflict my production database as I'm very inexperienced in databases and have too much to risk at the moment. Would appreciate any tips on best practices when dealing with such situations!
As docs says:
The migration files for each app live in a “migrations” directory inside of that app, and are designed to be committed to, and distributed as part of, its codebase. You should be making them once on your development machine and then running the same migrations on your colleagues’ machines, your staging machines, and eventually your production machines.
So it's best practice is to push your migration files and make your local and production migration files sync.
And if you got conflict when pushing migraions files and pulling them, the makemigrations --merge command is for solving that.
Also docs says:
Because migrations are stored in version control, you’ll occasionally come across situations where you and another developer have both committed a migration to the same app at the same time, resulting in two migrations with the same number.
Don’t worry - the numbers are just there for developers’ reference, Django just cares that each migration has a different name. Migrations specify which other migrations they depend on - including earlier migrations in the same app - in the file, so it’s possible to detect when there’s two new migrations for the same app that aren’t ordered.
When this happens, Django will prompt you and give you some options. If it thinks it’s safe enough, it will offer to automatically linearize the two migrations for you. If not, you’ll have to go in and modify the migrations yourself - don’t worry, this isn’t difficult, and is explained more in Migration files below.
Also be aware that in case of updating existed data in production, you can use RunPython in migration file. Read about it here.
Related
Conflicts occurring in migration files, Is it needed to commit the migration files or not? Is it needed to run makemigrations? (python,django 2.11)
Migrations are code. They must be in the repository. Don't generate migrations on server. On your developer machine only and don't forget to commit them. It's important to follow the rule: each commit that contains any model changes must contains appropriate migrations. Then when deploying you need to apply ones.
In other words, when someone pulls your code and runs manage.py makemigrations he or she doesn't produce any new migarions.
As of now, I'm using django 2. In dev we won't commit any migration files. We put the migration folder to gitignore, so that we won't commit.
When our code is pulled to production, they run makemigrations command, so that there won't be any migration issue.
With this production will have only one file for a model, where as in dev we may have multiple migration files same modal
I am not sure what you mean by production, but there is need to commit migrations. And you always need to run makemigrations before you migrate (if you are making changes to your model). And if there is any error, you might have to clearly state here.
I have accidently deleted one of migrations folders and and have no backup for it.
What are my options?
DB is postgres. Right now everything is OK.(I have moved instead migration folder I have on my DEV server with SQL lite) So I am just getting red message on server that not all migrations have been applied.
But next time if i run migration i will be in trouble.
What is my way out?
Migrations are mainly for backward compatibility, and tracking/versioning of the changes to models/database. If you do not really care about historical changes, etc.. then you can just delete the migrations directory and do:
python manage.py makemigrations <app_name>
This creates the initial migrations (like starting from a clean slate) - and moving forward you can track the historical migrations moving forward. More on this can be read here
Now, when you run the migrations, you can do
python manage.py migrate <app_name> --fake-initial
to fake the initial migration.
Now might be a good time to have a version control on your application
Use version control.
You are not the first developer to delete important files, but often recovery takes less than a second - thanks to version control systems (also called revision control systems). Please stop everything else and install and use one of Git, Mercury or Subversion.
Don't use FTP
It's totally. I mean totally insecure. Always use SFTP
Don't use sqlite3 for local with other db for production
sqlite doesn't enforce strict type checking. Postgresql on the other hand is very particular about it. Additionally sqlite only has a subset of the functionality that's found on postgresql. Last but not least different RDBMS have different intricacies. If you use one locally and another in production, there is always a chance that your code will break when you deploy to live
Managing without the migration files
This is not a big loss as long as your database is in sync with your models.
If you database is not in sync with your models, you can use
./manage.py inspectdb
to recreate local models that represent the actual structure in the db. Then you do makemigrations and migrate (as explained by karthik) on the generated models.
Then replace that with your live models and do the step again.
My team has had the Django migration files in .gitignore , and so hasn't been committing these files. Instead we've been making migrations on our production server. We discovered that this is not the recommended practice (upon encountering Should I be adding the Django migration files in the .gitignore file? ). Would it cause problems to - at this point in time - remove migrations from .gitignore, makemigrations on the development machine, commit, push, and apply the migrations on the production server? If so, how can we get around these problems?
This question is motivated largely because we need to apply a custom migration that we partially wrote ourselves. (As such there's actually an additional step between making migrations on the development machine and committing them, namely adding in our custom code to the migration file).
I think the only way is to copy over all the migrations from production server to your repo then commit them. This is most likely a manual process because your production server is the only place that tracks all migrations. You don't need to worry about how to migrate your production server because it keeps the original copy. However, all these should be done before any new migrations are added and applied.
After all is fixed, you should create new migrations in your local dev environment, add to git and push your migration to production then apply the migration. Remember to have your CI or something else check for duplicate migration files.
Background :-
I am using Django 1.3. We are using South as the module for DB migration and Git SCM.
Problem:-
What is the correct way to deal with the migrations Folder that is formed?
The main problem is I make changes in the DB schema in the development machine, when I upload it to the production server I have to migrate the existing schema. While doing that there is always some issue with the migration files.
Should I just add the migrations folder to the gitignore ? or is there a better way to go about it ?
You should add the migrations folder to your version control system and use the same files for production and development. You may run into some problems on your production system if you introduced your migrations not from the beginning and you have already existing tables.
Therefore you have to fake the first migration, which normally does the same thing as syncdb did when you created your database for the first time.
So when trying to apply migrations for your app for the first time on the production machine, execute manage.py migrate app_name 0001 --fake. This lets South know, that the first migration has already been applied (which already happend with syncdb) and when you run migrate again, it would continue with the following migrations.
I need some advice on pushing Django updates, specifically the database updates, from my development server to my production server. I believe that updating the scripts, files, and such will be easy -- simply copy the new files from the dev server to the production server. However, the database updates are what have me unsure.
For Django, I have been using South during initial web app creation to change the database schema. If I were to have some downtime on the production server for updates, I could copy all the files over to the production server. These would include and changed models.py files which describe the database tables. I could then perform a python manage.py schemamigration my_app --auto and then a python migrate my_app to update the database based on the new files/models.py I've copied over.
Is this an OK solution or are there more appropriate ways to go about updating a database from development to production servers?
Your thoughts?
Thanks
Actually, python manage.py schemamigration my_app --auto will only create the migration based on the changes in models.py. To actually apply the migration to the database, you need to run python manage.py migrate my_app. Another option would be to create migrations (by running schemamigration) on the development server, and then copy over the migration files to the production server and apply migrations by running migrate.
Of course, having a source code repository would be way better than copying the files around. You could create migrations on your development server, commit them to the repository, on the production server pull the new files from repository, and finally apply migrations.