I took over a Django project and discovered that migrations for various apps were not being tracked by Git. That seemed a bit problematic since my understanding was that one should always track the migrations. Is that pretty much the consensus or are there reasons not to do it?
The second part of the question has to do with the fact that I have done some work involving tweaking some migrations locally. I would now like to push the changes to production. However, I am unsure as to what the best way to combine the possible conflicts would be.
For instance, in production, I have the following:
Untracked files:
(use "git add <file>..." to include in what will be committed)
mysite/aldryn_forms/migrations/0019_auto_20200730_1455.py
mysite/apps/common/migrations/0016_auto_20200624_2028.py
mysite/apps/common/migrations/0016_auto_20200625_1125.py
mysite/apps/common/migrations/0017_merge_20200625_1129.py
mysite/apps/common/migrations/0018_auto_20200720_1743.py
mysite/apps/payment/migrations/0005_auto_20200624_2028.py
mysite/apps/payment/migrations/0005_auto_20200625_1125.py
mysite/apps/payment/migrations/0006_merge_20200625_1129.py
mysite/apps/payment/migrations/0007_auto_20200720_1743.py
mysite/apps/payment/migrations/0008_paymentmodel_course.py
mysite/apps/payment/migrations/0009_paymentmodel_user.py
mysite/apps/plugins/migrations/0016_auto_20200624_2028.py
mysite/apps/plugins/migrations/0016_auto_20200625_1125.py
mysite/apps/plugins/migrations/0017_merge_20200625_1129.py
mysite/apps/xyz/migrations/0005_auto_20200730_1455.py
Locally, I have the following:
Untracked files:
(use "git add <file>..." to include in what will be committed)
mysite/aldryn_forms/migrations/0019_auto_20201108_1623.py
mysite/apps/common/migrations/0016_auto_20201108_1623.py
mysite/apps/common/migrations/0017_auto_20201108_1806.py
mysite/apps/payment/migrations/0005_auto_20201108_1623.py
mysite/apps/plugins/migrations/0016_auto_20201108_1623.py
mysite/apps/xyz/migrations/0005_auto_20201108_1623.py
These are the files with custom work:
mysite/apps/common/migrations/0016_auto_20201108_1623.py
mysite/apps/common/migrations/0017_auto_20201108_1806.py
It appears that all the migrations existing on the production server have been applied to the production database. Hence, I have concluded that they correctly describe the state of the production DB.
What should I do to sync my local work with that currently in-production?
First I would review code for conflicts in your new migrations and migrations aplied in production.
Then backup evrything (your site and production).
if you don't finde logic errors, try copy migrations files from production to yours site and go with :
python manage.py makemigrations –merge
python manage.py migrate
if it works then it will propably work in production.
There is article about it :
https://www.algotech.solutions/blog/python/django-migrations-and-how-to-manage-conflicts/#:~:text=So%2C%20in%20order%20to%20allow,manage.py%20makemigrations%20%E2%80%93merge)
once i had it too.. annoing is that you will have to do this -merge evry new migration.
At some point I reset migrations.
When you will have database in order. You can do:
backup db and remove insert from db table : migrations
delete all migration files from porject dir
qoute all paterns in urls.py
go with makemigrations and migrate
populete new db with old data (exept migration table)
oh there is even simpler method with --fake:
https://simpleisbetterthancomplex.com/tutorial/2016/07/26/how-to-reset-migrations.html
that will put you with initial migration.
Related
I have a django REST project and a PostgreSQL database deployed to DigitalOcean. When I develop locally, I have a separate dockerized REST server and a separate PostgreSQL database to test backend features without touching production data.
My question arises when I'm adding/modifying model fields that require me to make migrations using python [manage.py](https://manage.py) makemigrations and python [manage.py](https://manage.py) migrate command. Here is my current situation so far:
What I was supposed to do
IN LOCAL ENV, to create the migration files,
python manage.py makemigrations
python manage.py migrate
Now commit these newly created files, something like below.
git add app/migrations/...
git commit -m 'add migration files' app/migrations/...
IN PRODUCTION ENV, run only the below command.
python manage.py migrate
What I did so far
IN LOCAL ENV, created the migration files,
python manage.py makemigrations
python manage.py migrate
I committed & pushed the changes to production WITHOUT the created migration file
IN PRODUCTION ENV, ran BOTH commands.
python manage.py makemigrations
python manage.py migrate
The production server successfully added the isActive field to the database and is working fine, but I still have a 0011_user_isActive.py migration file in my local changes that hasn't been staged/committed/pushed to github repo.
And because I ran makemigrations command in production env, it probably created the same migration file that I haven't pushed from local env.
My questions are:
What happens if I push the local migration file to production? Wouldn't it create a conflict when I run migration command on digitalocean console in the future?
How should I fix this situation?
I am just scared I'm going to corrupt/conflict my production database as I'm very inexperienced in databases and have too much to risk at the moment. Would appreciate any tips on best practices when dealing with such situations!
As docs says:
The migration files for each app live in a “migrations” directory inside of that app, and are designed to be committed to, and distributed as part of, its codebase. You should be making them once on your development machine and then running the same migrations on your colleagues’ machines, your staging machines, and eventually your production machines.
So it's best practice is to push your migration files and make your local and production migration files sync.
And if you got conflict when pushing migraions files and pulling them, the makemigrations --merge command is for solving that.
Also docs says:
Because migrations are stored in version control, you’ll occasionally come across situations where you and another developer have both committed a migration to the same app at the same time, resulting in two migrations with the same number.
Don’t worry - the numbers are just there for developers’ reference, Django just cares that each migration has a different name. Migrations specify which other migrations they depend on - including earlier migrations in the same app - in the file, so it’s possible to detect when there’s two new migrations for the same app that aren’t ordered.
When this happens, Django will prompt you and give you some options. If it thinks it’s safe enough, it will offer to automatically linearize the two migrations for you. If not, you’ll have to go in and modify the migrations yourself - don’t worry, this isn’t difficult, and is explained more in Migration files below.
Also be aware that in case of updating existed data in production, you can use RunPython in migration file. Read about it here.
I've got a Django app with a lot of out-of-date migrations. I'd like to remove the old migrations and start fresh.
The app has 14 different "migrations" folders.
Here is what a few of them look like:
Is it safe to remove all the contents from each of these folders? Or, do I have to make sure to only remove some of the files -- and if so which files?
You should never just delete migrations before unapplying them, or it will be a nightmare when you want to apply new migrations.
To unapply migrations you should do the following:
Use the python manage.py migrate your_app_name XXXX in case you want to unapply migrations after the XXXX migration. Otherwise use python manage.py migrate your_app_name zero to completely unapply all migrations.
Remove the .pyc files under /migrations/_pycache_/ that you have unapplied.
Remove the .py files under migrations/ that you have unapplied.
Now you can create new migrations without any headaches.
If what you're looking for is to squash all the migrations into one, do the steps above removing all migrations and then run python manage.py makemigrations your_app_name to create a single migration file. After that just run python manage.py migrate your_app_name and you're done.
That depends. If you have a production database (or any database you cannot simply drop and recreate), then the answer is no, you cannot safely remove migrations.
If you do not have any permanent databases, then yes, you can remove all migrations, run python manage.py makemigrations --initial and it will create fresh migrations based on your current models.
Also, you should check if any of the migrations are custom data migrations written by hand. If there are any, you might want to keep those.
The .pyc files are generally safe to remove, provided the related .py files are still there.
your first screenshot is not Django and looks like a JS project of some sort.
The json and js files are unrelated to the django migrations as well as __pycache__ folder. You can delete all off them.
If you mean "previously applied and no longer needed as the project only needs the latest version of the migrations" you don't want to remove but squash them instead with squashmigrations which reduces the files you have to two, init file and the initial migration file, this way your project still works.
If by remove you mean you no longer need them because you already changed the models so much that the previous migrations aren't even used other than being applied and unapplied without ever being used, doesn't matter, go to step 2 and do that instead of deleting the files manually. When you create migrations on your applications one by one, you also create migration dependency tree, well, django does. And it is really hard to keep track of after some point, if you try to delete everything thinking you can create new migration files with ease, trust me as someone who experienced otherwise, it does not work like that. It is way simpler to let django handle the migration squashing, it optimizes the migration meaning that it also deletes the unused ones in your final state.
More to read at: https://docs.djangoproject.com/en/2.2/topics/migrations/#migration-squashing
Having marked one of the answers provided previously as being accepted, here is a summary of a few things I learned:
Deleting Django migrations is generally a bad idea.
Django keeps track of what's in your db through these migration files, as well as through a table it creates in your db, and if you delete any of this Django will start throwing errors on migrate that can be hard to fix.
I was getting some of those hard-to-fix errors. Here is what I did to fix it:
Ran migrate on the production server.
When I got an error, it would tell me how the db was out of sync with what Django expected. I corrected that manually by directly editing the db with an sql client.
E.g. If it said a key existed that wasn't supposed to exist, I deleted the relevant index from the indicated table.
Or if it said a table existed that wasn't supposed to exist, I backed up the table to a file, and deleted the table. Migrate then created the table, and then I repopulated it with data from the backup.
In the case of many-to-many tables, once Django had re-created them, I deleted all the new Django-created tables, and restored them from a backup created on my local dev system, which had already had all the latest migrations run on it.
Eventually I was able to complete all migrations successfully.
I have a feeling I lucked out and the above won't work in all cases! I've learned a lot about Django and migrations and will be much more careful about this in the future.
when you import from third app:
there are 2 step uninstall it
there are use the 'django_celery_beat' app for example.
step1: clean table
python .\manage.py migrate django_celery_beat zero
step2: remove app from INSTALLED_APPS
there are done!!!
this is django document on this.
How to Reset Migrations
if you are using linux/unix os then you can fire this command. delete all migration directory.
find . -path "/migrations/.py" -not -name "init.py" -delete
find . -path "/migrations/.pyc" -delete
Conflicts occurring in migration files, Is it needed to commit the migration files or not? Is it needed to run makemigrations? (python,django 2.11)
Migrations are code. They must be in the repository. Don't generate migrations on server. On your developer machine only and don't forget to commit them. It's important to follow the rule: each commit that contains any model changes must contains appropriate migrations. Then when deploying you need to apply ones.
In other words, when someone pulls your code and runs manage.py makemigrations he or she doesn't produce any new migarions.
As of now, I'm using django 2. In dev we won't commit any migration files. We put the migration folder to gitignore, so that we won't commit.
When our code is pulled to production, they run makemigrations command, so that there won't be any migration issue.
With this production will have only one file for a model, where as in dev we may have multiple migration files same modal
I am not sure what you mean by production, but there is need to commit migrations. And you always need to run makemigrations before you migrate (if you are making changes to your model). And if there is any error, you might have to clearly state here.
My team has had the Django migration files in .gitignore , and so hasn't been committing these files. Instead we've been making migrations on our production server. We discovered that this is not the recommended practice (upon encountering Should I be adding the Django migration files in the .gitignore file? ). Would it cause problems to - at this point in time - remove migrations from .gitignore, makemigrations on the development machine, commit, push, and apply the migrations on the production server? If so, how can we get around these problems?
This question is motivated largely because we need to apply a custom migration that we partially wrote ourselves. (As such there's actually an additional step between making migrations on the development machine and committing them, namely adding in our custom code to the migration file).
I think the only way is to copy over all the migrations from production server to your repo then commit them. This is most likely a manual process because your production server is the only place that tracks all migrations. You don't need to worry about how to migrate your production server because it keeps the original copy. However, all these should be done before any new migrations are added and applied.
After all is fixed, you should create new migrations in your local dev environment, add to git and push your migration to production then apply the migration. Remember to have your CI or something else check for duplicate migration files.
Background :-
I am using Django 1.3. We are using South as the module for DB migration and Git SCM.
Problem:-
What is the correct way to deal with the migrations Folder that is formed?
The main problem is I make changes in the DB schema in the development machine, when I upload it to the production server I have to migrate the existing schema. While doing that there is always some issue with the migration files.
Should I just add the migrations folder to the gitignore ? or is there a better way to go about it ?
You should add the migrations folder to your version control system and use the same files for production and development. You may run into some problems on your production system if you introduced your migrations not from the beginning and you have already existing tables.
Therefore you have to fake the first migration, which normally does the same thing as syncdb did when you created your database for the first time.
So when trying to apply migrations for your app for the first time on the production machine, execute manage.py migrate app_name 0001 --fake. This lets South know, that the first migration has already been applied (which already happend with syncdb) and when you run migrate again, it would continue with the following migrations.