git and django migrations: ignore the migrations files - django

Should I keep django migrations files in a git repository? In a developers team, how they manages their database changes.
For example, Tom has made changes in their models, and ran makemigrations and migrate, his database now has changed, as well his migrations files, keeping his migration story.
Meantime, Bob has made changes too. He has his migration files that are about another models, He ran makemigrations and migrate commands, and his db changed.
Tom and Bob are working in the same app, so, They share the same migrations files. And the same db too.
So, what will happen when Bob push their code to git repo, and later Tom pull or fetch it from git repo? The migrations files will be mixed and their stories will be broken. Also, what about with the db itself, if it is a sqlite file, should I keep it in git repo?

You should absolutely keep migrations in your repo! With some projects, it makes sense to include initial data or other functions in the migrations by editing them, so setting up each developer's environment by automatically generating them from models.py won't work for every project.
If Tom and Bob both make changes that are independent of each other (Tom adds one field, Bob adds another), the migrations files will work when you create a merge migration. Tom and Bob may have to coordinate if they conflict:
$ python manage.py migrate
CommandError: Conflicting migrations detected; multiple leaf nodes in the migration graph: (0002_mymodel_my_field_tom, 0002_mymodel_my_field_bob in myapp).
To fix them run 'python manage.py makemigrations --merge'
$ python manage.py makemigrations --merge
Merging myapp
Branch 0002_mymodel_my_field_bob
- Add field my_field_bob to mymodel
Branch 0002_mymodel_my_field_tom
- Add field my_field_tom to mymodel
Merging will only work if the operations printed above do not conflict
with each other (working on different fields or models)
Do you want to merge these migration branches? [y/N] y
Created new merge migration /myapp/migrations/0003_merge_20170517_1445.py
Here's a good read:
https://www.algotech.solutions/blog/python/django-migrations-and-how-to-manage-conflicts/

You should keep migration files in git to make sure database schema of all developers are in sync. Later these same migration files are used to migrate production database.
Any developer can create their migration file and add to version control and push to remote server. Django's migrate command detects new migrations and applies them to database.
In case two developers are working on same app and both of them create migration, django will create a merge migration file just like git creates a merge commit. This merge migration will make sure there are no conflicts in database schema and their database is in sync with latest the commit.

https://gist.github.com/mhipo1364/a55da230e1ec80bfab70e9650637bb15/revisions
Re-Generate Migration
To merge exist migration files into one file:
Remove django_migration records table (manually)
Remove all migration files
run python manage.py migrate --fake command
run python manage.py makemigrations command
run python manage.py migrate --fake-initial command
run python manage.py migrate contenttypes command
and finally, for chacking if everything is just fine, run python manage.py migrate command

If changes are on different models best way to handle them would be
makemigrations with -merge flag

Related

How can I update sqlite database in django without original migration files and preserve the data in db.sqlite3 at the same time?

I am a beginner in django. I accidentally deleted the original migration files and the only remaining related file is db.sqlite3. Now I made some changes to models.py and want to update my database while preserving the previous data in db.sqlite3. How can I do this? Will python manage.py makemigrations work?
Nop.
What you can do is run make migrations before your changes.
Run
Python manage.py migrate --fake
This will make a migration for your table as it is and then mark it as applied. Now you can make your changes and run make migration... Migrate

Django manage.py: Migration applied before its dependency

When running python manage.py migrate I encounter this error:
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration
<appname>.0016_auto_<date2>_<time2> is applied before its dependency
<appname>.0001_squashed_0015_auto_<date1>_<time1>
running showmigrations returns:
<appname>
[X] 0001_squashed_0015_auto_<date1>_<time1> (15 squashed migrations)
[X] 0016_auto_<date2>_<time2>
[ ] 0017_<modelname>_squashed_0019_auto_<date3>_<time3> (3 squashed migrations)
I was trying out django-extensions yesterday, when it all got messed up after me running some direct SQL queries and I reset hard using git. I'm still learning about migrations, so I don't understand what is wrong, since it seems to me that both migrations already have been applied.
Thank you for your help!
This worked for me. I thank my coworker for sharing this knowledge after I searched online for many hours.
Start your db shell
python manage.py dbshell
Use the database you want. If you don't know, run .databases (SQLite) or SHOW databases
mysql>use <database_name>;
Retrieve all the migrations under your app
mysql> select * from django_migrations where app='<app>';
You will see the output with ids next to all migrations. Look at the migration you want to drop. Say the id is 361
mysql> delete from django_migrations where id=361;
You have squashed the migrations, so one of the dependencies that 0016_auto_<date2>_<time2> had is now part of the newly created squashed migrations. Meanwhile the 0016_auto_<date2>_<time2> has already been run and now you're trying to run the squashed migration.
I personally don't know if there's any way to fix this automatically. You will need to fix the issues yourself. If you have version control, revert these changes and try to rethink how you should squash the migration without affecting old ones.
I have solved this problem when i did (custom user model) by this steps:
delete this file :
migrations\0001_initial.py
delete this :
db.sqlite3
put this code in settings.py :
AUTH_USER_MODEL = 'users.CustomUser'
Then do (makemigrations) then (migrate )
run server .. the problem solved :)
i have used this link it is help me to solve the problem of dependency :
https://docs.djangoproject.com/en/3.1/topics/auth/customizing/
Due to limitations of Django’s dynamic dependency feature for swappable models, the model referenced by AUTH_USER_MODEL must be created in the first migration of its app (usually called 0001_initial); otherwise, you’ll have dependency issues.
In addition, you may run into a CircularDependencyError when running your migrations as Django won’t be able to automatically break the dependency loop due to the dynamic dependency. If you see this error, you should break the loop by moving the models depended on by your user model into a second migration. (You can try making two normal models that have a ForeignKey to each other and seeing how makemigrations resolves that circular dependency if you want to see how it’s usually done.)
run this python manage.py dbshell
INSERT INTO public.django_migrations(app, name, applied)
VALUES ('YOUR_APP_NAME, '0017_<modelname>_squashed_0019_auto_<date3>_<time3>', now());
and you should be fine. If Your migration was changing a lot to the database, then I am afraid it won't be that easy to fix it.
you need to fake migrations and migrate again
just make sure that you have a backup from your data because when you migrate again you need to delete apps table.
make sure that you look at show migrations and migrate un migrated apps by its sequence
Edit the dependencies of the conflicting migration, so that it no longer references the already applied migration.
Then run python manage.py migrate again and it should be fixed.
Warning: this only work suppossing that the state of the database matchs the state you get having applied the conflicting migration.
I had the same issue on 2020 with Django 3.0.6.
I tried all the relevant answers with no success. So I went in my database and deleted all the tables. You must export the relevant tables if you have done lot of work. I mainly delete django files in my database. And after, run:
python manage.py makemigrations <my-app>
And:
python manage.py migrate
Export your relevant tables if any.
First back up your database before resolving the conflicts, (Use "python manage.py dumpdata > db.json" for SQLite).
Execute python manage.py dbshell, to access the database.
Delete the migrations rows that are having conflicts from the django_migrations table.
Rename the tables conflicting in the database
Execute the makemigrations and migrate commands
After successful migrations, Drop the newly readded tables and finally restore the previously renamed tables to match the migrations need
I had the same problem, and here's how I solved it.
The following is my error message
File "/usr/local/lib/python3.11/site-packages/django/db/migrations/loader.py", line 327, in check_consistent_history
raise InconsistentMigrationHistory(
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration aaaa.0024_campaign_template is applied before its dependency bbbb.0005_templatemodel_from_template on database 'default'.
My solution
python manage.py migrate bbbb
python manage.py migrate
Because I changed the Django's app name in batches, the application order was not consistent when applied to the database. The bbbb that aaaa relies on was not created first, so I manually created the bbbb first
Migration file is not created for all app:
step 1:
create migration folder and add __init__.py file for all app
step 2:
delete db.sqlite3 database
step 3:
python manage.py migrate
python manage.py makemigrations
Delete all of your migrations folder
Delete the database(sqlite3)
Then run the makemigrations and migrate command
Delete the migration files.
Run:
python manage.py migrate
python manage.py makemigrations
python manage.py migrate
python manage.pyrunserver

makemigrations doesn't detect changes in model

I'm using django 1.9.6. I recently deleted my migrations and ran migrate --run-syncdb and makemigrations my_app. Today I added a new field to one of my models:
models.py:
value = models.PositiveSmallIntegerField(null=True)
I tried to migrate the changes, but makemigrations doesn't detect the change. It's just the development version, so I can resync (I don't have to preserve the data), but running --run-syncdb again doesn't detect it either.
Why isn't this migrating?
Delete every past migration files and __pycache__ files except __init__
then:
python manage.py makemigrations yourApp
After that make sure that the db is the same as the code in model.py (remove new changes) and run next line:
python manage.py migrate --fake-initial
Now add all your changes in the model.py and run next lines:
python manage.py makemigrations
python manage.py migrate
Best Regards,
Kristian
I had the same issue. I realised I had a property defined on the model with the same name as a field I was trying to add on the model. Ensure the model doesn't have a model property/method with the same name as the field you are trying to add.
You should not delete migrations, you should squash them. If you simply deleted the files you likely messed things up, the easiest way to recover is re-sync your code to get the files back. A more complex route is to delete all the records from the django_migrations table and re-init the migrations from scratch but there is more steps/issues than I can really get into and I don't recommend it.
The reason makemigrations is not detecting the change is likely because there is no migrations folder in that app. If you run python manage.py makemigrations your_app --initial it might detect and generate the migrations or it might freak out because of the difference in your files and the django_migrations table.
The --run-syncdb command is great when you don't care about the data, usually before you actually deploy but once you start using migrations you should not use the --run-syncdb command anymore. For instance, during initial development here is the code I run every model change instead of dealing with migrations:
dropdb mydb && createdb mydb && python manage.py migrate --run-syncdb && python manage.py loaddata initial
I store all the initial data in a fixtures file and the command wipes out the entire database, the --run-syncdb rebuilds the schema, and the initial data is loaded while skipping actual migration files.
So, if you don't care about any of your data or can easily move it to a fixture than you can drop and re-create the DB. You are then free to delete all migration folders and you can use the command above until you go live and need to move to using migrations.
*** UPDATE FOR DJANGO 1.11 *** STILL USING ON 3.2.5 ***
I started using Django 1.11 and noticed the test framework can fail if you have dependencies on framework models and don't actually have migrations. This is the new command i'm using to wipe everything out and start over while still developing.
set -e
dropdb yourdb
createdb yourdb
find . -name "migrations" -type d -prune -exec rm -rf {} \;
python manage.py makemigrations name every app you use seperated by space
python manage.py migrate
python manage.py loaddata initial
I put this in a builddb.sh at the root of my project (next to manage.py) so I can just run ./builddb.sh. Be sure to delete it on deploy so there is no accidents!
Try creating a migrations folder that contains an empty __init__.py file inside your app folder if there is no migrations folder already. And make migrations.
In my case, I created a new project and created an app. The python manage.py makemigrations returned no change detected although I had added a handful of models.
Found out that you need to manually add the name of your app in the INSTALLED_APPS list inside your settings.py file. Unless you do that, no change in the migration of that app can be detected.
If you're deleting a model, and expecting the changes to be picked up in migrations, ensure that the model doesn't have a *.pyc still lying around.
Another case is abstract classes. Make sure that your model’s metaclass does not have the abstract = True property.

500 error on production server after model update

I deployed a new django app on Heroku which worked out fine. However, today I changed my model a little bit (added a new field) and afterwards deleted my development server file db-sqlite3 and the migrations folder in order to reflect the changes in the development database. This worked out fine on the local server but when I pushed this to heroku I get a 500 error when trying to access the model in the django-admin section.
I tried to run some migrations via heroku but I get the following error message:
! These migrations are in the database but not on disk:
<joins: 0002_auto__add_field_join_ip_address>
<joins: 0003_auto__add_unique_join_email>
<joins: 0004_auto__add_field_join_ref_id>
<joins: 0005_auto__add_unique_join_email_ref_id>
<joins: 0006_auto__add_unique_join_ref_id>
<joins: 0007_auto__del_unique_join_ref_id>
<joins: 0008_auto__del_unique_join_email__add_unique_join_ref_id>
<joins: 0009_auto__add_field_join_friend>
! I'm not trusting myself; either fix this yourself by fiddling
! with the south_migrationhistory table, or pass --delete-ghost-migrations
! to South to have it delete ALL of these records (this may not be good).
(lwc) Daniels-MacBook-Pro:src danielrichter$ heroku run python manage.py migrate --delete-ghost-migrations
I can see that in my local migrations folder I only have the 0001_initial migration and somehow missing the other but I have no idea how to resolve the issue.
I have seen that others ran into the same error message, but I did not understand the proposed answers, since I am quite new to Django and coding in general. So if there is someone who could give me a hint how to resolve this I would be very thankful!
many many thanks!
Apparently the database thinks you have applied the migrations mentioned in the message, but it can't find the files on disk. Which you confirm. Maybe there was something wrong with your version management and you lost these files? I'd try to see if you can recover the files, then the problem would be over.
If not, it is a bit more difficult. The migrations mentioned have probably already been executed but the files are lost. Also you have made new changes which have not been applied. You should try to get your code state back to the state where the last missing migration was executed. You can then make a new migration file (python manage.py schemamigration --auto your_app_name) which can replace the missing migration files. This migration will be called 0002_something (0001 being present and 0002 being the next). After this replacement migration you can have new migrations (0003 and further).
On the server, before updating your version of the code (so you don't already have 0002) execute python manage.py migrate --delete-ghost-migrations. This will delete the references to the missing migration. Afterwards you can update your version and get the new 0002 etc migrations.
Call python manage.py migrate --fake your_app_name 0002. This will tell the database that the migration was applied, without actually applying anything. This is good because the changes were already applied by the lost migration files.
After this, you can run normal migrations: python manage.py migrate and it should be good.
Hope this helps.
and afterwards deleted my development server file db-sqlite3 and the
migrations folder in order to reflect the changes in the development
database
This is your issue, you shouldn't delete any migration files, if you make a change in your database, django (or south) will create a new migration file for those changes, then you run the migration command to apply those changes to your database, you have to commit those new migration files and send them to heroku, so the changes will be applied to the remote database too.
The workflow is like this:
You have some migration files or the initial one
Edit your models by adding/removing fields
Create migration file(s), in django (without South):
python manage.py makemigrations
Apply those changes to your local database by runnig:
heroku run python manage.py migrate
Add those model changes and migration files to your git index
Push the changes to heroku and run the migration command:
heroku run python manage.py migrate

django south on multiple databases

I am using two enviorments for my django project, dev and production. mostly I am working on dev, when I change fields in my models I am using south to migrate to the new models.
What I dont understand is how can I now transfter it to my production database ? What I am doing is committing my code into SVN and updating on production, then try to migrate.
Should I commit my migration directory and files ? or should I not ? it seems that it allways fail when trying to migrate on my production machine.
EDIT :
I have done additional tests. I have cleared my south histroy table and delete any migrations directories.
Now on my local enviorment I run : python manage.py convert_to_south app_name
everything seems to work perfectly. I have now add an additional field to my model and run : python manage.py schemamigration app_name --auto
The new field was migrate into my database as expected. Now, I commited my migrations folders with the .py files (contains 0001-initial and 0002 for my new field)
on my dev enviorment I updated my tree which includes the migrations files. I run python manage.py syncdb. the south_migrationhistory table was created but enpty.
now I run python manage.py schemamigration app_name --auto and got :
"Nothing seems to have changed."
the south_migrationhistroy is still empty..... I have no clue what I am doing wrong here
You Should commit your migration directory and files. When you do an svn update, south now knows what to do with your new migrations.
South looks up a database table called south_migrationhistory to see what migrations have already been applied, and based on that applies the new migrations specified in the migration folder. If you do not checkin the new migration files, it will always end up with "No new updates".