On Migration Files part of Django Official Document, it reads:
Paragraph 1: The operations are the key; they are a set of declarative instructions which tell Django what schema changes need to be made. Django scans them and builds an in-memory representation of all of the schema changes to all apps, and uses this to generate the SQL which makes the schema changes.
Paragraph 2: That in-memory structure is also used to work out what the differences are between your models and the current state of your migrations; Django runs through all the changes, in order, on an in-memory set of models to come up with the state of your models last time you ran makemigrations. It then uses these models to compare against the ones in your models.py files to work out what you have changed
Q1: In differences are between your models and the current state of your migrations, what exactly your models and current states of your migrations refer to? which one refers to the Database Version before this new migration file is applied and which one refers to the Database Version after new migration file is applied?
Q2: That in-memory structure is also used to work out what the differences are between your models and the current state of your migrations
Isn't this already accomplished by Django scans them and builds an in-memory representation of all of the schema changes to all apps, and uses this to generate the SQL which makes the schema changes in Paragraph 1? If so, why it says is also used to which makes audience think it is something different from paragraph 1?
Q3: As a best practice, should we delete all migrations files every time after successful migration? I just fixed some errors poping up when makemigrations, which has bothered me for days, by deleting all existing migration files and makemigrations again.
Related
In my company's Django project, our models are currently spread across about 20 "app" folders with little rhyme or reason. We'd like to consolidate them into a single new app (along with the rest of our code) so that in the future, we can refactor all parts of our system at-will without worrying about violating an "app" boundary.
Due to how Django works, simply moving the model breaks everything. I've spent hours reading everything I can about the various proposed solutions to this problem, with the best article being:
https://realpython.com/move-django-model/
So far, I'm not happy with any of the solutions I've come across. They're all just too onerous.
If I do the following:
Move all models to the new app. (Let's call it "newapp".)
Update all references in the code to the previous locations. (Including migration files.)
Take down the site for a minute.
Run a script to rename all database tables from someprevapp_model to newapp_model.
Run a script to update app_label for all rows in the django_content_type table.
Deploy latest codebase version, with updated model locations and references.
Turn site back on.
Have I succeeded, or am I missing something?
UPDATE
Just remembered - I'd also have to:
5.5) Run a script to update app for all rows in the in django_migrations table (easy part), and to renumber all migrations to form a single sequence (hard part), as well as modifying all migration files to match new sequence.
UPDATE - V2 - Full Reset
Given the need to re-sequence migrations, combined with having already been wanting to squash our huge migration stack, lead me to this new and improved recipe:
Move models to new app. (Let’s call it “newapp”.)
Delete previous migrations.
Update code to reflect new model locations.
Run makemigrations to generate single migration in new app with all models.
Turn site off.
Run script to rename my tables from prevapp_model to newapp_model.
Run script to update django_content_type.app_label.
Run script to replace contents of django_migrations with single row indicating migration newapp-0001 has already run.
Deploy new version.
Turn site back on.
I've heard the opinion that mix data migration and structure migrations is bad practice in Django. Even if you are specify atomic=False in your Migration class. But i could not find any information on this topic. Even my more expirience collegues could not answer this question.
So, is it bad to mix data and structure migrations? If so why? What exactly may happen if i do it?
There is an actual reason for not mixing data and schema migrations in one migration, mentioned in the entry for RunPython operation in Django docs:
On databases that do support DDL transactions (SQLite and PostgreSQL), RunPython operations do not have any transactions automatically added besides the transactions created for each migration. Thus, on PostgreSQL, for example, you should avoid combining schema changes and RunPython operations in the same migration or you may hit errors like OperationalError: cannot ALTER TABLE "mytable" because it has pending trigger events.
It should be also noted that for databases that do not support DDL transactions, it may be easier to fix the database after an unsuccessful migration attempt when data and schema migration operations are not mixed together, as data migration operations can be rolled back automatically in Django.
In the past the best practice was to keep them separate. The second sentence in this section in the docs says:
Migrations that alter data are usually called “data migrations”;
they’re best written as separate migrations, sitting alongside your
schema migrations.
But doesn't list any reasons why. Since Django ~2.0 I've been allowing small data migrations to occur with schema migrations. However there have been times when the migration simply couldn't run with the schema migration. There are two main cases that I've run into.
The data migration takes a long time and shouldn't be a migration in the first place. The resolution was to simply run a script that did what the data migration would have, but in batches.
Attempting to add/update data, then creating an index. This forced me into splitting the migrations into two separate files. I don't remember the exact error, but it simply wouldn't migrate. This shouldn't cause problems for you unless there are non-atomic migrations running which would leave your DB in an unexpected state.
Situation: a project I'm working in had a file corruption or something. Models are the latest version, but the SQLite DB had to be rolled back before some columns were added/removed/modified. Migration files are all gone. Trying to create migrations anew results in the new columns being present in the initial migration file, so I can neither migrate for real (due to the table existing) nor fake it (since the columns are missing in the DB).
Given those circumstances, how can I make an initial migration matching the columns currently present in the DB so I can fake it and then make a real second migration to bring the tables in line with the models? The only thing that comes to mind is manually tweaking the models to match the DB schema, making the initial migration, faking it and then restoring the new version of the models, but I'd much prefer having this done automatically.
django inpsectdb to the rescue.
But first, learn how to use git if you had used proper version control, you would not be facing this difficulty now.
First step, add the code to version control.
Delete the existing models files
Use inspectdb to generate a models.py from the tables in the database. This is not perfect, you will have to edit the file manually and you may have to spread it out between different models files manually.
Now delete the contents of the migrations table
do a ./manage.py makemigrations (yourapp)
Do the fake migration you mentioned
replace the generated models.py with your current models.py (a git checkout of that file will do the trick nicely)
do a makemigrations and migrate again.
good luck.
Background:
I'm dealing with quite complex django application and I'm looking for a way to make my life a little bit easier.
One of the models (tables) serves as source of "options" for most "select lists" in the system (simple field <-> dictionary construct). It should always be populated with the default values and/or extenden when the there are new entries provided with a new version of the system.
I manage it all now manually so:
I run migrations for the main app
Unload the dictionary tables from the dev enviroment
Insert them manually into the enviroment I'm currently upgrading
I migrate the rest of the apps which may contain refferences to the dictionary tables
All of this because of default values provided in ForeignKeys.
Actual Question:
Is it possible to add table entries (table content) to the makemigrations process for particular tables??
Yes, you can do anything you like in a migration via the RunPython operation. This is called a data migration and is fully documented.
I am new to Django. I have completed the tutorial and am reading the documentation for more learning. As I try to add to my understanding, say, in new Managers or ModelForms I am curious as to what needs South (or even just scrapping it and rewriting the app).
update django database to reflect changes in existing models
The link above says basically that any column change it is necessary, while the link below is more of what I am asking. Can someone generalize when it is not needed (eg: Adding a new Form/ModelForm based on an existing Model? Adding a Manager?) If no changes are made to the columns of the database South is not then not necessary?
Does changing a django models related_name attribute require a south migration?
There are two types of south migrations: schema and data.
Data migrations are used to change data in the DB and not the schema of the DB.
The schema migrations are the ones you are interested in. They are used to keep track of changes to the DB schema and one should accompany any changes to your models that result in DB schema change (create table, drop table, drop column, change null constrain e.g.)
Some great insights might be found if you read two consecutive migrations for a django app.
In each of them you can find code that applies the migreation, code that reverts the migrations and a snapshot of the DB schema.
P.S. It is quite easy to check if south migration is needed for a particular change in your models. Just run a schemamigration for the modified django app and delete the newly created migration if such was created. As creating a south migration is different that running it, this is a great way to test and learn.
Keep in mind that south is a piece of software like any other and it does 'support' bugs.
Related_name attribute changes only affect your project and django uses it to make queries.
Changes like blank = True/False, null = True/False, symmetrical = True/False require database changes although symmetrical = True/False does not trigger update by south, but the setting definitely makes difference at field creation.
Column changes, like the link in your post shows, require updates in database and that's what south does very good.