I have a rather complex K8s environment that one of the Deployments is a Django application. Currently we are having a very hard time whenever I need to update a model that has already been migrated to a PostgreSQL database.
Let's say for instance that I create an application named Sample, that has a simple table on the models.py. My development process (skaffold) builds the docker and apply it locally on the minikube, after this is done I connect to the pod via kubectl exec and execute the python manage.py makemigrations and python manage.py migrate, so far so good.
After some time, let's say I need to create a new table on the models.py file of the Sample application, the skaffold builds the docker, kills the old pod, and create the new pod. So I connect as usual via kubectl exec and try to execute the makemigrations and migrate command, lo and behold, there's no migration to apply. And of course no change is made on the PostgreSQL.
Upon further searching this, I believe that the reason for this is that since the docker is built without the Sample/migrations folder, and there's already a table (the original one) on the PostgreSQL, when I run the makemigrations it creates only the 0001_initial.py file, that has all the tables but, since the table already exists, when executing the migratethe django believes that the migration is already applied, therefore it won't apply.
If what I found out is true, how can I keep this files on a PVC, so that they are always kept between each pod recreation?
Thank you.
Related
I have a django REST project and a PostgreSQL database deployed to DigitalOcean. When I develop locally, I have a separate dockerized REST server and a separate PostgreSQL database to test backend features without touching production data.
My question arises when I'm adding/modifying model fields that require me to make migrations using python [manage.py](https://manage.py) makemigrations and python [manage.py](https://manage.py) migrate command. Here is my current situation so far:
What I was supposed to do
IN LOCAL ENV, to create the migration files,
python manage.py makemigrations
python manage.py migrate
Now commit these newly created files, something like below.
git add app/migrations/...
git commit -m 'add migration files' app/migrations/...
IN PRODUCTION ENV, run only the below command.
python manage.py migrate
What I did so far
IN LOCAL ENV, created the migration files,
python manage.py makemigrations
python manage.py migrate
I committed & pushed the changes to production WITHOUT the created migration file
IN PRODUCTION ENV, ran BOTH commands.
python manage.py makemigrations
python manage.py migrate
The production server successfully added the isActive field to the database and is working fine, but I still have a 0011_user_isActive.py migration file in my local changes that hasn't been staged/committed/pushed to github repo.
And because I ran makemigrations command in production env, it probably created the same migration file that I haven't pushed from local env.
My questions are:
What happens if I push the local migration file to production? Wouldn't it create a conflict when I run migration command on digitalocean console in the future?
How should I fix this situation?
I am just scared I'm going to corrupt/conflict my production database as I'm very inexperienced in databases and have too much to risk at the moment. Would appreciate any tips on best practices when dealing with such situations!
As docs says:
The migration files for each app live in a “migrations” directory inside of that app, and are designed to be committed to, and distributed as part of, its codebase. You should be making them once on your development machine and then running the same migrations on your colleagues’ machines, your staging machines, and eventually your production machines.
So it's best practice is to push your migration files and make your local and production migration files sync.
And if you got conflict when pushing migraions files and pulling them, the makemigrations --merge command is for solving that.
Also docs says:
Because migrations are stored in version control, you’ll occasionally come across situations where you and another developer have both committed a migration to the same app at the same time, resulting in two migrations with the same number.
Don’t worry - the numbers are just there for developers’ reference, Django just cares that each migration has a different name. Migrations specify which other migrations they depend on - including earlier migrations in the same app - in the file, so it’s possible to detect when there’s two new migrations for the same app that aren’t ordered.
When this happens, Django will prompt you and give you some options. If it thinks it’s safe enough, it will offer to automatically linearize the two migrations for you. If not, you’ll have to go in and modify the migrations yourself - don’t worry, this isn’t difficult, and is explained more in Migration files below.
Also be aware that in case of updating existed data in production, you can use RunPython in migration file. Read about it here.
I have a Django application and I'm using MariaDB as the database. Both of them have been deployed to a namespace on kubernetes. Now I want to add an additional field, so I made changes in the models.py file in the django app. These changes were made locally - I pulled the code from GIT and just made the changes locally. Normally to apply the changes, I have to run manage.py makemigrations and manage.py migrate and all the changes would've been reflected on to the DB if the DB was present locally.
So now my questions are
How can I apply the changes to MariaDb that is there on Kubernetes ?
Will running manage.py makemigrations and manage.py migrate locally and redeploying the django app to kubernetes solve this issue ?
TLDR;
If you don't require multiple replicas, then the simplest way to do it would be to run your migrations when the container starts.
If you require multiple replicas, then you'll have to get creative with jobs and init containers.There's a good article with more info on it here: https://andrewlock.net/deploying-asp-net-core-applications-to-kubernetes-part-7-running-database-migrations/
This is a Heroku-specific issue with a Django project 1.11.24 running Python 3.6.5 and a Heroku postgres database.
During testing of two different branches during development, different conflicting migration files were deployed at different times to the Heroku server. We recognized this, and have now merged the migrations, but the order the Heroku psql db schema was migrated is out of order with the current migration files.
As a result, specific tables already exist, so on deploy applying the updated merged migration files errs with:
psycopg2.errors.DuplicateTable: relation "table_foo" already exists
In heroku run python manage.py showmigrations -a appname all of the migrations are shown as having run.
We've followed Heroku's docs and done the following:
Rolled back the app itself to before when the conflicting migrations took place and were run (https://blog.heroku.com/releases-and-rollbacks)
Rolled back the postgres db itself to a datetime before when the conflicting migrations took place and were run (https://devcenter.heroku.com/articles/heroku-postgres-rollback)
However, despite both app and db rollbacks, when we check the db tables themselves in the rollback in pql shell with \dt, the table causing the DuplicateTable err still exists, so the db rollback doesn't actually seem to effect the django_migrations table.
It's Heroku, so we can't fake the migrations.
We could attempt to drop the specific db tables that already exist (or drop the entire db, since it's a test server), but that seems like bad practice. Is there any other way to address this in Heroku? thanks
I eventually fixed this by manually modifying migration files to align with the schema dependency order that was established. Very unsatisfying fix, wish Heroku offered a better solution for this (or a longer postgres database rollback window)
We are running our stack on Heroku. We are using Django 2.2 with a Postgres 11 DB. Our build pipeline (Github Actions) pushes to Heroku (git push https://git.heroku.com...) and immediately afterwards runs the migrations (heroku run python manage.py migrate --app heroku-app-name). All of that was working with a Postgres 9.6 database and is still working in our staging environment (Postgres 11). Now with production being on Postgres 11, the django migrate command is just stuck and doesn't produce any output, even so there are no actual migrations to apply.
The only differences between our production setup and our staging setup are a follower/slave in production attached to the master DB and "production workload".
In order to fix that deployment I have to run a
heroku pg:killall -a heroku-app-name
heroku restart -a heroku-app-name
At this point the migrations task in the build pipeline fails.
and afterwards migrations can be applied manually without problems:
heroku run python manage.py migrate --app heroku-app-name
So for some reason the migrations command is "waiting" for something, some database lock or whatever, yet I cannot put my finger on it. Especially odd for me is the fact that its also stuck where no migrations are to applied. Why would it be stuck there?
We found the solution. There are actually three things coming together.
We trigger a DB backup before running any migrations. We only do so in Production and not on staging, which was the reason why our staging environment had no issues while production had.
A DB migration (even so it looks like there is nothing to apply) is actually running some commands (besides regular SELECT, UPDATE, INSERTS). E.g. in our case there was a CREATE EXTENSION ... IF NOT EXISTS always executed at the beginning.
While it was possible with Postgres 9.6 to have a backup job running in parallel (I don't know what heroku is using under the hood, yet I assume a noraml pg_dump) the backup job on Postgres 11 (and others?) has now a more exclusiv lock on some operations. I assume that that a CREATE EXTENSION ... IF NOT EXISTS (even so the extension already exists) cannot be executed while having a backup job running in parallel.
(I am sure there are some Postgres internas missing to explain this more correctly)
As a result of these three things, the DB blocks the migrate operation, waiting for the backup job to finish. I have moved the daily backup job to a different time and reconfigured our pipeline to wait for the "pre-deploy"-backup to finish first.
I have a learning project deployed on Heroku. It had a Postgres database provisioned. I introduced some major changes in the models of my Django project and destroyed the old database and provisioned a new one, which is totally empty, but it is not working like an empty database.
When I run the command heroku run python manage.py makemigrations, I get the error message
You are trying to add a non-nullable field....
Why am I getting this message when I have destroyed the old database?
First of all, you should never run manage.py makemigrations on Heroku.
By the time your code gets there no model changes should exist to generate new migrations. Run makemigrations locally to create migration files. Run migrate locally and on Heroku to apply migrations to your database.
Now that that's out of the way, this is likely caused by existing migrations files, not anything in your database. If you truly want to start over you can delete the files from each of yours apps' migrations/ directories.
Finally, there is no need to destroy and reprovision your database to reset it. Instead you can use heroku pg:reset:
The PostgreSQL user your database is assigned doesn’t have permission to create or drop databases. To drop and recreate your database use pg:reset.
use this command
heroku pg:reset