I have a learning project deployed on Heroku. It had a Postgres database provisioned. I introduced some major changes in the models of my Django project and destroyed the old database and provisioned a new one, which is totally empty, but it is not working like an empty database.
When I run the command heroku run python manage.py makemigrations, I get the error message
You are trying to add a non-nullable field....
Why am I getting this message when I have destroyed the old database?
First of all, you should never run manage.py makemigrations on Heroku.
By the time your code gets there no model changes should exist to generate new migrations. Run makemigrations locally to create migration files. Run migrate locally and on Heroku to apply migrations to your database.
Now that that's out of the way, this is likely caused by existing migrations files, not anything in your database. If you truly want to start over you can delete the files from each of yours apps' migrations/ directories.
Finally, there is no need to destroy and reprovision your database to reset it. Instead you can use heroku pg:reset:
The PostgreSQL user your database is assigned doesn’t have permission to create or drop databases. To drop and recreate your database use pg:reset.
use this command
heroku pg:reset
Related
I have a django REST project and a PostgreSQL database deployed to DigitalOcean. When I develop locally, I have a separate dockerized REST server and a separate PostgreSQL database to test backend features without touching production data.
My question arises when I'm adding/modifying model fields that require me to make migrations using python [manage.py](https://manage.py) makemigrations and python [manage.py](https://manage.py) migrate command. Here is my current situation so far:
What I was supposed to do
IN LOCAL ENV, to create the migration files,
python manage.py makemigrations
python manage.py migrate
Now commit these newly created files, something like below.
git add app/migrations/...
git commit -m 'add migration files' app/migrations/...
IN PRODUCTION ENV, run only the below command.
python manage.py migrate
What I did so far
IN LOCAL ENV, created the migration files,
python manage.py makemigrations
python manage.py migrate
I committed & pushed the changes to production WITHOUT the created migration file
IN PRODUCTION ENV, ran BOTH commands.
python manage.py makemigrations
python manage.py migrate
The production server successfully added the isActive field to the database and is working fine, but I still have a 0011_user_isActive.py migration file in my local changes that hasn't been staged/committed/pushed to github repo.
And because I ran makemigrations command in production env, it probably created the same migration file that I haven't pushed from local env.
My questions are:
What happens if I push the local migration file to production? Wouldn't it create a conflict when I run migration command on digitalocean console in the future?
How should I fix this situation?
I am just scared I'm going to corrupt/conflict my production database as I'm very inexperienced in databases and have too much to risk at the moment. Would appreciate any tips on best practices when dealing with such situations!
As docs says:
The migration files for each app live in a “migrations” directory inside of that app, and are designed to be committed to, and distributed as part of, its codebase. You should be making them once on your development machine and then running the same migrations on your colleagues’ machines, your staging machines, and eventually your production machines.
So it's best practice is to push your migration files and make your local and production migration files sync.
And if you got conflict when pushing migraions files and pulling them, the makemigrations --merge command is for solving that.
Also docs says:
Because migrations are stored in version control, you’ll occasionally come across situations where you and another developer have both committed a migration to the same app at the same time, resulting in two migrations with the same number.
Don’t worry - the numbers are just there for developers’ reference, Django just cares that each migration has a different name. Migrations specify which other migrations they depend on - including earlier migrations in the same app - in the file, so it’s possible to detect when there’s two new migrations for the same app that aren’t ordered.
When this happens, Django will prompt you and give you some options. If it thinks it’s safe enough, it will offer to automatically linearize the two migrations for you. If not, you’ll have to go in and modify the migrations yourself - don’t worry, this isn’t difficult, and is explained more in Migration files below.
Also be aware that in case of updating existed data in production, you can use RunPython in migration file. Read about it here.
This is a Heroku-specific issue with a Django project 1.11.24 running Python 3.6.5 and a Heroku postgres database.
During testing of two different branches during development, different conflicting migration files were deployed at different times to the Heroku server. We recognized this, and have now merged the migrations, but the order the Heroku psql db schema was migrated is out of order with the current migration files.
As a result, specific tables already exist, so on deploy applying the updated merged migration files errs with:
psycopg2.errors.DuplicateTable: relation "table_foo" already exists
In heroku run python manage.py showmigrations -a appname all of the migrations are shown as having run.
We've followed Heroku's docs and done the following:
Rolled back the app itself to before when the conflicting migrations took place and were run (https://blog.heroku.com/releases-and-rollbacks)
Rolled back the postgres db itself to a datetime before when the conflicting migrations took place and were run (https://devcenter.heroku.com/articles/heroku-postgres-rollback)
However, despite both app and db rollbacks, when we check the db tables themselves in the rollback in pql shell with \dt, the table causing the DuplicateTable err still exists, so the db rollback doesn't actually seem to effect the django_migrations table.
It's Heroku, so we can't fake the migrations.
We could attempt to drop the specific db tables that already exist (or drop the entire db, since it's a test server), but that seems like bad practice. Is there any other way to address this in Heroku? thanks
I eventually fixed this by manually modifying migration files to align with the schema dependency order that was established. Very unsatisfying fix, wish Heroku offered a better solution for this (or a longer postgres database rollback window)
I have deployed a django application to heroku but I need to reset those migrations. Is that possible without deleting the entire project and redeploying it? I have some test data in that database that I would prefer not to have to enter all over again.
I'd like to delete all migration files and create new ones. Thing is, I deleted all migration files from my local machine and created new ones so now the migration files on my local machine are all 001. Pushing that to heroku says there were no changes because the 001 migration files already exist on there. Only deleting them would work. Basically something similar to
find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc" -delete
but for heroku.
Pushing to Heroku is part of the solution, but not all of it. This updates the migration files, but not the django_migrations table in your database.
If you still have the old code running on Heroku:
Back your database up
Reverse all of your migrations:
heroku run python manage.py migrate appname zero
Then deploy your new code:
git push heroku master
Then run the new migrations:
heroku run python manage.py migrate
If you've already deployed the new code to Heroku:
Back your database up
Manually delete your app's tables from your database, as well as rows from the django_migrations table where the app matches your app name e.g. via heroku pg:psql (this may get tricky if you have other apps installed and inter-app dependencies)
Run your new migrations:
heroku run python migrate
Another, probably safer, option if you've already deployed the new code is to rollback to a version that reflects your database schema, then use option 1 above. In this case you can rollback again to the new version instead of doing a regular release.
Yes it can be done by two ways.
You can generate fixture for your current database schema if no such column is deleted or altered.
You can dump your database than just remove database from your db server and migrate again! Than load dumped data into fresh db, this will keep your previous data safe.
I'm new to Django but I'm deploying a Django-based website to Heroku, using Postgresql. The deployment was successful, and the website is online and has established connection with the database. However, none of the data from my local database has migrated to the heroku database, causing it to be a blank database. If I go into the admin section and manually input a datapoint, it appears on my site, so I know that database is correctly serving data. What is the proper way for migrating data from your local database to your online, heroku version of the database? I thought the following code would migrate the data:
heroku run python manage.py makemigrations
heroku run python manage.py migrate
But apparently I'm missing something.
make migrations will create a migration that contains your schema, but no data. The migrate command applies the migration to the database.
In order to provide data to be sent over as part of the migrate command you need to either create a data migration or use a fixture.
Another option you have is to dump your local database and do an import into Heroku Postgres
All in all, it depends on how much local data you have that you want copied over. If its only a few rows, I would use either a data migration or a fixture, if its 100s or 1000s of rows an export/import of your dataset is your best bet.
I deployed a new django app on Heroku which worked out fine. However, today I changed my model a little bit (added a new field) and afterwards deleted my development server file db-sqlite3 and the migrations folder in order to reflect the changes in the development database. This worked out fine on the local server but when I pushed this to heroku I get a 500 error when trying to access the model in the django-admin section.
I tried to run some migrations via heroku but I get the following error message:
! These migrations are in the database but not on disk:
<joins: 0002_auto__add_field_join_ip_address>
<joins: 0003_auto__add_unique_join_email>
<joins: 0004_auto__add_field_join_ref_id>
<joins: 0005_auto__add_unique_join_email_ref_id>
<joins: 0006_auto__add_unique_join_ref_id>
<joins: 0007_auto__del_unique_join_ref_id>
<joins: 0008_auto__del_unique_join_email__add_unique_join_ref_id>
<joins: 0009_auto__add_field_join_friend>
! I'm not trusting myself; either fix this yourself by fiddling
! with the south_migrationhistory table, or pass --delete-ghost-migrations
! to South to have it delete ALL of these records (this may not be good).
(lwc) Daniels-MacBook-Pro:src danielrichter$ heroku run python manage.py migrate --delete-ghost-migrations
I can see that in my local migrations folder I only have the 0001_initial migration and somehow missing the other but I have no idea how to resolve the issue.
I have seen that others ran into the same error message, but I did not understand the proposed answers, since I am quite new to Django and coding in general. So if there is someone who could give me a hint how to resolve this I would be very thankful!
many many thanks!
Apparently the database thinks you have applied the migrations mentioned in the message, but it can't find the files on disk. Which you confirm. Maybe there was something wrong with your version management and you lost these files? I'd try to see if you can recover the files, then the problem would be over.
If not, it is a bit more difficult. The migrations mentioned have probably already been executed but the files are lost. Also you have made new changes which have not been applied. You should try to get your code state back to the state where the last missing migration was executed. You can then make a new migration file (python manage.py schemamigration --auto your_app_name) which can replace the missing migration files. This migration will be called 0002_something (0001 being present and 0002 being the next). After this replacement migration you can have new migrations (0003 and further).
On the server, before updating your version of the code (so you don't already have 0002) execute python manage.py migrate --delete-ghost-migrations. This will delete the references to the missing migration. Afterwards you can update your version and get the new 0002 etc migrations.
Call python manage.py migrate --fake your_app_name 0002. This will tell the database that the migration was applied, without actually applying anything. This is good because the changes were already applied by the lost migration files.
After this, you can run normal migrations: python manage.py migrate and it should be good.
Hope this helps.
and afterwards deleted my development server file db-sqlite3 and the
migrations folder in order to reflect the changes in the development
database
This is your issue, you shouldn't delete any migration files, if you make a change in your database, django (or south) will create a new migration file for those changes, then you run the migration command to apply those changes to your database, you have to commit those new migration files and send them to heroku, so the changes will be applied to the remote database too.
The workflow is like this:
You have some migration files or the initial one
Edit your models by adding/removing fields
Create migration file(s), in django (without South):
python manage.py makemigrations
Apply those changes to your local database by runnig:
heroku run python manage.py migrate
Add those model changes and migration files to your git index
Push the changes to heroku and run the migration command:
heroku run python manage.py migrate