I have deployed a django application to heroku but I need to reset those migrations. Is that possible without deleting the entire project and redeploying it? I have some test data in that database that I would prefer not to have to enter all over again.
I'd like to delete all migration files and create new ones. Thing is, I deleted all migration files from my local machine and created new ones so now the migration files on my local machine are all 001. Pushing that to heroku says there were no changes because the 001 migration files already exist on there. Only deleting them would work. Basically something similar to
find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc" -delete
but for heroku.
Pushing to Heroku is part of the solution, but not all of it. This updates the migration files, but not the django_migrations table in your database.
If you still have the old code running on Heroku:
Back your database up
Reverse all of your migrations:
heroku run python manage.py migrate appname zero
Then deploy your new code:
git push heroku master
Then run the new migrations:
heroku run python manage.py migrate
If you've already deployed the new code to Heroku:
Back your database up
Manually delete your app's tables from your database, as well as rows from the django_migrations table where the app matches your app name e.g. via heroku pg:psql (this may get tricky if you have other apps installed and inter-app dependencies)
Run your new migrations:
heroku run python migrate
Another, probably safer, option if you've already deployed the new code is to rollback to a version that reflects your database schema, then use option 1 above. In this case you can rollback again to the new version instead of doing a regular release.
Yes it can be done by two ways.
You can generate fixture for your current database schema if no such column is deleted or altered.
You can dump your database than just remove database from your db server and migrate again! Than load dumped data into fresh db, this will keep your previous data safe.
Related
This is a Heroku-specific issue with a Django project 1.11.24 running Python 3.6.5 and a Heroku postgres database.
During testing of two different branches during development, different conflicting migration files were deployed at different times to the Heroku server. We recognized this, and have now merged the migrations, but the order the Heroku psql db schema was migrated is out of order with the current migration files.
As a result, specific tables already exist, so on deploy applying the updated merged migration files errs with:
psycopg2.errors.DuplicateTable: relation "table_foo" already exists
In heroku run python manage.py showmigrations -a appname all of the migrations are shown as having run.
We've followed Heroku's docs and done the following:
Rolled back the app itself to before when the conflicting migrations took place and were run (https://blog.heroku.com/releases-and-rollbacks)
Rolled back the postgres db itself to a datetime before when the conflicting migrations took place and were run (https://devcenter.heroku.com/articles/heroku-postgres-rollback)
However, despite both app and db rollbacks, when we check the db tables themselves in the rollback in pql shell with \dt, the table causing the DuplicateTable err still exists, so the db rollback doesn't actually seem to effect the django_migrations table.
It's Heroku, so we can't fake the migrations.
We could attempt to drop the specific db tables that already exist (or drop the entire db, since it's a test server), but that seems like bad practice. Is there any other way to address this in Heroku? thanks
I eventually fixed this by manually modifying migration files to align with the schema dependency order that was established. Very unsatisfying fix, wish Heroku offered a better solution for this (or a longer postgres database rollback window)
I have a Django website served by Heroku with a model storing data about projects that I've worked on in the past. I already ran makemigrations and migrate locally before pushing to heroku with git and running heroku run python3 manage.py migrate. So, my database models and fields are synced, but I'm asking about the field values. Whenever I update the value of a field for a model instance locally, I want it (the data) to sync with Heroku, and vice versa–sync values I've updated on Heroku in the admin panel with my local sqlite3 database. Is there a command for updating the values of the database itself, or am I missing something? Because I've looked all over the internet for the last hour for how to do this one thing and I couldn't find the command to do it.
Side note: I also want the command I'm looking for to sync newly created instances, not just data for existing model instances.
Ok, I've figured it out.
First step is to run python3 manage.py dumpdata --exclude contenttypes > data.json. This copies the local database data into a file called data.json (that's created by > also if it doesn't exist).
Next, git push to heroku and run heroku run python3 manage.py migrate for good measure.
Finally, heroku run python3 manage.py loaddata data.json. This translates the data loaded from the local sqlite3 database and loads it to the heroku postgre database. Unless the translating is done when you dump the data. Regardless, this synchronizes the heroku data with the local data.
I haven't tested synchronizing the local data with the heroku data, but I'm sure it'll work the same way: heroku run python3 manage.py dumpdata --exclude contenttypes > data.json and then git fetch from heroku (I've never fetched before to sync a directory with what's on github but it should be straightforward).
That's all there is to it. If I locally change the name of a project that I worked on, update the date it was last worked on, and write a few more paragraphs on the work process, and I don't want to redo all of that in the heroku shell, then I just synchronize by dumping the data, pushing it to heroku, and loading it there.
I have a learning project deployed on Heroku. It had a Postgres database provisioned. I introduced some major changes in the models of my Django project and destroyed the old database and provisioned a new one, which is totally empty, but it is not working like an empty database.
When I run the command heroku run python manage.py makemigrations, I get the error message
You are trying to add a non-nullable field....
Why am I getting this message when I have destroyed the old database?
First of all, you should never run manage.py makemigrations on Heroku.
By the time your code gets there no model changes should exist to generate new migrations. Run makemigrations locally to create migration files. Run migrate locally and on Heroku to apply migrations to your database.
Now that that's out of the way, this is likely caused by existing migrations files, not anything in your database. If you truly want to start over you can delete the files from each of yours apps' migrations/ directories.
Finally, there is no need to destroy and reprovision your database to reset it. Instead you can use heroku pg:reset:
The PostgreSQL user your database is assigned doesn’t have permission to create or drop databases. To drop and recreate your database use pg:reset.
use this command
heroku pg:reset
I deployed a new django app on Heroku which worked out fine. However, today I changed my model a little bit (added a new field) and afterwards deleted my development server file db-sqlite3 and the migrations folder in order to reflect the changes in the development database. This worked out fine on the local server but when I pushed this to heroku I get a 500 error when trying to access the model in the django-admin section.
I tried to run some migrations via heroku but I get the following error message:
! These migrations are in the database but not on disk:
<joins: 0002_auto__add_field_join_ip_address>
<joins: 0003_auto__add_unique_join_email>
<joins: 0004_auto__add_field_join_ref_id>
<joins: 0005_auto__add_unique_join_email_ref_id>
<joins: 0006_auto__add_unique_join_ref_id>
<joins: 0007_auto__del_unique_join_ref_id>
<joins: 0008_auto__del_unique_join_email__add_unique_join_ref_id>
<joins: 0009_auto__add_field_join_friend>
! I'm not trusting myself; either fix this yourself by fiddling
! with the south_migrationhistory table, or pass --delete-ghost-migrations
! to South to have it delete ALL of these records (this may not be good).
(lwc) Daniels-MacBook-Pro:src danielrichter$ heroku run python manage.py migrate --delete-ghost-migrations
I can see that in my local migrations folder I only have the 0001_initial migration and somehow missing the other but I have no idea how to resolve the issue.
I have seen that others ran into the same error message, but I did not understand the proposed answers, since I am quite new to Django and coding in general. So if there is someone who could give me a hint how to resolve this I would be very thankful!
many many thanks!
Apparently the database thinks you have applied the migrations mentioned in the message, but it can't find the files on disk. Which you confirm. Maybe there was something wrong with your version management and you lost these files? I'd try to see if you can recover the files, then the problem would be over.
If not, it is a bit more difficult. The migrations mentioned have probably already been executed but the files are lost. Also you have made new changes which have not been applied. You should try to get your code state back to the state where the last missing migration was executed. You can then make a new migration file (python manage.py schemamigration --auto your_app_name) which can replace the missing migration files. This migration will be called 0002_something (0001 being present and 0002 being the next). After this replacement migration you can have new migrations (0003 and further).
On the server, before updating your version of the code (so you don't already have 0002) execute python manage.py migrate --delete-ghost-migrations. This will delete the references to the missing migration. Afterwards you can update your version and get the new 0002 etc migrations.
Call python manage.py migrate --fake your_app_name 0002. This will tell the database that the migration was applied, without actually applying anything. This is good because the changes were already applied by the lost migration files.
After this, you can run normal migrations: python manage.py migrate and it should be good.
Hope this helps.
and afterwards deleted my development server file db-sqlite3 and the
migrations folder in order to reflect the changes in the development
database
This is your issue, you shouldn't delete any migration files, if you make a change in your database, django (or south) will create a new migration file for those changes, then you run the migration command to apply those changes to your database, you have to commit those new migration files and send them to heroku, so the changes will be applied to the remote database too.
The workflow is like this:
You have some migration files or the initial one
Edit your models by adding/removing fields
Create migration file(s), in django (without South):
python manage.py makemigrations
Apply those changes to your local database by runnig:
heroku run python manage.py migrate
Add those model changes and migration files to your git index
Push the changes to heroku and run the migration command:
heroku run python manage.py migrate
My deployment strategy looks like this (using Fabric):
create a new virtualenv
deploy new code in new virtualenv
show a maintenance page
copy the current db to new db
migrate new db
point new code to new db
symlink current virtualenv to new venv
restart services
remove maintenance page
I want to iterate fast. Now, most of the code changes do not contain migrations. Also, the db is growing, so there is much overhead created by copying the database everytime I deploy a (mostly small) change. To avoid copying the database I want to check whether there are migrations that need to be deployed (prior to step 4). If there are no migrations, I can go straight from step 2 to step 7. If there are, I will follow all the steps. For this, I need to check programmatically whether there are migrations that need to be deployed. How can I do this?
In step 2 while deploying the new code, you could deploy a script which when run on the server will detect if there are new migrations.
Example code is as follows:
# copied mostly from south.management.commands.migrate
from south import migration
from south.models import MigrationHistory
apps = list(migration.all_migrations())
applied_migrations = MigrationHistory.objects.filter(app_name__in=[app.app_label() for app in apps])
applied_migrations = ['%s.%s' % (mi.app_name,mi.migration) for mi in applied_migrations]
num_new_migrations = 0
for app in apps:
for migration in app:
if migration.app_label() + "." + migration.name() not in applied_migrations:
num_new_migrations = num_new_migrations + 1
return num_new_migrations
If you wrap the above code up in a script, your fabric deployment script can use the run operation to get the number of new migrations.
If this returns zero, then you can skip the steps associated with copying the database.
./manage.py migrate --all --merge --list | grep "( )"
Will tell and show you which migrations. If you want a return code or count, use wc.
This has the advantages of not copying and pasting code like the accepted answer (violating DRY), and also if the internal south api changes your code will still work.
UPDATE:
Django 1.7 changed the output to use bracket instead of parenthesis and Django 1.8 introduced a showmigration command:
./manage.py showmigrations --list | grep '[ ]'
dalore's answer updated for Django 1.7+
./manage.py migrate --list | grep "\[ ]"
If you just want the count then:
./manage.py migrate --list | grep "\[ ]" | wc -l
Why are you moving the databases around? The whole point of migrations is to apply the changes you made in development to your production database in place.
Your steps should really be:
create a new virtualenv
deploy new code in new virtualenv
show a maintenance page
migrate new db
symlink current virtualenv to new venv
restart services
remove maintenance page
And the migration step doesn't take that long if there's no actual new migrations to run. It'll just run through each app saying it's already up to date.
If you're copying the database to have a backup, that's something that should be running anyways on cron or something, not as part of your deployment.
Actually, I'm confused on the creating a new virtualenv each time too. The normalized (read: most typical) deployment is:
deploy new code
migrate db
restart services
If you want to add back in the maintenance page stuff, you can, but the process takes only a minute or two total.