My project use an SQLite3 DB but now I need to do a migration from SQLite3 to PostgreSQL. I've used DBeaver for do this using the "Import Data" option. At the end of the importation of all the tables from SQLite3 to PostgreSQL I noticed this error when I try to add a new content:
IntegrityError at /engine/kernel/keyconcept/add/
duplicate key value violates unique constraint
"kernel_keyconcept_pkey" DETAIL: Key (id)=(4) already exists.
If I add a new model, into this project, that use the new DB(PostgreSQL) there aren't problems to add contents, but if I use the old models I see the IntegrityError above.
I've do a test with a model that have only two rows: with the first attempt I received a same error but for key id 1, with the second attempt I received a same error but for key id 2. When I've tried for the third time I was be able to add the content and the same think happens for the next attempts.
So I think that the row counter of the every tables after the migrations never start from the last content but from zero. It strange because previous I've do the same migrations but from SQLite3 to SQLite3 whitout problems.
How I can solve this?
You have to reset the sequences used to generate primary keys. There is a management command that prints the necessary SQL: sqlsequencereset
Note that it only prints out the SQL, you have to execute it manually (which you can probably do with DBeaver).
I've solved using dumpdata and loaddata instead of DBeaver and following this examples all work fine.
In my case I need to exclude some apps as auth and contenttypes, then with dumpdata I've used this:
python3 manage.py dumpdata --exclude contenttypes --exclude auth --indent 2 > db.json
To upload tha datas I've used this:
python3 manage.py loaddata db.json
Foreground
Hi. I´m uploading my Django app database to my Heroku production environment. When I try to manage.py loaddata I get the following error:
django.db.utils.IntegrityError: Problem installing fixtures: insert or update in table "catalog_contactos" violates key constraint.
I found this post where someone suggests that first you should load models that are the base for foreign key: related post
Also I found this post that refers to moving a database to another app (the production version for example). Basically it says:
When you backup whole database by using dumpdata command, it will
backup all the database tables
If you use this database dump to load the fresh database(in another
django project), it can be causes IntegrityError (If you loaddata in
same database it works fine)
To fix this problem, make sure to backup the database by excluding
contenttypes and auth.permissions tables
Link to the post
My code
Following that suggestions I do:
python manage.py dumpdata --exclude auth.permission --exclude contenttypes > data.json
Then on the server I run the following code and get the integrity error:
heroku run python manage.py loaddata data.json
Should I dumpdata model by model and loaddata them in a certaing order? Or I´m coding something wrong? Any clues?
Thanks in advance!
Update
Could this "Natural keys" issue be the solutuion? Django serializing objects
When I had integrity error with django/postgreSQL I did DROP TABLE table in postgreSQL.
I wanna use sorl.thumbnail django app for generating thumbnails in my site.
I followed the guide and I added sorl.thumbnail to INSTALLED_APPS.
Also, I want to use the default key-value store, so I need to use the migrate command, but this doesn't do anything, it doesn't create the required database table. I tried the following commands, but it still does not create the tables:
python manage.py makemigrations
python manage.py makemigrations sorl.thumbnail
python manage.py migrate
Whats going on? How can I migrate and add the required table in order to use default key-value store?
If I dont migrate the db it raises me a exception that says that the table thumbnail_kvstore wasn't found.
I run python manage.py makemigrations and I get:
No changes detected
Then, python manage.py migrate and I get:
No migrations to apply.
Then, I try to push the changes to production:
git push heroku master
Everything up-to-date
Then, in production, I repeat the command:
heroku run python manage.py migrate
No migrations to apply.
Just in case, I run makemigrations in production:
heroku run python manage.py makemigrations
No changes detected
WHY then I get a
ProgrammingError at ....
column .... does not exist
"No changes detected" means the database is coherent with the code.
How can I debug this?¡?
I got the same problem (column not exist) but when I try to run migrate not with makemigrations (it is the same issue I believe)
Cause: I removed the migration files and replaced them with single pretending intial migration file 0001 before running the migration for the last change
Solution:
Drop tables involved in that migration of that app (consider a backup workaround if any)
Delete the rows responsible of the migration of that app from the table django_migrations in which migrations are recorded, This is how Django knows which migrations have been applied and which still need to be applied.
And here is how solve this problem:
log in as postgres user (my user is called posgres):
sudo -i -u postgres
Open an sql terminal and connect to your database:
psql -d database_name
List your table and spot the tables related to that app:
\dt
Drop them (consider drop order with relations):
DROP TABLE tablename ;
List migration record, you will see migrations applied classified like so:
id | app | name | applied
--+------+--------+---------+
SELECT * FROM django_migrations;
Delete rows of migrations of that app (you can delete by id or by app, with app don't forget 'quotes'):
DELETE FROM django_migrations WHERE app='yourapp';
log out and run your migrations merely (maybe run makemigrations in your case):
python manage.py migrate --settings=your.settings.module_if_any
Note: it is possible that in your case will not have to drop all the tables of that app and not all the migrations, just the ones of the models causing the problem.
I wish this can help.
Django migrations are recorded in your database under the 'django_migrations' table. This is how Django knows which migrations have been applied and which still need to be applied.
Have a look at django_migrations table in your DB. It may be that something went wrong when your migration was applied. So, delete the row in the table which has the migration file name that is related to that column that 'does not exist'. Then, try to re-run a migration.
Here's what i tried and it worked:
Go and add manually the column to your table
run python manage.py makemigrations
go back drop that column you added
run python manage.py migrate
I had a similar issue - the error message appeared when I clicked on the model on the django-admin site. I solved it by commenting out the field in models.py, then running migrations. Following this I uncommented the field and re ran the migrations. After that the error message disappeared.
My case might be a bit obscure, but if it helps someone, it is worth documenting here.
I was calling a function in one of my migrations, which imported a Model of said migration regularly, i.e.
from myApp.models import ModelX
The only way models should be imported in migrations would be using e.g. RunPython:
def myFunc(apps, schema_editor):
MyModel = apps.get_model('myApp 'MyModel')
and then calling that function like so:
class Migration(migrations.Migration):
operations = [
migrations.RunPython(initialize_mhs, reverse_code=migrations.RunPython.noop),
]
Additionally the original import worked until I modified the model in a later migration, making this error harder to locate.
So, I always run into this sort of problem, so today I decided to try and work it out at the database level. Thing is, I changed a model field name which Django didn't bother reflecting in a migration file. I only found out later when I ran into problems. I later looked at the migration files and discovered there was no migration for that change. But I didn't notice because I made other changes as well, so once I saw a migration file I was happy.
My advice. Create migration for each change one at a time. That way you get to see if it happened or not.
So here's my working through it in MySQL.
open mysql console.
show databases; # see all my dbs. I deleted a few
drop database <db-name>; # if needed
use <db-name>; # the database name for your django project
show tables; # see all tables in the database
DESCRIBE <table-name>; # shows columns in the database
SHOW COLUMNS FROM <db-name>; # same thing as above
ALTER TABLE <table-name> CHANGE <old-column-name> <new-column-name> <col-type>; # now I manually updated my column name
If you're using postgresql, just google the corresponding commands.
The issue was in the Models for me, for some reason Django was adding '_id' to the end of my Foreign Key column. I had to explicitly set the related named to the Foreign Key. Here 'Cards' is the parent table and 'Prices' is the child table.
class Cards(models.Model):
unique_id = models.CharField(primary_key=True, max_length=45)
name = models.CharField(max_length=225)
class Prices(models.Model):
unique_id = models.ForeignKey(Cards, models.DO_NOTHING)
Works after changing to:
class Cards(models.Model):
unique_id = models.CharField(primary_key=True, max_length=45)
name = models.CharField(max_length=225)
class Prices(models.Model):
unique_id = models.ForeignKey(Cards, models.DO_NOTHING, db_column='unique_id')
When I get this error, my extreme way to solve it is to reset my database:
Reset your database
For Postgresql on Heroku:
Heroku > your_app > Resources > database > add-ons > click on your database and open it
For postgresql
settings > Reset database
Delete all files in your_app > migrations > __pycache__ except __init.py__
Delete all files in your_app > migrations except __pycache__ folder and __init.py__
Then run:
python manage.py makemigrations
python manage.py migrate
python manage.py createsuperuser
type in to create your superuser, then run:
python manage.py makemigrations
python manage.py migrate
python manage.py
If you are able to inspect your models from your admin section, then it should be all okay now.
Just remove corresponding row migrations for that model in 'django_migrations' model in database.
And re run python manage.py migrate app_name
I tried all these answers with not much luck! What I did to have this problem solved with no harm was to go back through the migration files and find where the actual model was being created for the first time then manually add the field (in the column not being existed error message). Until if you run makemigrations --dry-run you get/see "No changes detected" and that worked. Basically, in my case, I had to carefully take my desired db changes back in time in proper migration file, rather creating a new migration now at the end of migration dependency chain.
Open the latest py file created after running the makemigrations command inside migrations folder of that particular app.
Inside class Migration there is a list attribute called 'operations'.
Remove the particular elements migrations.RemoveField(...).
Save and run python manage.py migrate.
A easier solution to the problem is to make your models exactly like it is in the migration first. and run python manage.py migrate.
Then revert those changes
Run
python manage.py makemigrations
python manage.py migrate
To check for which migrations are applied and which are not, use -:
python manage.py showmigrations
I solved a similar problem by deleting all migrations files (Don't forget to make a backup) and python manage.py makemigrations all of them into one clean file in development and pulling new files on the server. Before then I had dropped existing tables on the PostgreSQL.
I was getting this error for some reason when Django was looking for a column of type ForeignKey named category_id when the actual name in the database was category. I tried every Django solution I could imagine (renaming field, explicitly setting column name, etc.). I didn't want to drop tables or rows as this was a production database. The solution was simply to rename the column manually using SQL. In my case:
ALTER TABLE table_name
RENAME COLUMN category TO category_id;
Make sure you backup your database, ensure this won't break any other applications consuming that particular table, and consider having a fallback column if necessary.
What helped me in the end was simply dropping the database and creating it again as well as deleting all migrations files (including cache). (only removing migrations file didn't work for me at all)
sudo su - postgres
psql
DROP DATABASE 'yourdatabase';
CREATE DATABASE 'yourdatabase';
GRANT ALL PRIVILEGES ON DATABASE 'yourdatabase' to 'yourdjangouser';
then just
python manage.py makemigrations
python manage.py migrate
python manage.py runserver
If you're in development and you make some examples of data that's not important, this step is beneficial for me: just flush your data, make migrations, and migrate:
python manage.py flush
python manage.py makemigrations
python manage.py migrate
After that, you may create a new database from scratch, I hope this information was helpful.
Solved this issue by running
python manage.py migrate
in Heroku Bash shell
I am trying to migrate a database, after modifying the schema in the models.py file of my app in Django. When I try to migrate using ./manage.py migrate <app-name>, it gives the following message:
! These migrations are in the database but not on disk:
As I went through the schemamigration_table and the files, I noticed that for my last schema migration, the file (which added a table and a column in another table successfully) is not present for some reason. Is there any way I could perform my migration without resetting the database?
Yes,
If you are absolutely confident about the migration to have already applied to the database, You can safely use the --fake option.
./manage.py migrate <app-name> --fake
This would forward the migrations to the most recent migration.
--fake: Records the migration sequence as having been applied, but doesn’t actually run it. Useful for Converting An App.