Django migration IntegrityError: invalid foreign key (but the data exists) - django

I am gradually updating a legacy Django application from 1.19 -> 2.2 and beyond. To upgrade to 2.2, I just added on_delete=models.CASCADE to all models.ForeignKey fields that did not have the error (which I also had to do retroactively for existing migrations, apparently...).
Possibly related/unrelated to that, when I run manage.py migrate, Django throws the following error (I shortened the table/field names for brevity):
django.db.utils.IntegrityError: The row in table 'X' with primary key '3' has an invalid foreign key: X.fieldname_id contains a value '4' that does not have a corresponding value in Y__old.id.
Note in particular the __old.id suffix for the db table that Django expects to contain a row with id 4. When manually inspecting the db, the table Y does really contain a valid row with id 4! I'm assuming, to support the migration, Django is making some temporary tables suffixed with __old and somehow it is unable to migrate said data?
The db row Y in question is really simple: a char, boolean, and number column.
Edit: seems to be related to an old Django bug with SQLite. Not sure how to solve. It does not seem to occur for Django 2.1.15, and starts to occur in Django 2.2.

This problem is caused by the mentioned Django bug, but if you get the migration error, your database is broken already.
When you dump the DB to SQL, you can see REFERENCES statements which point to tables ending in __old, but these tables do not actually exist:
$> sqlite3 mydb.db .dump | grep '__old'
CREATE TABLE IF NOT EXISTS "company" [...]"account_id" integer NULL REFERENCES "account__old" ("id") [...]
Fortunately, the DB can be fixed easily, by just removing the __old and dumping into a new database. This can be automated with sed:
sqlite3 broken.db .dump | sed 's/REFERENCES "\(.[^"]*\)__old"/REFERENCES "\1"/g' | sqlite3 fixed.db

It is not an ideal solution, but you can manually delete the row from the database or set the value of the foreign key to a temporary value, migrate and then restore the original value.

Related

what way can i fix django DB migration error without deleting migration files

what is the best way to fix this kind of DataBase Errors without having to delete my db and migration files and starting to enter data from scratch?
django.db.utils.IntegrityError: The row in table 'store_product_category' with primary key '1' has an invalid foreign key: store_product_category.category_id contains a value '1' that does not have a corresponding value in store_category.id.
while inspection the sqlit DB i observe that there is a mismatch in the IDs of the store_product_category and store_category.id.
is there anyway i can modify the id directly on the DB, i dont want to start deleting database files and migrations
If I've understood right:
The model StoreProductCategory has a FK - category, linking to a model StoreCategory.
You have a SPC record with category == 1 but no record in StoreCategory with this ID?
If so, the fix is reasonably simple.
Enter the DB shell using python manage.py dbshell and run an SQL INSERT command to add the appropriate record.
Change your model for StoreProductCategory and set on_delete for that FK. I would suggest maybe PROTECT might be appropriate here, but it's up to you - just make sure it's something that will keep things consistent.
If (2) is already done, I do question how this happened in the first place - that would kind of indicate somebody has messed directly with the DB. You may want to investigate who has access and what gets done there.

Changing primary key in Django caused constraint does not exist error

I have Django project with DigestIssue model among others and there were Django auto created primary key field id and my field number. I wanted to get rid of duplication and set number as PK because number is unique and with same values as id. But I have foreign keys referencing this model.
I doubt that they will migrate automatically after such operation. I tried, hoping for such automigration and got constraint "idx_16528_sqlite_autoindex_gatherer_digestissue_1" of relation "gatherer_digestissue" does not exist error ("sqlite" in constraint name is historical thing, I switched to PostgreSQL a time go). I tried more complicated way, following https://blog.hexack.fr/en/change-the-primary-key-of-a-django-model.html but got same error on PK switching step.
So the question is - how to replace in Django old primary key with new one with same values and referenced by other models?

Deletion of an object in SQLite3 outputs a different error than Postgres

I have a Django and pytest-django project and I'm encountering a weird issue. My prod env has a Postgres DB and when running local tests I use SQLite3.
Our DB schema look like this
class A(BaseModel):
# Some properties
class B(BaseModel):
a = models.ForeignKey(A, on_delete=DO_NOTHING, null=False)
# Some other properties
Assuming I have 2 objects (a which is an instance of A, and b which is an instance of B), deleting object a before deleting b will raise the following error on Postgres:
django.db.utils.IntegrityError: update or delete on table \"db_a\" violates
foreign key constraint \"db_a_id_32c056c8_fk\" on table \"db_b\"
However, running the same code locally, using the (in-memory version of) SQLite, will raise:
django.db.utils.IntegrityError: The row in table 'db_b' with primary
key '453c667321254983915068259f6a999f' has an invalid foreign key:
db_b.a_id contains a value 'b8fd641710be466ca8eab451faaed757' that does not
have a corresponding value in db_a.id.
So, in both cases, the deletion operation will output an error - but not the same one.
I have no issues with the fact that an error is raised, but I would like my local unit tests to have the same error outputted as my remote one without changing the DB locally to PG.
Any insights into this behaviour, or ideas on how to get consistent results will be very appreciated.
Thanks!
I've looked at:
https://stackoverflow.com/a/53194777/4890123 but the errors remained the same.

How to add unique_together constraint to a Django table with prior data

I am trying to implement a unique_together constraint on one of the models in my Django project. The decision to have a unique constraint was taken after test data was already created in the table.
Now while running migrations, I came across the following error:
django.db.utils.IntegrityError: UNIQUE constraint failed:
movt_frt.frt_group_id,
movt_frt.rec_loc_id,
movt_frt.dis_loc_id
I have tried creating similar unique constraints on table that previously held no data and migrations happened successfully.
My question is:
Am I right in concluding that the Migration is failing because the table already has data residing in it?
Is there a way to do some changes to the migration file on the lines as discussed here and attempt migration again, to run successfully?
I am using Django ver. 2.0.6
I don't know if this question is still relevant. It's not so much that the table is filled with data. These data already contains combinations that conflict with your specified 'unique_together'.
Overruling is not possible, but if the table is not too big you could correct it manually.

Received "ValueError: Found wrong number (0) of constraints for ..." during Django migration

While using Django 1.7 migrations, I came across a migration that worked in development, but not in production:
ValueError: Found wrong number (0) of constraints for table_name(a, b, c, d)
This is caused by an AlterUniqueTogether rule:
migrations.AlterUniqueTogether(
name='table_name',
unique_together=set([('a', 'b')]),
)
Reading up on bugs and such in the Django bug DB it seems to be about the existing unique_together in the db not matching the migration history.
How can I work around this error and finish my migrations?
(Postgres and MySQL Answer)
If you look at your actual table (use \d table_name) and look at the indexes, you'll find an entry for your unique constraint. This is what Django is trying to find and drop. But it can't find an exact match.
For example,
"table_name_...6cf2a9c6e98cbd0d_uniq" UNIQUE CONSTRAINT, btree (d, a, b, c)
In my case, the order of the keys (d, a, b, c) did not match the constraint it was looking to drop (a, b, c, d).
I went back into my migration history and changed the original AlterUniqueTogether to match the actual order in the database.
The migration then completed successfully.
I had a similar issue come up while I was switching over a CharField to become a ForeignKey. Everything worked with that process, but I was left with Django thinking it still needed to update the unique_together in a new migration. (Even though everything looked correct from inside postgres.) Unfortunately applying this new migration would then give a similar error:
ValueError: Found wrong number (0) of constraints for program(name, funder, payee, payer, location, category)
The fix that ultimately worked for me was to comment out all the previous AlterUniqueTogether operations for that model. The manage.py migrate worked without error after that.
"unique_together in the db not matching the migration history" - Every time an index is altered on a table it checks its previous index and drops it. In your case it is not able to fetch the previous index.
Solution-
1.Either you can generate it manually
2.Or revert to code where previous index is used and migrate.Then finally change to new index in your code and run migration.(django_migration files to be taken care of)
Also worth checking that you only have the expected number of Unique indexes for the table in question.
For example, if your table has multiple Unique indexes, then you should delete them to make sure you have only 1 (or whatever the number of expected Unique indexes is) pre-migration index present.
To check how many Unique indexes are there for a given table in PostgreSQL:
SELECT *
FROM information_schema.table_constraints AS c
WHERE
c.table_name = '<table_name>'
and c.constraint_type = 'UNIQUE'
Just in case someone runs into this and the previous answers haven't solved, In my case the issue was that when I modified the unique together constraint, the migration was attempted but the data didn't allow it (because of a more restrictive unique together constraint). However, the migration managed to delete the unique together constraint from the table leaving it in an inconsistent state. I had to migrate back to zero and re-apply the migration without data. Then it went through without problems.
In summary, make sure your data will be able to accept the new constraint before you apply the migration.
Find the latest migration file of the respective table, find unique
together, and replace current unique constraints fields.
Migrate database using ./manage.py migrate your_app_name.
Revert or undo the previous migrations file.
In my case problem was that the previous migration was not present in the table dajsngo_migrations. I added missing entry and then the new migration worked
Someone may get this issue while modifying the unique_together. Basically, the table state is not consistent with the migrations. You may need to add the previous constraints manually using MySQL shell.
incase one is using migrate with django and there is no data in the database, then drop the database and the do again python manage.py migrate`