Django keeps migrating the same foreign key - django

I am importing an existing database into it's own Django project. I have generated the initial models from the database, via inspectdb, and am enabling Django to control each table one at a time by commenting the managed=False lines in the table meta settings. I've started with the simple models and am hitting a snag when enabling tables with foreign keys. Django keeps generating the same migration for the foreign key DocTagID and I'm not sure why it is doing so ?
The table in question is shown below, everything is as generated by inspectdb with the exception of the commented line which is where I pass control of the table over to Django.
class Doctagversion(models.Model):
id = models.IntegerField(db_column='Id', primary_key=True, blank=True) # Field name made lowercase.
doctagid = models.ForeignKey(DocTag, models.DO_NOTHING, db_column='DocTagId') # Field name made lowercase.
groupname = models.TextField(db_column='GroupName') # Field name made lowercase.
name = models.TextField(db_column='Name') # Field name made lowercase.
creationdate = models.DateTimeField(db_column='CreationDate') # Field name made lowercase.
lasteditdate = models.DateTimeField(db_column='LastEditDate', blank=True, null=True) # Field name made lowercase.
lastedituserid = models.IntegerField(db_column='LastEditUserId') # Field name made lowercase.
lastedituserdisplayname = models.TextField(db_column='LastEditUserDisplayName') # Field name made lowercase.
releasedate = models.DateTimeField(db_column='ReleaseDate', blank=True, null=True) # Field name made lowercase.
class Meta:
# managed = False
db_table = 'DocTagVersion'
Before passing on this control an initial migration for the schema in question is generated using python -m manage.py makemigrations, and applied with python -m manage.py migrate. This initial migration for the table is as follows, managed is set to False initially and the commented line is an entry I believe I should add to inform Django of the foreign key (inspectdb states as much in the generated models.py).
migrations.CreateModel(
name='Doctagversion',
fields=[
('id', models.IntegerField(blank=True, db_column='Id', primary_key=True, serialize=False)),
# ('doctagid',models.ForeignKey(db_column='DocTagId', default=-1, on_delete=models.deletion.DO_NOTHING, to='DocTag')),
('groupname', models.TextField(db_column='GroupName')),
('name', models.TextField(db_column='Name')),
('creationdate', models.DateTimeField(db_column='CreationDate')),
('lasteditdate', models.DateTimeField(blank=True, db_column='LastEditDate', null=True)),
('lastedituserid', models.IntegerField(db_column='LastEditUserId')),
('lastedituserdisplayname', models.TextField(db_column='LastEditUserDisplayName')),
('releasedate', models.DateTimeField(blank=True, db_column='ReleaseDate', null=True)),
],
options={
'db_table': 'DocTagVersion',
'managed': False,
},
),
When I enable control over the table the first migration simply changes the table options.
migrations.AlterModelOptions(
name='doctagversion',
options={},
),
Django adds the foreign key in question if it's not present in the initial migration as follows.
migrations.AddField(
model_name='doctagversion',
name='doctagid',
field=models.ForeignKey(db_column='DocTagId', default=-1, on_delete=django.db.models.deletion.DO_NOTHING, to='docutoo.DocTag'),
preserve_default=False,
),
Thereafter it repeatedly generates the following migration as one cycles between python m manage.py makemigrations and python -m manage.py migrate.
migrations.AlterField(
model_name='doctagversion',
name='doctagid',
field=models.ForeignKey(db_column='DocTagId', on_delete=django.db.models.deletion.DO_NOTHING, to='docutoo.DocTag'),
),
Perhaps my strategy is wrong and I should simply enable all tables in one migration ?
As far as I can tell the following related question(s) do not account for my situation :
Assigning function result instead of function to field attribute

From this Bug Report it seems Django migrations are quite sensitive to naming, I had commented out the db_table in all of my models. I believe I did this after creating the initial migration unwittingly breaking later ones.
class Meta :
...
# db_table = "TableName"
Because I broke the naming makemigrations could not see that the table name had changed and was doing it's best to resolve it by repeatedly declaring the field.
Aside
Coincidentally when one creates the initial migration for an existing database as described in my question (inspectdb -> makemigrations -> migrate), Django traverses the models alphabetically creating tables in the same order and ignoring ones foreign keys. It creates the tables alphabetically and then modifies them later to include foreign keys, clobbering any data that might be present in a existing database. It seems one must define the foreign keys as integer fields initially and change them back as one lets Django manage the tables. Alternatively one can comment out all of their models and generate migrations as one uncomments them in a fashion that resolves the foreign keys; Django will then link them accordingly. One may then squash the migrations into a single migration and set this as the initial migration.

Related

Django - Adding a primary key to an existing model with data

This is the existing model in a djnago app:
class Task_Master_Data(models.Model):
project_id = models.ForeignKey(Project_Master_Data, on_delete=models.CASCADE)
task_created_at = models.DateTimeField(auto_now_add=True)
task_updated_at = models.DateTimeField(auto_now=True)
task_name = models.CharField(max_length=200, null=True)
I want to add a new field (which will be a primary key):
task_id = models.AutoField(primary_key=True, null=False)
When I am making migrations from the terminal, it will provide me with the option of adding a default value, but it will, understandably, bring an error of:
UNIQUE constraint failed: new__main_task_master_data.task_id
What would be the best way forward for this kind of a scenario without having to delete all the data.
Note, I am using the default sqlite3 database.
This is the result of an old bug that is still unfixed. See here for the ticket.
I have created a workaround.
First create an empty migration (change taskapp to the name of the app where your models.py with Task_Master_Data lives):
python manage.py makemigrations taskapp --empty
Then copy the below migrations into this file and run python manage.py migrate but be sure to adjust the name of the app (replace taskapp with the name of your app) and the dependencies (0010_task_master_data should be replaced by the name of the migration that comes before it).
First the existing rows in the database need values for the new task_id field. I solved this by creating an IntegerField with null=True, which I then manually fill using a RunPython command. I then remove the id field (my previous primary key), and use AlterField to change the field into the AutoField. A default value is required, but it should never be used. When you create a new Task_Master_Data the task_id should be auto-incrementing.
# Generated by Django 3.0.4 on 2022-05-11 07:13
from django.db import migrations, models
def fill_taskid(apps, schema_editor):
# be sure to change it to your app here
Task_Master_Data = apps.get_model("taskapp", "Task_Master_Data")
db_alias = schema_editor.connection.alias
tasks = Task_Master_Data.objects.using(db_alias).all()
for i, task in enumerate(tasks):
task.task_id = i
Task_Master_Data.objects.using(db_alias).bulk_update(tasks, ["task_id"])
class Migration(migrations.Migration):
dependencies = [
('taskapp', '0010_task_master_data'),
]
operations = [
migrations.AddField(
model_name='task_master_data',
name='task_id',
field=models.IntegerField(null=True),
),
migrations.RunPython(fill_taskid),
migrations.RemoveField(
model_name='task_master_data',
name='id',
),
migrations.AlterField(
model_name='task_master_data',
name='task_id',
field=models.AutoField(default=-1, primary_key=True, serialize=False),
preserve_default=False,
),
]
Note that depending on the state of your database you might need to rollback some of your migrations. The migration I created should work if your database is in a state where your task_id field does not exist yet.

Foreign Key with db_constraint= False and postresql: foreign key appears to refer to wrong parent field

I am preparing tables for use in an analytics tool (Microsof Power BI).
It would be helpful to show relationships between the tables to assist in report building.
However, I don't want to use foreign keys with constraints because the method of data updating and integrity checking doesn't need constraints, and it will actually get in the way. Some of the alternate backends this code supports are for cloud databases that don't offer real foreign keys.
I mention that because it means I am trying to define a foreignkey field like this:
order_guid = models.ForeignKey("Dear_sales_header",to_field="order_guid",db_constraint=False,on_delete=models.DO_NOTHING)
The migration file has this:
operations = [
migrations.AlterField(
model_name='sales_fact',
name='order_guid',
field=models.ForeignKey(db_constraint=False, on_delete=django.db.models.deletion.DO_NOTHING, to='dear_zoho_analytics.Dear_sales_header', to_field='order_guid'),
),
]
This table is routed to a different database
python manage.py migrate --database=dear_analytics
does indicate that the migration file was applied (it is 0026)
Applying dear_zoho_analytics.0026_auto_20210217_2245... OK
But when I inspect the postgresql schema in my IDE, the column in sales_fact is renamed to
order_guid_id
so it looks like I have done something wrong because this seems to reference the id field of the "parent table" dear_sales_header yet I need it to refer to dear_sales_header.order_guid which is unique but not the primary key.
Partial extract of Models:
class AnalyticsTable:
# a dummy class used to mark the DearAnalytics tables. Creating a subclass of models.Model breaks things in the ORM such as bulk update
pass
class Sales_fact(models.Model, AnalyticsTable):
analytics_table_name = "sales_fact"
#order_guid = models.CharField(max_length=1024, null=True, blank=True, help_text="")
order_guid = models.ForeignKey("Dear_sales_header",to_field="order_guid",db_constraint=False,on_delete=models.DO_NOTHING)
source_dear = models.CharField(max_length=1024, null=True, blank=True,
help_text="Link to Dear instance which is the source of the data")
class Dear_sales_header(models.Model, AnalyticsTable):
analytics_table_name = "dear_sales_header"
source_dear = models.CharField(max_length=1024, null=True, blank=True,
help_text="Link to Dear instance which is the source of the data")
order_guid = models.CharField(max_length=1024, unique=True, help_text="")
global_sale_status = models.CharField(max_length=1024, null=True, blank=True, help_text="SO Header status")
order_status = models.CharField(max_length=1024, null=True, blank=True, help_text="")
Django adds suffix _id to ForeignKey name as documented
Behind the scenes, Django appends "_id" to the field name to create
its database column name.
If you want name not to include any additions you should set db_column
The name of the database column to use for this field. If this isn’t
given, Django will use the field’s name.
order_guid = models.ForeignKey(
"Dear_sales_header", to_field="order_guid", db_constraint=False,
db_column='order_guid',on_delete=models.DO_NOTHING
)
In MRE I am not able to reproduce any undesired behavior,
Django does properly map ForeignKey to column you set also when settingdb_constraint=True properly ads back right constraint and drops it as backward operation
constraint testbench_sales_fact_order_guid_f2dab1c5_fk_testbench
references testbench_dear_sales_header (order_guid)
deferrable initially deferred

Django - Existing DB Views and Foreign Keys

I have a simple view on the DB which selects from other DB's tables located on the same MSSQL Server to ultimately serve the collected info as a dropdown to the user.
So far I've added the Model with inspectdb:
class AutPricePlanView(models.Model):
priceplan_name = models.CharField(db_column='PricePlan', max_length=50, blank=True, unique=True)
class Meta:
managed = False # Created from a view. Don't remove.
db_table = 'AUT_PricePlanView'
Also I have a second existing (Django Native) Model where I want to use the values from the view for a Dropdown Field (to keep everything in sync):
class PricePlanDownload(models.Model):
requesting_user = models.CharField(blank=True, default=None, max_length=50, null=True)
requested_at = models.DateTimeField(auto_now_add=True)
document = models.FileField(upload_to='documents/price_plan_uploads/%Y/%m/%d', blank=True)
priceplan = models.ForeignKey(AutPricePlanView, null=True, on_delete=models.DO_NOTHING)
Makemigrations works fine but when I try to actually migrate I get the following issue: (shortened it a little bit)
django.db.utils.ProgrammingError: ('42000', "[42000] [FreeTDS][SQL Server]Foreign key references object 'AUT_PricePlanView' which is not a user table. (1768) (SQLExecDirectW)")
I would be really grateful if someone had an idea or a workaround since I can't figure out what the heck this has to do with a "user" table...
Since the view is not actually a table, you cannot set Foreign Key constraints. Since ForeignKey's default db_constraint value is True, Django tries to set Foreign Key constraints when performing migrations. This is the reason the migration fails.
So, you can turn off the db_constraint option. And you can remove the existing migration file, and re-create the migration file. Then, the migration will success and you can keep everything in sync.
class PricePlanDownload(models.Model):
... other fields ...
priceplan = models.ForeignKey(AutPricePlanView, null=True, on_delete=models.DO_NOTHING, db_constraint=False)
Pro Tip: You can review migration's SQL using python manage.py sqlmigrate <appname> <migration number>, like python manage.py sqlmigrate yourapp 0002.
Update: You can define __str__ to display the correct value at the dropdown menu.
class AutPricePlanView(models.Model):
priceplan_name = models.CharField(db_column='PricePlan', max_length=50, blank=True, unique=True, primary_key=True)
# null=False by default. See https://github.com/django/django/blob/master/django/db/models/fields/__init__.py#L132
def __str__(self):
return self.priceplan_name
class Meta:
managed = False # Created from a view. Don't remove.
db_table = 'AUT_PricePlanView'

How can one change the type of a Django model field from CharField to ForeignKey?

I need to change the type of a field in one of my Django models from CharField to ForeignKey. The fields are already populated with data, so I was wondering what is the best or right way to do this. Can I just update the field type and migrate, or are there any possible 'gotchas' to be aware of? N.B.: I just use vanilla Django management operations (makemigrations and migrate), not South.
This is likely a case where you want to do a multi-stage migration. My recommendation for this would look something like the following.
First off, let's assume this is your initial model, inside an application called discography:
from django.db import models
class Album(models.Model):
name = models.CharField(max_length=255)
artist = models.CharField(max_length=255)
Now, you realize that you want to use a ForeignKey for the artist instead. Well, as mentioned, this is not just a simple process for this. It has to be done in several steps.
Step 1, add a new field for the ForeignKey, making sure to mark it as null:
from django.db import models
class Album(models.Model):
name = models.CharField(max_length=255)
artist = models.CharField(max_length=255)
artist_link = models.ForeignKey('Artist', null=True)
class Artist(models.Model):
name = models.CharField(max_length=255)
...and create a migration for this change.
./manage.py makemigrations discography
Step 2, populate your new field. In order to do this, you have to create an empty migration.
./manage.py makemigrations --empty --name transfer_artists discography
Once you have this empty migration, you want to add a single RunPython operation to it in order to link your records. In this case, it could look something like this:
def link_artists(apps, schema_editor):
Album = apps.get_model('discography', 'Album')
Artist = apps.get_model('discography', 'Artist')
for album in Album.objects.all():
artist, created = Artist.objects.get_or_create(name=album.artist)
album.artist_link = artist
album.save()
Now that your data is transferred to the new field, you could actually be done and leave everything as is, using the new field for everything. Or, if you want to do a bit of cleanup, you want to create two more migrations.
For your first migration, you will want to delete your original field, artist. For your second migration, rename the new field artist_link to artist.
This is done in multiple steps to ensure that Django recognizes the operations properly. You could create a migration manually to handle this, but I will leave that to you to figure out.
Adding on top of Joey's answer, detailed steps for Django 2.2.11.
Here are the models from my use case, that consists of a Company and Employee model. We have to convert designation to a foreign key field. The app name is called core
class Company(CommonFields):
name = models.CharField(max_length=255, blank=True, null=True
class Employee(CommonFields):
company = models.ForeignKey("Company", on_delete=models.CASCADE, blank=True, null=True)
designation = models.CharField(max_length=100, blank=True, null=True)
Step 1
Create a foreign key designation_link in Employee and mark it as null=True
class Designation(CommonFields):
name = models.CharField(max_length=255)
company = models.ForeignKey("Company", on_delete=models.CASCADE, blank=True, null=True)
class Employee(CommonFields):
company = models.ForeignKey("Company", on_delete=models.CASCADE, blank=True, null=True)
designation = models.CharField(max_length=100, blank=True, null=True)
designation_link = models.ForeignKey("Designation", on_delete=models.CASCADE, blank=True, null=True)
Step 2
Create empty migration. Using the command:
python app_code/manage.py makemigrations --empty --name transfer_designations core
This will create a following file in migrations directory.
# Generated by Django 2.2.11 on 2020-04-02 05:56
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('core', '0006_auto_20200402_1119'),
]
operations = [
]
Step 3
Populate the empty migration with a function that loops over all Employees, creates a Designation and links it to the Employee.
In my use case each Designation is also linked to a Company. Which means that Designation may contain two rows for "managers", one for company A, another for company B.
Final migration would look something like this:
# core/migrations/0007_transfer_designations.py
# Generated by Django 2.2.11 on 2020-04-02 05:56
from django.db import migrations
def link_designation(apps, schema_editor):
Employee = apps.get_model('core', 'Employee')
Designation = apps.get_model('core', 'Designation')
for emp in Employee.objects.all():
if(emp.designation is not None and emp.company is not None):
desig, created = Designation.objects.get_or_create(name=emp.designation, company=emp.company)
emp.designation_link = desig
emp.save()
class Migration(migrations.Migration):
dependencies = [
('core', '0006_auto_20200402_1119'),
]
operations = [
migrations.RunPython(link_designation),
]
Step 4
Finally run this migration using:
python app_code/manage.py migrate core 0007
That's a continuation of the great answer by Joey.
How to rename the new field to the original name?
If the field has data, it probably means that you are using it elsewhere in your project, therefore this solution will leave you with a field named differently, and you have to either refactor the project to use the new field or delete the old field and rename the new one.
Be aware that this process is not going to prevent you to refactor code. If you where using a CharField with CHOICES, you were accessing its content with get_filename_display(), for example.
If you try to delete the field to make a migration, for then renaming the other field and make another migration, you'll see Django complaining because you cannot delete a field that you are using in the project.
Just create an empty migration as Joey explained, and put this in operations:
operations = [
migrations.RemoveField(
model_name='app_name',
name='old_field_name',
),
migrations.RenameField(
model_name='app_name',
old_name='old_field_name_link',
new_name='old_field_name',
),
]
Then run migrate and you'll have the changes made in your database, but obviously not in your model, it's time now to delete the old field and to rename new ForeignKey field to the original name.
I don't think that doing this is particularly hacky, but still, only do this kind of things if you are fully understanding what are you messing with.

How to fake migrations for not to create a specific existing intermediary table

I have following models
class VucutBolgesi(models.Model):
site = models.ForeignKey(Site)
bolge = models.CharField(verbose_name="Bölge", max_length=75)
hareketler = models.ManyToManyField("Hareket", verbose_name="Hareketler", null=True, blank=True, help_text="Bölgeyi çalıştıran hareketler")
class Hareket(models.Model):
site = models.ForeignKey(Site)
hareket = models.CharField(verbose_name="Hareket", max_length=75 )
bolgeler = models.ManyToManyField(VucutBolgesi, verbose_name="Çalıştırdığı Bölgeler", null=True, blank=True,
help_text="Hareketin çalıştırdığı bölgeler")
I have the same M2M on both table since I wish to display same intermediate table on both admin forms. They also have to use the same table (not create two separate tables) since one change in one admin form must be reflected to the other. Like, If I add a new Hareket to VucutBolgesi through HareketAdmin then the same result shoudl be visible on VucutBolgesiAdmin too.
For achieving this, I first remove hareketler M2M field from VucutBolgesi so Hareketler model would create the intermediate table. I migrate this and then add hareketler to VucutBolgesi with db_table attribute so it will recognize the same intermediary table.
final look of the field is as folows
hareketler = models.ManyToManyField("Hareket", verbose_name="Hareketler", db_table="antrenman_hareket_bolgeler",
null=True, blank=True, help_text="Bölgeyi çalıştıran hareketler")
When I try to migrate this, django throw following exception
django.db.utils.OperationalError: table "antrenman_hareket_bolgeler" already exists
How should I fake this migration?
Following is the migration django creates each time I run makemigrations
dependencies = [
('antrenman', '0005_vucutbolgesi_hareketler'),
]
operations = [
migrations.AddField(
model_name='vucutbolgesi',
name='hareketler',
field=models.ManyToManyField(to='antrenman.Hareket', db_table=b'antrenman_hareket_bolgeler', blank=True, help_text=b'B\xc3\xb6lgeyi \xc3\xa7al\xc4\xb1\xc5\x9ft\xc4\xb1ran hareketler', null=True, verbose_name=b'Hareketler'),
preserve_default=True,
),
]
Note: Editing related migration file and removing migrations.AddField fo not work since django creates the same migrations.AddField with each makemigrations
Is it possible to make a migration always to be faked, just override the apply and unapply methods. The consequences of this are not sufficiently investigated, but this far it works for me.
In the following example we create a migration that reuses django.contrib.auth.User.group's M2M table b'profile_user_groups:
from __future__ import unicode_literals
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('profile', '0001_initial'),
]
operations = [
migrations.AddField(
model_name='user',
name='organizations',
field=models.ManyToManyField(db_column=b'group_id', db_table=b'profile_user_groups', related_name='members', to='profile.Organization'),
),
]
def apply(self, project_state, schema_editor, collect_sql=False):
return project_state.clone()
def unapply(self, project_state, schema_editor, collect_sql=False):
return project_state.clone()
Solution was so simple.
You must be sure related migration is the only migration operation that needed to be faked. You must first create the migration with
python manage.py makemigrations antrenman
Then apply that migration with --fake
python manage.py migrate --fake antrenman
Handicap is, other developers should know that they have to fake related migration. If there are others migrations alongside with this one, they should make them first and then fake this one.
It is too bad there is no parameter that tells related migration should be real or fake.