This is the existing model in a djnago app:
class Task_Master_Data(models.Model):
project_id = models.ForeignKey(Project_Master_Data, on_delete=models.CASCADE)
task_created_at = models.DateTimeField(auto_now_add=True)
task_updated_at = models.DateTimeField(auto_now=True)
task_name = models.CharField(max_length=200, null=True)
I want to add a new field (which will be a primary key):
task_id = models.AutoField(primary_key=True, null=False)
When I am making migrations from the terminal, it will provide me with the option of adding a default value, but it will, understandably, bring an error of:
UNIQUE constraint failed: new__main_task_master_data.task_id
What would be the best way forward for this kind of a scenario without having to delete all the data.
Note, I am using the default sqlite3 database.
This is the result of an old bug that is still unfixed. See here for the ticket.
I have created a workaround.
First create an empty migration (change taskapp to the name of the app where your models.py with Task_Master_Data lives):
python manage.py makemigrations taskapp --empty
Then copy the below migrations into this file and run python manage.py migrate but be sure to adjust the name of the app (replace taskapp with the name of your app) and the dependencies (0010_task_master_data should be replaced by the name of the migration that comes before it).
First the existing rows in the database need values for the new task_id field. I solved this by creating an IntegerField with null=True, which I then manually fill using a RunPython command. I then remove the id field (my previous primary key), and use AlterField to change the field into the AutoField. A default value is required, but it should never be used. When you create a new Task_Master_Data the task_id should be auto-incrementing.
# Generated by Django 3.0.4 on 2022-05-11 07:13
from django.db import migrations, models
def fill_taskid(apps, schema_editor):
# be sure to change it to your app here
Task_Master_Data = apps.get_model("taskapp", "Task_Master_Data")
db_alias = schema_editor.connection.alias
tasks = Task_Master_Data.objects.using(db_alias).all()
for i, task in enumerate(tasks):
task.task_id = i
Task_Master_Data.objects.using(db_alias).bulk_update(tasks, ["task_id"])
class Migration(migrations.Migration):
dependencies = [
('taskapp', '0010_task_master_data'),
]
operations = [
migrations.AddField(
model_name='task_master_data',
name='task_id',
field=models.IntegerField(null=True),
),
migrations.RunPython(fill_taskid),
migrations.RemoveField(
model_name='task_master_data',
name='id',
),
migrations.AlterField(
model_name='task_master_data',
name='task_id',
field=models.AutoField(default=-1, primary_key=True, serialize=False),
preserve_default=False,
),
]
Note that depending on the state of your database you might need to rollback some of your migrations. The migration I created should work if your database is in a state where your task_id field does not exist yet.
Related
I have a model (devices) with a CharField primary key containing integers that I would like to convert to an IntegerField. The problem is the devices are also used as a foreign key in another tables meaning when I change the field in my models.py and run the makemigrations/migrate I keep getting an error prompting me to drop the dependent objects first. How can I change the device primary key to an IntegerField without dropping the dependent objects?
models.py
class Device(models.Model):
id = models.CharField(primary_key=True, unique=True, max_length=255)
date_created = models.DateField()
class Station(models.Model):
id = models.AutoField(primary_key=True)
device = models.ForeignKey('Device', on_delete=models.PROTECT)
error when running the migration
constraint api_station_device_id_117642ec_fk_api_device_id on table api_station depends on index api_device_pkey
HINT: Use DROP ... CASCADE to drop the dependent objects too.
The error message you are receiving is due to the fact that changing the primary key type of the Device model will require changing the foreign key type of the device field in the Station model as well. This will create a dependency that requires the api_station_device_id_117642ec_fk_api_device_id constraint on the api_station table to be dropped, which is not allowed by the database without first dropping the dependent objects.
You can create a new id field of type IntegerField in the Device model and set it as the primary key. You can then populate this new field with the integer values of the old id field and remove the old id field after updating the Station model to reference the new id field. eg in your models.py file
class Device(models.Model):
id = models.AutoField(primary_key=True)
id_char = models.CharField(unique=True, max_length=255)
date_created = models.DateField()
def save(self, *args, **kwargs):
# set the integer value of the new id field to the integer value of the old id field
if self.id_char is not None:
self.id = int(self.id_char)
super().save(*args, **kwargs)
class Station(models.Model):
id = models.AutoField(primary_key=True)
device = models.ForeignKey('Device', on_delete=models.PROTECT, to_field='id')
Write a script to populate the id field with the integer values of
the id_char field for all existing Device objects.
Here are the steps to create a custom Django management command that runs the script:
Create a new Python module (e.g., update_device_ids.py) in a Django
app that has the Device model. You can put this module in the
management/commands/ directory within your app to define a new
management command.
Import the relevant Django models and the BaseCommand class from the
django.core.management.base module. Your module may look like this
from django.core.management.base import BaseCommand
from myapp.models import Device
class Command(BaseCommand):
help = 'Updates the Device id field with the integer value of the id_char field for all devices'
def handle(self, *args, **options):
devices = Device.objects.all()
for device in devices:
if device.id_char is not None:
device.id = int(device.id_char)
device.save()
Then follow the below process:
Update the id field in your Device model from CharField to
IntegerField.
Create a new migration by running python manage.py makemigrations.
Before running the migration, create a new temporary field in your
Device model that will store the integer value of the old id_char
field. For example, you could add the following line to your Device
model:
id_int = models.IntegerField(null=True, blank=True)
Run the python manage.py update_device_ids command to populate the
id_int field with the integer values of the old id_char field.
Create a data migration by running python manage.py makemigrations --empty yourappname.
Edit the data migration file (in the migrations/ directory) and add
the following code to the forwards() method:
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('yourappname', 'the_previous_migration'),
]
operations = [
migrations.AddField(
model_name='device',
name='id_int',
field=models.IntegerField(null=True, blank=True),
),
migrations.RunSQL('UPDATE yourappname_device SET id_int = CAST(id as integer)'),
migrations.RemoveField(
model_name='device',
name='id',
),
migrations.RenameField(
model_name='device',
old_name='id_int',
new_name='id',
),
]
This code adds a new IntegerField field called id_int, populates it with the integer values of the old id_char field using a SQL command, removes the old id_char field, and renames the id_int field to id.
Finally, run the data migration by running python manage.py migrate.
I have two models which are Contact and UpdateInfo. Update model is a foreignkey relationship with Contact like below:
class Contacts(models.Model):
full_name = models.CharField(max_length=100, blank=True)
notes = RichTextField(blank=True)
/............./
class UpdateInfo(models.Model):
contacts = models.ForeignKey(Contacts,on_delete=models.CASCADE, related_name='update_info')
updated_at = models.DateTimeField(auto_now=True)
modified_by = models.CharField(max_length=100, blank=True)
Now what I need is to put the notes field into UpdateInfo model and remove from Contacts model. Because now the businees requirement has changed and what they need is they need to add notes in the existing notes and want to see them who added the notes and when they added it with the notes appended.
But the problem is if I remove the notes field from Contacts and add it to Updateinfo model , there are many data into the production and the notes field data will be lost.
Now how can we run migrate in such a way that I add notes field into the Updatinfo model and also migrating the data at the same time.?? Is this what is done by dev ops or we should do it?? Because usually dev ops are migrating the data to the production server from staging area.
I need expert opinion on this.
You can create two migration files: 0002_add_notes_to_updateinfo.py and 0003_remove_notes_from_contact.py the first one will add the field to UpdateInfo model and then execute a function to migrate old data, after that in the second migration file you can remove the old field.
0002_add_notes_to_updateinfo.py:
from django.db import migrations
def migrate_old_data(apps, schema_editor):
Contacts = apps.get_model('yourappname', 'Contacts')
UpdateInfo = apps.get_model('yourappname', 'UpdateInfo')
for contacts in Contacts.objects.all():
UpdateInfo.objects.filter(contacts=contacts).update(notes=contacts.notes)
class Migration(migrations.Migration):
dependencies = [
#dependencies here,
]
operations = [
migrations.AddField(
model_name='updateinfo',
name='notes',
field=RichTextField(blank=True),
preserve_default=False,
),
migrations.RunPython(migrate_old_data),
]
0003_remove_notes_from_contact.py:
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('yourapp', '0002_add_notes_to_updateinfo'),
]
operations = [
migrations.RemoveField(
model_name='contacts',
name='notes',
),
]
I am importing an existing database into it's own Django project. I have generated the initial models from the database, via inspectdb, and am enabling Django to control each table one at a time by commenting the managed=False lines in the table meta settings. I've started with the simple models and am hitting a snag when enabling tables with foreign keys. Django keeps generating the same migration for the foreign key DocTagID and I'm not sure why it is doing so ?
The table in question is shown below, everything is as generated by inspectdb with the exception of the commented line which is where I pass control of the table over to Django.
class Doctagversion(models.Model):
id = models.IntegerField(db_column='Id', primary_key=True, blank=True) # Field name made lowercase.
doctagid = models.ForeignKey(DocTag, models.DO_NOTHING, db_column='DocTagId') # Field name made lowercase.
groupname = models.TextField(db_column='GroupName') # Field name made lowercase.
name = models.TextField(db_column='Name') # Field name made lowercase.
creationdate = models.DateTimeField(db_column='CreationDate') # Field name made lowercase.
lasteditdate = models.DateTimeField(db_column='LastEditDate', blank=True, null=True) # Field name made lowercase.
lastedituserid = models.IntegerField(db_column='LastEditUserId') # Field name made lowercase.
lastedituserdisplayname = models.TextField(db_column='LastEditUserDisplayName') # Field name made lowercase.
releasedate = models.DateTimeField(db_column='ReleaseDate', blank=True, null=True) # Field name made lowercase.
class Meta:
# managed = False
db_table = 'DocTagVersion'
Before passing on this control an initial migration for the schema in question is generated using python -m manage.py makemigrations, and applied with python -m manage.py migrate. This initial migration for the table is as follows, managed is set to False initially and the commented line is an entry I believe I should add to inform Django of the foreign key (inspectdb states as much in the generated models.py).
migrations.CreateModel(
name='Doctagversion',
fields=[
('id', models.IntegerField(blank=True, db_column='Id', primary_key=True, serialize=False)),
# ('doctagid',models.ForeignKey(db_column='DocTagId', default=-1, on_delete=models.deletion.DO_NOTHING, to='DocTag')),
('groupname', models.TextField(db_column='GroupName')),
('name', models.TextField(db_column='Name')),
('creationdate', models.DateTimeField(db_column='CreationDate')),
('lasteditdate', models.DateTimeField(blank=True, db_column='LastEditDate', null=True)),
('lastedituserid', models.IntegerField(db_column='LastEditUserId')),
('lastedituserdisplayname', models.TextField(db_column='LastEditUserDisplayName')),
('releasedate', models.DateTimeField(blank=True, db_column='ReleaseDate', null=True)),
],
options={
'db_table': 'DocTagVersion',
'managed': False,
},
),
When I enable control over the table the first migration simply changes the table options.
migrations.AlterModelOptions(
name='doctagversion',
options={},
),
Django adds the foreign key in question if it's not present in the initial migration as follows.
migrations.AddField(
model_name='doctagversion',
name='doctagid',
field=models.ForeignKey(db_column='DocTagId', default=-1, on_delete=django.db.models.deletion.DO_NOTHING, to='docutoo.DocTag'),
preserve_default=False,
),
Thereafter it repeatedly generates the following migration as one cycles between python m manage.py makemigrations and python -m manage.py migrate.
migrations.AlterField(
model_name='doctagversion',
name='doctagid',
field=models.ForeignKey(db_column='DocTagId', on_delete=django.db.models.deletion.DO_NOTHING, to='docutoo.DocTag'),
),
Perhaps my strategy is wrong and I should simply enable all tables in one migration ?
As far as I can tell the following related question(s) do not account for my situation :
Assigning function result instead of function to field attribute
From this Bug Report it seems Django migrations are quite sensitive to naming, I had commented out the db_table in all of my models. I believe I did this after creating the initial migration unwittingly breaking later ones.
class Meta :
...
# db_table = "TableName"
Because I broke the naming makemigrations could not see that the table name had changed and was doing it's best to resolve it by repeatedly declaring the field.
Aside
Coincidentally when one creates the initial migration for an existing database as described in my question (inspectdb -> makemigrations -> migrate), Django traverses the models alphabetically creating tables in the same order and ignoring ones foreign keys. It creates the tables alphabetically and then modifies them later to include foreign keys, clobbering any data that might be present in a existing database. It seems one must define the foreign keys as integer fields initially and change them back as one lets Django manage the tables. Alternatively one can comment out all of their models and generate migrations as one uncomments them in a fashion that resolves the foreign keys; Django will then link them accordingly. One may then squash the migrations into a single migration and set this as the initial migration.
I need to change the type of a field in one of my Django models from CharField to ForeignKey. The fields are already populated with data, so I was wondering what is the best or right way to do this. Can I just update the field type and migrate, or are there any possible 'gotchas' to be aware of? N.B.: I just use vanilla Django management operations (makemigrations and migrate), not South.
This is likely a case where you want to do a multi-stage migration. My recommendation for this would look something like the following.
First off, let's assume this is your initial model, inside an application called discography:
from django.db import models
class Album(models.Model):
name = models.CharField(max_length=255)
artist = models.CharField(max_length=255)
Now, you realize that you want to use a ForeignKey for the artist instead. Well, as mentioned, this is not just a simple process for this. It has to be done in several steps.
Step 1, add a new field for the ForeignKey, making sure to mark it as null:
from django.db import models
class Album(models.Model):
name = models.CharField(max_length=255)
artist = models.CharField(max_length=255)
artist_link = models.ForeignKey('Artist', null=True)
class Artist(models.Model):
name = models.CharField(max_length=255)
...and create a migration for this change.
./manage.py makemigrations discography
Step 2, populate your new field. In order to do this, you have to create an empty migration.
./manage.py makemigrations --empty --name transfer_artists discography
Once you have this empty migration, you want to add a single RunPython operation to it in order to link your records. In this case, it could look something like this:
def link_artists(apps, schema_editor):
Album = apps.get_model('discography', 'Album')
Artist = apps.get_model('discography', 'Artist')
for album in Album.objects.all():
artist, created = Artist.objects.get_or_create(name=album.artist)
album.artist_link = artist
album.save()
Now that your data is transferred to the new field, you could actually be done and leave everything as is, using the new field for everything. Or, if you want to do a bit of cleanup, you want to create two more migrations.
For your first migration, you will want to delete your original field, artist. For your second migration, rename the new field artist_link to artist.
This is done in multiple steps to ensure that Django recognizes the operations properly. You could create a migration manually to handle this, but I will leave that to you to figure out.
Adding on top of Joey's answer, detailed steps for Django 2.2.11.
Here are the models from my use case, that consists of a Company and Employee model. We have to convert designation to a foreign key field. The app name is called core
class Company(CommonFields):
name = models.CharField(max_length=255, blank=True, null=True
class Employee(CommonFields):
company = models.ForeignKey("Company", on_delete=models.CASCADE, blank=True, null=True)
designation = models.CharField(max_length=100, blank=True, null=True)
Step 1
Create a foreign key designation_link in Employee and mark it as null=True
class Designation(CommonFields):
name = models.CharField(max_length=255)
company = models.ForeignKey("Company", on_delete=models.CASCADE, blank=True, null=True)
class Employee(CommonFields):
company = models.ForeignKey("Company", on_delete=models.CASCADE, blank=True, null=True)
designation = models.CharField(max_length=100, blank=True, null=True)
designation_link = models.ForeignKey("Designation", on_delete=models.CASCADE, blank=True, null=True)
Step 2
Create empty migration. Using the command:
python app_code/manage.py makemigrations --empty --name transfer_designations core
This will create a following file in migrations directory.
# Generated by Django 2.2.11 on 2020-04-02 05:56
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('core', '0006_auto_20200402_1119'),
]
operations = [
]
Step 3
Populate the empty migration with a function that loops over all Employees, creates a Designation and links it to the Employee.
In my use case each Designation is also linked to a Company. Which means that Designation may contain two rows for "managers", one for company A, another for company B.
Final migration would look something like this:
# core/migrations/0007_transfer_designations.py
# Generated by Django 2.2.11 on 2020-04-02 05:56
from django.db import migrations
def link_designation(apps, schema_editor):
Employee = apps.get_model('core', 'Employee')
Designation = apps.get_model('core', 'Designation')
for emp in Employee.objects.all():
if(emp.designation is not None and emp.company is not None):
desig, created = Designation.objects.get_or_create(name=emp.designation, company=emp.company)
emp.designation_link = desig
emp.save()
class Migration(migrations.Migration):
dependencies = [
('core', '0006_auto_20200402_1119'),
]
operations = [
migrations.RunPython(link_designation),
]
Step 4
Finally run this migration using:
python app_code/manage.py migrate core 0007
That's a continuation of the great answer by Joey.
How to rename the new field to the original name?
If the field has data, it probably means that you are using it elsewhere in your project, therefore this solution will leave you with a field named differently, and you have to either refactor the project to use the new field or delete the old field and rename the new one.
Be aware that this process is not going to prevent you to refactor code. If you where using a CharField with CHOICES, you were accessing its content with get_filename_display(), for example.
If you try to delete the field to make a migration, for then renaming the other field and make another migration, you'll see Django complaining because you cannot delete a field that you are using in the project.
Just create an empty migration as Joey explained, and put this in operations:
operations = [
migrations.RemoveField(
model_name='app_name',
name='old_field_name',
),
migrations.RenameField(
model_name='app_name',
old_name='old_field_name_link',
new_name='old_field_name',
),
]
Then run migrate and you'll have the changes made in your database, but obviously not in your model, it's time now to delete the old field and to rename new ForeignKey field to the original name.
I don't think that doing this is particularly hacky, but still, only do this kind of things if you are fully understanding what are you messing with.
I have following models
class VucutBolgesi(models.Model):
site = models.ForeignKey(Site)
bolge = models.CharField(verbose_name="Bölge", max_length=75)
hareketler = models.ManyToManyField("Hareket", verbose_name="Hareketler", null=True, blank=True, help_text="Bölgeyi çalıştıran hareketler")
class Hareket(models.Model):
site = models.ForeignKey(Site)
hareket = models.CharField(verbose_name="Hareket", max_length=75 )
bolgeler = models.ManyToManyField(VucutBolgesi, verbose_name="Çalıştırdığı Bölgeler", null=True, blank=True,
help_text="Hareketin çalıştırdığı bölgeler")
I have the same M2M on both table since I wish to display same intermediate table on both admin forms. They also have to use the same table (not create two separate tables) since one change in one admin form must be reflected to the other. Like, If I add a new Hareket to VucutBolgesi through HareketAdmin then the same result shoudl be visible on VucutBolgesiAdmin too.
For achieving this, I first remove hareketler M2M field from VucutBolgesi so Hareketler model would create the intermediate table. I migrate this and then add hareketler to VucutBolgesi with db_table attribute so it will recognize the same intermediary table.
final look of the field is as folows
hareketler = models.ManyToManyField("Hareket", verbose_name="Hareketler", db_table="antrenman_hareket_bolgeler",
null=True, blank=True, help_text="Bölgeyi çalıştıran hareketler")
When I try to migrate this, django throw following exception
django.db.utils.OperationalError: table "antrenman_hareket_bolgeler" already exists
How should I fake this migration?
Following is the migration django creates each time I run makemigrations
dependencies = [
('antrenman', '0005_vucutbolgesi_hareketler'),
]
operations = [
migrations.AddField(
model_name='vucutbolgesi',
name='hareketler',
field=models.ManyToManyField(to='antrenman.Hareket', db_table=b'antrenman_hareket_bolgeler', blank=True, help_text=b'B\xc3\xb6lgeyi \xc3\xa7al\xc4\xb1\xc5\x9ft\xc4\xb1ran hareketler', null=True, verbose_name=b'Hareketler'),
preserve_default=True,
),
]
Note: Editing related migration file and removing migrations.AddField fo not work since django creates the same migrations.AddField with each makemigrations
Is it possible to make a migration always to be faked, just override the apply and unapply methods. The consequences of this are not sufficiently investigated, but this far it works for me.
In the following example we create a migration that reuses django.contrib.auth.User.group's M2M table b'profile_user_groups:
from __future__ import unicode_literals
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('profile', '0001_initial'),
]
operations = [
migrations.AddField(
model_name='user',
name='organizations',
field=models.ManyToManyField(db_column=b'group_id', db_table=b'profile_user_groups', related_name='members', to='profile.Organization'),
),
]
def apply(self, project_state, schema_editor, collect_sql=False):
return project_state.clone()
def unapply(self, project_state, schema_editor, collect_sql=False):
return project_state.clone()
Solution was so simple.
You must be sure related migration is the only migration operation that needed to be faked. You must first create the migration with
python manage.py makemigrations antrenman
Then apply that migration with --fake
python manage.py migrate --fake antrenman
Handicap is, other developers should know that they have to fake related migration. If there are others migrations alongside with this one, they should make them first and then fake this one.
It is too bad there is no parameter that tells related migration should be real or fake.