preserve data while making changes to fields of model in django - django

I have two models which are Contact and UpdateInfo. Update model is a foreignkey relationship with Contact like below:
class Contacts(models.Model):
full_name = models.CharField(max_length=100, blank=True)
notes = RichTextField(blank=True)
/............./
class UpdateInfo(models.Model):
contacts = models.ForeignKey(Contacts,on_delete=models.CASCADE, related_name='update_info')
updated_at = models.DateTimeField(auto_now=True)
modified_by = models.CharField(max_length=100, blank=True)
Now what I need is to put the notes field into UpdateInfo model and remove from Contacts model. Because now the businees requirement has changed and what they need is they need to add notes in the existing notes and want to see them who added the notes and when they added it with the notes appended.
But the problem is if I remove the notes field from Contacts and add it to Updateinfo model , there are many data into the production and the notes field data will be lost.
Now how can we run migrate in such a way that I add notes field into the Updatinfo model and also migrating the data at the same time.?? Is this what is done by dev ops or we should do it?? Because usually dev ops are migrating the data to the production server from staging area.
I need expert opinion on this.

You can create two migration files: 0002_add_notes_to_updateinfo.py and 0003_remove_notes_from_contact.py the first one will add the field to UpdateInfo model and then execute a function to migrate old data, after that in the second migration file you can remove the old field.
0002_add_notes_to_updateinfo.py:
from django.db import migrations
def migrate_old_data(apps, schema_editor):
Contacts = apps.get_model('yourappname', 'Contacts')
UpdateInfo = apps.get_model('yourappname', 'UpdateInfo')
for contacts in Contacts.objects.all():
UpdateInfo.objects.filter(contacts=contacts).update(notes=contacts.notes)
class Migration(migrations.Migration):
dependencies = [
#dependencies here,
]
operations = [
migrations.AddField(
model_name='updateinfo',
name='notes',
field=RichTextField(blank=True),
preserve_default=False,
),
migrations.RunPython(migrate_old_data),
]
0003_remove_notes_from_contact.py:
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('yourapp', '0002_add_notes_to_updateinfo'),
]
operations = [
migrations.RemoveField(
model_name='contacts',
name='notes',
),
]

Related

Django - Adding a primary key to an existing model with data

This is the existing model in a djnago app:
class Task_Master_Data(models.Model):
project_id = models.ForeignKey(Project_Master_Data, on_delete=models.CASCADE)
task_created_at = models.DateTimeField(auto_now_add=True)
task_updated_at = models.DateTimeField(auto_now=True)
task_name = models.CharField(max_length=200, null=True)
I want to add a new field (which will be a primary key):
task_id = models.AutoField(primary_key=True, null=False)
When I am making migrations from the terminal, it will provide me with the option of adding a default value, but it will, understandably, bring an error of:
UNIQUE constraint failed: new__main_task_master_data.task_id
What would be the best way forward for this kind of a scenario without having to delete all the data.
Note, I am using the default sqlite3 database.
This is the result of an old bug that is still unfixed. See here for the ticket.
I have created a workaround.
First create an empty migration (change taskapp to the name of the app where your models.py with Task_Master_Data lives):
python manage.py makemigrations taskapp --empty
Then copy the below migrations into this file and run python manage.py migrate but be sure to adjust the name of the app (replace taskapp with the name of your app) and the dependencies (0010_task_master_data should be replaced by the name of the migration that comes before it).
First the existing rows in the database need values for the new task_id field. I solved this by creating an IntegerField with null=True, which I then manually fill using a RunPython command. I then remove the id field (my previous primary key), and use AlterField to change the field into the AutoField. A default value is required, but it should never be used. When you create a new Task_Master_Data the task_id should be auto-incrementing.
# Generated by Django 3.0.4 on 2022-05-11 07:13
from django.db import migrations, models
def fill_taskid(apps, schema_editor):
# be sure to change it to your app here
Task_Master_Data = apps.get_model("taskapp", "Task_Master_Data")
db_alias = schema_editor.connection.alias
tasks = Task_Master_Data.objects.using(db_alias).all()
for i, task in enumerate(tasks):
task.task_id = i
Task_Master_Data.objects.using(db_alias).bulk_update(tasks, ["task_id"])
class Migration(migrations.Migration):
dependencies = [
('taskapp', '0010_task_master_data'),
]
operations = [
migrations.AddField(
model_name='task_master_data',
name='task_id',
field=models.IntegerField(null=True),
),
migrations.RunPython(fill_taskid),
migrations.RemoveField(
model_name='task_master_data',
name='id',
),
migrations.AlterField(
model_name='task_master_data',
name='task_id',
field=models.AutoField(default=-1, primary_key=True, serialize=False),
preserve_default=False,
),
]
Note that depending on the state of your database you might need to rollback some of your migrations. The migration I created should work if your database is in a state where your task_id field does not exist yet.

How can I undo this (accidental) table inheritance in Django? [duplicate]

My current project uses multi-table inheritance models:
from django.db import models
class Place(models.Model):
name = models.CharField(max_length=50)
address = models.CharField(max_length=80)
class Restaurant(Place):
serves_hot_dogs = models.BooleanField(default=False)
serves_pizza = models.BooleanField(default=False)
class Cinema(Place):
sells_tickets = models.BooleanField(default=False)
sells_popcorn = models.BooleanField(default=False)
I want to switch to abstract base classes instead. Since my model is already deployed I need to write some custom migrations to convert the above schema to this one:
from django.db import models
class AbstractPlace(models.Model):
name = models.CharField(max_length=50)
address = models.CharField(max_length=80)
class Meta:
abstract = True
class Restaurant(AbstractPlace):
serves_hot_dogs = models.BooleanField(default=False)
serves_pizza = models.BooleanField(default=False)
class Cinema(AbstractPlace):
sells_tickets = models.BooleanField(default=False)
sells_popcorn = models.BooleanField(default=False)
Does anyone have any advice on the steps to take to achieve this?
I recently tackled this exact problem, which I solved by writing and running the migration in the code block below - loosely translated to fit the models in your case.
I'm pretty sure that it's not possible to alter the tables of the old Restaurant and Cinema models directly, as if you try to add fields to them, they will collide with the existing fields of the base model, and if you try to "decouple" the derived models from the base model by e.g. by manually setting abstract=True in the base model's options, Django reports that it's unable to find the base models of Restaurant and Cinema. (These issues might be caused by a bug, for all I know.) To circumvent this problem, I created new tables for the derived models, copied the data from the old tables to the new ones, deleted the old tables, and renamed the new tables to match the names of the old ones.
I got large parts of the code below from code generated by Django, which can be reproduced by creating a temporary migration (before creating one with the code below) which only deletes Restaurant, Cinema and Place, running makemigrations, and copying the CreateModel()s and AlterField()s (for related fields pointing to Restaurant or Cinema) from the generated migration.
For the record, I'm using Django 3.1.4.
from django.db import migrations, models
def copy_objects_from_restaurant_and_cinema_to_restaurant_tmp_and_cinema_tmp(apps, schema_editor):
Restaurant_Tmp = apps.get_model('<app name>', 'Restaurant_Tmp')
Cinema_Tmp = apps.get_model('<app name>', 'Cinema_Tmp')
Restaurant = apps.get_model('<app name>', 'Restaurant')
Cinema = apps.get_model('<app name>', 'Cinema')
# The `_meta.fields` list includes the PK
copy_objects_from_old_model_to_new_model(Restaurant, Restaurant_Tmp, Restaurant_Tmp._meta.fields)
copy_objects_from_old_model_to_new_model(Cinema, Cinema_Tmp, Cinema_Tmp._meta.fields)
def copy_objects_from_old_model_to_new_model(old_model, new_model, fields_to_copy):
field_names = [field.name for field in fields_to_copy]
for old_obj in old_model.objects.all():
old_obj_field_dict = {
field_name: getattr(old_obj, field_name)
for field_name in field_names
}
new_model.objects.create(**old_obj_field_dict)
def copy_objects_from_restaurant_tmp_and_cinema_tmp_to_restaurant_and_cinema(apps, schema_editor):
Restaurant_Tmp = apps.get_model('<app name>', 'Restaurant_Tmp')
Cinema_Tmp = apps.get_model('<app name>', 'Cinema_Tmp')
Restaurant = apps.get_model('<app name>', 'Restaurant')
Cinema = apps.get_model('<app name>', 'Cinema')
copy_objects_from_old_model_to_new_model(Restaurant_Tmp, Restaurant, Restaurant_Tmp._meta.fields)
copy_objects_from_old_model_to_new_model(Cinema_Tmp, Cinema, Cinema_Tmp._meta.fields)
class Migration(migrations.Migration):
dependencies = [
('<app name>', '<last migration>'),
]
operations = [
migrations.CreateModel(
name='Restaurant_Tmp',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=50)),
('address', models.CharField(max_length=80)),
('serves_hot_dogs', models.BooleanField(default=False)),
('serves_pizza', models.BooleanField(default=False)),
],
options={
'abstract': False,
},
),
migrations.CreateModel(
name='Cinema_Tmp',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=50)),
('address', models.CharField(max_length=80)),
('sells_tickets', models.BooleanField(default=False)),
('sells_popcorn', models.BooleanField(default=False)),
],
options={
'abstract': False,
},
),
migrations.RunPython(copy_objects_from_restaurant_and_cinema_to_restaurant_tmp_and_cinema_tmp, migrations.RunPython.noop),
# Update foreign keys to reference the non-abstract models directly,
# instead of through the (automatically generated) `place_ptr` field of the old models
<
Run `migrations.AlterField()` here for each related field (like ForeignKey) of other models that point to Restaurant or Cinema,
but change their `to` argument from e.g. `<app name>.restaurant` to `<app name>.restaurant_tmp`
>
migrations.RunPython(migrations.RunPython.noop, copy_objects_from_restaurant_tmp_and_cinema_tmp_to_restaurant_and_cinema),
migrations.DeleteModel(
name='Restaurant',
),
migrations.DeleteModel(
name='Cinema',
),
migrations.DeleteModel(
name='Place',
),
migrations.RenameModel(
old_name='Restaurant_Tmp',
new_name='Restaurant',
),
migrations.RenameModel(
old_name='Cinema_Tmp',
new_name='Cinema',
),
]
Note that the migration I originally wrote was only tested to work using SQLite; other database management systems might not accept such a large variety of migration operations, and you might have to split it into multiple migrations. (I'm somewhat unsure what exactly could cause this problem, but I can recall that I've experienced it with PostgreSQL.)
Please let me know if this solves your problem! 😊

How to start Django models auto increment key from a certain number like 2456?

In mysql if you want to start your auto-increment key from 2456 you can write AUTO_INCREMENT=2456
Also in PostgreSQL we can do this by using Sequence
CREATE SEQUENCE serial START 2456;
INSERT INTO distributors VALUES (nextval('serial'), 'nothing');
But in Django Rest Framework or Django by default auto-increment starts from 1. If I have a model like this
class Content(models.Model):
title = models.CharField(max_length=70, blank=False, default='')
description = models.CharField(max_length=200,blank=False, default='')
published = models.BooleanField(default=False)
How can I make sure that the first Content starts from 2456?
Create a RunSQL--(Django Doc) operation
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('sample', '0003_content'),
]
operations = [
migrations.RunSQL(
"ALTER SEQUENCE sample_content_id_seq RESTART WITH 1453"
),
]

How can one change the type of a Django model field from CharField to ForeignKey?

I need to change the type of a field in one of my Django models from CharField to ForeignKey. The fields are already populated with data, so I was wondering what is the best or right way to do this. Can I just update the field type and migrate, or are there any possible 'gotchas' to be aware of? N.B.: I just use vanilla Django management operations (makemigrations and migrate), not South.
This is likely a case where you want to do a multi-stage migration. My recommendation for this would look something like the following.
First off, let's assume this is your initial model, inside an application called discography:
from django.db import models
class Album(models.Model):
name = models.CharField(max_length=255)
artist = models.CharField(max_length=255)
Now, you realize that you want to use a ForeignKey for the artist instead. Well, as mentioned, this is not just a simple process for this. It has to be done in several steps.
Step 1, add a new field for the ForeignKey, making sure to mark it as null:
from django.db import models
class Album(models.Model):
name = models.CharField(max_length=255)
artist = models.CharField(max_length=255)
artist_link = models.ForeignKey('Artist', null=True)
class Artist(models.Model):
name = models.CharField(max_length=255)
...and create a migration for this change.
./manage.py makemigrations discography
Step 2, populate your new field. In order to do this, you have to create an empty migration.
./manage.py makemigrations --empty --name transfer_artists discography
Once you have this empty migration, you want to add a single RunPython operation to it in order to link your records. In this case, it could look something like this:
def link_artists(apps, schema_editor):
Album = apps.get_model('discography', 'Album')
Artist = apps.get_model('discography', 'Artist')
for album in Album.objects.all():
artist, created = Artist.objects.get_or_create(name=album.artist)
album.artist_link = artist
album.save()
Now that your data is transferred to the new field, you could actually be done and leave everything as is, using the new field for everything. Or, if you want to do a bit of cleanup, you want to create two more migrations.
For your first migration, you will want to delete your original field, artist. For your second migration, rename the new field artist_link to artist.
This is done in multiple steps to ensure that Django recognizes the operations properly. You could create a migration manually to handle this, but I will leave that to you to figure out.
Adding on top of Joey's answer, detailed steps for Django 2.2.11.
Here are the models from my use case, that consists of a Company and Employee model. We have to convert designation to a foreign key field. The app name is called core
class Company(CommonFields):
name = models.CharField(max_length=255, blank=True, null=True
class Employee(CommonFields):
company = models.ForeignKey("Company", on_delete=models.CASCADE, blank=True, null=True)
designation = models.CharField(max_length=100, blank=True, null=True)
Step 1
Create a foreign key designation_link in Employee and mark it as null=True
class Designation(CommonFields):
name = models.CharField(max_length=255)
company = models.ForeignKey("Company", on_delete=models.CASCADE, blank=True, null=True)
class Employee(CommonFields):
company = models.ForeignKey("Company", on_delete=models.CASCADE, blank=True, null=True)
designation = models.CharField(max_length=100, blank=True, null=True)
designation_link = models.ForeignKey("Designation", on_delete=models.CASCADE, blank=True, null=True)
Step 2
Create empty migration. Using the command:
python app_code/manage.py makemigrations --empty --name transfer_designations core
This will create a following file in migrations directory.
# Generated by Django 2.2.11 on 2020-04-02 05:56
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('core', '0006_auto_20200402_1119'),
]
operations = [
]
Step 3
Populate the empty migration with a function that loops over all Employees, creates a Designation and links it to the Employee.
In my use case each Designation is also linked to a Company. Which means that Designation may contain two rows for "managers", one for company A, another for company B.
Final migration would look something like this:
# core/migrations/0007_transfer_designations.py
# Generated by Django 2.2.11 on 2020-04-02 05:56
from django.db import migrations
def link_designation(apps, schema_editor):
Employee = apps.get_model('core', 'Employee')
Designation = apps.get_model('core', 'Designation')
for emp in Employee.objects.all():
if(emp.designation is not None and emp.company is not None):
desig, created = Designation.objects.get_or_create(name=emp.designation, company=emp.company)
emp.designation_link = desig
emp.save()
class Migration(migrations.Migration):
dependencies = [
('core', '0006_auto_20200402_1119'),
]
operations = [
migrations.RunPython(link_designation),
]
Step 4
Finally run this migration using:
python app_code/manage.py migrate core 0007
That's a continuation of the great answer by Joey.
How to rename the new field to the original name?
If the field has data, it probably means that you are using it elsewhere in your project, therefore this solution will leave you with a field named differently, and you have to either refactor the project to use the new field or delete the old field and rename the new one.
Be aware that this process is not going to prevent you to refactor code. If you where using a CharField with CHOICES, you were accessing its content with get_filename_display(), for example.
If you try to delete the field to make a migration, for then renaming the other field and make another migration, you'll see Django complaining because you cannot delete a field that you are using in the project.
Just create an empty migration as Joey explained, and put this in operations:
operations = [
migrations.RemoveField(
model_name='app_name',
name='old_field_name',
),
migrations.RenameField(
model_name='app_name',
old_name='old_field_name_link',
new_name='old_field_name',
),
]
Then run migrate and you'll have the changes made in your database, but obviously not in your model, it's time now to delete the old field and to rename new ForeignKey field to the original name.
I don't think that doing this is particularly hacky, but still, only do this kind of things if you are fully understanding what are you messing with.

How to add through option to existing ManyToManyField with migrations and data in django

I can't find reference to particular issue in docs or online.
I have an existing many to many relation.
class Books(models.Model):
name = models.CharField(max_length=100)
class Authors(models.Model):
name = models.CharField(max_length=100)
books = models.ManyToManyField(Books)
This has migrations and data. Now I need to use through option in order to add one extra field in table holding many to many relation.
class Authorship(models.Model):
book = models.ForeignKey(Books)
author = models.ForeignKey(Authors)
ordering = models.PositiveIntegerField(default=1)
class Authors(models.Model):
name = models.CharField(max_length=100)
books = models.ManyToManyField(Books, through=Authorship)
When I run migrations, django creates fresh migration for Authorship model. I tried to create migration file manually by adding ordering column in Authorship table and altering books column in Authors table but I get some migration issues.
operations = [
migrations.AddField(
model_name='authorship',
name='ordering',
field=models.PositiveIntegerField(default=1),
),
migrations.AlterField(
model_name='authors',
name='books',
field=models.ManyToManyField(to='app_name.Books', through='app_name.Authorship'),
),
]
When trying to migrate, it gives KeyError: ('app_name', u'authorship') I bet there are other things that are affected and thus errors.
What things am I missing? Is there any other approach to work with this?
There is a way to add "through" without data migrations.
I managed to do it based on this #MatthewWilkes' answer.
So, to translate it to your data model:
Create the Authorship model only with book and author fields. Specify the table name to use the same name as the auto-generated M2M table you already have. Add the 'through' parameter.
class Authorship(models.Model):
book = models.ForeignKey(Books)
author = models.ForeignKey(Authors)
class Meta:
db_table = 'app_name_authors_books'
class Authors(models.Model):
name = models.CharField(max_length=100)
books = models.ManyToManyField(Books, through=Authorship)
Generate a migration, but don't run it yet.
Edit the generated migration and wrap the migration operations into a migrations. SeparateDatabaseAndState operation with all the operations inside state_operations field (with database_operations left empty). You will end up with something like this:
operations = [
migrations.SeparateDatabaseAndState(state_operations=[
migrations.CreateModel(
name='Authorship',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('book', models.ForeignKey(to='app_name.Books')),
],
options={
'db_table': 'app_name_authors_books',
},
),
migrations.AlterField(
model_name='authors',
name='books',
field=models.ManyToManyField(through='app_name.Authorship', to='app_name.Books'),
),
migrations.AddField(
model_name='authorship',
name='author',
field=models.ForeignKey( to='app_name.Author'),
),
])
]
You can now run the migration and add the extra ordering field to your M2M table.
Edit:
Apparently, column names in the DB are generated slightly differently for automatic M2M tables as for models-defined tables. (I am using Django 1.9.3.)
After the described procedure, I also had to manually change the column names of a field with a 2-word name (two_words=models.ForeignKey(...)) from twowords_id to two_words_id.
Looks like there is no way to use through option without having to do data migrations. So had to resort to data migration approach, I took some ideas from #pista329's answer and solved the issue using following steps.
Create Authorship model
class Authorship(models.Model):
book = models.ForeignKey(Books)
author = models.ForeignKey(Authors)
ordering = models.PositiveIntegerField(default=1)
Keep original ManyToManyField relation, but add another field using above defined model as through model:
class Authors(models.Model):
name = models.CharField(max_length=100)
books = models.ManyToManyField(Books)
published_books = models.ManyToManyField(
to=Books,
through=Authorship,
related_name='authors_lst' # different related name is needed.
)
IMPORTANT: You must use a different related_name to the existing ManyToManyField. If you don't do this then Django may lose the data in the original field.
Add data migration to copy all data from old table to new Authorship table.
After this, books field on Authors model can be removed as we have new field named published_books.
Migrations can be messy sometimes.
If you want to alter m2m field with through, I would suggest to rename altered field Authors.books to Authors.book. When asked by makemigrations, if you changed name from books to book? [yN], choose "N" as No. Django will delete books and create book field instead of altering.
class Authorship(models.Model):
book = models.ForeignKey("Books")
author = models.ForeignKey("Authors")
ordering = models.PositiveIntegerField(default=1)
class Authors(models.Model):
name = models.CharField(max_length=100)
book = models.ManyToManyField("Books", through="Authorship")
If you want to use books anyway, change book to books and repeat migration process with y as answer to makemigrations question about renaming.