I have created a Django web application that has models and many instances of models created. The problem is that I want to add another field to one of the models already in use (has over 1000 instances). I have had difficult experiences with Django Migrations in the past and am scared that if I modify the model by adding the field, my web application will not function.
How do I change the model in such a way that it will not affect the functionality of my web application?
the following should solve your problem without introducing any issues.
You can check that the data has been populated correctly by checking the fields in django admin if need be.
I would suggest, however, that you set yourself up with a dev environment so that you can test this kind of thing without worrying about whether it's going to cause issues on your production environment.
I hope this helps!
Initial:
Models.py
class ExampleModel(models.Model):
name = models.CharField(blank=True, null=True)
First change:
Models.py
class ExampleModel(models.Model):
name = models.CharField(blank=True, null=True)
unique_str = models.CharField(blank=True, null=True) # add this here
Run:
python manage.py makemigrations
python manage.py migrate
Then you can run a script to populate your unique_str:
# get all model instances
example_models = ExampleModel.objects.all()
# update each unique_str as required
for e in example_models:
e.unique_str = e.pk # add any required logic here to assign value to this variable
e.save()
Finally:
Models.py
class ExampleModel(models.Model):
name = models.CharField(blank=True, null=True)
unique_str = models.CharField(blank=True, null=True, unique=True) # add unique
and then again, Run:
python manage.py makemigrations
python manage.py migrate
Related
I have a simple view on the DB which selects from other DB's tables located on the same MSSQL Server to ultimately serve the collected info as a dropdown to the user.
So far I've added the Model with inspectdb:
class AutPricePlanView(models.Model):
priceplan_name = models.CharField(db_column='PricePlan', max_length=50, blank=True, unique=True)
class Meta:
managed = False # Created from a view. Don't remove.
db_table = 'AUT_PricePlanView'
Also I have a second existing (Django Native) Model where I want to use the values from the view for a Dropdown Field (to keep everything in sync):
class PricePlanDownload(models.Model):
requesting_user = models.CharField(blank=True, default=None, max_length=50, null=True)
requested_at = models.DateTimeField(auto_now_add=True)
document = models.FileField(upload_to='documents/price_plan_uploads/%Y/%m/%d', blank=True)
priceplan = models.ForeignKey(AutPricePlanView, null=True, on_delete=models.DO_NOTHING)
Makemigrations works fine but when I try to actually migrate I get the following issue: (shortened it a little bit)
django.db.utils.ProgrammingError: ('42000', "[42000] [FreeTDS][SQL Server]Foreign key references object 'AUT_PricePlanView' which is not a user table. (1768) (SQLExecDirectW)")
I would be really grateful if someone had an idea or a workaround since I can't figure out what the heck this has to do with a "user" table...
Since the view is not actually a table, you cannot set Foreign Key constraints. Since ForeignKey's default db_constraint value is True, Django tries to set Foreign Key constraints when performing migrations. This is the reason the migration fails.
So, you can turn off the db_constraint option. And you can remove the existing migration file, and re-create the migration file. Then, the migration will success and you can keep everything in sync.
class PricePlanDownload(models.Model):
... other fields ...
priceplan = models.ForeignKey(AutPricePlanView, null=True, on_delete=models.DO_NOTHING, db_constraint=False)
Pro Tip: You can review migration's SQL using python manage.py sqlmigrate <appname> <migration number>, like python manage.py sqlmigrate yourapp 0002.
Update: You can define __str__ to display the correct value at the dropdown menu.
class AutPricePlanView(models.Model):
priceplan_name = models.CharField(db_column='PricePlan', max_length=50, blank=True, unique=True, primary_key=True)
# null=False by default. See https://github.com/django/django/blob/master/django/db/models/fields/__init__.py#L132
def __str__(self):
return self.priceplan_name
class Meta:
managed = False # Created from a view. Don't remove.
db_table = 'AUT_PricePlanView'
I need to change the type of a field in one of my Django models from CharField to ForeignKey. The fields are already populated with data, so I was wondering what is the best or right way to do this. Can I just update the field type and migrate, or are there any possible 'gotchas' to be aware of? N.B.: I just use vanilla Django management operations (makemigrations and migrate), not South.
This is likely a case where you want to do a multi-stage migration. My recommendation for this would look something like the following.
First off, let's assume this is your initial model, inside an application called discography:
from django.db import models
class Album(models.Model):
name = models.CharField(max_length=255)
artist = models.CharField(max_length=255)
Now, you realize that you want to use a ForeignKey for the artist instead. Well, as mentioned, this is not just a simple process for this. It has to be done in several steps.
Step 1, add a new field for the ForeignKey, making sure to mark it as null:
from django.db import models
class Album(models.Model):
name = models.CharField(max_length=255)
artist = models.CharField(max_length=255)
artist_link = models.ForeignKey('Artist', null=True)
class Artist(models.Model):
name = models.CharField(max_length=255)
...and create a migration for this change.
./manage.py makemigrations discography
Step 2, populate your new field. In order to do this, you have to create an empty migration.
./manage.py makemigrations --empty --name transfer_artists discography
Once you have this empty migration, you want to add a single RunPython operation to it in order to link your records. In this case, it could look something like this:
def link_artists(apps, schema_editor):
Album = apps.get_model('discography', 'Album')
Artist = apps.get_model('discography', 'Artist')
for album in Album.objects.all():
artist, created = Artist.objects.get_or_create(name=album.artist)
album.artist_link = artist
album.save()
Now that your data is transferred to the new field, you could actually be done and leave everything as is, using the new field for everything. Or, if you want to do a bit of cleanup, you want to create two more migrations.
For your first migration, you will want to delete your original field, artist. For your second migration, rename the new field artist_link to artist.
This is done in multiple steps to ensure that Django recognizes the operations properly. You could create a migration manually to handle this, but I will leave that to you to figure out.
Adding on top of Joey's answer, detailed steps for Django 2.2.11.
Here are the models from my use case, that consists of a Company and Employee model. We have to convert designation to a foreign key field. The app name is called core
class Company(CommonFields):
name = models.CharField(max_length=255, blank=True, null=True
class Employee(CommonFields):
company = models.ForeignKey("Company", on_delete=models.CASCADE, blank=True, null=True)
designation = models.CharField(max_length=100, blank=True, null=True)
Step 1
Create a foreign key designation_link in Employee and mark it as null=True
class Designation(CommonFields):
name = models.CharField(max_length=255)
company = models.ForeignKey("Company", on_delete=models.CASCADE, blank=True, null=True)
class Employee(CommonFields):
company = models.ForeignKey("Company", on_delete=models.CASCADE, blank=True, null=True)
designation = models.CharField(max_length=100, blank=True, null=True)
designation_link = models.ForeignKey("Designation", on_delete=models.CASCADE, blank=True, null=True)
Step 2
Create empty migration. Using the command:
python app_code/manage.py makemigrations --empty --name transfer_designations core
This will create a following file in migrations directory.
# Generated by Django 2.2.11 on 2020-04-02 05:56
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('core', '0006_auto_20200402_1119'),
]
operations = [
]
Step 3
Populate the empty migration with a function that loops over all Employees, creates a Designation and links it to the Employee.
In my use case each Designation is also linked to a Company. Which means that Designation may contain two rows for "managers", one for company A, another for company B.
Final migration would look something like this:
# core/migrations/0007_transfer_designations.py
# Generated by Django 2.2.11 on 2020-04-02 05:56
from django.db import migrations
def link_designation(apps, schema_editor):
Employee = apps.get_model('core', 'Employee')
Designation = apps.get_model('core', 'Designation')
for emp in Employee.objects.all():
if(emp.designation is not None and emp.company is not None):
desig, created = Designation.objects.get_or_create(name=emp.designation, company=emp.company)
emp.designation_link = desig
emp.save()
class Migration(migrations.Migration):
dependencies = [
('core', '0006_auto_20200402_1119'),
]
operations = [
migrations.RunPython(link_designation),
]
Step 4
Finally run this migration using:
python app_code/manage.py migrate core 0007
That's a continuation of the great answer by Joey.
How to rename the new field to the original name?
If the field has data, it probably means that you are using it elsewhere in your project, therefore this solution will leave you with a field named differently, and you have to either refactor the project to use the new field or delete the old field and rename the new one.
Be aware that this process is not going to prevent you to refactor code. If you where using a CharField with CHOICES, you were accessing its content with get_filename_display(), for example.
If you try to delete the field to make a migration, for then renaming the other field and make another migration, you'll see Django complaining because you cannot delete a field that you are using in the project.
Just create an empty migration as Joey explained, and put this in operations:
operations = [
migrations.RemoveField(
model_name='app_name',
name='old_field_name',
),
migrations.RenameField(
model_name='app_name',
old_name='old_field_name_link',
new_name='old_field_name',
),
]
Then run migrate and you'll have the changes made in your database, but obviously not in your model, it's time now to delete the old field and to rename new ForeignKey field to the original name.
I don't think that doing this is particularly hacky, but still, only do this kind of things if you are fully understanding what are you messing with.
models.py:
class Time(models.Model):
id = models.AutoField(primary_key=True)
who= models.TextField(db_column='who')
class Task(models.Model):
id = models.AutoField(primary_key=True)
when= models.ForeignKey(Time)
I need to record Time and Task but my problem is that there can be:
1) a Time without Task,
2) a Task without Time,
3) a Time with multiple Task
I've tried adding ForeignKey to Task as null and blank = True but django says that a "foreignKey can't be null"
Are there other way to make this "optional" relation?
You should do like this ::
It seems most likely that you added null=True after running manage.py makemigrations. You'll need to modify your database schema to allow null values in that column.
when = models.ForeignKey(Time, default=None, blank=True, null=True)
and make new migration to change the schema:
python manage.py makemigrations
python manage.py migrate
you are getting
foreignKey can't be null
because your schema doesnot allow fk to be null
I'm trying to modify a M2M field to a ForeignKey field. The command validate shows me no issues and when I run syncdb :
ValueError: Cannot alter field xxx into yyy they are not compatible types (you cannot alter to or from M2M fields, or add or remove through= on M2M fields)
So I can't make the migration.
class InstituteStaff(Person):
user = models.OneToOneField(User, blank=True, null=True)
investigation_area = models.ManyToManyField(InvestigationArea, blank=True,)
investigation_group = models.ManyToManyField(InvestigationGroup, blank=True)
council_group = models.ForeignKey(CouncilGroup, null=True, blank=True)
#profiles = models.ManyToManyField(Profiles, null = True, blank = True)
profiles = models.ForeignKey(Profiles, null = True, blank = True)
Any suggestions?
I stumbled upon this and although I didn't care about my data much, I still didn't want to delete the whole DB. So I opened the migration file and changed the AlterField() command to a RemoveField() and an AddField() command that worked well. I lost my data on the specific field, but nothing else.
I.e.
migrations.AlterField(
model_name='player',
name='teams',
field=models.ManyToManyField(related_name='players', through='players.TeamPlayer', to='players.Team'),
),
to
migrations.RemoveField(
model_name='player',
name='teams',
),
migrations.AddField(
model_name='player',
name='teams',
field=models.ManyToManyField(related_name='players', through='players.TeamPlayer', to='players.Team'),
),
NO DATA LOSS EXAMPLE
I would say: If machine cannot do something for us, then let's help it!
Because the problem that OP put here can have multiple mutations, I will try to explain how to struggle with that kind of problem in a simple way.
Let's assume we have a model (in the app called users) like this:
from django.db import models
class Person(models.Model):
name = models.CharField(max_length=128)
def __str__(self):
return self.name
class Group(models.Model):
name = models.CharField(max_length=128)
members = models.ManyToManyField(Person)
def __str__(self):
return self.name
but after some while we need to add a date of a member join. So we want this:
class Group(models.Model):
name = models.CharField(max_length=128)
members = models.ManyToManyField(Person, through='Membership') # <-- through model
def __str__(self):
return self.name
# and through Model itself
class Membership(models.Model):
person = models.ForeignKey(Person, on_delete=models.CASCADE)
group = models.ForeignKey(Group, on_delete=models.CASCADE)
date_joined = models.DateField()
Now, normally you will hit the same problem as OP wrote. To solve it, follow these steps:
start from this point:
from django.db import models
class Person(models.Model):
name = models.CharField(max_length=128)
def __str__(self):
return self.name
class Group(models.Model):
name = models.CharField(max_length=128)
members = models.ManyToManyField(Person)
def __str__(self):
return self.name
create through model and run python manage.py makemigrations (but don't put through property in the Group.members field yet):
from django.db import models
class Person(models.Model):
name = models.CharField(max_length=128)
def __str__(self):
return self.name
class Group(models.Model):
name = models.CharField(max_length=128)
members = models.ManyToManyField(Person) # <-- no through property yet!
def __str__(self):
return self.name
class Membership(models.Model): # <--- through model
person = models.ForeignKey(Person, on_delete=models.CASCADE)
group = models.ForeignKey(Group, on_delete=models.CASCADE)
date_joined = models.DateField()
create an empty migration using python manage.py makemigrations users --empty command and create conversion script in python (more about the python migrations here) which creates new relations (Membership) for an old field (Group.members). It could look like this:
# Generated by Django A.B on YYYY-MM-DD HH:MM
import datetime
from django.db import migrations
def create_through_relations(apps, schema_editor):
Group = apps.get_model('users', 'Group')
Membership = apps.get_model('users', 'Membership')
for group in Group.objects.all():
for member in group.members.all():
Membership(
person=member,
group=group,
date_joined=datetime.date.today()
).save()
class Migration(migrations.Migration):
dependencies = [
('myapp', '0005_create_models'),
]
operations = [
migrations.RunPython(create_through_relations, reverse_code=migrations.RunPython.noop),
]
remove members field in the Group model and run python manage.py makemigrations, so our Group will look like this:
class Group(models.Model):
name = models.CharField(max_length=128)
add members field the the Group model, but now with through property and run python manage.py makemigrations:
class Group(models.Model):
name = models.CharField(max_length=128)
members = models.ManyToManyField(Person, through='Membership')
and that's it!
Now you need to change creation of members in a new way in your code - by through model. More about here.
You can also optionally tidy it up, by squashing these migrations.
Potential workarounds:
Create a new field with the ForeignKey relationship called profiles1 and DO NOT modify profiles. Make and run the migration. You might need a related_name parameter to prevent conflicts. Do a subsequent migration that drops the original field. Then do another migration that renames profiles1 back to profiles. Obviously, you won't have data in the new ForeignKey field.
Write a custom migration: https://docs.djangoproject.com/en/1.7/ref/migration-operations/
You might want to use makemigration and migration rather than syncdb.
Does your InstituteStaff have data that you want to retain?
If you're still developing the application, and don't need to preserve your existing data, you can get around this issue by doing the following:
Delete and re-create the db.
go to your project/app/migrations folder
Delete everything in that folder with the exception of the init.py file. Make sure you also delete the pycache dir.
Run syncdb, makemigrations, and migrate.
Another approach that worked for me:
Delete the existing M2M field and run migrations.
Add the FK field and run migrations again.
FK field added in this case has no relation to the previously used M2M field and hence should not create any problems.
This link helps you resolve all problems related to this
The one which worked for me is python3 backend/manage.py migrate --fake "app_name"
I literally had the same error for days and i had tried everything i saw here but still didn'y work.
This is what worked for me:
I deleted all the files in migrations folder exceps init.py
I also deleted my database in my case it was the preinstalled db.sqlite3
After this, i wrote my models from the scratch, although i didn't change anything but i did write it again.
Apply migrations then on the models and this time it worked and no errors.
This worked for Me as well
Delete last migrations
run command python manage.py migrate --fake <application name>
run command 'python manage.py makemigrations '
run command 'python manage.py migrate'
Hope this will solve your problem with deleting database/migrations
First delete the migrations in your app (the folders/ files under 'migrations'
folder)
Showing the 'migrations' folder
Then delete the 'db.sqlite3' file
Showing the 'db.sqlite3' file
And run python manage.py makemigrations name_of_app
Finally run python manage.py migrate
I had the same problem and found this How to Migrate a ‘through’ to a many to many relation in Django article which is really really helped me to solve this problem. Please have a look. I'll summarize his answer here,
There is three model and one(CollectionProduct) is going to connect as many-to-many relationship.
This is the final output,
class Product(models.Model):
pass
class Collection(models.Model):
products = models.ManyToManyField(
Product,
blank=True,
related_name="collections",
through="CollectionProduct",
through_fields=["collection", "product"],
)
class CollectionProduct(models.Model):
collection = models.ForeignKey(Collection, on_delete=models.CASCADE)
product = models.ForeignKey(Product, on_delete=models.CASCADE)
class Meta:
db_table = "product_collection_products"
and here is the solution,
The solution
Take your app label (the package name, e.g. ‘product’) and your M2M field name, and combine them together with and underscore:
APPLABEL + _ + M2M TABLE NAME + _ + M2M FIELD NAME
For example in our case, it’s this:
product_collection_products
This is your M2M’s through database table name. Now you need to edit your M2M’s through model to this:
Also found another solution in In Django you cannot add or remove through= on M2M fields article which is going to edit migration files. I didn't try this, but have a look if you don't have any other solution.
this happens when adding 'through' attribute to an existing M2M field:
as M2M fields are by default handled by model they are defined in (if through is set).
although when through is set to new model the M2M field is handled by that new model, hence the error in alter
solutions:-
you can reset db or
remove those m2m fields and run migration as explained above then create them again
*IF YOU ARE IN THE INITIAL STAGES OF DEVELOPMENT AND CAN AFFORD TO LOOSE DATA :)
delete all the migration files except init.py
then apply the migrations.
python manage.py makemigrations
python manage.py migrate
this will create new tables.
I changed my model from this:
class DistList(models.Model):
creator = models.ForeignKey(User, related_name='creator')
created_date = models.DateTimeField(auto_now_add=True)
name = models.CharField(max_length=200, unique=True)
description = models.TextField(blank=True, null=True)
company = models.ForeignKey(Company, blank=True, null=True)
To this:
class DistList(models.Model):
creator = models.ForeignKey(User, related_name='creator')
created_date = models.DateTimeField(auto_now_add=True)
name = models.CharField(max_length=200, unique=True)
description = models.TextField(blank=True, null=True)
company = models.ForeignKey(Company)
The only change was turning the company FK relationship from not required to required.
When I run the migration I specify a one off value that corresponds to the pk of the first company.
./manage.py schemamigration distlist --auto
? The field 'DistList.company' does not have a default specified, yet is NOT NULL.
? Since you are making this field non-nullable, you MUST specify a default
? value to use for existing rows. Would you like to:
? 1. Quit now, and add a default to the field in models.py
? 2. Specify a one-off value to use for existing columns now
? Please select a choice: 2
? Please enter Python code for your one-off default value.
? The datetime module is available, so you can do e.g. datetime.date.today()
>>> 1
But when I run the migration I get an error because it has a pending trigger event?
./manage.py migrate distlist
Running migrations for distlist:
- Migrating forwards to 0005_auto__chg_field_distlist_company.
> distlist:0005_auto__chg_field_distlist_company
FATAL ERROR - The following SQL query failed: ALTER TABLE "distlist_distlist" ALTER COLUMN "company_id" SET NOT NULL;
The error was: cannot ALTER TABLE "distlist_distlist" because it has pending trigger events
I'm not doing anything that seems weird from my point of view so I don't understand this error at all. Can anyone offer insight? I can post the full stack trace if it'll help but I feel like theres something obvious about south and postgresql that perhaps I'm missing?
So I believe I've found the answer. I think postgresql doesn't like altering schemas and adding data at the same time. I first created a datamigration:
./manage.py datamigration distlist add_default_values_to_existing_companies
Then I added this to the forwards method:
def forwards(self, orm):
"Write your forwards methods here."
for distlist in orm['distlist.Distlist'].objects.all():
distlist.company = orm['userprofile.Company'].objects.get(id=1)
distlist.save()
Then I altered the model to remove the blank and null from company.
Then I ran the schema migration and chose to specify a one off for the value as 1 (as I did in the question).
Then I edited that migration file thusly:
def forwards(self, orm):
# Changing field 'DistList.company'
# db.alter_column(u'distlist_distlist', 'company_id', self.gf('django.db.models.fields.related.ForeignKey')(default=1, to=orm['userprofile.Company']))
db.alter_column(u'distlist_distlist', 'company_id', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['userprofile.Company']))
I just commented out the generated line and removed the default=1 arg.
I don't know... maybe this isn't right but it seemed to work. Hopefully this will help someone.