Django AutoField wrong migration for "id" field - django

I have quite simple Django model:
class MyModel(models.Model):
user = models.ForeignKey(User)
somestring = models.CharField(
max_length=250
)
... some other string fields...
There is no declared "id" field in model, so it has automatic primary key assigned by Django.
Initial migration looked like this:
migrations.CreateModel(
name='MyModel',
fields=[
('user', models.ForeignKey(to=settings.AUTH_USER_MODEL)),
('somestring', models.CharField(max_length=250))
... some other string fields...
],
options={},
bases=(models.Model,),
),
It was successfully applied and database table contains fields like:
- id <== autogenerated
- user
- somestring
etc...
Occasionally I've started to get messages like
Your models have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
After running manage.py makemigrations it generated very strange migration:
$ ./manage.py makemigrations
You are trying to add a non-nullable field 'id' to mymodel without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows)
2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now()
>>>
Please enter some code, or 'exit' (with no quotes) to exit.
>>> ''
Migrations for 'myapp':
0036_auto_20150623_1535.py:
- Add field id to mymodel
It loosk like this:
class Migration(migrations.Migration):
dependencies = [
('foo', 'bar'),
]
operations = [
migrations.AddField(
model_name='mymodel',
name='id', <== !!!
field=models.AutoField(auto_created=True, primary_key=True, default='', serialize=False, verbose_name='ID'),
preserve_default=False,
),
]
But this migration doesn't makes sense and sure it fails, because there is already "id" field in appropriate DB table.
The quick and dirty solution is to --fake this migration. Which should work locally on dev machine, but can result into migration errors on other environments (test/staging/prod).
Looks like the model's old/new fields state was calculated incorrectly, so that "id" wasn't included in old model but it is included in new one, so django descided that id field should be added.
Not sure about the cause, but my main question is - What is the proper approach to reset the model or migration state and return it to the previous one, when there is no any need in such migration?

Alright, found a viable workaround for this.
Added "id" field declaration into original migration, so it is no longer reported as missing.
migrations.CreateModel(
name='MyModel',
fields=[
!!===> ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
('user', models.ForeignKey(to=settings.AUTH_USER_MODEL)),
('somestring', models.CharField(max_length=250))
... some other string fields...
],
options={},
bases=(models.Model,),
),
This change will not be applied on environments where we already have this migration and new environments should get the proper db table structure from the start.
TIL: Django DOES NOT use SQL to fetch the current model state from DB, - but create mock of the model and applies the migrations to that mock until it gets the latest version of model according all available migration operations. And then that reconstructed model is compared to real one from models.py.
Adjusting the original migration solved the issue. Thanks to everyone who participated in discussions.

Related

django simple history field indexing

How can I tell DSH to index particular field? Some queries that I do to historical models take too much time
I've base abstract model and all my models inherit from that model. The history field is also defined in this base model:
class BaseModel(models.Model):
class PublishingStatus(models.TextChoices):
DRAFT = 'draft', _('Draft')
ACCEPTED = 'accepted', _('Accepted'),
REJECTED = 'rejected', _('Rejected'),
MODIFIED = 'modified', _('Modified')
publishing_status = models.CharField(
max_length=9,
choices=PublishingStatus.choices,
default=PublishingStatus.DRAFT,
help_text=_("Publishing status represents the state of the object. By default it is 'draft'")
)
history = HistoricalRecords(inherit=True)
And I've also added indexes in this base model
class Meta:
abstract = True
indexes = [models.Index(fields=[
'publishing_status',
])]
It would be nice if Django Simple History could check which fields are indexed and create the same indexes in the historical models
Maybe there is a way to tell django simple history explicitly which field must be indexed additionally?
I hope there is a better solution, but I've ended up creating a custom migration file. First, I created a blank migration file for the app:
python manage.py makemigrations dosh --empty
I then modified this newly created empty migration file and added AddIndex for each historical model.
# Generated by Django 3.1.1 on 2020-10-15 06:49
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('dosh', '0002_auto_20201008_2155'),
]
operations = [
# ...
migrations.AddIndex(
model_name='historicaltense',
index=models.Index(fields=['publishing_status'], name='htense_publish_status_idx')
),
migrations.AddIndex(
model_name='historicalsynonym',
index=models.Index(fields=['publishing_status'], name='hsynonym_publish_status_idx')
),
# ...
]
And finally, I've migrated those changes
python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, authtoken, comment, contenttypes, dosh, pinax_badges, sessions, social, users, vote
Running migrations:
Applying dosh.0003_auto_20201015_0849... OK

Where are the historical models?

The doc says:
When you run migrations, Django is working from historical versions of your models stored in the migration files.
But I can't see them there. I have data migrations with RunPython operations, but no historical models there as well. Could it be that Django generates them on the fly? How does it do it?
And while we're at it, let me confirm if I understand it correctly. Historical models are models as they were when a migration was written? Except for some limitations, like no custom methods.
Whenever you create a model and run makemigrations. Django creates a migration file for that model which signifies the creation of such Model.
operations = [
migrations.CreateModel(
name='Book',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=50)),
],
)
]
When you edit that model like adding a field and again run makemigrations, a new migration file is created for that change. This is the way Django stores history of that model.
operations = [
migrations.AddField(
model_name='book',
name='author',
field=models.CharField(blank=True, max_length=50, null=True),
),
]
The migration files are the historical models.
Now when you add some custom Python code with RunPython in migration it will be a part of the historical context for future migrations.
Migration files are set of operations that are needed to be done on database for a model. They do not have the customization of the model like save() method because this model is created from the history i.e. migration files.

Django makemigrations make change for 'auth.user' every time

I upgrade my system's django version from 1.6.10 to 1.8.16 for test.
On before version, i use South for migration.
So, I followed 'https://docs.djangoproject.com/en/1.7/topics/migrations/#upgrading-from-south' this documentation.
My problem is every makemigrations are check same field, then make migration file.
That field is 'auth.User' foreign key field. like user = models.ForeignKey('auth.User').
here are my screenshot for that problem.
This is Sample code for that foreign key field.
cancelled_by = models.ForeignKey(
'auth.User',
verbose_name=_("Cancelled by"),
related_name='project_cancel',
blank=True,
null=True
)
How can i fix it?
edited:
This is my migration file created by makemigrations after all migration.
class Migration(migrations.Migration):
dependencies = [
('meeting', '0003_meeting_proposal'),
]
operations = [
migrations.AlterField(
model_name='meeting',
name='manager',
field=models.ForeignKey(verbose_name='Manager', blank=True, to=settings.AUTH_USER_MODEL, null=True),
),
]
I would first delete the recently made migration files and try makemigration again.
deleting migration files is a common solution since Django is smart enough to recreate them easily regardless of what has been changed in your folder.
you can fake these migrations too, but i prefer to keep my migration folder clean and tidy.

How to properly make migrations when adding a new unique field

I added a new field to one of my models:
class Agency(models.Model):
email = models.EmailField(unique=True, verbose_name=_("e-mail"))
As this field cannot be blank, django-admin makemigrations requested me to provide one-off default, which I did. Here is the generated migration:
# Generated by Django 1.9.4 on 2016-03-20 10:38
from __future__ import unicode_literals
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('accounts', '0008_auto_20160226_1226'),
]
operations = [
migrations.AddField(
model_name='agency',
name='email',
field=models.EmailField(default='example#example.fr', max_length=254, unique=True, verbose_name='e-mail'),
preserve_default=False,
),
]
As expected, django-admin migrate throwed an error:
psycopg2.IntegrityError: could not create unique index "accounts_agency_email_key"
DETAIL: Key (email)=(example#example.fr) is duplicate.
I thought I could edit the migration to set unique values before making the field unique. So I tried:
# -*- coding: utf-8 -*-
# Generated by Django 1.9.4 on 2016-03-20 10:38
from __future__ import unicode_literals
from django.db import migrations, models
from django.utils.text import slugify
def set_email(apps, schema_editor):
Agency = apps.get_model('accounts', 'Agency')
for agency in Agency.objects.all():
agency.email = '{}#example.fr'.format(slugify(agency.name))
agency.save()
class Migration(migrations.Migration):
dependencies = [
('accounts', '0008_auto_20160226_1226'),
]
operations = [
migrations.AddField(
model_name='agency',
name='email',
field=models.EmailField(default='', max_length=254, blank=True, verbose_name='e-mail'),
preserve_default=False,
),
migrations.RunPython(set_email),
migrations.AlterField(
model_name='agency',
name='email',
field=models.EmailField(max_length=254, unique=True, verbose_name='e-mail'),
preserve_default=False,
),
]
Unfortunately I get this error when running django-admin migrate:
django.db.utils.OperationalError: cannot ALTER TABLE "accounts_agency" because it has pending trigger events
My guess is that operations are not executed synchronously.
I think I could fix the issue by splitting the migration into two migrations, but I'd like to know if I can do it in only one migration. What is the common way to create migrations when adding a new unique field in a model?
PS: I also tried to use an F expression as default (default=models.F('name') + '#example.fr') but it failed:
django.db.utils.IntegrityError: could not create unique index "accounts_agency_email_key"
DETAIL: Key (email)=(F(name) + Vallu(#example.fr)) is duplicated.
Maybe it's too late but maybe it could work for someone else
You can do this in one migration via using migrations.RunSQL method
For your example code after you added the new field to your model and run the python manage.py makemigrations command (here if you have existing rows in your table command wants to choice default value you can choice "Provide a one-off default now" option and give some string value it is not important because actually we did not use it) then go to migration file and change operations part with this (Note i use postgresql you can change SQL for your database)
operations = [
migrations.RunSQL(
'ALTER TABLE "agency" ADD COLUMN "email" varchar(254) NULL;ALTER TABLE "agency" ALTER COLUMN "email" DROP DEFAULT;COMMIT;',
),
migrations.RunSQL(
"UPDATE agency SET email= Concat(country_code, '#example.fr');COMMIT;",
),
migrations.RunSQL(
'ALTER TABLE "agency" ALTER COLUMN "email" SET NOT NULL;ALTER TABLE "agency" ADD CONSTRAINT "agency_email_b551ad2a_uniq" UNIQUE ("email");ALTER TABLE "agency" ALTER COLUMN "email" DROP DEFAULT;CREATE INDEX "agency_email_b551ad2a_like" ON "agency" ("email" varchar_pattern_ops);COMMIT;'
)
]
then run "python manage.py migrate" command
that is it.

Using django-custom-user in existing django-cms project

I have an existing django-cms ver. 3.1.3 project where I would like to replace the default django User model with the one found in django-custom-user (to have email as user name for my users). I have added the custom_user app to my INSTALLED_APPS and set AUTH_USER_MODEL = 'custom_user.EmailUser'. Finally I applied the migrations.
Everything seems to work fine in my custom models, but the django-cms models that have a reference to the auth_user table (GlobalPagePermission, PagePermission, PageUser and UserSettings) are not updated to have a foreign-key reference to the new custom user table.
The django-cms documentation says that is generally advisable to add custom user models in the beginning of a project, but here I am, in the middle of a project, and would very much like to avoid having to delete my cms models (with data) to make it work.
If I were in the beginning of a project, and had added the custom user model before migrating the django-cms models, would the django-cms models actually get a reference to the custom user model table instead of the default one?
Is there any way for me to make migrations for the django-cms models, so they use the new custom user model instead of the default one?
UPDATE
Im trying to implement what yakky suggested:
from __future__ import unicode_literals
from django.conf import settings
from django.db import models, migrations
import django.db.models.deletion
from django.utils.translation import ugettext_lazy as _
def up(apps, schema_editor):
PageUser = apps.get_model("cms", "PageUser")
db_alias = schema_editor.connection.alias
for pu in PageUser.objects.all():
pu['emailuser_ptr_id'] = pu['user_ptr_id']
pu.save()
def down(apps, schema_editor):
PageUser = apps.get_model("cms", "PageUser")
db_alias = schema_editor.connection.alias
for pu in PageUser.objects.all():
pu['user_ptr_id'] = pu['emailuser_ptr_id']
pu.save()
class Migration(migrations.Migration):
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
(‘myapp’, ‘previous-migration’),
]
operations = [
migrations.RenameField(
model_name='PageUser',
old_name='user_ptr_id',
new_name='user_ptr_id_old',
),
migrations.AddField(
model_name='PageUser',
name='emailuser_ptr_id',
field=models.ForeignKey(null=True, to=settings.AUTH_USER_MODEL, verbose_name=_('user'), blank=True),
preserve_default=True,
),
migrations.RunPython(up, down),
migrations.RemoveField(
model_name='PageUser',
name='user_ptr_id_old',
),
]
It fails with KeyError: ('myapp', u'pageuser') suggesting that it looks for the PageUser model in the my custom app and not in the cms app. How do I apply these migrations to the cms models?
Being said that replacing the Django User model it's not advisable in the middle of a project lifetime (see https://docs.djangoproject.com/en/1.9/topics/auth/customizing/#substituting-a-custom-user-model) as Django does not provide any way to modify the relations after their are being created, to replace the custom user model you can write a custom migration that alter the foreignkey in the django CMS models to point to theand create a new one to the new user model.
Something like:
...
migrations.RenameField(
model_name='globalpagepermission',
old_name='user',
new_name='user_old',
),
migrations.AddField(
model_name='globalpagepermission',
name='user',
field=models.ForeignKey(null=True, to=settings.AUTH_USER_MODEL, verbose_name=_('user'), blank=True),
preserve_default=True,
),
migrations.RunPython(copy_data),
migrations.RemoveField(
model_name='globalpagepermission',
name='user_old',
),
...
Repeat for any relevant model / field.
copy_data method will copy the value for the foreign keys from user_old to user
Code is completely untested but the general idea should work.
Alternatively you can write a RunSQL migration to get the same result.
Users with the same primary keys must exist also for the new model for this to work.
Thanks to https://code.djangoproject.com/ticket/25313 I managed to make it work. This is what I did:
Backed up my database!
Created a new app email_username to hold my user model (named User as recommended in the link). A new app was necessary because my main app (myapp) is dependent on django-cms, and if I put my new user model in myapp, django-cms would become dependent on my app - this would create a circular reference.
class User(AbstractEmailUser):
# Needed to add username, first_name and last_name here, because cms.PageUser model depends on these fields
username = models.CharField(max_length=100, verbose_name=_('username'))
first_name = models.CharField(max_length=100, verbose_name=_('first_name'))
last_name = models.CharField(max_length=100, verbose_name=_('last_name'))
class Meta:
db_table = 'auth_user' # the model will use the existing auth_user table (at first)
Set AUTH_USER_MODEL = ‘email_username.User’ in settings.py
Deleted all migrations of myapp and truncated the django_migrations table
Ran manage.py makemigrations and got
Cannot resolve bases for [] This can
happen if you are inheriting models from an app with migrations (e.g.
contrib.auth) in an app with no migrations; see
https://docs.djangoproject.com/en/1.8/topics/migrations/#dependencies
for more
... so I removed the cms app from settings.py, ran makemigrations again which now suceeded, then re-added and ran makemigrations again.
Ran manage.py migrate --fake, but got
Cannot resolve bases for [ModelState: 'djangocms_file.File', ModelState: 'djangocms_video.Video', ModelState: 'djangocms_link.Link', ModelState: 'djangocms_googlemap.GoogleMap', ModelState: 'djangocms_picture.Picture', ModelState: 'djangocms_teaser.Teaser']
... so I removed these apps as well from settings.py, ran migrate --fake, re-added the apps and migrate --fake again.
Removed the db_table setting from the new User model, ran makemigrations followed by migrate (without fake!)
Everything now seems to be in order. All foreign keys that previously referenced the standard auth_user table is now referincing the table of my new user model. Existing users have even been transferred because of the db_table trick.