How can I tell DSH to index particular field? Some queries that I do to historical models take too much time
I've base abstract model and all my models inherit from that model. The history field is also defined in this base model:
class BaseModel(models.Model):
class PublishingStatus(models.TextChoices):
DRAFT = 'draft', _('Draft')
ACCEPTED = 'accepted', _('Accepted'),
REJECTED = 'rejected', _('Rejected'),
MODIFIED = 'modified', _('Modified')
publishing_status = models.CharField(
max_length=9,
choices=PublishingStatus.choices,
default=PublishingStatus.DRAFT,
help_text=_("Publishing status represents the state of the object. By default it is 'draft'")
)
history = HistoricalRecords(inherit=True)
And I've also added indexes in this base model
class Meta:
abstract = True
indexes = [models.Index(fields=[
'publishing_status',
])]
It would be nice if Django Simple History could check which fields are indexed and create the same indexes in the historical models
Maybe there is a way to tell django simple history explicitly which field must be indexed additionally?
I hope there is a better solution, but I've ended up creating a custom migration file. First, I created a blank migration file for the app:
python manage.py makemigrations dosh --empty
I then modified this newly created empty migration file and added AddIndex for each historical model.
# Generated by Django 3.1.1 on 2020-10-15 06:49
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('dosh', '0002_auto_20201008_2155'),
]
operations = [
# ...
migrations.AddIndex(
model_name='historicaltense',
index=models.Index(fields=['publishing_status'], name='htense_publish_status_idx')
),
migrations.AddIndex(
model_name='historicalsynonym',
index=models.Index(fields=['publishing_status'], name='hsynonym_publish_status_idx')
),
# ...
]
And finally, I've migrated those changes
python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, authtoken, comment, contenttypes, dosh, pinax_badges, sessions, social, users, vote
Running migrations:
Applying dosh.0003_auto_20201015_0849... OK
Related
I have a Django application in which I want to change a field from a ForeignKey to a ManyToManyField. I want to preserve my old data. What is the simplest/best process to follow for this? If it matters, I use sqlite3 as my database back-end.
If my summary of the problem isn't clear, here is an example. Say I have two models:
class Author(models.Model):
author = models.CharField(max_length=100)
class Book(models.Model):
author = models.ForeignKey(Author)
title = models.CharField(max_length=100)
Say I have a lot of data in my database. Now, I want to change the Book model as follows:
class Book(models.Model):
author = models.ManyToManyField(Author)
title = models.CharField(max_length=100)
I don't want to "lose" all my prior data.
What is the best/simplest way to accomplish this?
Ken
I realize this question is old and at the time the best option for Data Migrations was using South. Now Django has its own migrate command, and the process is slightly different.
I've added these models to an app called books -- adjust accordingly if that's not your case.
First, add the field to Book and a related_name to at least one, or both of them (or they'll clash):
class Book(models.Model):
author = models.ForeignKey(Author, related_name='book')
authors = models.ManyToManyField(Author, related_name='books')
title = models.CharField(max_length=100)
Generate the migration:
$ ./manage.py makemigrations
Migrations for 'books':
0002_auto_20151222_1457.py:
- Add field authors to book
- Alter field author on book
Now, create an empty migration to hold the migration of the data itself:
./manage.py makemigrations books --empty
Migrations for 'books':
0003_auto_20151222_1459.py:
And add the following content to it. To understand exactly how this works, check the documentation on Data Migrations. Be careful not to overwrite the migration dependency.
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import models, migrations
def make_many_authors(apps, schema_editor):
"""
Adds the Author object in Book.author to the
many-to-many relationship in Book.authors
"""
Book = apps.get_model('books', 'Book')
for book in Book.objects.all():
book.authors.add(book.author)
class Migration(migrations.Migration):
dependencies = [
('books', '0002_auto_20151222_1457'),
]
operations = [
migrations.RunPython(make_many_authors),
]
Now remove the author field from the Model -- it should look like this:
class Book(models.Model):
authors = models.ManyToManyField(Author, related_name='books')
title = models.CharField(max_length=100)
Create a new migration for that, and run them all:
$ ./manage.py makemigrations
Migrations for 'books':
0004_remove_book_author.py:
- Remove field author from book
$ ./manage.py migrate
Operations to perform:
Synchronize unmigrated apps: messages, staticfiles
Apply all migrations: admin, auth, sessions, books, contenttypes
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying books.0002_auto_20151222_1457... OK
Applying books.0003_auto_20151222_1459... OK
Applying books.0004_remove_book_author... OK
And that's it. The authors previously available at book.author now should be in the queryset you get from book.authors.all().
Probably the best and easiest thing you should do would be:
Create the Many to many field with a different name say
authors = models.ManyToManyField(Author)
write a small function to convert foreignkey values to M2M values:
def convert():
books = Book.objects.all()
for book in books:
if book.author:
li = [book.author.id]
book.authors.append(li)
book.save()
Once it is run, you can delete the author field from the table and run migration again.
I have added a UUID to the following model:
class Post(models.Model):
uuid = models.UUIDField(default=uuid.uuid4, editable=False)
...
But there already are entries in the database that were created without the uuid field.
When I run migrate, it adds the same UUID to all my previous objects.
Is there an easy way of populating the existing objects with a different UUID automatically?
I think the easiest way to fix this is to make a data migration [Django-doc]. You create a new migration file with:
python manage.py makemigrations --empty yourappname
This will create a new file in the migrations/ directory that does nothing. We can alter that migration file with something that looks like:
# Generated by Django 3.2 on 2021-05-01 12:49
from django.db import migrations
from uuid import uuid4
class Migration(migrations.Migration):
def populate_uuid(apps, schema_editor):
Post = apps.get_model('yourappname', 'Post')
posts = list(Post.objects.all())
for post in posts:
post.uuid = uuid4()
Post.objects.bulk_update(posts, ['uuid'])
dependencies = [
('yourappname', 'previous_migrationname'),
]
operations = [
migrations.RunPython(populate_uuid)
]
If you then migrate, it will load all Post objects, provide these each a unique uuid, and then update the items in bulk.
I'm sure this has been asked before, but I cannot find the answer. In django, if I have this model
from django.db import models
class Person(models.Model):
first_name = models.CharField(max_length=30)
last_name = models.CharField(max_length=30)
How would one populate this model with a default Person entry when the first migrating the field?
I'm not talking about default values for the fields, but for default entries in the database.
thanks
You can make a data migration, that follows on the migration where you create the Person object. You can first let Django write the "skeleton" of the migration, this can be done with:
python manage.py makemigrations --empty appname
Next Django will make a file. In that file you can add RunPython item to the operations list. This then obtain the historical model (the model at that moment of the migration), where you then create a Person object in the database. For example with:
from django.db import migrations
class Migration(migrations.Migration):
def create_person(apps, schema_editor):
Person = apps.get_model('appname', 'Person')
Person.objects.create(first_name='will', last_name='mendil')
dependencies = [
('appname', 'migrationname'),
]
operations = [
migrations.RunPython(create_person)
]
I have an existing django-cms ver. 3.1.3 project where I would like to replace the default django User model with the one found in django-custom-user (to have email as user name for my users). I have added the custom_user app to my INSTALLED_APPS and set AUTH_USER_MODEL = 'custom_user.EmailUser'. Finally I applied the migrations.
Everything seems to work fine in my custom models, but the django-cms models that have a reference to the auth_user table (GlobalPagePermission, PagePermission, PageUser and UserSettings) are not updated to have a foreign-key reference to the new custom user table.
The django-cms documentation says that is generally advisable to add custom user models in the beginning of a project, but here I am, in the middle of a project, and would very much like to avoid having to delete my cms models (with data) to make it work.
If I were in the beginning of a project, and had added the custom user model before migrating the django-cms models, would the django-cms models actually get a reference to the custom user model table instead of the default one?
Is there any way for me to make migrations for the django-cms models, so they use the new custom user model instead of the default one?
UPDATE
Im trying to implement what yakky suggested:
from __future__ import unicode_literals
from django.conf import settings
from django.db import models, migrations
import django.db.models.deletion
from django.utils.translation import ugettext_lazy as _
def up(apps, schema_editor):
PageUser = apps.get_model("cms", "PageUser")
db_alias = schema_editor.connection.alias
for pu in PageUser.objects.all():
pu['emailuser_ptr_id'] = pu['user_ptr_id']
pu.save()
def down(apps, schema_editor):
PageUser = apps.get_model("cms", "PageUser")
db_alias = schema_editor.connection.alias
for pu in PageUser.objects.all():
pu['user_ptr_id'] = pu['emailuser_ptr_id']
pu.save()
class Migration(migrations.Migration):
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
(‘myapp’, ‘previous-migration’),
]
operations = [
migrations.RenameField(
model_name='PageUser',
old_name='user_ptr_id',
new_name='user_ptr_id_old',
),
migrations.AddField(
model_name='PageUser',
name='emailuser_ptr_id',
field=models.ForeignKey(null=True, to=settings.AUTH_USER_MODEL, verbose_name=_('user'), blank=True),
preserve_default=True,
),
migrations.RunPython(up, down),
migrations.RemoveField(
model_name='PageUser',
name='user_ptr_id_old',
),
]
It fails with KeyError: ('myapp', u'pageuser') suggesting that it looks for the PageUser model in the my custom app and not in the cms app. How do I apply these migrations to the cms models?
Being said that replacing the Django User model it's not advisable in the middle of a project lifetime (see https://docs.djangoproject.com/en/1.9/topics/auth/customizing/#substituting-a-custom-user-model) as Django does not provide any way to modify the relations after their are being created, to replace the custom user model you can write a custom migration that alter the foreignkey in the django CMS models to point to theand create a new one to the new user model.
Something like:
...
migrations.RenameField(
model_name='globalpagepermission',
old_name='user',
new_name='user_old',
),
migrations.AddField(
model_name='globalpagepermission',
name='user',
field=models.ForeignKey(null=True, to=settings.AUTH_USER_MODEL, verbose_name=_('user'), blank=True),
preserve_default=True,
),
migrations.RunPython(copy_data),
migrations.RemoveField(
model_name='globalpagepermission',
name='user_old',
),
...
Repeat for any relevant model / field.
copy_data method will copy the value for the foreign keys from user_old to user
Code is completely untested but the general idea should work.
Alternatively you can write a RunSQL migration to get the same result.
Users with the same primary keys must exist also for the new model for this to work.
Thanks to https://code.djangoproject.com/ticket/25313 I managed to make it work. This is what I did:
Backed up my database!
Created a new app email_username to hold my user model (named User as recommended in the link). A new app was necessary because my main app (myapp) is dependent on django-cms, and if I put my new user model in myapp, django-cms would become dependent on my app - this would create a circular reference.
class User(AbstractEmailUser):
# Needed to add username, first_name and last_name here, because cms.PageUser model depends on these fields
username = models.CharField(max_length=100, verbose_name=_('username'))
first_name = models.CharField(max_length=100, verbose_name=_('first_name'))
last_name = models.CharField(max_length=100, verbose_name=_('last_name'))
class Meta:
db_table = 'auth_user' # the model will use the existing auth_user table (at first)
Set AUTH_USER_MODEL = ‘email_username.User’ in settings.py
Deleted all migrations of myapp and truncated the django_migrations table
Ran manage.py makemigrations and got
Cannot resolve bases for [] This can
happen if you are inheriting models from an app with migrations (e.g.
contrib.auth) in an app with no migrations; see
https://docs.djangoproject.com/en/1.8/topics/migrations/#dependencies
for more
... so I removed the cms app from settings.py, ran makemigrations again which now suceeded, then re-added and ran makemigrations again.
Ran manage.py migrate --fake, but got
Cannot resolve bases for [ModelState: 'djangocms_file.File', ModelState: 'djangocms_video.Video', ModelState: 'djangocms_link.Link', ModelState: 'djangocms_googlemap.GoogleMap', ModelState: 'djangocms_picture.Picture', ModelState: 'djangocms_teaser.Teaser']
... so I removed these apps as well from settings.py, ran migrate --fake, re-added the apps and migrate --fake again.
Removed the db_table setting from the new User model, ran makemigrations followed by migrate (without fake!)
Everything now seems to be in order. All foreign keys that previously referenced the standard auth_user table is now referincing the table of my new user model. Existing users have even been transferred because of the db_table trick.
I'm trying to rename a model and I would like to write the migration in the way that it doesn't depend on the old name still present while it being applied. Can I somehow get data from a database table that no longer has a model in my migration code?
Details:
I have a Region model that I want to move into a more generic GeoObject model and remove from the models.py. If I write my migration code that creates GeoObjects from existing Regions with from models import Region I'll have to keep Region model until my main database will migrate. But I'd like to write a migration so that it doesn't depend on Region model being present, just check that the database table exists and use it. Is it possible to do it using Django instruments, without depending on a specific database type if possible?
Yes, you can.
But first of all, you really shouldn't import any model inside migration.
Take look at RunPython operation, that will allow you to run any python code inside your migration. RunPython will pass to your function 2 parameters: apps and schema_editor. First parameter contains structure of your models at stage of applying that migration, so if actual removing of model is later on that migration, you can still access that model using apps passed into function.
Let's say your model looked like this:
class SomeModel(models.Model):
some_field = models.CharField(max_length=32)
Now you're deleting that model, automatically created migration will contain:
class Migration(migrations.Migration):
dependencies = [
('yourapp', '0001_initial'), # or any other dependencies
]
operations = [
migrations.DeleteModel(
name='Main',
),
]
You can modify that migration by injecting RunPython just above DeleteModel operation:
operations = [
migrations.RunPython(
move_data_to_other_model,
move_data_back, # for backwards migration - if you won't ever want to undo this migration, just don't pass that function at all
),
migrations.DeleteModel(
name='SomeModel',
),
]
and creating 2 functions before Migration class:
def move_data_to_other_model(apps, schema_editor):
SomeModel = apps.get_model('yourapp', 'SomeModel')
for something in SomeModel.objects.all():
# do your data migration here
o = OtherModel.objects.get(condition=True)
o.other_field = something.some_field
def move_data_back(apps, schema_editor):
SomeModel = apps.get_model('yourapp', 'SomeModel')
for something in OtherModel.objects.all():
# move back your data here
SomeModel(
some_field=something.other_field,
).save()
It doesn't matter that your model is no longer defined in models.py, django can rebuild that model based on migration history. But remember: save method from your models (and other customized methods) won't be called in migrations. Also any pre_save or post_save signals won't be triggered.