I've run into this a couple other times and can't figure out why it happens. When I run the migrations all together through ./manage.py migrate then the last migration (a data migration) fails. The solution is to run the data migration on it's own after the other migrations have been completed. How can I run them all automatically with no errors?
I have a series of migrations:
fulfillment/0001.py
order/0041.py (dependency: fulfillment/0001.py)
order/0042.py
order/0043.py
I followed this RealPython article to move a model to a new app which which works perfectly and is covered by migrations #1 to #3. Migration #3 also adds a GenericForeignKey field. Migration #4 is a data migration that simply populates the GenericForeignKey field from the existing ForeignKey field.
from django.db import migrations, models
def copy_to_generic_fk(apps, schema_editor):
ContentType = apps.get_model('contenttypes.ContentType')
Order = apps.get_model('order.Order')
pickup_point_type = ContentType.objects.get(
app_label='fulfillment',
model='pickuppoint'
)
Order.objects.filter(pickup_point__isnull=False).update(
content_type=pickup_point_type,
object_id=models.F('pickup_point_id')
)
class Migration(migrations.Migration):
dependencies = [
('order', '0042'),
]
operations = [
migrations.RunPython(copy_to_generic_fk, reverse_code=migrations.RunPython.noop)
]
Running the sequence together I get an error:
fake.DoesNotExist: ContentType matching query does not exist.
If I run the migration to #3 then run #4 by itself everything works properly. How can I get them to run in sequence with no errors?
There is two things that might fix the problem, first look into run_before https://docs.djangoproject.com/en/3.1/howto/writing-migrations/#controlling-the-order-of-migrations
if you add it to fulfillment #1, and make sure it runs before orders #4, it should fix the problem.
Another thing that you can do is to move your data migrations to fulfillment #2, that way you know for sure all orders are finished and fulfillment #1 is also finished.
Instead of getting the ContentType through .get() you have to retrieve the model through the apps argument then use get_for_model().
def copy_to_generic_fk(apps, schema_editor):
ContentType = apps.get_model('contenttypes', 'ContentType')
PickupPoint = apps.get_model('fulfillment', 'pickuppoint')
pickup_point_type = ContentType.objects.get_for_model(PickupPoint)
...
Related
I read the following text on docs:
"""
Django’s default behavior is to run in autocommit mode. Each query is immediately committed to the database, unless a transaction is active. See below for details.
"""
and I'm running the following data migration:
def fill_query(apps, schema_editor):
Result = apps.get_model('monitoring', 'Result')
for r in Result.objects.all():
r.query = r.monitored_search.query
r.user_id = r.monitored_search.user_id
r.save()
class Migration(migrations.Migration):
dependencies = [
('monitoring', '0006_searchresult_user_id'),
]
operations = [
migrations.RunPython(fill_query),
]
But when I try to find objects from Result I found that all still have query and user_id as null. And my data migration keep running (more than 2 millions registers on database)
maybe the changes will be applied when data migration stop running or my data migration is not working?
I have this code and I want it to just create the groups every time the program runs so that if the database is deleted it will still be a sufficient program itself and someone won't have to create groups again, do you know an easy way to do this?
system_administrator = Group.objects.get_or_create(name='system_administrator')
manager = Group.objects.get_or_create(name='manager')
travel_advisor = Group.objects.get_or_create(name='travel_advisor')
If you lose your DB, you'd have to rerun migrations on a fresh db before the program could run again. So I think data migrations might be a good solution for this? A data migration, is a migration that runs python code to alter the data in the DB, not the schema as a normal migration does.
You could do something like this:
In a new migration file (you can run python manage.py makemigrations --empty yourappname to create an empty migration file for an app)
def generate_groups(apps, schema_editor):
Group = apps.get_model('yourappname', 'Group')
Group.objects.get_or_create(name="system_administrator")
Group.objects.get_or_create(name="manager")
Group.objects.get_or_create(name="travel_advisor")
class Migration(migrations.Migration):
dependencies = [
('yourappname', 'previous migration'),
]
operations = [
migrations.RunPython(generate_groups),
]
Worth reading the docs on this https://docs.djangoproject.com/en/3.0/topics/migrations/#data-migrations
You can do it in the ready method of one of your apps.
class YourApp(Appconfig):
def ready(self):
# important do the import inside the method
from something import Group
Group.objects.get_or_create(name='system_administrator')
Group.objects.get_or_create(name='manager')
Group.objects.get_or_create(name='travel_advisor')
The problem with the data migrations approach is that it is useful for populate the database the first time. But if the groups are deleted once the data migration has run, you will need to populate them again.
Also remember that get_or_create return a tuple.
group, created = Group.objects.get_or_create(name='manager')
# group if an instance of Group
# created is a boolean
I have a simple data migration, which creates a Group, and which looks like this :
def make_manager_group(apps, schema_editor):
Group = apps.get_model("auth", "Group")
managers_group = Group(name="managers")
managers_group.save()
class Migration(migrations.Migration):
dependencies = [
('my_app', '0001_initial'),
('auth', '0006_require_contenttypes_0002'),
]
operations = [
migrations.RunPython(make_manager_group, reverse_code=lambda *args, **kwargs: True)
]
and a simple functional test app containing the following tests :
from django.contrib.auth.models import Group
from django.contrib.staticfiles.testing import StaticLiveServerTestCase
class FunctionalTest(StaticLiveServerTestCase):
def setUp(self):
print("Groups : {}".format(Group.objects.all()))
def test_2(self):
pass
def test_1(self):
pass
When I run the tests, I get :
Creating test database for alias 'default'...
Groups : [<Group: managers>]
.Groups : []
.
Clearly, the group is being created when the test db is created, but when this db is reset between tests, it is reset to an empty db and not to the state it was after all the migrations were applied.
The model itself doesn't contain anything special (I only created one for the migration not to be the first, as in the project I'm working on, but I'm not sure it is needed at all).
Is this a bug, or am I missing something about data migration, to be able to have my group created when every single test starts?
Edit 1 : I'm using Django 1.8.3
Edit 2 : Quick'n'dirty hack added to the setUp of the test class :
from django.contrib.auth.models import Group, Permission
if not Group.objects.all():
managers_group = Group(name="managers")
managers_group.save()
managers_group.permissions.add(
Permission.objects.get(codename='add_news'),
Permission.objects.get(codename='change_news'),
Permission.objects.get(codename='delete_news')
)
This is all but DRY, but until now, I couldn't find another way...
I answer my own question :
It seems to be a filed bug which has became documented
It says that using TransactionTestCase and its subclasses (like in my case LiveServerTestCase) doesn't insert the data migrations before every test. It is just done once for the first of them.
It also says that we could set serialized_rollback to True, which should force the rollback to the filled database. But in my case, I'm having the same error as the last message in the bug report.
So I'm going to stick with my dirty hack for now, and maybe create a data fixture, as it says that fixtures are used everytime.
I'm running into an issue in Django 1.7 when attempting to write multiple migrations in a row. Here's the basic setup of the migrations:
Initial schema migration to create the models for an app
Data migration that calls loaddata on a specific fixture that contains one-time default data
A new optional field was added to one of the models, so it's a schemamigration to add the field
If I generate the first migration, run it, generate the second, run it, and then add the new field, generate the third migration, and run it, everything is fine. However, if my database were on migration #1 and then I pulled down from a source repository, migration 2 would fail because it uses the models from models.py when calling loaddata rather than the models as of the time of that migration. It then produces the following error:
"Unknown column '[field]' in 'field list'"
In this case, [field] is the new field that I added for migration #3. The error makes sense, because my database doesn't have the new field yet but loaddata expects it to be there (even though the fixture doesn't reference the new field), but is there any way to make loaddata use the database at the time of the migration rather than the current state in models.py? Or are there any other ways to get around this issue?
Thanks.
I ended up writing a hack to get around this for now, but I feel like there has to be a better way. Instead of calling loaddata in the migration, I now call this function:
def load_fixture_in_data_migration(apps, schema_editor, fixture_filename, migration_file):
"""
Load fixture data in data migrations without breaking everything
when the models change later on
"""
fixture_dir = os.path.abspath(os.path.join(os.path.dirname(migration_file), '../fixtures'))
fixture_file = os.path.join(fixture_dir, fixture_filename)
fixture = open(fixture_file, 'rb')
objects = serializers.deserialize('json', fixture, ignorenonexistent=True)
for obj in objects:
ObjApp = apps.get_model(obj.object._meta.app_label, obj.object._meta.object_name)
new_obj = ObjApp(pk=obj.object.pk)
for field in ObjApp._meta.fields:
setattr(new_obj, field.name, getattr(obj.object, field.name))
new_obj.save()
fixture.close()
And I call it like this from the data migration:
load_fixture_in_data_migration(apps, schema_editor, 'initial_add_ons.json', __file__)
Does anyone know a better way to do this? It feels really like a hack since I have to access object meta data to accomplish this.
Edit: I understand the reason why this happened. It was because of the existence of `initial_data.json` file. Apparently, south wants to add those fixtures after migration but failing because of the unique property of a field.
I changed my model from this:
class Setting(models.Model):
anahtar = models.CharField(max_length=20,unique=True)
deger = models.CharField(max_length=40)
def __unicode__(self):
return self.anahtar
To this,
class Setting(models.Model):
anahtar = models.CharField(max_length=20,unique=True)
deger = models.CharField(max_length=100)
def __unicode__(self):
return self.anahtar
Schema migration command completed successfully, but, trying to migrate gives me this error:
IntegrityError: duplicate key value violates unique constraint
"blog_setting_anahtar_key" DETAIL: Key (anahtar)=(blog_baslik) already
exists.
I want to keep that field unique, but still migrate the field. By the way, data loss on that table is acceptable, so long as other tables in DB stay intact.
It's actually the default behavior of syncdb to run initial_data.json each time. From the Django docs:
If you create a fixture named initial_data.[xml/yaml/json], that fixture will be loaded every time you run syncdb. This is extremely convenient, but be careful: remember that the data will be refreshed every time you run syncdb. So don't use initial_data for data you'll want to edit.
See: docs
Personally, I think the use-case for initial data that needs to be reloaded each and every time a change occurs is retarded, so I never use initial_data.json.
The better method, since you're using South, is to manually call loaddata on a specific fixture necessary for your migration. In the case of initial data, that would go in your 0001_initial.py migration.
def forwards(self, orm):
from django.core.management import call_command
call_command("loaddata", "my_fixture.json")
See: http://south.aeracode.org/docs/fixtures.html
Also, remember that the path to your fixture is relative to the project root. So, if your fixture is at "myproject/myapp/fixtures/my_fixture.json" call_command would actually look like:
call_command('loaddata', 'myapp/fixtures/my_fixture.json')
And, of course, your fixture can't be named 'initial_data.json', otherwise, the default behavior will take over.