I am using Django 1.5b1 and south migrations and life has generally been great. I have some schema updates which create my database, with a User table among others. I then load a fixture for ff.User (my custom user model):
def forwards(self, orm):
from django.core.management import call_command
fixture_path = "/absolute/path/to/my/fixture/load_initial_users.json"
call_command("loaddata", fixture_path)
All has been working great until I have added another field to my ff.User model, much further down the migration line. My fixture load now breaks:
DatabaseError: Problem installing fixture 'C:\<redacted>create_users.json':
Could not load ff.User(pk=1): (1054, "Unknown column 'timezone_id' in 'field list'")
Timezone is the field (ForeignKey) which I added to my user model.
The ff.User differs from what is in the database, so the Django ORM gives up with a DB error. Unfortunately, I cannot specify my model in my fixture as orm['ff.User'], which seems to be the south way of doing things.
How should I load fixtures properly using south so that they do not break once the models for which these fixtures are for gets modified?
I found a Django snippet that does the job!
https://djangosnippets.org/snippets/2897/
It load the data according to the models frozen in the fixture rather than the actual model definition in your apps code! Works perfect for me.
I proposed a solution that might interest you too:
https://stackoverflow.com/a/21631815/797941
Basicly, this is how I load my fixture:
from south.v2 import DataMigration
import json
class Migration(DataMigration):
def forwards(self, orm):
json_data=open("path/to/your/fixture.json")
items = json.load(json_data)
for item in items:
# Be carefull, this lazy line won't resolve foreign keys
obj = orm[item["model"]](**item["fields"])
obj.save()
json_data.close()
This was a frustrating part of using fixtures for me as well. My solution was to make a few helper tools. One which creates fixtures by sampling data from a database and includes South migration history in the fixtures.
There's also a tool to add South migration history to existing fixtures.
The third tool checks out the commit when this fixture was modified, loads the fixture, then checks out the most recent commit and does a south migration and dumps the migrated db back to the fixture. This is done in a separate database so your default db doesn't get stomped on.
The first two can be considered beta code, and the third please treat as usable alpha, but they're already being quite helpful to me.
Would love to get some feedback from others:
git#github.com:JivanAmara/django_fixture_tools.git
Currently, it only supports projects using git as the RCS.
The most elegant solution I've found is here where by your app model's get_model function is switched out to instead supply the model from the supplied orm. It's then set back after the fixture is applied.
from django.db import models
from django.core.management import call_command
def load_fixture(file_name, orm):
original_get_model = models.get_model
def get_model_southern_style(*args):
try:
return orm['.'.join(args)]
except:
return original_get_model(*args)
models.get_model = get_model_southern_style
call_command('loaddata', file_name)
models.get_model = original_get_model
You call it with load_fixture('my_fixture.json', orm) from within you forwards definition.
Generally South handles migrations using forwards() and backwards() functions. In your case you should either:
alter the fixtures to contain proper data, or
import fixture before migration that breaks it (or within the same migration, but before altering the schema),
In the second case, before migration adding (or, as in your case, removing) the column, you should perform the migration that will explicitly load the fixtures similarly to this (docs):
def forwards(self, orm):
from django.core.management import call_command
call_command("loaddata", "create_users.json")
I believe this is the easiest way to accomplish what you needed. Also make sure you do not do some simple mistakes like trying to import data with new structure before applying older migrations.
Reading the following two posts has helped me come up with a solution:
http://andrewingram.net/2012/dec/common-pitfalls-django-south/#be-careful-with-fixtures
http://news.ycombinator.com/item?id=4872596
Specifically, I rewrote my data migrations to use output from 'dumpscript'
I needed to modify the resulting script a bit to work with south. Instead of doing
from ff.models import User
I do
User = orm['ff.User']
This works exactly like I wanted it to. Additionally, it has the benefit of not hard-coding IDs, like fixtures require.
Related
I need to do some unit tests in my Django project. The problem is that almost every use case depends on prepopulated database objects.
For example, I want to create a product and test, if there were all pre_save signals successful.
from django.contrib.auth.models import User
from django.test import TestCase
from .models import Product
class ProductTestCase(TestCase):
def setUp(self):
self.user = User.objects.create(username='test_user')
self.product = Product.objects.create(name='Test product',user=self.user)
def test_product_exists(self):
self.assertIsNotNone(self.product)
def product_is_active_by_default(self):
...
I can't do that because product has to have User object related. But I can't create a User object because User has to have related plan object. There are multiple plans in my production database from which one is default but there are no plans inside test database.
So to be able to do unit tests I need to prepopulate test database with multiple objects from multiple apps.
How can I do that?
you can simply use django fixtures for that :-)
first populate a sample db with data then export data with python manage.py dumpdata
then in one of your apps create a directory named fixtures and put exported json file there (named tests.json or something else)
in your test class load fixtures like this
class ProductTestCase(TestCase):
fixtures = ['tests.json', ]
checkout django docs
PS: checkout factory boy too (#Gabriel Muj) answer
I don't recommend using fixture since you will need to maintain them each time you make changes to the model. Here is a better approach on creating objects for tests by using this library https://factoryboy.readthedocs.io/en/latest/ which is more flexible.
I need to add a many-to-many relationship in my setUpTestData sequence so that my tests will run correctly.
According to the docs, Many-to-Many relationships cannot be created until an object has been saved, as the primary key must first exist. This means that I cannot set the relationship in setUpTestData with Model.objects.create() as I do with other variables.
Is there any way to include a Many-to-Many relationship in setUpTestData?
As a partial answer, a solution I found in the docs seems to work half way. Here is my code:
(I'm linking a social app to Django's Sites framework so that my tests will be able to run).
SetupTestData.py
from django.test import TestCase
from django.contrib.sites.models import Site
from allauth.socialaccount.models import SocialApp
class TestData(TestCase):
#classmethod
def setUpTestData(cls):
cls.current_site = Site.objects.get_current()
cls.SocialApp1 = cls.current_site.socialapp_set.create(
provider="facebook",
name="facebook",
client_id="123456789",
secret="0987654321",
)
test_sandbox.py
from MyApp.tests.SetupTestData import TestData
from django.contrib.sites.models import Site
from allauth.socialaccount.models import SocialApp
class TestHomePage(TestData):
def test_sandbox(self):
print(Site.objects.all())
print(Site.objects.get_current())
print(Site.objects.get_current().socialapp_set.all())
print(self.SocialApp1)
print(self.SocialApp1.sites)
print(SocialApp.objects.get(provider='facebook').sites)
Test Output
Creating test database for alias 'default'...
[<Site: example.com>]
example.com
[<SocialApp: facebook>]
facebook
sites.Site.None
sites.Site.None
Destroying test database for alias 'default'...
You can see from the results that I can locate the app via Sites, but it doesn't seem to work backwards. I suppose if I somehow reversed the relationship in setUpTestData I could make it work the opposite way instead. I'll leave the question open for a better solution.
I've Django project which is using South application to handle schema and data migration. In one of my applications I have migration (number 0004) which is responsible for loading data fixtures from json file:
class Migration(DataMigration):
def forwards(self, orm):
from django.core.management import call_command
call_command("loaddata", "dummy_data.json")
In the same project I try to add functionality of 'soft delete' which needs adding one more filed, defined as:
deleted_at = models.DateTimeField(blank=True, null=True)
Based on this change I've added new migration, which has number 0009. After that I start migrate command which give me error:
DatabaseError: Problem installing fixture 'C:/Users/Grzegorz/PycharmProjects/Dummy Project/Dummy\app_subapp\fixtures\dummy_data.json': Could not load app_subapp.DummyData(pk=1): (1054, "Unknown column 'deleted_at' in 'field list'")
It's quite strange, because this error occurs while applying migration 0004 which earlier worked ok and from point of South process in this step filed deleted_at shouldn't and doesn't exists in my database.I've found that moving migration with loading fixture from step 0004 after 0009 resolves problem, but it looks like very dirty and not good approach to resolve this issue.
Do you have any advices how can I resolve this problem and properly handle migrations and fixture loading with South?
I found a Django snippet that does the job!
https://djangosnippets.org/snippets/2897/
It load the data according to the models frozen in the fixture rather the actual model definition in your apps code!
Works perfect for me.
I've found workaround to my problem. Finally I've extracted load fixtures from South migration and delegated this action to Fabric. Now I have separated migration and load initial data, so everything works as I expect.
An other solution is to load the fixture file and insert it using the migration's orm :
from south.v2 import DataMigration
import json
class Migration(DataMigration):
def forwards(self, orm):
json_data=open("path/to/your/fixture.json")
items = json.load(json_data)
for item in items:
# Be carefull, this lazy line won't resolve foreign keys
obj = orm[item["model"]](**item["fields"])
obj.save()
json_data.close()
Using this method, you're fixture will be loaded within the current database structure.
I use Django and South for my database. Now I want to add a new Model and a field in an existing model, referencing the new model. For example:
class NewModel(models.Model):
# a new model
# ...
class ExistingModel(models.Model):
# ... existing fields
new_field = models.ForeignKey(NewModel) # adding this now
Now South obviously complains that I added a non-null field and asks me to enter a one-off value. But what I really want is to create a new NewModel instance for every existing ExistingModel instance, thus fulfilling the database requirements. Is that possible somehow?
The easiest way to do this is to write a schema migration that makes the column change, and then write a datamigration to correctly fill in the value. Depending on the database you're using you'll have to do this in slightly different ways.
Sqlite
For Sqlite, you can add a sentinel value for the relation and use a datamigration to fill it in without any issue:
0001_schema_migration_add_foreign_key_to_new_model_from_existing_model.py
from south.db import db
from south.v2 import SchemaMigration
from django.db import models
class Migration(SchemaMigration):
def forwards(self, orm):
db.add_column('existing_model_table', 'new_model',
self.gf('django.db.models.fields.related.ForeignKey')(default=0, to=orm['appname.new_model']), keep_default=False)
0002_data_migration_for_new_model.py:
class Migration(DataMigration):
def forwards(self, orm):
for m in orm['appname.existing_model'].objects.all():
m.new_model = #custom criteria here
m.save()
This will work just fine, with no issues.
Postgres and MySQL
With MySql, you have to give it a valid default. If 0 isn't actually a valid Foreignkey, you'll get errors telling you so.
You could default to 1, but there are instances where that isn't a valid foreign key (happened to me because we have different environments, and some environments publish to other databases, so the IDs rarely match up (we use UUIDs for cross-database identification, as God intended).
The second issue you get is that South and MySQL don't play well together. Partially because MySQL doesn't have the concept of DDL transactions.
In order to get around some issues you will inevitably face (including the error I mentioned above and from South asking you to mark orm items in a SchemaMigration as no-dry-run), you need to change the above 0001 script to do the following:
0001_schema_migration_add_foreign_key_to_new_model_from_existing_model.py
from south.db import db
from south.v2 import SchemaMigration
from django.db import models
class Migration(SchemaMigration):
def forwards(self, orm):
id = 0
if not db.dry_run:
new_model = orm['appname.new_model'].objects.all()[0]
if new_model:
id = new_model.id
db.add_column('existing_model_table', 'new_model',
self.gf('django.db.models.fields.related.ForeignKey')(default=id, to=orm['appname.new_model']), keep_default=False)
And then you can run the 0002_data_migration_for_new_model.py file as normal.
I advise using the same example above for Postgres and for MySql. I don't remember any issues offhand with Postgres with the first example, but I'm certain the second example works for both (tested).
You want a data migration to supplement your schema migration in this scenario.
South has a nice step by step tutorial on how to achieve this in the docs, here.
It's not uncommon in South to have the desired outcome spread over two or three schema/data migrations as its not always possible to do it in one big hit (sometimes depends on the underlying db if it will tolerate adding a non null column with no default). So in this case you might add a schema migration that has a default, then a data migration with your object manipulation then a final schema migration.
I'm trying to get my app to run some custom SQL on syncdb with the official method of placing some INSERT statements into /sql/.sql
Now, when I run "manage.py sqlall ", all the SQL I want run is there.
However, after running syncdb, the data I want is nowhere to be found in the database! Am I missing something?
EDIT: The app I want to make the inserts for is using South migrations, and this is maybe why the initial SQL is skipped. Does anyone know how I can force it to run the SQL after migration, perhaps?
Initial data sql doesn't get run for South-managed applications. From what I understood this is by design, though I couldn't find any formal proof of that, except this mailing list post.
To achieve the same behavior you can create a migration and convert your SQL to Python. This is how I did it for installing a view:
Create and edit the migration:
$ python manage.py schemamigration app1 install_foo_view --empty
$ vim app1/migrations/*_install_foo_view.py
Here's the migration itself:
from south.db import db
from south.v2 import SchemaMigration
class Migration(SchemaMigration):
def forwards(self, orm):
db.execute("CREATE VIEW app1_foo AS SELECT col1, col2 FROM app1_bar")
def backwards(self, orm):
db.execute("DROP VIEW app1_foo")
# autogenerated models attibute goes here
complete_apps = ['app1']
For data migrations the flow is similar:
$ python manage.py datamigration app1 install_bars
$ vim app1/migrations/*_install_bars.py
Here's the migration itself:
from south.db import db
from south.v2 import DataMigration
class Migration(DataMigration):
def forwards(self, orm):
orm.Bar.objects.create(col1='val1', col2='val2')
def backwards(self, orm):
orm.Bar.objects.filter(col1='val1', col2='val2').delete()
# autogenerated models attibute goes here
complete_apps = ['app1']
The official South doc for data fixtures is broken:
It doesn't scale well with respect to schema changes.
You should copy and adjust the fixture file each time you change the Bar model fields. So you would end up with "my_fixture_v1.json", "my_fixture_v2.json" and so on.
Otherwise, if you don't keep versioning them and just keep the latest version, there will be mismatches between previous versions of model and the fixture and you won't be able to navigate migrations back and forth.
It doesn't support backwards migration.
The suggested approach takes care of both of these issues:
Since you use orm.Bar you have the set of fields that is applicable to Bar right at that point in history. You won't get any missing or extra fields.
There's no need to keep copying anything with Bar changes.
It is possible to migrate backwards.
Note that filter().delete() combination is used instead of get().delete(). This way you won't delete the Bars that has been changed by user, neither you fail with Bar.DoesNotExist when not finding them.
I am not exactly sure what you are doing, but if you are wanting to prepopulate the database you need to use fixtures.
https://docs.djangoproject.com/en/1.3/howto/initial-data/