So i have this model where I added user field as foreign key to User model
from django.db import models
from django.contrib.auth.models import User
from django.conf import settings
class Post(models.Model):
url = models.URLField()
title = models.CharField(max_length=140)
user = models.ForeignKey(settings.AUTH_USER_MODEL)
Than I run
python3 manage.py makemigrations
And get:
You are trying to add a non-nullable field 'user' to comment without a default;
we can't do that (the database needs something to populate existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows)
2) Quit, and let me add a default in models.py
I understand that I need to give it default user but I don't get how to do that.
P.S. Django version - 1.8.3
Ok, so it happens migration utility's question hadn't appeared intuitive enough for me. What migration utility was really asking were required fields of User model. So basically you need to type in some data 3 times (for username, password and email). The problem is utility isn't really saying what field is it expecting.
I'd prefer not to destroy all the users on my site. But I want to take advantage of Django 1.5's custom pluggable user model. Here's my new user model:
class SiteUser(AbstractUser):
site = models.ForeignKey(Site, null=True)
Everything works with my new model on a new install (I've got other code, along with a good reason for doing this--all of which are irrelevant here). But if I put this on my live site and syncdb & migrate, I'll lose all my users or at least they'll be in a different, orphaned table than the new table created for my new model.
I'm familiar with South, but based on this post and some trials on my part, it seems its data migrations are not currently a fit for this specific migration. So I'm looking for some way to either make South work for this or for some non-South migration (raw SQL, dumpdata/loaddata, or otherwise) that I can run on each of my servers (Postgres 9.2) to migrate the users once the new table has been created while the old auth.User table is still in the database.
South is more than able to do this migration for you, but you need to be smart and do it in stages. Here's the step-by-step guide: (This guide presupposed you subclass AbstractUser, not AbstractBaseUser)
Before making the switch, make sure that south support is enabled in the application
that contains your custom user model (for the sake of the guide, we'll call it accounts and the model User).
At this point you should not yet have a custom user model.
$ ./manage.py schemamigration accounts --initial
Creating migrations directory at 'accounts/migrations'...
Creating __init__.py in 'accounts/migrations'...
Created 0001_initial.py.
$ ./manage.py migrate accounts [--fake if you've already syncdb'd this app]
Running migrations for accounts:
- Migrating forwards to 0001_initial.
> accounts:0001_initial
- Loading initial data for accounts.
Create a new, blank user migration in the accounts app.
$ ./manage.py schemamigration accounts --empty switch_to_custom_user
Created 0002_switch_to_custom_user.py.
Create your custom User model in the accounts app, but make sure it is defined as:
class SiteUser(AbstractUser): pass
Fill in the blank migration with the following code.
# encoding: utf-8
from south.db import db
from south.v2 import SchemaMigration
class Migration(SchemaMigration):
def forwards(self, orm):
# Fill in the destination name with the table name of your model
db.rename_table('auth_user', 'accounts_user')
db.rename_table('auth_user_groups', 'accounts_user_groups')
db.rename_table('auth_user_user_permissions', 'accounts_user_user_permissions')
def backwards(self, orm):
db.rename_table('accounts_user', 'auth_user')
db.rename_table('accounts_user_groups', 'auth_user_groups')
db.rename_table('accounts_user_user_permissions', 'auth_user_user_permissions')
models = { ....... } # Leave this alone
Run the migration
$ ./manage.py migrate accounts
- Migrating forwards to 0002_switch_to_custom_user.
> accounts:0002_switch_to_custom_user
- Loading initial data for accounts.
Make any changes to your user model now.
# settings.py
AUTH_USER_MODEL = 'accounts.User'
# accounts/models.py
class SiteUser(AbstractUser):
site = models.ForeignKey(Site, null=True)
create and run migrations for this change
$ ./manage.py schemamigration accounts --auto
+ Added field site on accounts.User
Created 0003_auto__add_field_user_site.py.
$ ./manage.py migrate accounts
- Migrating forwards to 0003_auto__add_field_user_site.
> accounts:0003_auto__add_field_user_site
- Loading initial data for accounts.
Honestly, If you already have good knowledge of your setup and already use south, It should be as simple as adding the following migration to your accounts module.
# encoding: utf-8
from south.db import db
from south.v2 import SchemaMigration
from django.db import models
class Migration(SchemaMigration):
def forwards(self, orm):
# Fill in the destination name with the table name of your model
db.rename_table('auth_user', 'accounts_user')
db.rename_table('auth_user_groups', 'accounts_user_groups')
db.rename_table('auth_user_permissions', 'accounts_user_permissions')
# == YOUR CUSTOM COLUMNS ==
db.add_column('accounts_user', 'site_id',
models.ForeignKey(orm['sites.Site'], null=True, blank=False)))
def backwards(self, orm):
db.rename_table('accounts_user', 'auth_user')
db.rename_table('accounts_user_groups', 'auth_user_groups')
db.rename_table('accounts_user_user_permissions', 'auth_user_user_permissions')
# == YOUR CUSTOM COLUMNS ==
db.remove_column('accounts_user', 'site_id')
models = { ....... } # Leave this alone
EDIT 2/5/13: added rename for auth_user_group table. FKs will auto update to point at the correct table due to db constraints, but M2M fields' table names are generated from the names of the 2 end tables and will need manual updating in this manner.
EDIT 2: Thanks to #Tuttle & #pix0r for the corrections.
My incredibly lazy way of doing this:
Create a new model (User), extending AbstractUser. Within new model, in it's Meta, override db_table and set to 'auth_user'.
Create an initial migration using South.
Migrate, but fake the migration, using --fake when running migrate.
Add new fields, create migration, run it normally.
This is beyond lazy, but works. You now have a 1.5 compliant User model, which just uses the old table of users. You also have a proper migration history.
You can fix this later on with manual migrations to rename the table.
I think you've correctly identified that a migration framework like South is the right way to go here. Assuming you're using South, you should be able to use the Data Migrations functionality to port the old users to your new model.
Specifically, I would add a forwards method to copy all rows in your user table to the new table. Something along the lines of:
def forwards(self, orm):
for user in orm.User.objects.all():
new_user = SiteUser(<initialize your properties here>)
new_user.save()
You could also use the bulk_create method to speed things up.
I got tired of struggling with South so I actually ended up doing this differently and it worked out nicely for my particular situation:
First, I made it work with ./manage.py dumpdata, fixing up the dump, and then ./manage.py loaddata, which worked. Then I realized I could do basically the same thing with a single, self-contained script that only loads necessary django settings and does the serialization/deserialization directly.
Self-contained python script
## userconverter.py ##
import json
from django.conf import settings
settings.configure(
DATABASES={
# copy DATABASES configuration from your settings file here, or import it directly from your settings file (but not from django.conf.settings) or use dj_database_url
},
SITE_ID = 1, # because my custom user implicates contrib.sites (which is why it's in INSTALLED_APPS too)
INSTALLED_APPS = ['django.contrib.sites', 'django.contrib.auth', 'myapp'])
# some things you have to import after you configure the settings
from django.core import serializers
from django.contrib.auth.models import User
# this isn't optimized for huge amounts of data -- use streaming techniques rather than loads/dumps if that is your case
old_users = json.loads(serializers.serialize('json', User.objects.all()))
for user in old_users:
user['pk'] = None
user['model'] = "myapp.siteuser"
user['fields']["site"] = settings['SITE_ID']
for new_user in serializers.deserialize('json', json.dumps(old_users)):
new_user.save()
With dumpdata/loaddata
I did the following:
1) ./manage.py dumpdata auth.User
2) Script to convert auth.user data to new user. (or just manually search and replace in your favorite text editor or grep) Mine looked something like:
def convert_user_dump(filename, site_id):
file = open(filename, 'r')
contents = file.read()
file.close()
user_list = json.loads(contents)
for user in user_list:
user['pk'] = None # it will auto-increment
user['model'] = "myapp.siteuser"
user['fields']["site"] = side_id
contents = json.dumps(user_list)
file = open(filename, 'w')
file.write(contents)
file.close()
3) ./manage.py loaddata filename
4) set AUTH_USER_MODEL
*Side Note: One critical part of doing this type of migration, regardless of which technique you use (South, serialization/modification/deserialization, or otherwise) is that as soon as you set AUTH_USER_MODEL to your custom model in the current settings, django cuts you off from auth.User, even if the table still exists.*
We decided to switch to a custom user model in our Django 1.6/Django-CMS 3 project, perhaps a little bit late because we had data in our database that we didn't want to lose (some CMS pages, etc).
After we switched AUTH_USER_MODEL to our custom model, we had a lot of problems that we hadn't anticipated, because a lot of other tables had foreign keys to the old auth_user table, which wasn't deleted. So although things appeared to work on the surface, a lot of things broke underneath: publishing pages, adding images to pages, adding users, etc. because they tried to create an entry in a table that still had a foreign key to auth_user, without actually inserting a matching record into auth_user.
We found a quick and dirty way to rebuild all the tables and relations, and copy our old data across (except for users):
do a full backup of your database with mysqldump
do another backup with no CREATE TABLE statements, and excluding a few tables that won't exist after the rebuild, or will be populated by syncdb --migrate on a fresh database:
south_migrationhistory
auth_user
auth_user_groups
auth_user_user_permissions
auth_permission
django_content_types
django_site
any other tables that belong to apps that you removed from your project (you might only find this out by experimenting)
drop the database
recreate the database (e.g. manage.py syncdb --migrate)
create a dump of the empty database (to make it faster to go round this loop again)
attempt to load the data dump that you created above
if it fails to load because of a duplicate primary key or a missing table, then:
edit the dump with a text editor
remove the statements that lock, dump and unlock that table
reload the empty database dump
try to load the data dump again
repeat until the data dump loads without errors
The commands that we ran (for MySQL) were:
mysqldump <database> > ~/full-backup.sql
mysqldump <database> \
--no-create-info \
--ignore-table=<database>.south_migrationhistory \
--ignore-table=<database>.auth_user \
--ignore-table=<database>.auth_user_groups \
--ignore-table=<database>.auth_user_user_permissions \
--ignore-table=<database>.auth_permission \
--ignore-table=<database>.django_content_types \
--ignore-table=<database>.django_site \
> ~/data-backup.sql
./manage.py sqlclear
./manage.py syncdb --migrate
mysqldump <database> > ~/empty-database.sql
./manage.py dbshell < ~/data-backup.sql
(edit ~/data-backup.sql to remove data dumped from a table that no longer exists)
./manage.py dbshell < ~/empty-database.sql
./manage.py dbshell < ~/data-backup.sql
(repeat until clean)
Previously I had a django model like this
class Review(models.Model):
reviewdate=models.DateField(default=date.today)
description=models.TextField()
author=models.ForeignKey(User,null=True)
I have some 500 records of Review in db.
I added a field to model
from django.core.validators import MinValueValidator,MaxValueValidator
class Review(models.Model):
reviewdate=models.DateField(default=date.today)
description=models.TextField()
author=models.ForeignKey(User,null=True)
rating= models.IntegerField(validators=[MinValueValidator(1), MaxValueValidator(10)], default=5, help_text='integers 1 to 10')
I ran python manage.py schemamigration myapp --auto successfully ,which created a 0002_auto__add_field_review_rating.py file
Now, I need to do the datamigration for the existing records in db. Do I have to run
python manage.py datamigration myapp somechanges
and then implement the functions in the created somechanges.py ? Since I have already defined in the new field difficulty a default value of 5, will that not be taken when migrate command is run? Do I have to explicitly set it in the somechanges.py functions?
You do not need to do a data migration for this case. As you have specified a default for your new field, that will be used when you apply the schema migration.
Use the following command to apply the migration:
./manage.py migrate myapp
See the advanced changes South tutorial for more information on default values for new fields.
I have a Django project for a simple blog/forum website I’m building.
I’m using the syndication feed framework, which seems to generate the URLs for items in the feed using the domain of the current site from the Sites framework.
I was previously unaware of the Sites framework. My project isn’t going to be used for multiple sites, just one.
What I want to do is set the domain property of the current site. Where in my Django project should I do that? Somewhere in /settings.py?
If I understand correctly, Sites framework data is stored in the database, so if I want to store this permanently, I guess it’s appropriate in an initial_data fixture.
I fired up the Django shell, and did the following:
>>> from django.contrib.sites.models import Site
>>> one = Site.objects.all()[0]
>>> one.domain = 'myveryspecialdomain.com'
>>> one.name = 'My Special Site Name'
>>> one.save()
I then grabbed just this data at the command line:
python manage.py dumpdata sites
And pasted it into my pre-existing initial_data fixture.
The other answers suggest to manually update the site in the admin, shell, or your DB. That's a bad idea—it should be automatic.
You can create a migration that'll do this automatically when you run your migrations, so you can be assured it's always applied (such as when you deploy to production). This is also recommended in the documentation, but it doesn't list instructions.
First, run ./manage.py makemigrations --empty --name UPDATE_SITE_NAME myapp to create an empty migration. Then add the following code:
from django.db import migrations
from django.conf import settings
def update_site_name(apps, schema_editor):
SiteModel = apps.get_model('sites', 'Site')
domain = 'mydomain.com'
SiteModel.objects.update_or_create(
pk=settings.SITE_ID,
defaults={'domain': domain,
'name': domain}
)
class Migration(migrations.Migration):
dependencies = [
# Make sure the dependency that was here by default is also included here
('sites', '0002_alter_domain_unique'), # Required to reference `sites` in `apps.get_model()`
]
operations = [
migrations.RunPython(update_site_name),
]
Make sure you've set SITE_ID in your settings. Then run ./manage.py migrate to apply the changes :)
You can change it using django admin site.
Just go to 127.0.0.1:8000/admin/sites/
For those who are struggling to find "Sites" section on Django's admin page, for newer Django versions you need to enable the optional Sites Framework like so:
On your settings.py file, add this to your "INSTALLED_APPS":
'django.contrib.sites'
Then specify an ID for the default site (since it's probably your first site to be specified, you can use ID 1):
SITE_ID = 1
Run your migrations and check if the "Sites" section is available on your Django's admin page.
More details at https://docs.djangoproject.com/en/3.0/ref/contrib/sites/#enabling-the-sites-framework
You can modify the Site entry in your database manually. Navigate to the table called 'django_site'. Then, you should only see one entry (row). You'll want to modify the field (column) named 'domain'.
so i'm trying to do a data migration where i take the "listings" from a realestate app into a new "listings" app that i've created.
i did startmigration like this:
python manage.py startmigration listings migrate_listings --freeze realestate
created a blank migration, which i populated with this:
def forwards(self, orm):
"Write your forwards migration here"
for listing in orm['realestate.RealEstateListing'].objects.all():
sub_type = orm.SubType.objects.get(slug_url=slugify(listing.listing_type.name))
lt = orm.Listing(listing_type=sub_type.parent,
sub_type=sub_type,
expiration_date=listing.expiration_date,
title=listing.title,
slug_url = listing.slug_url,
description = listing.description,
contact_person=listing.contact_person,
secondary_contact=listing.secondary_contact,
address=listing.address,
location=listing.location,
price=listing.price,
pricing_option=listing.pricing_option,
display_picture=listing.display_picture,
image_gallery=listing.image_gallery,
date_added=listing.date_added,
status=listing.status,
featured_on_homepage=listing.featured_on_homepage,
)
lt.save()
lt.features.clear()
for ft in listing.property_features.all:
lt.features.add(ft)
for cft in listing.community_features.all:
lt.features.add(cft)
lt.restrictions.clear()
for na in listing.not_allowed.all:
lt.restrictions.add(na)
however when i run the migration is still get this error:
whiney_method
ValueError("you cannot instantiate a stub model")
from what i understand you can't access a "stub" model using the fakeorm but freezing additional apps is not allowed. how do i go about using the "stub" models without freezing them?
ok so i'm answering my own question, since apparantly i'm the only django south user here. i had to figure it out by myself.
what i wasn't doing, was freezing all the apps that were required in the above migration. since i didn't freeze it created the stub models.
the proper syntax for freezing multiple apps is:
python manage.py startmigration listings migrate_listings --freeze realestate --freeze logistics --freeze media --freeze upload
and everything works after that!