Previously I had a django model like this
class Review(models.Model):
reviewdate=models.DateField(default=date.today)
description=models.TextField()
author=models.ForeignKey(User,null=True)
I have some 500 records of Review in db.
I added a field to model
from django.core.validators import MinValueValidator,MaxValueValidator
class Review(models.Model):
reviewdate=models.DateField(default=date.today)
description=models.TextField()
author=models.ForeignKey(User,null=True)
rating= models.IntegerField(validators=[MinValueValidator(1), MaxValueValidator(10)], default=5, help_text='integers 1 to 10')
I ran python manage.py schemamigration myapp --auto successfully ,which created a 0002_auto__add_field_review_rating.py file
Now, I need to do the datamigration for the existing records in db. Do I have to run
python manage.py datamigration myapp somechanges
and then implement the functions in the created somechanges.py ? Since I have already defined in the new field difficulty a default value of 5, will that not be taken when migrate command is run? Do I have to explicitly set it in the somechanges.py functions?
You do not need to do a data migration for this case. As you have specified a default for your new field, that will be used when you apply the schema migration.
Use the following command to apply the migration:
./manage.py migrate myapp
See the advanced changes South tutorial for more information on default values for new fields.
Related
I am following these two references (one and two) to have a custom user model in order to authenticate via email and also to add an extra field to it.
class User(AbstractBaseUser, PermissionsMixin):
email = models.EmailField(
unique=True,
max_length=254,
)
mobile_number = models.IntegerField(unique=True)
is_active = models.BooleanField(default=True)
is_admin = models.BooleanField(default=False)
objects = UserManager()
...
...
class Meta:
db_table = 'auth_user'
...
...
As you can see, I have added the db_table='auth_user' into the Meta fields of the class. Also, I have included AUTH_USER_MODEL = 'accounts.User' and User model app (i.e., accounts)into the INSTALLED_APPS in settings.py. Further more, I deleted the migrations folder from the app.
Then tried migrating:
$ python manage.py makemigrations accounts
Migrations for 'accounts':
accounts/migrations/0001_initial.py:
- Create model User
$ python manage.py migrate accounts
Which gives me an error:
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration admin.0001_initial is applied before its dependency accounts.0001_initial on database 'default'.
How can I migrate from the existing django user model into a custom user model?
You have to clear admin, auth, contenttypes, and sessions from the migration history and also drop the tables. First, remove the migration folders of your apps and then type the following:
python manage.py migrate admin zero
python manage.py migrate auth zero
python manage.py migrate contenttypes zero
python manage.py migrate sessions zero
Afterwards, you can run makemigrations accounts and migrate accounts.
The solution is to undo your existing migrations that depend on AUTH_USER_MODEL as mentioned in this answer. In case you are trying to undo migrations for admin, auth, contenttypes and sessions and you get an error like:
ERRORS:
auth.User.groups: (fields.E304) Reverse accessor for 'User.groups' clashes with reverse accessor for 'Profile.groups'.
....
First of all comment out/undo AUTH_USER_MODEL in settings.py if you had changed that.
Secondly comment out/undo your django app that contains new AUTH_MODEL from INSTALLED_APPS in settings.py.
Now you should be able to undo migrations for auth, admin, contenttypes and sessions as:
python manage.py migrate admin zero
python manage.py migrate auth zero
python manage.py migrate contenttypes zero
python manage.py migrate sessions zero
Now add your auth model app to INSTALLED_APPS and set AUTH_USER_MODEL in your settings.py again.
Run: python manage.py migrate AUTH_APP, you may need to make migrations for your auth model app as well: python manage.py makemigrations AUTH_APP
Apply all migrations that you undo by: python manage.py migrate.
You are all done.
Note: You will lose all existing users present in database.
As in my particular case, the other answers did not help (the error still occured even after I tried to drop the tables with migrate ... zero and even after I deleted the migrations folder), the following helped, but I was at the very beginning and therefore it was no problem to just delete the db.sqlite3 file which is created whenever you migrate the first time. (Depending on your settings.py you might have a different database-file).
You really can only do this if you are sure that you don't lose important data from your database file (e.g. you do not yet have much information stored in the database and it is not difficult to start over again), and you will need to migrate everything again.
Delete the existing all the tables from data base.[Note : data will be lost]
Delete pycache and migrations from all the apps.
Run migrations for your relative app
python manage.py makemigrations users
Migrate the tables to database
python manage.py migrate
You need to run:
python manage.py makemigrations accounts
Before executing the initial manage.py migrate (by initial I mean at the very first time you run migrate on your project)
it is recommended to set up your custom User model at the start of your project so you'll have the "accounts" app migrated at the same time as the admin,auth,contenttypes,sessions tables are created.
but if you have created your tables already, then you should follow the instructions as #krishna-chandak described: https://stackoverflow.com/a/53599345/5950111
you can read the docs : https://docs.djangoproject.com/en/2.0/topics/auth/customizing/#using-a-custom-user-model-when-starting-a-project
There's a django_migrations table in your database after your previous migration which is the cause of this inconsistency.
Solution: Deleting the django_migrations table from your database.
delete the migration folder from your apps
and then perform
python3 manage.py makemigrations
python3 manage.py migrate
I know it's rather an old question, but for people googling this topic like me today, here is a solution without deleting migrations, dropping the tables, and other nasty stuff)
https://www.caktusgroup.com/blog/2019/04/26/how-switch-custom-django-user-model-mid-project/
I also had the same problem. I followed the steps:
In models.py, i setup basic User model
# accounts/models.py
class User(AbstractBaseUser):
class Meta:
db_table = 'auth_user'
Then, i ran makemigrations command to generate migration file
$ python manage.py makemigrations accounts
Migrations for 'accounts':
accounts/migrations/0001_initial.py:
- Create model User
Next step, i inserted record has 0001_initial todjango_migrations table
$ echo "INSERT INTO django_migrations (app, name, applied) VALUES ('accounts', '0001_initial', CURRENT_TIMESTAMP);" | python manage.py dbshell
Update lastest in model
# accounts/models.py
class User(AbstractBaseUser, PermissionsMixin):
email = models.EmailField(
unique=True,
max_length=254,
)
mobile_number = models.IntegerField(unique=True)
is_active = models.BooleanField(default=True)
is_admin = models.BooleanField(default=False)
objects = UserManager()
...
...
class Meta:
db_table = 'auth_user'
...
...
I need makemigrations again
After run makemigrations, i had the next migration file.
0002_....py
Migrate agains
python manage.py migrate.
What worked for me was a solution that I pieced togeather from all the diffretent solutions given here.
I check if the database exists since I don't have an issue with an existing database, only when the database is empty.
# check if the database exists
db_ok=false
if python ./manage.py check; then
db_ok=true
fi
if [ $db_ok = true ]; then
# database exists: do a normal migrate
python ./manage.py migrate
else
# database does not exists, make and migrate users then a migrate and cleanup of the users migraton
python ./manage.py makemigrations users
python ./manage.py migrate users
python ./manage.py migrate
rm -r users/migrations/
fi
I had similar problem, where I have to introduce the custom user model in the middle of the project. So following steps helped me to solve the issue without table drop or data loss.
(1) Create an initial empty migration for the app ('accounts')
python manage.py makemigrations accounts --empty
(2) Run migrate
python manage.py migrate
(3) Update the timestamp in the django_migrations table in the database , for 'accounts' initial migration to the 'admin' initial timestamp.
UPDATE django_migrations SET applied=<<admin 0001_initial date>> WHERE app='accounts' and name='0001_initial';
(4) Now create your Custom User model (derived from AbstractUser) with out any fields, and set the table name to auth_user. You are basically re-using the existing auth user table in database.
class User(AbstractUser):
class Meta:
db_table = 'auth_user'
(4) Now run migration, and copy the migration to 0001_initial and remove '0001_initial' from the dependency array. Also remove the newly created migration file.
python manage.py makemigrations accounts
cp accounts/migrations/0002_user.py accounts/migrations/0001_initial.py
edit the file 0001_initial.py
rm accounts/migrations/0002_user.py (remove migration file)
(5) Now add your custom fields, run makemigrations and migrate as usual.
Migrate zeroing didn't help me. I had to drop the whole database:
sudo -u postgres psql
drop database YOURDATABASENAME;
create database YOURDATABASENAME;
Then:
python ./manage.py makemigrations MYAPPNAME
python ./manage.py migrate MYAPPNAME
python ./manage.py migrate
And after these I got forward..
I have a Django application with a My-SQL database. recently I alter the table_name with the help of MySQL query in the MySQL-shell, after this when I run makemigration and migrate command terminal says "No changes detected". how can i resolve this issue and create again this table with help of Django makemigration and migrate?
can I delete a table from MySQL, any possibility will Django create it again?
If you renamed your table outside Django - you will have to tell Django the new table name like so (using the Meta class):
class Model(models.Model):
name = models.CharField(max_length=255)
class Meta:
db_table = 'new_table_name'
To re-create your table using existing model you need to reset migration for that app to zero and then run migration again
python manage.py migrate APP_NAME zero
python manage.py migrate APP_NAME
It's because the migrations table managed by django doesn't reflect the correct db schema since it was already modified outside of django. If you don't have any important data you can do a migration rollback or recreate the table by hand.
The best way to dela with this is to rename your table back to the original name. Then create a blank migration inside your app and recreate the sql commands you did in the shell inside that migration file. That way django can keep track of the database schema.
You should change the name of the table in models.py not in MySQL shell.
From
class MyModel(models.Model):
...
To
class ThisModel(models.Model):
...
Or Create Proxy Model :
class ThisModel(MyModel):
class Meta:
proxy = True
verbose_name = "ThisModel"
When I use manage.py makemigrations <app> only columns with relations are migrated to pg database.
How to instruct django to migrate new basic non-relational columns, like:
title = models.CharField(max_length=20, null=True)
I'm using django 1.8.2. on Ubuntu
No fields at all are being migrated when you run manage.py makemigrations <app>, as that command only creates a new migration in your <app>/migrations/ directory. It's only when you run manage.py migrate that changes are written to the database.
If you are not getting the expected results, have a look at the newest migration and determine which fields it affects. All fields should be targeted by migrations, not relational fields only. Changing properties like max_length, blank, required etc. should trigger migrations. If they don't it's probably because your changes doesn't require any database schema modification.
If you are still having problems, please post:
Your models prior to model change
Your models after model change
The migration generated by makemigrations
our website lists local city events. It's Django-based so there's a lot of code related to 'event' model. Until now, we work only in one city and so all the events mean to be local.
Now we need to extend the website to another city. This means 'event' model gets a new attribute 'city', and our middleware will set a global value CurrentCity based on geoip.
We need to extend 'event' model so it would filter only records where 'city' attribute equals to CurrentCity value. There is too much code in different views and models working with the 'event' so we can't update each module.
Is there any single place to patch that would make our 'event' model aware of the CurrentCity value?
Depending a lot in your structure and Django version I think you have 2 options.
South
The best one is to install the application "South". I don't know if you already know it or you're using it but I think it should be your first option.
In case you're not using it, you should do this steps:
Install with pip install South
Create your first migration with:
python manage.py schemamigration YOURAPPNAME --initial
You need to fake this migration, because you have already the fullfilled database so you need to do:
python manage.py migrate YOURAPPNAME --fake
Add the new field to the model Event in the file models.py
Generate the new migration to make South create the new field in your database with:
python manage.py schemamigration YOURAPPNAME --auto
Final step, execute the migration created with:
python manage.py migrate YOURAPPNAME
Tips
--initial for the first migration --auto for the rest
The initial migration is faked because you already have tables in your database, if you try to migrate without the fake it will return error "Table already exists"
New Model City
Another option, in case you can't modify your actual Model, or maybe if it's too messy, another option is to generate an externa Model City like this:
class City(models.Model):
event_foreign = models.ForeignKey(Event)
event_many = models.ManyToManyField(Event, blank=True, null=True)
name = models.CharField....
postal_code = models.CharField....
# etc...
I don't know wich is optimal for you, a Foreign Key or a ManyToMany, depends if a City can have more than 1 Event or no, it's your choice.
When you have a model like this you can access from this City model to the Event (because of the ForeignKEy or ManyToMany) but this relation goes also in the other direction, if you have an Event you can get the City/cities related to it I'm gonna show two examples:
Example 1 using Foreign Key
city = City.objects.get(id=1)
city.event # Returns event
event = Event.objects.get(id=1)
event.city # Returns city
Example 2 using Many to Many
city = City.objects.get(id=1)
city.event.all() # Returns a list of events
event = Event.objects.get(id=1)
event.city_set.all() # Returns a list of cities
I'd prefer not to destroy all the users on my site. But I want to take advantage of Django 1.5's custom pluggable user model. Here's my new user model:
class SiteUser(AbstractUser):
site = models.ForeignKey(Site, null=True)
Everything works with my new model on a new install (I've got other code, along with a good reason for doing this--all of which are irrelevant here). But if I put this on my live site and syncdb & migrate, I'll lose all my users or at least they'll be in a different, orphaned table than the new table created for my new model.
I'm familiar with South, but based on this post and some trials on my part, it seems its data migrations are not currently a fit for this specific migration. So I'm looking for some way to either make South work for this or for some non-South migration (raw SQL, dumpdata/loaddata, or otherwise) that I can run on each of my servers (Postgres 9.2) to migrate the users once the new table has been created while the old auth.User table is still in the database.
South is more than able to do this migration for you, but you need to be smart and do it in stages. Here's the step-by-step guide: (This guide presupposed you subclass AbstractUser, not AbstractBaseUser)
Before making the switch, make sure that south support is enabled in the application
that contains your custom user model (for the sake of the guide, we'll call it accounts and the model User).
At this point you should not yet have a custom user model.
$ ./manage.py schemamigration accounts --initial
Creating migrations directory at 'accounts/migrations'...
Creating __init__.py in 'accounts/migrations'...
Created 0001_initial.py.
$ ./manage.py migrate accounts [--fake if you've already syncdb'd this app]
Running migrations for accounts:
- Migrating forwards to 0001_initial.
> accounts:0001_initial
- Loading initial data for accounts.
Create a new, blank user migration in the accounts app.
$ ./manage.py schemamigration accounts --empty switch_to_custom_user
Created 0002_switch_to_custom_user.py.
Create your custom User model in the accounts app, but make sure it is defined as:
class SiteUser(AbstractUser): pass
Fill in the blank migration with the following code.
# encoding: utf-8
from south.db import db
from south.v2 import SchemaMigration
class Migration(SchemaMigration):
def forwards(self, orm):
# Fill in the destination name with the table name of your model
db.rename_table('auth_user', 'accounts_user')
db.rename_table('auth_user_groups', 'accounts_user_groups')
db.rename_table('auth_user_user_permissions', 'accounts_user_user_permissions')
def backwards(self, orm):
db.rename_table('accounts_user', 'auth_user')
db.rename_table('accounts_user_groups', 'auth_user_groups')
db.rename_table('accounts_user_user_permissions', 'auth_user_user_permissions')
models = { ....... } # Leave this alone
Run the migration
$ ./manage.py migrate accounts
- Migrating forwards to 0002_switch_to_custom_user.
> accounts:0002_switch_to_custom_user
- Loading initial data for accounts.
Make any changes to your user model now.
# settings.py
AUTH_USER_MODEL = 'accounts.User'
# accounts/models.py
class SiteUser(AbstractUser):
site = models.ForeignKey(Site, null=True)
create and run migrations for this change
$ ./manage.py schemamigration accounts --auto
+ Added field site on accounts.User
Created 0003_auto__add_field_user_site.py.
$ ./manage.py migrate accounts
- Migrating forwards to 0003_auto__add_field_user_site.
> accounts:0003_auto__add_field_user_site
- Loading initial data for accounts.
Honestly, If you already have good knowledge of your setup and already use south, It should be as simple as adding the following migration to your accounts module.
# encoding: utf-8
from south.db import db
from south.v2 import SchemaMigration
from django.db import models
class Migration(SchemaMigration):
def forwards(self, orm):
# Fill in the destination name with the table name of your model
db.rename_table('auth_user', 'accounts_user')
db.rename_table('auth_user_groups', 'accounts_user_groups')
db.rename_table('auth_user_permissions', 'accounts_user_permissions')
# == YOUR CUSTOM COLUMNS ==
db.add_column('accounts_user', 'site_id',
models.ForeignKey(orm['sites.Site'], null=True, blank=False)))
def backwards(self, orm):
db.rename_table('accounts_user', 'auth_user')
db.rename_table('accounts_user_groups', 'auth_user_groups')
db.rename_table('accounts_user_user_permissions', 'auth_user_user_permissions')
# == YOUR CUSTOM COLUMNS ==
db.remove_column('accounts_user', 'site_id')
models = { ....... } # Leave this alone
EDIT 2/5/13: added rename for auth_user_group table. FKs will auto update to point at the correct table due to db constraints, but M2M fields' table names are generated from the names of the 2 end tables and will need manual updating in this manner.
EDIT 2: Thanks to #Tuttle & #pix0r for the corrections.
My incredibly lazy way of doing this:
Create a new model (User), extending AbstractUser. Within new model, in it's Meta, override db_table and set to 'auth_user'.
Create an initial migration using South.
Migrate, but fake the migration, using --fake when running migrate.
Add new fields, create migration, run it normally.
This is beyond lazy, but works. You now have a 1.5 compliant User model, which just uses the old table of users. You also have a proper migration history.
You can fix this later on with manual migrations to rename the table.
I think you've correctly identified that a migration framework like South is the right way to go here. Assuming you're using South, you should be able to use the Data Migrations functionality to port the old users to your new model.
Specifically, I would add a forwards method to copy all rows in your user table to the new table. Something along the lines of:
def forwards(self, orm):
for user in orm.User.objects.all():
new_user = SiteUser(<initialize your properties here>)
new_user.save()
You could also use the bulk_create method to speed things up.
I got tired of struggling with South so I actually ended up doing this differently and it worked out nicely for my particular situation:
First, I made it work with ./manage.py dumpdata, fixing up the dump, and then ./manage.py loaddata, which worked. Then I realized I could do basically the same thing with a single, self-contained script that only loads necessary django settings and does the serialization/deserialization directly.
Self-contained python script
## userconverter.py ##
import json
from django.conf import settings
settings.configure(
DATABASES={
# copy DATABASES configuration from your settings file here, or import it directly from your settings file (but not from django.conf.settings) or use dj_database_url
},
SITE_ID = 1, # because my custom user implicates contrib.sites (which is why it's in INSTALLED_APPS too)
INSTALLED_APPS = ['django.contrib.sites', 'django.contrib.auth', 'myapp'])
# some things you have to import after you configure the settings
from django.core import serializers
from django.contrib.auth.models import User
# this isn't optimized for huge amounts of data -- use streaming techniques rather than loads/dumps if that is your case
old_users = json.loads(serializers.serialize('json', User.objects.all()))
for user in old_users:
user['pk'] = None
user['model'] = "myapp.siteuser"
user['fields']["site"] = settings['SITE_ID']
for new_user in serializers.deserialize('json', json.dumps(old_users)):
new_user.save()
With dumpdata/loaddata
I did the following:
1) ./manage.py dumpdata auth.User
2) Script to convert auth.user data to new user. (or just manually search and replace in your favorite text editor or grep) Mine looked something like:
def convert_user_dump(filename, site_id):
file = open(filename, 'r')
contents = file.read()
file.close()
user_list = json.loads(contents)
for user in user_list:
user['pk'] = None # it will auto-increment
user['model'] = "myapp.siteuser"
user['fields']["site"] = side_id
contents = json.dumps(user_list)
file = open(filename, 'w')
file.write(contents)
file.close()
3) ./manage.py loaddata filename
4) set AUTH_USER_MODEL
*Side Note: One critical part of doing this type of migration, regardless of which technique you use (South, serialization/modification/deserialization, or otherwise) is that as soon as you set AUTH_USER_MODEL to your custom model in the current settings, django cuts you off from auth.User, even if the table still exists.*
We decided to switch to a custom user model in our Django 1.6/Django-CMS 3 project, perhaps a little bit late because we had data in our database that we didn't want to lose (some CMS pages, etc).
After we switched AUTH_USER_MODEL to our custom model, we had a lot of problems that we hadn't anticipated, because a lot of other tables had foreign keys to the old auth_user table, which wasn't deleted. So although things appeared to work on the surface, a lot of things broke underneath: publishing pages, adding images to pages, adding users, etc. because they tried to create an entry in a table that still had a foreign key to auth_user, without actually inserting a matching record into auth_user.
We found a quick and dirty way to rebuild all the tables and relations, and copy our old data across (except for users):
do a full backup of your database with mysqldump
do another backup with no CREATE TABLE statements, and excluding a few tables that won't exist after the rebuild, or will be populated by syncdb --migrate on a fresh database:
south_migrationhistory
auth_user
auth_user_groups
auth_user_user_permissions
auth_permission
django_content_types
django_site
any other tables that belong to apps that you removed from your project (you might only find this out by experimenting)
drop the database
recreate the database (e.g. manage.py syncdb --migrate)
create a dump of the empty database (to make it faster to go round this loop again)
attempt to load the data dump that you created above
if it fails to load because of a duplicate primary key or a missing table, then:
edit the dump with a text editor
remove the statements that lock, dump and unlock that table
reload the empty database dump
try to load the data dump again
repeat until the data dump loads without errors
The commands that we ran (for MySQL) were:
mysqldump <database> > ~/full-backup.sql
mysqldump <database> \
--no-create-info \
--ignore-table=<database>.south_migrationhistory \
--ignore-table=<database>.auth_user \
--ignore-table=<database>.auth_user_groups \
--ignore-table=<database>.auth_user_user_permissions \
--ignore-table=<database>.auth_permission \
--ignore-table=<database>.django_content_types \
--ignore-table=<database>.django_site \
> ~/data-backup.sql
./manage.py sqlclear
./manage.py syncdb --migrate
mysqldump <database> > ~/empty-database.sql
./manage.py dbshell < ~/data-backup.sql
(edit ~/data-backup.sql to remove data dumped from a table that no longer exists)
./manage.py dbshell < ~/empty-database.sql
./manage.py dbshell < ~/data-backup.sql
(repeat until clean)