I am trying to change the field type of one of attributes from CharField to DecimalField by doing an empty migrations and populate the new field by filling the migrations log with the following:
from __future__ import unicode_literals
from django.db import migrations
from decimal import Decimal
def populate_new_col(apps, schema_editor): #Plug data from 'LastPrice' into 'LastPrice_v1' in the same model class 'all_ks'.
all_ks = apps.get_model('blog', 'all_ks')
for ks in all_ks.objects.all():
if float(ks.LastPrice): #Check if conversion to float type is possible...
print ks.LastPrice
ks.LastPrice_v1, created = all_ks.objects.get_or_create(LastPrice_v1=Decimal(float(ks.LastPrice)*1.0))
else: #...else insert None.
ks.LastPrice_v1, created = all_ks.objects.get_or_create(LastPrice_v1=None)
ks.save()
class Migration(migrations.Migration):
dependencies = [
('blog', '0027_auto_20190301_1600'),
]
operations = [
migrations.RunPython(populate_new_col),
]
But I kept getting an error when I tried to migrate:
TypeError: Tried to update field blog.All_ks.LastPrice_v1 with a model instance, <All_ks: All_ks object>. Use a value compatible with DecimalField.
Is there something I missed converting string to Decimal?
FYI, ‘LastPrice’ is the old attribute with CharField, and ‘LastPrice_v1’ is the new attribute with DecimalField.
all_ks.objects.get_or_create() returns an All_ks object which you assign to the DecimalField LastPrice_v1. So obviously Django complains. Why don't you assign the same ks's LastPrice?
ks.LastPrice_v1 = float(ks.LastPrice)
That said, fiddling around with manual migrations seems a lot of trouble for what you want to achieve (unless you're very familiar with migration code). If you're not, you're usually better off
creating the new field in code
migrating
populating the new field
renaming the old field
renaming the new field to the original name
removing the old field
migrating again
All steps are vanilla Django operations, with the bonus that you can revert until the very last step (nice to have when things can take unexpected turns as you've just experienced).
Related
When using models in migrations, in Django we can use apps.get_model() to make sure that the migration will use the right "historical" version of the model (as it was when the migration was defined).
But how do we deal with "regular" variables (not models) imported from the codebase?
If we import a variable from another module, we will probably face issues in the future. For example:
If someday in the future we delete the variable (because we changed the implementation) this will break migrations. So we won't be able to re-run migrations locally to re-create a database from scratch.
If we modify the variable (e.g. we change the values in a list) this will produce unexpected effects when we run the reverse operation on an existing db.
So the question is: what's the best practice for writing migrations? Should we always hard-code values without importing external variables?
Example
Suppose I want to simply modify the value of a field with a variable that I've defined somewhere in the codebase. For example, I want to turn all normal users into admins. I stored user roles in an enum (UserRoles). One way to write the migration would be this:
from django.db import migrations
from user_roles import UserRoles
def change_user_role(apps, schema_editor):
User = apps.get_model('users', 'User')
users = User.objects.filter(role=UserRoles.NORMAL_USER.value)
for user in users:
user.role = UserRoles.ADMIN.value
User.objects.bulk_update(users, ["role"])
def revert_user_role_changes(apps, schema_editor):
User = apps.get_model('users', 'User')
users = User.objects.filter(role=UserRoles.ADMIN.value)
for user in users:
user.role = UserRoles.NORMAL_USER.value
User.objects.bulk_update(users, ["role"])
class Migration(migrations.Migration):
dependencies = [
('users', '0015_auto_20220612_0824'),
]
operations = [
migrations.RunPython(change_user_role, revert_user_role_changes)
]
As you see, this will have the issues I mentioned above.
I've used the enum example, but this can be applied to every variable referenced inside the migrations.
So the question again: what's the best practice for migrations? Should we always hard-code values without referencing external variables that might change?
I am adding a new field to an existing db table. it is to be auto-generated with strings.
Here is my code:
from django.utils.crypto import get_random_string
...
Model:
verification_token = models.CharField(max_length=60, null=False, blank=False, default=get_random_string)
I generate my migration file with ./manage.py makemigrations and a file is generated.
I verify the new file has default set to field=models.CharField(default=django.utils.crypto.get_random_string, max_length=60)
so all seems fine.
Proceed with ./manage.py migrate it goes with no error from terminal.
However when i check my table i see all the token fields are filled with identical values.
Is this something i am doing wrong?
How can i fix this within migrations?
When a new column is added to a table, and the column is NOT NULL, each entry in the column must be filled with a valid value during the creation of the column. Django does this by adding a DEFAULT clause to the column definition. Since this is a single default value for the whole column, your function will only be called once.
You can populate the column with unique values using a data migration. The procedure for a slightly different use-case is described in the documentation, but the basics of the data migrations are as follows:
from django.db import migrations, models
from django.utils.crypto import get_random_string
def generate_verification_token(apps, schema_editor):
MyModel = apps.get_model('myapp', 'MyModel')
for row in MyModel.objects.all():
row.verification_token = get_random_string()
row.save()
class Migration(migrations.Migration):
dependencies = [
('myapp', '0004_add_verification_token_field'),
]
operations = [
# omit reverse_code=... if you don't want the migration to be reversible.
migrations.RunPython(generate_verification_token, reverse_code=migrations.RunPython.noop),
]
Just add this in a new migration file, change the apps.get_model() call and change the dependencies to point to the previous migration in the app.
It maybe the token string to sort, so django will save some duplicates values. But, i'm not sure it is your main problem.
Anyway, I suggest you to handle duplicates values using while, then filter your model by generated token, makesure that token isn't used yet. I'll give you exampe such as below..
from django.utils.crypto import get_random_string
def generate_token():
token = get_random_string()
number = 2
while YourModel.objects.filter(verification_token=token).exists():
token = '%s-%d' % (token, number)
number += 1
return token
in your field of verification_token;
verification_token = models.CharField(max_length=60, unique=True, default=generate_token)
I also suggest you to using unique=True to handle duplicated values.
Consider this structure:
some_table(id: small int)
and I want change it to this:
some_table(id: string)
Now I do this with three migrations:
Create a new column _id with type string
(datamigration) Copy data from id to _id with string conversion
Remove id and rename _id to id
Is there a way to do this with only one migration?
You can directly change the type of a column from int to string. Note that, unless strict sql mode is enabled, integers will be truncated to the maximum string length and data is possibly lost, so always make a backup and choose a max_length that's high enough. Also, the migration can't easily be reversed (sql doesn't directly support changing a string column to an int column), so a backup is really important in this one.
Django pre 1.7 / South
You can use db.alter_column. First, create a migration, but don't apply it yet, or you'll lose the data:
>>> python manage.py schemamigration my_app --auto
Then, change the forwards method into this:
class Migration(SchemaMigration):
def forwards(self, orm):
db.alter_column('some_table', 'id', models.CharField(max_length=255))
def backwards(self, orm):
raise RuntimeError('Cannot reverse this migration.')
This will alter the column to match the new CharField field. Now apply the migration and you're done.
Django 1.7
You can use the AlterField operation to change the column. First, create an empty migration:
>>> python manage.py makemigrations --empty my_app
Then, add the following operation:
class Migration(migrations.Migration):
operations = [
migrations.AlterField('some_model', 'id', models.CharField(max_length=255))
]
Now run the migration, and Django will alter the field to match the new CharField.
I'm writing a Django Model that will link to another app Model. I know that we should link the ForeginKeys with "nameoftheapp.nameofthemodel" but I'm not successful doing it with this use case.
Here is my installed_apps:
INSTALLED_APPS = (
...
'signup',
'paypal.standard.ipn',
...
)
Basically I'm creating a model in the app "signup" and I need to do a foreignkey to "paypal.standard.ipn".
Here is my model:
class SignupPaymentPayPal(models.Model):
gettingstarted = models.ForeignKey('GettingStarted')
paypalipn = models.ForeignKey('paypal.standard.ipn.PayPalIPN')
The model I need to link is this one, https://github.com/spookylukey/django-paypal/blob/master/paypal/standard/ipn/models.py
When I try to do a shemamigration I got this:
$ python manage.py schemamigration signup --auto
Here is the error I got:
CommandError: One or more models did not validate:
signup.signuppaymentpaypal: 'paypalipn' has a relation with model paypal.standard.ipn.PayPalIPN, which has either not been installed or is abstract.
Any clues on what I'm doing wrong?
Best Regards,
from paypal.standard.ipn.models import PayPalIPN
class SignupPaymentPayPal(models.Model):
gettingstarted = models.ForeignKey('GettingStarted')
paypalipn = models.ForeignKey(PayPalIPN)
André solution works, and in fact I'd recommend using the actual model instead of a string whenever possible, to avoid any unexpected errors when the string can't be resolved to a model, but here's an explanation why your previous method didn't work:
Django has a model cache that keeps track of all Model subclasses that are created. Each model is uniquely identified by an appname and modelname, but not by a fully qualified import path.
Let's say I have two models, one is myapp.reviews.Review, the other is myapp.jobs.reviews.Review. If myapp.reviews.Review comes first in my INSTALLED_APPS, both classes will actually be the myapp.reviews.Review model:
>>> from myapp.jobs.reviews import Review
>>> Review
<class 'myapp.reviews.Review'>
To specify a ForeignKey using a string instead of a class (e.g. to avoid circular imports), you need to follow the '<appname>.<modelname>' format, i.e.:
class SignupPaymentPayPal(models.Model):
paypalipn = models.ForeignKey('ipn.PayPalIPN')
I have a bit of a problem using django-registration and signals.
The basic setup is that I have a django 1.4.3 setup, with django-south and django-registration (and the db is SQLite for what it's worth).
EDIT: I changed the question a bit because the effect is the same in a shell, so the registration is not in cause (edits are in italic).
I have a one of my model that is related to the User model in the following way:
class MyUserProfile(models.Model):
user = models.OneToOneFiled(User)
#additional fields
I initialized the base using south.
When I do a little sqlall to check the sql that should be in it, I can clearly see:
CREATE TABLE "myApp_myuserprofile" (
"id" integer NOT NULL PRIMARY KEY
"user_id" integer NOT NULL UNIQUE REFERENCES "auth_user" ("id"),
#other fields
)
After that, I wanted to initialize the data if the user activated its account.
So in models.py I put
from django.dispatch import receiver
from registration.signals import user_activated
#Models....
#receiver(user_activated)
def createMyProfile(sender, **kwargs):
currentUser = kwargs['user']
profile = Profile(user = currentUser, #other fields default value)
profile.save()
#And now the reverse relation:
currentUser.myuserprofile = profile
currentUser.save()
While I am in there, everything seems alright, if I print the ids (both for the user and the profile), and if I travel back and forth between the 2, I see something that seems correct.
If I disable this part of the code and do the same kind of initialization using the shell, I get the same result.
But after that, everyhting is wrong.
If I open a shell and import the relevant matters, I have the following for every X value
MyUserProfile.objects.get(pk=X)
#DoesNotExist Exception
User.objects.get(pk=X).myuserprofile.pk
1
MyUserProfile.objects.all()[X].pk
1
Seems a bit weird no?
And now if I go to the sql shell
select id from myApp_myuserprofile;
1
1
1
1
...
So I have a primary column which is filled with the same value all over the place. Which is well... embarrassing to say the least (and does lead to problem, because everyone has a profile with the same Id).
Any idea to what could be the cause of the problem and how I could solve it?
P.S: Note that the foreign key from the related relation are correct and their uniqueness is preserved.
Well looks like the problem was indeed coming from the use of SQLite and South.
The doc states that:
SQLite doesn’t natively support much schema altering at all, but South has workarounds to allow deletion/altering of columns. Unique indexes are still unsupported, however; South will silently ignore any such commands.
Which was (I think) the case, as I hadn't created this relation from the start but with a latter migration. I just reseted the base and migrations and voilà.
See What's the recommended approach to resetting migration history using Django South? for the migrations and a simple ./manage.py reset myApp for the base reset.