Django: convert CharField primary key to IntegerField - django

I have a model (devices) with a CharField primary key containing integers that I would like to convert to an IntegerField. The problem is the devices are also used as a foreign key in another tables meaning when I change the field in my models.py and run the makemigrations/migrate I keep getting an error prompting me to drop the dependent objects first. How can I change the device primary key to an IntegerField without dropping the dependent objects?
models.py
class Device(models.Model):
id = models.CharField(primary_key=True, unique=True, max_length=255)
date_created = models.DateField()
class Station(models.Model):
id = models.AutoField(primary_key=True)
device = models.ForeignKey('Device', on_delete=models.PROTECT)
error when running the migration
constraint api_station_device_id_117642ec_fk_api_device_id on table api_station depends on index api_device_pkey
HINT: Use DROP ... CASCADE to drop the dependent objects too.

The error message you are receiving is due to the fact that changing the primary key type of the Device model will require changing the foreign key type of the device field in the Station model as well. This will create a dependency that requires the api_station_device_id_117642ec_fk_api_device_id constraint on the api_station table to be dropped, which is not allowed by the database without first dropping the dependent objects.
You can create a new id field of type IntegerField in the Device model and set it as the primary key. You can then populate this new field with the integer values of the old id field and remove the old id field after updating the Station model to reference the new id field. eg in your models.py file
class Device(models.Model):
id = models.AutoField(primary_key=True)
id_char = models.CharField(unique=True, max_length=255)
date_created = models.DateField()
def save(self, *args, **kwargs):
# set the integer value of the new id field to the integer value of the old id field
if self.id_char is not None:
self.id = int(self.id_char)
super().save(*args, **kwargs)
class Station(models.Model):
id = models.AutoField(primary_key=True)
device = models.ForeignKey('Device', on_delete=models.PROTECT, to_field='id')
Write a script to populate the id field with the integer values of
the id_char field for all existing Device objects.
Here are the steps to create a custom Django management command that runs the script:
Create a new Python module (e.g., update_device_ids.py) in a Django
app that has the Device model. You can put this module in the
management/commands/ directory within your app to define a new
management command.
Import the relevant Django models and the BaseCommand class from the
django.core.management.base module. Your module may look like this
from django.core.management.base import BaseCommand
from myapp.models import Device
class Command(BaseCommand):
help = 'Updates the Device id field with the integer value of the id_char field for all devices'
def handle(self, *args, **options):
devices = Device.objects.all()
for device in devices:
if device.id_char is not None:
device.id = int(device.id_char)
device.save()
Then follow the below process:
Update the id field in your Device model from CharField to
IntegerField.
Create a new migration by running python manage.py makemigrations.
Before running the migration, create a new temporary field in your
Device model that will store the integer value of the old id_char
field. For example, you could add the following line to your Device
model:
id_int = models.IntegerField(null=True, blank=True)
Run the python manage.py update_device_ids command to populate the
id_int field with the integer values of the old id_char field.
Create a data migration by running python manage.py makemigrations --empty yourappname.
Edit the data migration file (in the migrations/ directory) and add
the following code to the forwards() method:
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('yourappname', 'the_previous_migration'),
]
operations = [
migrations.AddField(
model_name='device',
name='id_int',
field=models.IntegerField(null=True, blank=True),
),
migrations.RunSQL('UPDATE yourappname_device SET id_int = CAST(id as integer)'),
migrations.RemoveField(
model_name='device',
name='id',
),
migrations.RenameField(
model_name='device',
old_name='id_int',
new_name='id',
),
]
This code adds a new IntegerField field called id_int, populates it with the integer values of the old id_char field using a SQL command, removes the old id_char field, and renames the id_int field to id.
Finally, run the data migration by running python manage.py migrate.

Related

Django - Adding a primary key to an existing model with data

This is the existing model in a djnago app:
class Task_Master_Data(models.Model):
project_id = models.ForeignKey(Project_Master_Data, on_delete=models.CASCADE)
task_created_at = models.DateTimeField(auto_now_add=True)
task_updated_at = models.DateTimeField(auto_now=True)
task_name = models.CharField(max_length=200, null=True)
I want to add a new field (which will be a primary key):
task_id = models.AutoField(primary_key=True, null=False)
When I am making migrations from the terminal, it will provide me with the option of adding a default value, but it will, understandably, bring an error of:
UNIQUE constraint failed: new__main_task_master_data.task_id
What would be the best way forward for this kind of a scenario without having to delete all the data.
Note, I am using the default sqlite3 database.
This is the result of an old bug that is still unfixed. See here for the ticket.
I have created a workaround.
First create an empty migration (change taskapp to the name of the app where your models.py with Task_Master_Data lives):
python manage.py makemigrations taskapp --empty
Then copy the below migrations into this file and run python manage.py migrate but be sure to adjust the name of the app (replace taskapp with the name of your app) and the dependencies (0010_task_master_data should be replaced by the name of the migration that comes before it).
First the existing rows in the database need values for the new task_id field. I solved this by creating an IntegerField with null=True, which I then manually fill using a RunPython command. I then remove the id field (my previous primary key), and use AlterField to change the field into the AutoField. A default value is required, but it should never be used. When you create a new Task_Master_Data the task_id should be auto-incrementing.
# Generated by Django 3.0.4 on 2022-05-11 07:13
from django.db import migrations, models
def fill_taskid(apps, schema_editor):
# be sure to change it to your app here
Task_Master_Data = apps.get_model("taskapp", "Task_Master_Data")
db_alias = schema_editor.connection.alias
tasks = Task_Master_Data.objects.using(db_alias).all()
for i, task in enumerate(tasks):
task.task_id = i
Task_Master_Data.objects.using(db_alias).bulk_update(tasks, ["task_id"])
class Migration(migrations.Migration):
dependencies = [
('taskapp', '0010_task_master_data'),
]
operations = [
migrations.AddField(
model_name='task_master_data',
name='task_id',
field=models.IntegerField(null=True),
),
migrations.RunPython(fill_taskid),
migrations.RemoveField(
model_name='task_master_data',
name='id',
),
migrations.AlterField(
model_name='task_master_data',
name='task_id',
field=models.AutoField(default=-1, primary_key=True, serialize=False),
preserve_default=False,
),
]
Note that depending on the state of your database you might need to rollback some of your migrations. The migration I created should work if your database is in a state where your task_id field does not exist yet.

Making use of the users table, causing an error

In Django (2.x) I have an entry form, the model is here:
from django.db import models
from django.contrib.auth.models import User
from django.conf import settings
class Sample(models.Model):
sample_id = models.AutoField(primary_key=True)
area_easting = models.IntegerField()
area_northing = models.IntegerField()
context_number = models.IntegerField()
sample_number = models.IntegerField()
# taken_by = models.IntegerField()
taken_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete = models.PROTECT)
def __str__(self):
return str(self.sample_id)
class Meta:
db_table = 'samples\".\"sample'
#ordering = ["sample_id"]
managed = False
#verbose_name_plural = "samples"
This works as expected, a list of usernames drops down (while I would like to format - firstname lastname). However, when I return to the main viewing page I see an error.
django.db.utils.ProgrammingError: column sample.taken_by_id does not exist
LINE 1: ...text_number", "samples"."sample"."sample_number", "samples"....
^
HINT: Perhaps you meant to reference the column "sample.taken_by".
Clearly Django is adding the _id to the table name causing the error, I expect because it is a foreign key.
Any ideas how to remedy this behaviour?
You can explicitly set the underlying db column via the db_column attribute:
taken_by = models.ForeignKey(settings.AUTH_USER_MODEL, db_column='taken_by', on_delete=models.PROTECT)
https://docs.djangoproject.com/en/2.1/ref/models/fields/#database-representation
^ link to the docs where it specifies that it creates a _id field.
based from the error message you have posted. It seems that your database schema is not updated.
you might need to do manage makemigrations and migrate to apply your model changes to your db schema
e.g
$ python manage.py makemigrations
# to apply the new migrations file
$ python manage.py migrate

unique_together does not work in Django shell

I have below model:
class Property(models.Model):
job = models.ForeignKey(Job, on_delete=models.CASCADE)
app = models.ForeignKey(App, on_delete=models.CASCADE)
name = models.CharField(max_length=120)
value = models.CharField(max_length=350, blank=True)
description = models.TextField(blank=True)
pub_date = models.DateTimeField('date_published', default=timezone.now)
class Meta:
verbose_name_plural = "properties"
unique_together = (('value', 'name'),)
def __str__(self):
return self.name
When I try to create a Property object in admin page (I'm using Django Suit) with name/value which are already exist I get the exception: "Property with this Value and Name already exists." So it works perfect.
But in manage.py shell:
>>>from myapp.models import App, Property, Job
>>>from django.shortcuts import get_object_or_404
>>>app = get_object_or_404(App, app_name='BLABLA')
>>>job = get_object_or_404(Job, job_name='BLABLA2')
>>> Property.objects.create(job=job, app=app, name='1', value='1')
<Property: 1>
>>> Property.objects.create(job=job, app=app, name='1', value='1')
<Property: 1>
In this case I do not get any exceptions and objects are added in database.
I tried makemigrations, migrate and migrate --run-syncdb.
Django 1.9.12, sqlite3
The unique constraints are enforced at database level. You're not getting any error probably because SQLite doesn't support this type of constraint. You cannot add constraint to existing table in SQLite. If you're in early development stage, drop the table and recreate it with updated constraints. Then it should work fine in shell.
Check SQLite alter table docs for allowed updates on an existing table.
The admin form throws error because it checks uniqueness by itself without relying on database constraints.

Django Introduce a foreign key from a model CharField

I have migrated over 10,000 records from my old mySQL database to Django/sqlite. In my old mysql schema's Song table, the artist field was not a 1 to many field but was just a mysql varchar field. In my new Django model, I converted the artist field to a ForeignKey and used temp_artist to temporarily store the artist's name from the old database.
How do I create each Song instance's artist foreignkey based on the temp_artist field? I'm assuming I should use the manager's get_or_create method but where and how do I write the code?
my model below:
class Artist (models.Model):
name = models.CharField(max_length=100)
class Song (models.Model):
artist = models.ForeignKey(Artist, blank=True, null=True, on_delete=models.CASCADE, verbose_name="Artist")
temp_artist = models.CharField(null=True, blank=True, max_length=100)
title = models.CharField(max_length=100, verbose_name="Title")
duration = models.DurationField(null=True, blank=True, verbose_name="Duration")
You can write a custom management command that performs this logic for you. The docs provide good instructions on how to set it up. Your command code would look something like this:
# e.g., migrateauthors.py
from django.core.management.base import BaseCommand
from myapp import models
class Command(BaseCommand):
help = 'Migrate authors from old schema'
def handle(self, *args, **options):
for song in myapp.models.Song.objects.all():
song.artist, _ = models.Artist.objects.get_or_create(name=song.temp_artist)
song.save()
Then you simply run the management command with manage.py migrateauthors. Once this is done and verified you can remove the temporary field from your model.
Since you don't have a usable foreign key at the moment you would have to dig down to raw_sql. If you were still on mysql you could have used the UPDATE JOIN syntax. But unfortunately Sqlite does not support UPDATE JOIN.
Luckily for you you have only a few thousand rows and that makes it possible to iterate through them and update each row individually.
raw_query = '''SELECT s.*, a.id as fkid
FROM myapp_song s
INNER JOIN myapp_artist a on s.temp_artist = a.name'''
for song in Song.objects.raw(raw_query)
song.artist_id = s.fkid
song.save()
This might take a few minutes to complete because you don't have an index on temp_artist and name. Take care to replace myapp with the actual name of your app.
Edit1:
Though Sqlite doesn't have update JOIN, it does allow you to SET a value with a subquery. So this will also work.
UPDATE myapp_song set artist_id =
(SELECT id from myapp_artist WHERE name = myapp_song.temp_artist)
type it in the sqlite console or GUI. Make sure to replace myapp with your own app name. This will be very quick because it's a single query. All other solutions including my alternative solution in this answer involve 10,000 queries.
Edit 2
If your Artist table is empty at the moment, before you do all this you will have to populate it, here is an easy query that does it
INSERT INTO stackoverflow_artist(name)
SELECT distinct temp_artist from stackoverflow_song
note that you should have a unique index on Artist.name

Add non-null and unique field with already populated model

I have one model in my app running in a server with a few entries. I need to add a SlugField, unique and not-null for this model. The SlugField will be populated based on trading_name. I've changed my model in order to add this new field and modified save method:
class Supplier(StatusModel):
SLUG_MAX_LENGTH = 210
slug = models.SlugField(unique=True, max_length=SLUG_MAX_LENGTH)
trading_name = models.CharField(max_length=200, verbose_name=_('trading name'))
...
def save(self, *args, **kwargs):
self.slug = orig = slugify(self.trading_name)[:Supplier.SLUG_MAX_LENGTH]
for x in itertools.count(1):
if not Supplier.objects.filter(slug=self.slug).exists():
break
# Truncate the original slug dynamically. Minus 1 for the hyphen.
self.slug = "%s-%d" % (orig[:Supplier.SLUG_MAX_LENGTH - len(str(x)) - 1], x)
self.full_clean()
super(Supplier, self).save(*args, **kwargs)
After changing the model, I've run manage.py makemigrations and got this migration as output:
class Migration(migrations.Migration):
dependencies = [
('opti', '0003_auto_20141226_1755'),
]
operations = [
migrations.AddField(
model_name='supplier',
name='slug',
field=models.SlugField(unique=True, default='', max_length=210),
preserve_default=False,
),
]
I can't run manage.py migrate because the default value wont work due to the unique constrant.
My question is: How can I do this with Django 1.7? I need to apply the schema change and keep the current entries in my database.
Unfortunatelly, I found no answer but I could create one solution:
First I have created a migration that allows the slug field to be nullable;
Then I have created another migration that populates the slug column with proper values to every row in the model;
Then another migration which adds the not-null constraint in the column.
You do your model changes (add field, change, etc), then you call manage.py makemigrations, then apply the migrations with manage.py migrate
You can add the field with null=True, then you e.g. make a script to populate it one time
Otherwise, if you need to populate the field within the migration you can write a custom one, see https://docs.djangoproject.com/en/1.7/ref/migration-operations/#writing-your-own