Porting (postgresql) SQL into django model classes - django

I am porting a database schema (by hand), from (postgresql) SQL to django (1.10) model.
Here is my SQL:
CREATE TABLE ref_continent (
id SERIAL PRIMARY KEY,
name VARCHAR(64)
);
CREATE UNIQUE INDEX idxu_continent_nm ON ref_continent (name);
CREATE TABLE ref_geographic_region (
id SERIAL PRIMARY KEY,
continent_id INTEGER REFERENCES ref_continent(id) ON DELETE RESTRICT ON UPDATE CASCADE,
name VARCHAR(256)
);
CREATE UNIQUE INDEX idxu_geogreg_nm ON ref_geographic_region (name);
I am particularly interested in how to create the UNIQUE INDEX and how to link the FK to the PKey, since (AFAIK), django creates the primary key id behind the scenes?

According to docs
Note that when unique is True, you don’t need to specify db_index,
because unique implies the creation of an index.
Well, when you link a model as a foreign key, it links to default FK(that's id by default).
models.ForeignKey('app_name.ref_continent', )
there is another class class META that is used to fix these things. You don't have to almost ever specify to link FK to PK of other table, this is all done by Django. Have a at source code
models.py
class ref_continent(models.Model):
# Fields
name = models.CharField(max_length=64,unique=True)#making unique=True will generate query that will have UNIQUE INDEX both in PLSQL
def __unicode__(self):
return u'%s' % self.pk
class ref_geographic_region(models.Model):
# Fields
name = models.CharField(max_length=64,unique=True)
# Relationship Fields
continent_id = models.ForeignKey('app_name.ref_continent', )
def __unicode__(self):
return u'%s' % self.pk

Related

Django: use update_or_create having the foreign key value but not a foreign key object instance

I have a data from a raw sql query which has records that look like this:
"product_id": 12345
"date": 2022-12-25
"qty": 10
this is to go into a table with model "ProductMovement".
Its unique key is product_id and date.
product_movement has product_id as a foreign key to the Product model.
ProductMovement.objects.update_or_create() requires me to provide an instance of the Product model, not merely the unique key. This is inconvenient.
I guess I can use raw sql (backend is postgresql) but I wonder if there is another way. Can I add something to the model manager that intercepts the key value and replaces it with the instance, so at least I can hide this complexity from this part of the code? (I have never worked with model manager overrides).
Try this first
If you're command looks like this ProductMovement.objects.update_or_create(product=productObj) has no _id
Try running it like ProductMovement.objects.update_or_create(product_id=12345) with _id
Manager
Yea! You can override the update_or_create function- I've done it for filters/gets to fetch M2M relations easier
class ProductMovementManager(models.Manager):
def update_or_create(self, *args, **kwargs):
product = kwargs.pop('product')
if type(product) == int:
kwargs['product'] = Product.objects.filter(pk=product).first()
else:
# oops! It was already the object
kwargs['product'] = product
super().update_or_create(*args, **kwargs)
class ProductMovement(models.Model)
product = models.ForeignKey(product, on_delete=models.PROTECT)
objects = ProductMovementManager()

Maintain uniqueness on django-localized-fields

I'm trying to avoid having duplicate localized items stored in a Django-rest-framework app, django-localalized-fields package with a PostgreSQL database I can't find any way to make this work.
(https://pypi.org/project/django-localized-fields/)
I've tried writing custom duplicate detection logic in the Serializer, which works for create, but for update the localized fields become null (they are required fields, so I receive a not null constraint error). It seems to be django-localized-fields utility which is causing this problem.
The serializer runs correctly (create/update) when I'm not overriding create/update in the serializer by defining them separately.
I've also tried adding unique options to the database in the model, which does not work - duplicates are still created. Using the standard unique methods, or the method in the django-localized-fields documentation (uniqueness=['en', 'ro']).
I've also tried the UniqueTogetherValidator in Django, which also doesn't seem to support HStore/localizedfields.
I'd appreciate some help in tracking down either how to fix the update in the serializer or place a unique constraint in the database. Since django-localized-fields uses hstore in PostgreSQL it must be a common enough problem for applications using hstore to maintain uniqueness.
For those who aren't familiar, Hstore stores items as key/value pairs within a database. Here's an example of how django-localized-fields stores language data within the database:
"en"=>"english word!", "es"=>"", "fr"=>"", "frqc"=>"", "fr-ca"=>""
django-localized-fields constraint unique values only per the same language. If you want to achieve that values in a row don't collide with values in another row, you have to validate them on Django and database level.
Validation in Django
In Django you can create custom function validate_hstore_uniqueness, which is called everytime is model validated.
def validate_hstore_uniqueness(obj, field_name):
value_dict = getattr(obj, field_name)
cls = obj.__class__
values = list(value_dict.values())
# find all duplicite existing objects
duplicite_objs = cls.objects.filter(**{field_name+'__values__overlap':values})
if obj.pk:
duplicite_objs = duplicite_objs.exclude(pk=obj.pk)
if len(duplicite_objs):
# extract duplicite values
existing_values = []
for obj2 in duplicite_objs:
existing_values.extend(getattr(obj2, field_name).values())
duplicate_values = list(set(values) & set(existing_values))
# raise error for field
raise ValidationError({
field_name: ValidationError(
_('Values %(values)s already exist.'),
code='unique',
params={'values': duplicate_values}
),
})
class Test(models.Model):
slug = LocalizedField(blank=True, null=True, required=False)
def validate_unique(self, exclude=None):
super().validate_unique(exclude)
validate_hstore_uniqueness(self, 'slug')
Constraint in DB
For DB constraint you have to use constraint trigger.
def slug_uniqueness_constraint(apps, schema_editor):
print('Recreating trigger quotes.slug_uniqueness_constraint')
# define trigger
trigger_sql = """
-- slug_hstore_unique
CREATE OR REPLACE FUNCTION slug_uniqueness_constraint() RETURNS TRIGGER
AS $$
DECLARE
duplicite_count INT;
BEGIN
EXECUTE format('SELECT count(*) FROM %I.%I ' ||
'WHERE id != $1 and avals("slug") && avals($2)', TG_TABLE_SCHEMA, TG_TABLE_NAME)
INTO duplicite_count
USING NEW.id, NEW.slug;
IF duplicite_count > 0 THEN
RAISE EXCEPTION 'Duplicate slug value %', avals(NEW.slug);
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS slug_uniqueness_constraint on quotes_author;
CREATE CONSTRAINT TRIGGER slug_uniqueness_constraint
AFTER INSERT OR UPDATE OF slug ON quotes_author
FOR EACH ROW EXECUTE PROCEDURE slug_uniqueness_constraint();
"""
cursor = connection.cursor()
cursor.execute(trigger_sql)
And enable it in migrations:
class Migration(migrations.Migration):
dependencies = [
('quotes', '0031_auto_20200109_1432'),
]
operations = [
migrations.RunPython(slug_uniqueness_constraint)
]
Probably is a good idea to also create GIN db index for speeding up lookups:
CREATE INDEX ON test_table using GIN (avals("slug"));

Getting bigint id fields when using Django and Postgres?

Djangos ORM uses a integer datatype for the automatically created ID column, but I need them to be bigint (using postgres backend). Is there a way of doing this?
Django 1.10
Use the newly added BigAutoField
A 64-bit integer, much like an AutoField except that it is guaranteed
to fit numbers from 1 to 9223372036854775807.
Older versions of django
You need to create your model like this
class MyModel(models.Model):
id = models.BigIntegerField(primary_key=True)
name = models.CharField(max_length=100)
Then after ./manage.py makemigrations has been run, open the generated migration and add the following into he operations list:
migrations.RunSQL("CREATE SEQUENCE myapp_seq"),
migrations.RunSQL("ALTER TABLE myapp_mymodel ALTER COLUMN id SET DEFAULT NEXTVAL('myapp_seq')");
Update
A valid point was raised by Daniel Roseman in the comments. In postgreql the following query works
INSERT INTO myapp_mymodel(name) values('some name');
but the following doesn't because primary keys are not null
INSERT INTO myapp_mymodel(id, name) values(null,'some name');
unfortunately it's the second form of the query that's passed through by django. This can still be solved with a bit of work.
def save(self, *args, **kwargs):
if not self.id :
cursor = connection.cursor();
cursor.execute("SELECT NEXTVAL('myapp_seq')")
id = cursor.fetchone()
self.id = id[0]
Model(MyModel,self).save(*args, **kwargs)

Adding values in my database via a ManyToMany relationship represented in admin.py

I've got a tiny little problem that, unfortunately, is taking all my time.
It is really simple, I already have my database and I created then modified models.py, and admin.py. Some staff users, who will need to enter values in my database, need the simpliest form to do so.
Here is my database :
-- Table NGSdb.line
CREATE TABLE IF NOT EXISTS `NGSdb`.`line` (
`id` INT NOT NULL AUTO_INCREMENT ,
`value` INT NOT NULL ,
PRIMARY KEY (`id`) )
ENGINE = InnoDB;
CREATE UNIQUE INDEX `value_UNIQUE` ON `NGSdb`.`line` (`value` ASC) ;
-- Table NGSdb.run_has_sample_lines
CREATE TABLE IF NOT EXISTS `NGSdb`.`run_has_sample_lines` (
`line_id` INT NOT NULL ,
`runhassample_id` INT NOT NULL ,
PRIMARY KEY (`line_id`, `runhassample_id`) ,
CONSTRAINT `fk_sample_has_line_line1`
FOREIGN KEY (`line_id` )
REFERENCES `NGSdb`.`line` (`id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION,
CONSTRAINT `fk_sample_has_line_run_has_sample1`
FOREIGN KEY (`runhassample_id` )
REFERENCES `NGSdb`.`run_has_sample` (`id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION)
-- Table NGSdb.run_has_sample
CREATE TABLE IF NOT EXISTS `NGSdb`.`run_has_sample` (
`id` INT NOT NULL AUTO_INCREMENT ,
`run_id` INT NOT NULL ,
`sample_id` INT NOT NULL ,
`dna_quantification_ng_per_ul` FLOAT NULL ,
PRIMARY KEY (`id`, `run_id`, `sample_id`) ,
CONSTRAINT `fk_run_has_sample_run1`
FOREIGN KEY (`run_id` )
REFERENCES `NGSdb`.`run` (`id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION,
CONSTRAINT `fk_run_has_sample_sample1`
FOREIGN KEY (`sample_id` )
REFERENCES `NGSdb`.`sample` (`id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION)
Here is my models.py :
class Run(models.Model):
id = models.AutoField(primary_key=True)
start_date = models.DateField(null=True, blank=True, verbose_name='start date')
end_date = models.DateField(null=True, blank=True, verbose_name='end date')
project = models.ForeignKey(Project)
sequencing_type = models.ForeignKey(SequencingType)
def __unicode__(self):
return u"run started %s from the project %s" % (self.start_date,self.project)
class Line(models.Model):
id = models.AutoField(primary_key=True)
value = models.IntegerField()
def __unicode__(self):
return u"%s" % str(self.value)
class RunHasSample(models.Model):
id = models.AutoField(primary_key=True)
run = models.ForeignKey(Run)
sample = models.ForeignKey(Sample)
dna_quantification_ng_per_ul = models.FloatField(null=True, blank=True)
lines = models.ManyToManyField(Line)
def __unicode__(self):
return u"Sample %s from run %s" % (self.sample, self.run)
And here is my admin.py :
class RunHasSamplesInLine(admin.TabularInline):
model = RunHasSample
fields = ['sample', 'dna_quantification_ng_per_ul', 'lines']
extra = 6
class RunAdmin(admin.ModelAdmin):
fields = ['project', 'start_date', 'end_date', 'sequencing_type']
inlines = [RunHasSamplesInLine]
list_display = ('project', 'start_date', 'end_date', 'sequencing_type')
As you can see, my samples are displayed in lines in the run form so that the staff can easily fullfill the database.
When I try to fill the database I have this error :
(1054, "Unknown column 'run_has_sample_lines.id' in 'field list'")
Of course, there are no field "lines" in my database ! It is a many to many field so I already created my intermediate table !
Okay okay ! So I tried to create the model for the intermediate table (run_has_sample_lines) and add a "through" to the ManyToManyField in the RunHasSample model. But, as I add manually the "through", I cannot use the ManyToMany field. The only way to add lines to the admin view is to stack them in lines... As you can see the samples are already in lines, it is impossible to put a new "inlines" in the already in lines samples...
Finally, I just tried to see what django had created with the manage.py sqlall.
I see that :
CREATE TABLE `run_has_sample_lines` (
`id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
`runhassample_id` integer NOT NULL,
`line_id` integer NOT NULL,
UNIQUE (`runhassample_id`, `line_id`)
)
;
ALTER TABLE `run_has_sample_lines` ADD CONSTRAINT `line_id_refs_id_4f0766aa` FOREIGN KEY (`line_id`) REFERENCES `line` (`id`);
It seems that there are no foreign key on the run_has_sample table whereas I created it in the database in the first place. I guess that the problem is coming from here but I cannot resolve it and I really hope that you can...
Thank you very much !
you may wish to try a 'through' attribute on the many-to-many relationship and declare your intermediate table in Django.
I found where the problem is...
It is not a problem in the ManyToManyField but in the intermediate table. Django refused that my intermediate table doesn't have an unique id !
So, in the sql which created django, it created automatically an unique id named "id", but in my database I didn't create one (because the couple of two foreign key is usually enough).
Next time, I'll be more carefull.

Django - How to prevent database foreign key constraint creation

I have a model that is backed by a database view.
class OrgCode(models.Model):
org_code = models.CharField(db_column=u'code',max_length=15)
org_description = models.CharField(max_length=250)
org_level_num = models.IntegerField()
class Meta:
db_table = u'view_FSS_ORG_PROFILE'
I need to reference this in another model
class AssessmentLocation(models.Model):
name = models.CharField(max_length=150)
org = models.ForeignKey(OrgCode)
I can't run syncdb because foreign key constraints cannot be created referencing a view.
u"Foreign key 'FK__main_asse__org__1D114BD1'
references object 'view_FSS_ORG_PROFILE'
which is not a user table.", None, 0, -214
7217900), None)
Command:
CREATE TABLE [main_assessmentlocation] (
[id] int IDENTITY (1, 1) NOT NULL PRIMARY KEY,
[name] nvarchar(150) NOT NULL,
[org] int NOT NULL REFERENCES [view_FSS_ORG_PROFILE] ([id]),
)
The workaround is to take out the Meta:db_table pointing to the view and let sync db create the the OrgCode table, then put the Meta:db_table back in after syncdb.
Is there a way to prevent the creation of foreign key constraints for certain models or fields?
Update: I added a static method to the related model indicating it's a view
class OrgCode(models.Model):
org_code = models.CharField(max_length=15)
org_description = models.CharField(max_length=250)
#staticmethod
def is_backend_view():
return True
Then overrode DatabaseCreation.sql_for_inline_foreign_key_references in django_mssql creation.py:
def sql_for_inline_foreign_key_references(self, field, known_models, style):
try:
field.rel.to.is_backend_view()
return "", False
except:
return super(DatabaseCreation,self).sql_for_inline_foreign_key_references(field, known_models, style)
The generated sql from syncdb leaves out the constraint:
CREATE TABLE [main_assessmentlocation] (
[id] int IDENTITY (1, 1) NOT NULL PRIMARY KEY,
[name] nvarchar(150) NOT NULL,
[org] int, -- NO FK CONSTRAINT ANYMORE --
);
It does involve hacking django_mssql so I'm going to keep on trying, maybe hooking into the django.db.backends.signals.connection_created signal will work...
django development version has a db_constraint field for ForeignKey model field - docs.
If you set managed=False (Django docs) in your model's Meta class, Django will not create the table when you run syncdb.
class AssessmentLocation(models.Model):
name = models.CharField(max_length=150)
org = models.ForeignKey(OrgCode)
class Meta:
managed = False
Django has a hook to provide initial sql data. We can (ab?)use this to get Django to create the table immediately after running syncdb.
Create a file myapp/sql/assessmentlocation.sql, containing the create table statement:
CREATE TABLE [main_assessmentlocation] (
[id] int IDENTITY (1, 1) NOT NULL PRIMARY KEY,
[name] nvarchar(150) NOT NULL,
[org] int, -- NO FK CONSTRAINT ANYMORE --
);
If you have other models with foreign keys to the AssessmentLocation model, you may have problems if Django tries to apply the foreign key constraint before executing the custom sql to create the table. Otherwise, I think this approach will work.