Unique model field in Django and case sensitivity (postgres) - django

Consider the following situation: -
Suppose my app allows users to create the states / provinces in their
country. Just for clarity, we are considering only ASCII characters
here.
In the US, a user could create the state called "Texas". If this app
is being used internally, let's say the user doesn't care if it is
spelled "texas" or "Texas" or "teXas"
But importantly, the system should prevent creation of "texas" if
"Texas" is already in the database.
If the model is like the following:
class State(models.Model):
name = models.CharField(max_length=50, unique=True)
The uniqueness would be case-sensitive in postgres; that is, postgres
would allow the user to create both "texas" and "Texas" as they are
considered unique.
What can be done in this situation to prevent such behavior. How does
one go about providing case-insenstitive uniqueness with Django and
Postgres
Right now I'm doing the following to prevent creation of case-
insensitive duplicates.
class CreateStateForm(forms.ModelForm):
def clean_name(self):
name = self.cleaned_data['name']
try:
State.objects.get(name__iexact=name)
except ObjectDoesNotExist:
return name
raise forms.ValidationError('State already exists.')
class Meta:
model = State
There are a number of cases where I will have to do this check and I'm not keen on having to write similar iexact checks everywhere.
Just wondering if there is a built-in or
better way? Perhaps db_type would help? Maybe some other solution exists?

You could define a custom model field derived from models.CharField.
This field could check for duplicate values, ignoring the case.
Custom fields documentation is here http://docs.djangoproject.com/en/dev/howto/custom-model-fields/
Look at http://code.djangoproject.com/browser/django/trunk/django/db/models/fields/files.py for an example of how to create a custom field by subclassing an existing field.
You could use the citext module of PostgreSQL https://www.postgresql.org/docs/current/static/citext.html
If you use this module, the the custom field could define "db_type" as CITEXT for PostgreSQL databases.
This would lead to case insensitive comparison for unique values in the custom field.

Alternatively you can change the default Query Set Manager to do case insensitive look-ups on the field. In trying to solve a similar problem I came across:
http://djangosnippets.org/snippets/305/
Code pasted here for convenience:
from django.db.models import Manager
from django.db.models.query import QuerySet
class CaseInsensitiveQuerySet(QuerySet):
def _filter_or_exclude(self, mapper, *args, **kwargs):
# 'name' is a field in your Model whose lookups you want case-insensitive by default
if 'name' in kwargs:
kwargs['name__iexact'] = kwargs['name']
del kwargs['name']
return super(CaseInsensitiveQuerySet, self)._filter_or_exclude(mapper, *args, **kwargs)
# custom manager that overrides the initial query set
class TagManager(Manager):
def get_query_set(self):
return CaseInsensitiveQuerySet(self.model)
# and the model itself
class Tag(models.Model):
name = models.CharField(maxlength=50, unique=True, db_index=True)
objects = TagManager()
def __str__(self):
return self.name

a very simple solution:
class State(models.Model):
name = models.CharField(max_length=50, unique=True)
def clean(self):
self.name = self.name.capitalize()

Explicit steps for Mayuresh's answer:
in postgres do: CREATE EXTENSION citext;
in your models.py add:
from django.db.models import fields
class CaseInsensitiveTextField(fields.TextField):
def db_type(self, connection):
return "citext"
reference: https://github.com/zacharyvoase/django-postgres/blob/master/django_postgres/citext.py
in your model use: name = CaseInsensitiveTextField(unique=True)

On the Postgres side of things, a functional unique index will let you enforce unique values without case. citext is also noted, but this will work with older versions of PostgreSQL and is a useful technique in general.
Example:
# create table foo(bar text);
CREATE TABLE
# create unique index foo_bar on foo(lower(bar));
CREATE INDEX
# insert into foo values ('Texas');
INSERT 0 1
# insert into foo values ('texas');
ERROR: duplicate key value violates unique constraint "foo_bar"

Besides already mentioned option to override save, you can simply store all text in lower case in database and capitalize them on displaying.
class State(models.Model):
name = models.CharField(max_length=50, unique=True)
def save(self, force_insert=False, force_update=False):
self.name = self.name.lower()
super(State, self).save(force_insert, force_update)

You can use lookup='iexact' in UniqueValidator on serializer, like this:
class StateSerializer(serializers.ModelSerializer):
name = serializers.CharField(validators=[
UniqueValidator(
queryset=models.State.objects.all(),lookup='iexact'
)]
django version: 1.11.6

If you don't want to use a postgres-specific solution, you can create a unique index on the field with upper() to enforce uniqueness at the database level, then create a custom Field mixin that overrides get_lookup() to convert case-sensitive lookups to their case-insensitive versions. The mixin looks like this:
class CaseInsensitiveFieldMixin:
"""
Field mixin that uses case-insensitive lookup alternatives if they exist.
"""
LOOKUP_CONVERSIONS = {
'exact': 'iexact',
'contains': 'icontains',
'startswith': 'istartswith',
'endswith': 'iendswith',
'regex': 'iregex',
}
def get_lookup(self, lookup_name):
converted = self.LOOKUP_CONVERSIONS.get(lookup_name, lookup_name)
return super().get_lookup(converted)
And you use it like this:
from django.db import models
class CICharField(CaseInsensitiveFieldMixin, models.CharField):
pass
class CIEmailField(CaseInsensitiveFieldMixin, models.EmailField):
pass
class TestModel(models.Model):
name = CICharField(unique=True, max_length=20)
email = CIEmailField(unique=True)
You can read more about this approach here.

You can do this by overwriting the Model's save method - see the docs. You'd basically do something like:
class State(models.Model):
name = models.CharField(max_length=50, unique=True)
def save(self, force_insert=False, force_update=False):
if State.objects.get(name__iexact = self.name):
return
else:
super(State, self).save(force_insert, force_update)
Also, I may be wrong about this, but the upcoming model-validation SoC branch will allow us to do this more easily.

Solution from suhail worked for me without the need to enable citext, pretty easy solution only a clean function and instead of capitalize I used upper(). Mayuresh's solution also works but changed the field from CharField to TextField.
class State(models.Model):
name = models.CharField(max_length=50, unique=True)
def clean(self):
self.name = self.name.upper()

Related

django admin add data with fixed value in some field

class Facilites(models.Model):
id = models.CharField(max_length=32, primary_key=True)
name = models.CharField(max_length=128)
class Objects(models.Model):
name = models.CharField(max_length=64)
facilityid = models.ForeignKey(Facilities)
class Admins(models.Model):
user = models.OneToOneField(User)
facilities = models.ManyToManyField(Facilities)
def create_user_profile(sender, instance, created, **kwargs):
if created:
Admins.objects.create(user=instance)
post_save.connect(create_user_profile, sender=User)
What i want is to have users (admins) only be able to add or modify "facilityid" in Objects to values specified in their Admins.facilities.
So if some user is named UserA and has facilities = ('FacA', 'FacB'), when he is adding a new object to DB, he shoudln't be able to add something like Object('Random object', 'FacC')
Also, he shouldn't be able to modify existing objects to facilities he doesn't belong to.
I have filtered the Objects with:
def queryset(self, request):
qs = super(ObjectsAdmin, self).queryset(request)
if request.user.is_superuser:
return qs
return qs.filter(facitityid__id__in = request.user.get_profile().facilities.all())
so users can only see the object that belong to their facilities. But i have no idea how to prevent them from adding/editing object out of their facilities.
edit:
found the answer here: https://stackoverflow.com/a/3048563/1421572
It turns out that ModelAdmin.formfield_for_foreignkey was the right answer in this situation: http://docs.djangoproject.com/en/dev/ref/contrib/admin/#django.contrib.admin.ModelAdmin.formfield_for_foreignkey
I would do this with either a pre-made facility list (i.e. You could create an integer field that is hooked to FACILITY_CHOICES for the user to select from.)
If only admins can do it then permissions sounds quite viable. You can also do form validation to check for errors against the db. Depending on how many facilities you have you may want a different approach.
You can do this same technique with a models.CharField as well. So perhaps assign a 3 letter facility code to each facility and require the entry to match one of the 3 letter strings. You could even have the list in a .txt file to read from. There are really so many ways to do this. I will provide an example of a pre-made facility list and accessing the facility a particular user belongs to from the api / template:
NYC_FACILITY = 0
LA_FACILITY = 1
ATL_FACILITY = 2
FACILITY_CHOICES = (
(NYC_FACILITY, 'NYC'),
(LA_FACILITY, 'LA'),
(ATL_FACILITY, 'ATL'),
class Facility(models.Model):
name = models.IntegerField(choices=FACILITY_CHOICES, default="NYC")
class Meta:
order_by = ['name']
verbose_name_plural = "facilities"
verbose_name = "facility"
def __unicode__(self):
return self.name
As far as viewing the facilities page that a particular user belongs to you will have a m2m one to one or FK relationship between the objects. If FK or m2m relationship then you will have access to additional methods of that model type. get_related However, I'm not going to use get_related in my example. Once you are in an instance you then have access to entry_set.
# models.py
from django.auth import User
class Person(User):
first_name = models.CharField(max_length=50)
last_name = models.CharField(max_length=50)
facility_loc = models.ForeignKey('Facility') # ForeignKey used assuming only one person can belong to a facility.
slug = models.SlugField(unique=True)
def get_absolute_url(self):
return "/%s/%s/" % self.facility_loc % self.slug
# views.py - TemplateView is automatically given a context variable called params which parses data from the URL. So, I'll leave the regex in the URLConf up to you.
class UserFacilityView(TemplateView):
model = Facility
template_name = "user_facility.html"
Now in your template you should be able to access facility_set from a User instance or user_set from a facility instance.

Setting default value for Foreign Key attribute

What is the best way to set a default value for a foreign key field in a model? Suppose I have two models, Student and Exam with student having exam_taken as foreign key. How would I ideally set a default value for it? Here's a log of my effort
class Student(models.Model):
....
.....
exam_taken = models.ForeignKey("Exam", default=1)
Works, but have a hunch there's a better way.
def get_exam():
return Exam.objects.get(id=1)
class Student(models.Model):
....
.....
exam_taken = models.ForeignKey("Exam", default=get_exam)
But this fails with tables does not exist error while syncing.
Any help would be appreciated.
I would modify #vault's answer above slightly (this may be a new feature). It is definitely desirable to refer to the field by a natural name. However instead of overriding the Manager I would simply use the to_field param of ForeignKey:
class Country(models.Model):
sigla = models.CharField(max_length=5, unique=True)
def __unicode__(self):
return u'%s' % self.sigla
class City(models.Model):
nome = models.CharField(max_length=64, unique=True)
nation = models.ForeignKey(Country, to_field='sigla', default='IT')
As already implied in #gareth's answer, hard-coding a default id value might not always be the best idea:
If the id value does not exist in the database, you're in trouble. Even if that specific id value does exist, the corresponding object may change. In any case, when using a hard-coded id value, you'd have to resort to things like data-migrations or manual editing of existing database content.
To prevent that, you could use get_or_create() in combination with a unique field (other than id).
Here's one way to do it:
from django.db import models
class Exam(models.Model):
title = models.CharField(max_length=255, unique=True)
description = models.CharField(max_length=255)
#classmethod
def get_default_pk(cls):
exam, created = cls.objects.get_or_create(
title='default exam',
defaults=dict(description='this is not an exam'),
)
return exam.pk
class Student(models.Model):
exam_taken = models.ForeignKey(
to=Exam, on_delete=models.CASCADE, default=Exam.get_default_pk
)
Here an Exam.title field is used to get a unique object, and an Exam.description field illustrates how we can use the defaults argument (for get_or_create) to fully specify the default Exam object.
Note that we return a pk, as suggested by the docs:
For fields like ForeignKey that map to model instances, defaults should be the value of the field they reference (pk unless to_field is set) instead of model instances.
Also note that default callables are evaluated in Model.__init__() (source). So, if your default value depends on another field of the same model, or on the request context, or on the state of the client-side form, you should probably look elsewhere.
I use natural keys to adopt a more natural approach:
<app>/models.py
from django.db import models
class CountryManager(models.Manager):
"""Enable fixtures using self.sigla instead of `id`"""
def get_by_natural_key(self, sigla):
return self.get(sigla=sigla)
class Country(models.Model):
objects = CountryManager()
sigla = models.CharField(max_length=5, unique=True)
def __unicode__(self):
return u'%s' % self.sigla
class City(models.Model):
nome = models.CharField(max_length=64, unique=True)
nation = models.ForeignKey(Country, default='IT')
In my case, I wanted to set the default to any existing instance of the related model. Because it's possible that the Exam with id 1 has been deleted, I've done the following:
class Student(models.Model):
exam_taken = models.ForeignKey("Exam", blank=True)
def save(self, *args, **kwargs):
try:
self.exam_taken
except:
self.exam_taken = Exam.objects.first()
super().save(*args, **kwargs)
If exam_taken doesn't exist, django.db.models.fields.related_descriptors.RelatedObjectDoesNotExist will be raised when a attempting to access it.
The issue with most of these approaches are that they use HARD CODED values or lambda methods inside the Model which are not supported anymore since Django Version 1.7.
In my opinion, the best approach here is to use a sentinel method which can also be used for the on_delete argument.
So, in your case, I would do
# Create or retrieve a placeholder
def get_sentinel_exam():
return Exam.objects.get_or_create(name="deleted",grade="N/A")[0]
# Create an additional method to return only the id - default expects an id and not a Model object
def get_sentinel_exam_id():
return get_sentinel_exam().id
class Exam(models.Model):
....
# Making some madeup values
name=models.CharField(max_length=200) # "English", "Chemistry",...
year=models.CharField(max_length=200) # "2012", "2022",...
class Student(models.Model):
....
.....
exam_taken = models.ForeignKey("Exam",
on_delete=models.SET(get_sentinel_exam),
default=get_sentinel_exam_id
)
Now, when you just added the exam_taken field uses a guaranteed existing value while also, when deleting the exam, the Student themself are not deleted and have a foreign key to a deleted value.
You could use this pattern:
class Other(models.Model):
DEFAULT_PK=1
name=models.CharField(max_length=1024)
class FooModel(models.Model):
other=models.ForeignKey(Other, default=Other.DEFAULT_PK)
Of course you need to be sure that there is a row in the table of Other. You should use a datamigration to be sure it exists.
I'm looking for the solution in Django Admin, then I found this:
class YourAdmin(admin.ModelAdmin)
def get_changeform_initial_data(self, request):
return {'owner': request.user}
this also allows me to use the current user.
see django docs
the best way I know is to use lambdas
class TblSearchCase(models.Model):
weights = models.ForeignKey('TblSearchWeights', models.DO_NOTHING, default=lambda: TblSearchWeights.objects.get(weight_name='value_you_want'))
so you can specify the default row..
default=lambda: TblSearchWeights.objects.get(weight_name='value_you_want')

Django unique_together doesn't work with ForeignKey=None

I saw some ppl had this problem before me, but on older versions of Django, and I'm running on 1.2.1.
I have a model that looks like:
class Category(models.Model):
objects = CategoryManager()
name = models.CharField(max_length=30, blank=False, null=False)
parent = models.ForeignKey('self', null=True, blank=True, help_text=_('The direct parent category.'))
class Meta:
unique_together = ('name', 'parent')
Whenever i try to save in the admin a category with a parent set to None, it still works when there's another category with the SAME name and parent set to None.
Ideas on how to solve this gracefully?
The unique together constraint is enforced at the database level, and it appears that your database engine does not apply the constraint for null values.
In Django 1.2, you can define a clean method for your model to provide custom validation. In your case, you need something that checks for other categories with the same name whenever the parent is None.
class Category(models.Model):
...
def clean(self):
"""
Checks that we do not create multiple categories with
no parent and the same name.
"""
from django.core.exceptions import ValidationError
if self.parent is None and Category.objects.filter(name=self.name, parent=None).exists():
raise ValidationError("Another Category with name=%s and no parent already exists" % self.name)
If you are editing categories through the Django admin, the clean method will be called automatically. In your own views, you must call category.fullclean().
I had that problem too and solved it by creating a supermodel with clean method (like Alasdair suggested) and use it as base class for all my models:
class Base_model(models.Model):
class Meta:
abstract=True
def clean(self):
"""
Check for instances with null values in unique_together fields.
"""
from django.core.exceptions import ValidationError
super(Base_model, self).clean()
for field_tuple in self._meta.unique_together[:]:
unique_filter = {}
unique_fields = []
null_found = False
for field_name in field_tuple:
field_value = getattr(self, field_name)
if getattr(self, field_name) is None:
unique_filter['%s__isnull'%field_name] = True
null_found = True
else:
unique_filter['%s'%field_name] = field_value
unique_fields.append(field_name)
if null_found:
unique_queryset = self.__class__.objects.filter(**unique_filter)
if self.pk:
unique_queryset = unique_queryset.exclude(pk=self.pk)
if unique_queryset.exists():
msg = self.unique_error_message(self.__class__, tuple(unique_fields))
raise ValidationError(msg)
Unfortunately, for those of us using PostgreSQL as our backend database engine, there will never have a fix for this issue:
"Currently, only B-tree indexes can be declared unique.
When an index is declared unique, multiple table rows with equal indexed values are not allowed. Null values are not considered equal. A multicolumn unique index will only reject cases where all indexed columns are equal in multiple rows.
PostgreSQL automatically creates a unique index when a unique constraint or primary key is defined for a table. The index covers the columns that make up the primary key or unique constraint (a multicolumn index, if appropriate), and is the mechanism that enforces the constraint."
Source: https://www.postgresql.org/docs/9.0/indexes-unique.html

Setting a preferred item of a many-to-one in Django

I'm trying to create a Django model that handles the following:
An Item can have several Names.
One of the Names for an Item is its primary Name, i.e. the Name displayed given an Item.
(The model names were changed to protect the innocent.)
The models.py I've got looks like:
class Item(models.Model):
primaryName = models.OneToOneField("Name", verbose_name="Primary Name",
related_name="_unused")
def __unicode__(self):
return self.primaryName.name
class Name(models.Model):
item = models.ForeignKey(Item)
name = models.CharField(max_length=32, unique=True)
def __unicode__(self):
return self.name
class Meta:
ordering = [ 'name' ]
The admin.py looks like:
class NameInline(admin.TabularInline):
model = Name
class ItemAdmin(admin.ModelAdmin):
inlines = [ NameInline ]
admin.site.register(Item, ItemAdmin)
It looks like the database schema is working fine, but I'm having trouble with the admin, so I'm not sure of anything at this point. My main questions are:
How do I explain to the admin that primaryName needs to be one of the Names of the item being edited?
Is there a way to automatically set primaryName to the first Name found, if primaryName is not set, since I'm using inline admin for the names?
EDIT: Dang, I forgot this was still open. Anyway, I wound up redoing the model, replacing primaryName with just name (a CharField) in Item and renaming Name to Alias. This can't do what I wanted to (just search one table for a name), but I couldn't make the primaryName work if it didn't have null=True, since Item gets saved before any Names were created, meaning that any attempt to auto-assign a Name would see an empty QuerySet and fail.
The only way I could see it working was to have Name's save routine auto-set itself as its parent's primaryName if primaryName was NULL, which just didn't sit well with me.
i think you will want to take a look here.
http://docs.djangoproject.com/en/dev/ref/contrib/admin/#adding-custom-validation-to-the-admin
define your own form to take care of any custom validation that might be required.
How do I explain to the admin that primaryName needs to be one of the
Names of the item being edited?
Check out formfield_for_foreignkey() in the ModelAdmin docs.
class ItemAdmin(admin.ModelAdmin):
def formfield_for_foreignkey(self, db_field, request, **kwargs):
if db_field.name == "primaryName":
# tweak the filter to your liking.
kwargs["queryset"] = Name.objects.filter(item=...)
return db_field.formfield(**kwargs)
return super(ItemAdmin, self).formfield_for_foreignkey(db_field, request, **kwargs)
Is there a way to automatically set primaryName to the first Name found,
if primaryName is not set, since I'm
using inline admin for the names?
You could do this in your model's save method, something like:
class Item(models.Model):
...(snip)...
def save(self,force_insert=False,force_update=False):
if self.primaryName is None:
self.primaryName = self.name_set.all()[0]
# will want to handle the case that no names are set, etc
super(Item,self).save(force_insert,force_update)

Django: list_filter and foreign key fields

Django doesn't support getting foreign key values from list_display or list_filter (e.g foo__bar). I know you can create a module method as a workaround for list_display, but how would I go about to do the same for list_filter? Thanks.
Django supports list_filter with foreign key fields
# models.py:
class Foo(models.Model):
name = models.CharField(max_length=255)
def __unicode__(self):
return self.name
class Bar(models.Model):
name = models.CharField(max_length=255)
foo = models.ForeignKey(Foo)
# admin.py:
class BarAdmin(admin.ModelAdmin):
list_filter = ('foo__name')
From documentation: Field names in list_filter can also span relations using the __ lookup
Well, the docs say that you can may use ForeignKey field types in list_filter:
http://docs.djangoproject.com/en/dev/ref/contrib/admin/#django.contrib.admin.ModelAdmin.list_filter
An example:
# models.py:
class Foo(models.Model):
name = models.CharField(max_length=255)
def __unicode__(self):
return self.name
class Bar(models.Model):
name = models.CharField(max_length=255)
foo = models.ForeignKey(Foo)
# admin.py:
class BarAdmin(admin.ModelAdmin):
list_filter = ('foo')
If you want to filter by a field from the related model, there's a patch on the way to make this work (will probably be merged into 1.2 as it seems):
http://code.djangoproject.com/ticket/3400
solution from this page worked for me http://www.hoboes.com/Mimsy/hacks/fixing-django-124s-suspiciousoperation-filtering/
define
class SmarterModelAdmin(admin.ModelAdmin):
valid_lookups = ()
def lookup_allowed(self, lookup, *args, **kwargs):
if lookup.startswith(self.valid_lookups):
return True
return super(SmarterModelAdmin, self).lookup_allowed(lookup, *args, **kwargs)
then allow the lookup for certain foreign key field
class PageAdmin(SmarterModelAdmin):
valid_lookups = ('parent')
If you construct the URL for the changelist manually then Django has no problems following relationships. For example:
/admin/contact/contact/?participant__event=8
or
/admin/contact/contact/?participant__event__name__icontains=er
Both work fine (although the latter doesn't add 'distinct()' so might have duplicates but that won't usually be an issue for filters)
So you just need to add something to the page that creates the correct links. You can do this either with by overriding the changelist template or by writing a custom filterspec. There are several examples I found by Googling - particularly on Django Snippets
You can easily create custom filters since Django 1.4 by overriding django.contrib.admin.SimpleListFilter class.
More information :
Admin list_filter documentation ;
Django-1.4 release note.
I ran into the same problem and really needed a solution. I have a workaround that lets you create a filter on a FK related model property. You can even traverse more than one FK relationship. It creates a new FilterSpec subclass that subclasses the default RelatedFilterSpec used to give you a filter on a ForeignKey field.
See http://djangosnippets.org/snippets/2260/
The Haes answer works perfectly, but if the __ looks up to another ForeignKey field, you end up with a blank result. You must place another __ lookup, until it points to the real field.
In my case: list_filter = ('place__condo__name', )
my models.py:
class Condo(models.Model):
name = models.CharField(max_length=70)
...
class Place(models.Model):
condo = models.ForeignKey(Condo)
...
class Actions(models.Model):
place = models.ForeignKey(Place)
...