Django: Querying read-only view with no primary key - django

class dbview(models.Model):
# field definitions omitted for brevity
class Meta:
db_table = 'read_only_view'
def main(request):
result = dbview.objects.all()
Caught an exception while rendering: (1054, "Unknown column 'read_only_view.id' in 'field list'")
There is no primary key I can see in the view. Is there a workaround?
Comment:
I have no control over the view I am accessing with Django. MySQL browser shows columns there but no primary key.

When you say 'I have no control over the view I am accessing with Django. MySQL browser shows columns there but no primary key.'
I assume you mean that this is a legacy table and you are not allowed to add or change columns?
If so and there really isn't a primary key (even a string or non-int column*) then the table hasn't been set up very well and performance might well stink.
It doesn't matter to you though. All you need is a column that is guaranteed to be unique for every row. Set that to be 'primary_key = True in your model and Django will be happy.
There is one other possibility that would be problemmatic. If there is no column that is guaranteed to be unique then the table might be using composite primary keys. That is - it is specifying that two columns taken together will provide a unique primary key. This is perfectly valid relational modelling but unfortunatly unsupported by Django. In that case you can't do much besides raw SQL unless you can get another column added.

I have this issue all the time. I have a view that I can't or don't want to change, but I want to have a page to display composite information (maybe in the admin section). I just override the save and raise a NotImplementedError:
def save(self, **kwargs):
raise NotImplementedError()
(although this is probably not needed in most cases, but it makes me feel a bit better)
I also set managed to False in the Meta class.
class Meta:
managed = False
Then I just pick any field and tag it as the primary key. It doesn't matter if it's really unique with you are just doing filters for displaying information on a page, etc.
Seems to work fine for me. Please commment if there are any problems with this technique that I'm overlooking.

If there really is no primary key in the view, then there is no workaround.
Django requires each model to have exactly one field primary_key=True.

There should have been an auto-generated id field when you ran syncdb (if there is no primary key defined in your model, then Django will insert an AutoField for you).
This error means that Django is asking your database for the id field, but none exists. Can you run django manage.py dbshell and then DESCRIBE read_only_view; and post the result? This will show all of the columns that are in the database.
Alternatively, can you include the model definition you excluded? (and confirm that you haven't altered the model definition since you ran syncdb?)

I know this post is over a decade old, but I ran into this recently and came to SO looking for a good answer. I had to come up with a solution that addresses the OP's original question, and, additionally, allows for us to add new objects to the model for unit testing purposes, which is a problem I still had with all of the provided solutions.
main.py
from django.db import models
def in_unit_test_mode():
"""some code to detect if you're running unit tests with a temp SQLite DB, like..."""
import sys
return "test" in sys.argv
"""You wouldn't want to actually implement it with the import inside here. We have a setting in our django.conf.settings that tests to see if we're running unit tests when the project starts."""
class AbstractReadOnlyModel(models.Model):
class Meta(object):
abstract = True
managed = in_unit_test_mode()
"""This is just to help you fail fast in case a new developer, or future you, doesn't realize this is a database view and not an actual table and tries to update it."""
def save(self, *args, **kwargs):
if not in_unit_test_mode():
raise NotImplementedError(
"This is a read only model. We shouldn't be writing "
"to the {0} table.".format(self.__class__.__name__)
)
else:
super(AbstractReadOnlyModel, self).save(*args, **kwargs)
class DbViewBaseModel(AbstractReadOnlyModel):
not_actually_unique_field = IntegerField(primary_key=True)
# the rest of your field definitions
class Meta:
db_table = 'read_only_view'
if in_unit_test_mode():
class DbView(DbViewBaseModel):
not_actually_unique_field = IntegerField()
"""This line removes the primary key property from the 'not_actually_unique_field' when running unit tests, so Django will create an AutoField named 'id' on the table it creates in the temp DB that it creates for running unit tests."""
else:
class DbView(DbViewBaseModel):
pass
class MainClass(object):
#staticmethod
def main_method(request):
return DbView.objects.all()
test.py
from django.test import TestCase
from main import DbView
from main import MainClass
class TestMain(TestCase):
#classmethod
def setUpTestData(cls):
cls.object_in_view = DbView.objects.create(
"""Enter fields here to create test data you expect to be returned from your method."""
)
def testMain(self):
objects_from_view = MainClass.main_method()
returned_ids = [object.id for object in objects_from_view]
self.assertIn(self.object_in_view.id, returned_ids)

Related

How can I support AutoField(primary_key=False) in django?

I need to add an autoinc field that is not the primary key. I am in the process of migrating a very large production database that uses autoincrementing fields to models.UUIDField. I have been doing a piecewise migration, and all of my relationships are now duplicated with both field types. I'm ready to make the primary key swap, but unfortunately I still need to keep the auto incrementing integer field for old clients as it becomes deprecated.
Since django will not allow me to define an autofield with primary_key=False (even though this is fully supported at the db layer), i'm looking for a simple solution. My initial strategy would be to simply change the field to models.BigIntegerField('GUID', db_index=True, null=True, unique=True) and then manually set the default nextval('my_guid_seq'::regclass) using migrations.RunSQL. So far so good, except not. It turns out, because of my null=True declaration, django at the ORM layer is taking over and inserting null which will not allow defaults at the database layer to do it's job.
The core developers are fast to reject this request because of bad design, which I most definetly agree with, but there are very valid use cases such as this. https://code.djangoproject.com/ticket/8576
I am a very weak django developer so I don't want to get in the weeds metaprogramming at the ORM layer. This is by definition a hack, so i'm looking for the least complex, creative solution that gets me around this limitation
You could subclass AutoField and override the _check_primary_key method.
from django.db.models.fields import AutoField
from django.db.models.fields import checks
class AutoFieldNonPrimary(AutoField):
def _check_primary_key(self):
if self.primary_key:
return [
checks.Error(
"AutoFieldNonPrimary must not set primary_key=True.",
obj=self,
id="fields.E100",
)
]
else:
return []
See AutoField source code here
Edit: Updated Link
I know, changing the primary key to UUID is such a pain.Hence the simple and better solution that I think of is to add another integer field that is auto-incrementing in nature.
Here is my solution:
class ModelName(models.Model):
auto_inc_id = models.IntegerField()
Then override the save model:
def save(self, *args, **kwargs):
self.object_list = ModelName.objects.order_by('auto_inc_id')
if len(self.object_list) == 0: # if there are no objects
self.auto_inc_id = 1
else:
self.auto_inc_id = self.object_list.last().auto_inc_id + 1
super(ModelName, self).save()
Couldn't format this as a comment, but modifying #Abhimanyu's answer to make the save method more concise (and issue only one query). Same model property:
class ModelName(models.Model):
auto_inc_id = models.IntegerField()
And here's the save method on the model:
def save(self, *args, **kwargs):
self.auto_inc_id = ModelName.objects.all().count() + 1
super(ModelName, self).save()

Correct Way to Validate Django Model Objects?

I'm still trying to understand the correct way to validate a Django model object using a custom validator at the model level. I know that validation is usually done within a form or model form. However, I want to ensure the integrity of my data at the model level if I'm interacting with it via the ORM in the Python shell. Here's my current approach:
from django.db import models
from django.core import validators
from django.core exceptions import ValidationError
def validate_gender(value):
""" Custom validator """
if not value in ('m', 'f', 'M', 'F'):
raise ValidationError(u'%s is not a valid value for gender.' % value)
class Person(models.Model):
name = models.CharField(max_length=128)
age = models.IntegerField()
gender = models.CharField(maxlength=1, validators=[validate_gender])
def save(self, *args, **kwargs):
""" Override Person's save """
self.full_clean(exclude=None)
super(Person, self).save(*args, **kwargs)
Here are my questions:
Should I create a custom validation function, designate it as a validator, and then override the Person's save() function as I've done above? (By the way, I know I could validate my gender choices using the 'choices' field option but I created 'validate_gender' for the purpose of illustration).
If I really want to ensure the integrity of my data, should I not only write Django unit tests for testing at the model layer but also equivalent database-level unit tests using Python/Psycopg? I've noticed that Django unit tests, which raise ValidationErrors, only test the model's understanding of the database schema using a copy of the database. Even if I were to use South for migrations, any database-level constraints are limited to what Django can understand and translate into a Postgres constraint. If I need a custom constraint that Django can't replicate, I could potentially enter data into my database that violates that constraint if I'm interacting with the database directly via the psql terminal.
Thanks!
I had a similar misunderstanding of the ORM when I first started with Django.
No, don't put self.full_clean() inside of save. Either
A) use a ModelForm (which will cause all the same validation to occur - note: ModelForm.is_valid() won't call Model.full_clean explicitly, but will perform the exact same checks as Model.full_clean). Example:
class PersonForm(forms.ModelForm):
class Meta:
model = Person
def add_person(request):
if request.method == 'POST':
form = PersonForm(request.POST, request.FILES)
if form.is_valid(): # Performs your validation, including ``validate_gender``
person = form.save()
return redirect('some-other-view')
else:
form = PersonForm()
# ... return response with ``form`` in the context for rendering in a template
Also note, forms aren't for use only in views that render them in templates - they're great for any sort of use, including an API, etc. After running form.is_valid() and getting errors, you'll have form.errors which is a dictionary containing all the errors in the form, including a key called '__all__' which will contain non-field errors.
B) Simply use model_instance.full_clean() in your view (or other logical application layer), instead of using a form, but forms are a nice abstraction for this.
I don't really have a solution to, but I've never run into such a problem, even in large projects (the current project I work with my company on has 146 tables) and I don't suspect it'll be a concern in your case either.

Django: making a custom PK auto-increment?

I've been using custom primary keys for a model in Django. (This was because I was importing values into the database and they already had ID's attached, and it made sense to preserve the existing values.)
class Transaction(models.Model):
id = models.IntegerField(primary_key=True)
transaction_type = models.IntegerField(choices=TRANSACTION_TYPES)
date_added = models.DateTimeField(auto_now_add=True)
However, now I want to add new instances of the model to the database, and I'd like to autogenerate a unique primary key. But if I don't specify the ID at the time of creating the instance, I get an error:
t = Transaction(transaction_type=0)
t.save()
gives:
IntegrityError at /page
(1048, "Column 'id' cannot be null")
How can I autogenerate a unique ID to specify for new values, without having to alter the way I import the existing values?
UPDATE
I've written this custom method, which seems to work...
class Transaction(models.Model):
def save(self, *args, **kwargs):
if not self.id:
i = Transaction.objects.all().order_by('-id')[0]
self.id = i.id+1
super(Transaction, self).save(*args, **kwargs)
You can use AutoField for the column id instead of IntegerField. The following should work for you:
id = models.AutoField(primary_key=True)
id will now increase automatically and won't have concurrency problems as it may encounter in save method.
I've ended up using very similar piece of code, but have made it slightly more generic:
def save(self, *args, **kwargs):
if self.id is None:
self.id = self.__class__.objects.all().order_by("-id")[0].id + 1
super(self.__class__, self).save(*args, **kwargs)
it uses self.__class__ so you can just copy paste this code to any model class without changing anything.
How are you importing the existing values? It would be trivial to write something into your Transactions __init__ to generate a new ID for you, but without knowing how you're importing the other values I can't say for sure whether it will alter the way you work with them.
If you remove your declared id field, django will automatically assume this:
id = models.AutoField(primary_key=True)
In Django 1.8, inspectdb will automatically detect auto_increment and use an AutoField when generating models.
Django migrations will do most of the hard work for you here.
Firstly, stop any access to your app so users can't change the database whilst you are working on it.
It would then be very wise to backup your database, before performing any work, as a precaution.
Remove your manually declared id field from your models.py (i.e. delete it).
Run makemigrations and then migrate. Django will modify the id field to the correct implementation for your database version.
Run this (example) command in psql adapting, if need be, to your table names:
select setval(pg_get_serial_sequence('transactions_transaction', 'id'), max(id)) from transactions_transaction;
This will set your id field to the correct serial sequence value in postgres for your table (i.e. the largest value of the id field of your existing records). This is crucial, as otherwise the value will be 1!
And that's it: from now on everything will be automatic again.

How to add Check Constraints for Django Model fields?

While subclassing db.models.Model, sometimes it's essential to add extra checks/constraints.
For example, I have an Event model with start_date and end_date: I want to add validation into the fields or the model so that end_date > start_date.
At least I know this can be done outside the models.Model inside the ModelForm validation. But how to attach to the fields and the models.Model?
I would not put constraints like these in the save method, it's too late. Raising an exception there, doesn't help the user who entered the data in the wrong way, because it will end up as a 500 and the user won't get the form with errors back etc.
You should really check for this in the Forms/ModelForms clean method and raise a ValidationError, so form.is_valid() returns false and you can send the errors in the form back to the user for correction.
Also note that since version 1.2, Django has had Model Validation.
It would look something like this:
class Foo(models.Model):
# ... model stuff...
def clean(self):
if self.start_date > self.end_date:
raise ValidationError('Start date is after end date')
As of Django 2.2, database level constraints are supported:
from django.db import models
from django.db.models import CheckConstraint, Q, F
class Event(models.Model):
start_date = models.DatetimeField()
end_date = models.DatetimeField()
class Meta:
constraints = [
CheckConstraint(
check = Q(end_date__gt=F('start_date')),
name = 'check_start_date',
),
]
Do it inside your save method of your model:
def save(self, *args, **kwargs):
if(self.end_date > self.start_date):
super(Foo, self).save(*args, **kwargs)
else:
raise Exception, "end_date should be greater than start_date"
As #stefanw says, it's better user experience to check in the form's clean method.
This is enough if you're very sure that there isn't, and never will be, another way to change the value. But since you can rarely be sure of that, if database consistency is important, you can add another check (in addition to the form), one of:
The easier and database-independent way is in the model's save method as #umnik700 said. Note that this still doesn't prevent other users of the database (another app, or the admin interface) from creating an inconsistent state.
To be 'completely' sure the database is consistent, you can add a database level constraint. E.g. you can create a migration with RunSQL and SQL, something like (not tested):
migrations.RunSQL('ALTER TABLE app_event ADD CONSTRAINT chronology CHECK (start_date > end_date);')
(Not tested). This may be database dependent, which is a downside of course.
In your example, it's probably not worth it (incorrect start/end times just look a bit weird, but affect only the one inconsistent event), and you don't want manual schema changes. But it's useful in cases where consistency is critical.
EDIT: You can also just save the start time and the duration, instead of the start and end times.
As of today, both postgres 9.4 and MS SQL Server >= 2008 support check constraints in sql. On top of this, there is django issue 11964 which seems to be ready for review since yesterday, so hopefully we'll see this integrated into django 2. The project rapilabs/django-db-constraints seems to implement this too.
Summarizing the answers from before, here is a complete solution I used for a project:
from django.db import models
from django.db.models import CheckConstraint, Q, F
from django.utils.translation import gettext_lazy as _
class Event(models.Model):
start_date = models.DatetimeField()
end_date = models.DatetimeField()
class Meta:
constraints = [
# Ensures constraint on DB level, raises IntegrityError (500 on debug=False)
CheckConstraint(
check=Q(end_date__gt=F('start_date')), name='check_start_date',
),
]
def clean(self):
# Ensures constraint on model level, raises ValidationError
if self.start_date > self.end_date:
# raise error for field
raise ValidationError({'end_date': _('End date cannot be smaller then start date.')})
Too bad there is no django.core.validators that can handle this :(

What is the canonical way to find out if a Django model is saved to db?

I generally check if obj.pk to knwo if the objects is saved. This wont work however, if you have primary_key = True set on some fields. Eg I set user = models.OneToOneField(User, primary_key=True) on my UserProfile.
What is the canonical way to find out if a Django model is saved to db?
Nowadays you can check for:
self._state.adding
This value is set by the QuerySet.iterator() for objects which are not added yet in the database. You can't use this value in the __init__() method yet, as it's set after the object is constructed.
Important Note (as of 6 May '19): If your models use UUID fields (or other method of internal ID generation, use self._state.adding as mentioned in the comments.
Actually,obj.pk is the most canonical way. Django itself often doesn't "know" if the object is saved or not. According to the django model instance reference, if there is a primary key set already, it checks onsave() calls by selecting for the id in the database before any insert.
Even if you set user = models.OneToOneField(..., primary_key=True) the .pk attribute will still point to the correct primary key (most likely user_id) and you can use it and set it as if it was the same property.
If you want to know after an object has been saved, you can catch the post_save signal. This signal is fired on model saves, and if you want you can add your own application-specific attribute to the model, for example obj.was_saved = True. I think django avoids this to keep their instances clean, but there's no real reason why you couldn't do this for yourself. Here is a minimal example:
from django.db.models.signals import post_save
from myapp.models import MyModel
def save_handler(sender, instance, **kwargs):
instance.was_saved = True
post_save.connect(save_handler, sender=MyModel)
You can alternately have this function work for all models in your app by simply connecting the signal without specifying the sender= argument. Beware though, you can create undefined behaviours if you override a property on someone else's model instance that you are importing.
Lets say obj is an instance of MyModel. Then we could use the following block of code to check if there already is an instance with that primary key in the database:
if obj.pk is None:
# Definitely doesn't exist, since there's no `pk`.
exists = False
else:
# The `pk` is set, but it doesn't guarantee exists in db.
try:
obj_from_db = MyModel.objects.get(pk=obj.pk)
exists = True
except MyModel.DoesNotExist:
exists = False
This is better than checking whether obj.pk is None, because you could do
obj = MyModel()
obj.pk = 123
then
obj.pk is None # False
This is even very likely when you don't use the autoincrement id field as the primary key but a natural one instead.
Or, as Matthew pointed out in the comments, you could do
obj.delete()
after which you still have
obj.pk is None # False
#Crast's answer was good, but I think incomplete. The code I use in my unit tests for determining if an object is in the database is as follows. Below it, I will explain why I think it is superior to checking if obj.pk is None.
My solution
from django.test import TestCase
class TestCase(TestCase):
def assertInDB(self, obj, msg=None):
"""Test for obj's presence in the database."""
fullmsg = "Object %r unexpectedly not found in the database" % obj
fullmsg += ": " + msg if msg else ""
try:
type(obj).objects.get(pk=obj.pk)
except obj.DoesNotExist:
self.fail(fullmsg)
def assertNotInDB(self, obj, msg=None):
"""Test for obj's absence from the database."""
fullmsg = "Object %r unexpectedly found in the database" % obj
fullmsg += ": " + msg if msg else ""
try:
type(obj).objects.get(pk=obj.pk)
except obj.DoesNotExist:
return
else:
self.fail(fullmsg)
Notes: Use the above code with care if you use custom managers on your models name something other than objects. (I'm sure there's a way to get Django to tell you what the default manager is.) Further, I know that /assert(Not)?InDB/ are not a PEP 8 method names, but I used the style the rest of the unittest package used.
Justification
The reason I think assertInDB(obj) is better than assertIsNotNone(obj.pk) is because of the following case. Suppose you have the following model.
from django.db import models
class Node(models.Model):
next = models.OneToOneField('self', null=True, related_name='prev')
Node models a doubly linked list: you can attach arbitrary data to each node using foreign keys and the tail is the Node obj such that obj.next is None. By default, Django adds the SQL constraint ON DELETE CASCADE to the primary key of Node. Now, suppose you have a list nodes of length n such that nodes[i].next == nodes[i + 1] for i in [0, n - 1). Suppose you call nodes[0].delete(). In my tests on Django 1.5.1 on Python 3.3, I found that nodes[i].pk is not None for i in [1, n) and only nodes[0].pk is None. However, my /assert(Not)?InDB/ methods above correctly detected that nodes[i] for i in [1, n) had indeed been deleted.