Given RangeA of auto-incrementing primary key integers (1 to 32767) and RangeB (32768 to 2147483647). If a condition is true, save an object with a primary key assigned in RangeA, else save it in RangeB.
The admin (me) would be the only user saving to RangeA. If the above is not possible: It would be not ideal but still usable if Django always saved in RangeB and it required going into shell to save in RangeA.
How can this be done using Django and Postgres?
Quite possible. First thing is to change your model so that it doesn't use the standard AutoField as the primary key
class MyModel(models.Model):
id = models.IntegerField(primary_key=True)
Then you need to connect to postgresql and create two different sequences.
CREATE SEQUENCE small START 1;
CREATE SEQUENCE big START 32768;
Instead of typing that into the PSQL console, you might also consider editing the django migration (using a RunSQL directive) to create create these.
Next step is to override the save method
def save(self,*args, **kwargs)
if not self.id :
cursor = connection.cursor()
if small condition:
cursor.execute("select nextval('small')")
else:
cursor.execute("select nextval('big')")
self.id = cursor.fetchone()[0]
super(MyModel,self).save(*args,**kwargs)
Alternative to overriding the save method is to create a postgresql BEFORE INSERT Trigger. The round trip to get the nextval isn't very costly but if this is a concern creating a trigger is the option to choose. In that case you don't need to over ride the save method but you have to implement the same logic inside the trigger.
Related
is there any possibility to define a Field which gets incremented, but starts with a letter? So the values should be like: S123, S124, S125.
I' d like to use the id field for it, but some reason i' d like to have it as a CharField.
As a workaround i could use:
id = models.CharField(max_length = 32, primary_key = True)
and redefine the save method, so i always precalculate its value, but this won' t be that robust than a "real" solution, and also my solution would too slow with the calculation.
Is there a proper solution for my problem?
Django: 1.9.2
Python: 3.4.2
.
I don't think you should manually define a primary key. Django usually uses relational database to build an app, which means it would rely on some key field to join other tables to do the lookup when it needs to. Having primary keys like S123 makes it extra hard to maintain because you need to store the same thing as a reference in other tables.
What I would suggest is storing the letter part and the digits separately. You could use the default id field django created as the digit part and create your own field to store the letter part. Then you would use a property method to return the value you want to have. Roughly:
class Foo(models.Model):
letter = models.CharField(max_length=1)
#property
def symbol(self):
return '%s%s' % (self.letter, self.id)
Then you could do:
foo = Foo.objects.create(letter='S')
print foo.symbol # this would print S1, S2, etc.
In case you don't know, here's an explanation of #property in python.
**
Difference between creating a foreign key for consistency and for joins
**
I am fine to use Foreignkey and Queryset API with Django.
I just want to understand little bit more deeply how it works behind the scenes.
In Django manual, it says
a database index is automatically created on the ForeignKey. You can
disable this by setting db_index to False. You may want to avoid the
overhead of an index if you are creating a foreign key for consistency
rather than joins, or if you will be creating an alternative index
like a partial of multiple column index.
creating for a foreign key for consistency rather than joins
this part is confusing me.
I expected that you use Join keyword if you do query with Foreign key like below.
SELECT
*
FROM
vehicles
INNER JOIN users ON vehicles.car_owner = users.user_id
For example,
class Place(models.Model):
name = models.Charfield(max_length=50)
address = models.Charfield(max_length=50)
class Comment(models.Model):
place = models.ForeignKeyField(Place)
content = models.Charfield(max_length=50)
if you use queryset like Comment.objects.filter(place=1), i expected using Join Keyword in low level SQL command.
but, when I checked it by printing out queryset.query in console, it showed like below.
(I simplified with Model just to explains. below, it shows all attributes in my model. you can ignore attributes)
SELECT
"bfm_comment"."id", "bfm_comment"."content", "bfm_comment"."user_id", "bfm_comment"."place_id", "bfm_comment"."created_at"
FROM "bfm_comment" WHERE "bfm_comment"."place_id" = 1
creating a foreign key for consistency vs creating a foreign key for joins
simply, I thought if you use any queryset, it means using foreign key for joins. Because you can get parent's table data by c = Comment.objects.get(id=1) c.place.name easily. I thought it joins two tables behind scenes. But result of Print(queryset.query) didn't how Join Keyword but Find it by Where keyword.
The way I understood from an answer
Case 1:
Comment.objects.filter(place=1)
result
SELECT
"bfm_comment"."id", "bfm_comment"."content", "bfm_comment"."user_id", "bfm_comment"."place_id", "bfm_comment"."created_at"
FROM "bfm_comment"
WHERE "bfm_comment"."id" = 1
Case 2:
Comment.objects.filter(place__name="df")
result
SELECT "bfm_comment"."id", "bfm_comment"."content", "bfm_comment"."user_id", "bfm_comment"."place_id", "bfm_comment"."created_at"
FROM "bfm_comment" INNER JOIN "bfm_place" ON ("bfm_comment"."place_id" = "bfm_place"."id")
WHERE "bfm_place"."name" = df
Case1 is searching rows which has comment.id column is 1 in just Comment table.
But in Case 2, it needs to know Place table's attribute 'name', so It has to use JOIN keyword to check values in column of Place table. Right?
So Is it alright to think that I create a foreign key for joins if i use queryset like Case2 and that it is better to create index on the Foreign Key?
for above question, I think I can take the answer from Django Manual
Consider adding indexes to fields that you frequently query using
filter(), exclude(), order_by(), etc. as indexes may help to speed up
lookups. Note that determining the best indexes is a complex
database-dependent topic that will depend on your particular
application. The overhead of maintaining an index may outweigh any
gains in query speed
In conclusion, it really depends on how my application work with it.
If you execute the following command the mystery will be revealed
./manage.py sqlmigrate myapp 0001
Take care to replace myapp with your app name (bfm I think) and 0001 with the actual migration where the Comment model is created.
The generated sql will reveal that the actual table is created with place_id int rather than a place Place that is because the RDBMS doesn't know anything about models, the models are only in the application level. It's the job of the django orm to fetch the data from the RDBMS and convert them into model instances. That's why you always get a place member in each of your Comment instances and that place member gives you access to the members of the related Place instance in turn.
So what happens when you do?
Comment.objects.filter(place=1)
Django is smart enough to know that you are referring to a place_id because 1 is obviously not an instance of a Place. But if you used a Place instance the result would be the same. So there is no join here. The above query would definitely benefit from having an index on the place_id, but it wouldn't benefit from having a foreign key constraint!! Only the Comment table is queried.
If you want a join, try this:
Comment.objects.filter(place__name='my home')
Queries of this nature with the __ often result in joins, but sometimes it results in a sub query.
Querysets are lazy.
https://docs.djangoproject.com/en/1.10/topics/db/queries/#querysets-are-lazy
QuerySets are lazy – the act of creating a QuerySet doesn’t involve
any database activity. You can stack filters together all day long,
and Django won’t actually run the query until the QuerySet is
evaluated. Take a look at this example:
How to get the value of the counter of an AutoField, such as the usual id field of most models?
At the moment, I do:
MyModel.objects.latest('id').id
But that does not work when all the objects have been deleted from the database.
Of course, a database-agnostic answer would be best.
EDIT
The accepted answer in Model next available primary key is not very relevant to my question, as I do not intend to use the counter value to create a new object. Also I don't mind if the value I get is not super accurate.
Background.
AFAIK there isn't a database agnostic query. Different databases handle auto increment differently and there rarely is a use case for django to find out what the next possible auto increment ID is.
To elaborate further, in postgresql you could do select nextval('my_sequence') while in mysql you would need to use the last_insert_id() but what this returns is the ID for the last insert and not the next one these two may actually be very different! To get the actual value you would need to use 'SHOW TABLE STATUS'
Solution.
Create a record, save it, inspect it's ID and delete it.
This will change the next id but you have indicated that you need only an approximation.
The alternative is to do a manual transaction with a rollback. This too would alter the next id in case of mysql.
from django.db import transaction
#transaction.atomic
def find_next_val(mymodel):
try:
# ...
obj = mymoel.objects.create(....)
print obj.id
raise IntegrityError
except IntegrityError:
pass
I've always found the Django orm's handling of subclassing models to be pretty spiffy. That's probably why I run into problems like this one.
Take three models:
class A(models.Model):
field1 = models.CharField(max_length=255)
class B(A):
fk_field = models.ForeignKey('C')
class C(models.Model):
field2 = models.CharField(max_length=255)
So now you can query the A model and get all the B models, where available:
the_as = A.objects.all()
for a in the_as:
print a.b.fk_field.field2 #Note that this throws an error if there is no B record
The problem with this is that you are looking at a huge number of database calls to retrieve all of the data.
Now suppose you wanted to retrieve a QuerySet of all A models in the database, but with all of the subclass records and the subclass's foreign key records as well, using select_related() to limit your app to a single database call. You would write a query like this:
the_as = A.objects.select_related("b", "b__fk_field").all()
One query returns all of the data needed! Awesome.
Except not. Because this version of the query is doing its own filtering, even though select_related is not supposed to filter any results at all:
set_1 = A.objects.select_related("b", "b__fk_field").all() #Only returns A objects with associated B objects
set_2 = A.objects.all() #Returns all A objects
len(set_1) > len(set_2) #Will always be False
I used the django-debug-toolbar to inspect the query and found the problem. The generated SQL query uses an INNER JOIN to join the C table to the query, instead of a LEFT OUTER JOIN like other subclassed fields:
SELECT "app_a"."field1", "app_b"."fk_field_id", "app_c"."field2"
FROM "app_a"
LEFT OUTER JOIN "app_b" ON ("app_a"."id" = "app_b"."a_ptr_id")
INNER JOIN "app_c" ON ("app_b"."fk_field_id" = "app_c"."id");
And it seems if I simply change the INNER JOIN to LEFT OUTER JOIN, then I get the records that I want, but that doesn't help me when using Django's ORM.
Is this a bug in select_related() in Django's ORM? Is there any work around for this, or am I simply going to have to do a direct query of the database and map the results myself? Should I be using something like Django-Polymorphic to do this?
It looks like a bug, specifically it seems to be ignoring the nullable nature of the A->B relationship, if for example you had a foreign key reference to B in A instead of the subclassing, that foreign key would of course be nullable and django would use a left join for it. You should probably raise this in the django issue tracker. You could also try using prefetch_related instead of select_related that might get around your issue.
I found a work around for this, but I will wait a while to accept it in hopes that I can get some better answers.
The INNER JOIN created by the select_related('b__fk_field') needs to be removed from the underlying SQL so that the results aren't filtered by the B records in the database. So the new query needs to leave the b__fk_field parameter in select_related out:
the_as = A.objects.select_related('b')
However, this forces us to call the database everytime a C object is accessed from the A object.
for a in the_as:
#Note that this throws an DoesNotExist error if a doesn't have an
#associated b
print a.b.fk_field.field2 #Hits the database everytime.
The hack to work around this is to get all of the C objects we need from the database from one query and then have each B object reference them manually. We can do this because the database call that accesses the B objects retrieved will have the fk_field_id that references their associated C object:
c_ids = [a.b.fk_field_id for a in the_as] #Get all the C ids
the_cs = C.objects.filter(pk__in=c_ids) #Run a query to get all of the needed C records
for c in the_cs:
for a in the_as:
if a.b.fk_field_id == c.pk: #Throws DoesNotExist if no b associated with a
a.b.fk_field = c
break
I'm sure there's a functional way to write that without the nested loop, but this illustrates what's happening. It's not ideal, but it provides all of the data with the absolute minimum number of database hits - which is what I wanted.
I need to set variable in db scope what will containt django user id what making DML query.
something like this :
CREATE OR REPLACE FUNCTION set_user_id(auser_id integer)
RETURNS void AS
'GD["user_id"] = auser_id'
LANGUAGE plpythonu VOLATILE
COST 100;
and call this function before every DML to pass that user id into audit trigger;
Is there is an easy way to do it?
If you need this to be done only for certain models, then you should override the save method of the model.
If you need this to be done irrespective of the models being using but across any model that accesses your particular database, then you should create your own variant of the db backend by adapting one for your particular database (in this case, I suspect its postgresql).
Something like this should do it:
from django.db.backends.postgresql_psycopg2.base import *
class DatabaseWrapper(DatabaseWrapper):
def _cursor(self):
cursor = super(DatabaseWrapper, self)._cursor()
q = " ... enter your query here ..."
cursor.execute(q % (arg1,arg2))
return cursor