Can I have an unsigned AutoField? - django

I want a primary key for my model to be unsigned. Therefore I do something like this:
class MyModel(models.Model):
id = models.PositiveIntegerField(primary_key=True)
This gets me an UNSIGNED column in the resulting MySQL table, which I want. However, I believe I will not get the automatic assigning to id each time I create a new object, will I? This seems to require the use of AutoField instead. Problem is, AutoField is signed. Is there a way to create an unsigned AutoField?

The actual type of the field is specified in the backend. In the case of MySQL, the backend is django.db.backends.mysql. This extract from django/db/backends/mysql/creation.py shows this translation:
class DatabaseCreation(BaseDatabaseCreation):
# This dictionary maps Field objects to their associated MySQL column
# types, as strings. Column-type strings can contain format strings; they'll
# be interpolated against the values of Field.__dict__ before being output.
# If a column type is set to None, it won't be included in the output.
data_types = {
'AutoField': 'integer AUTO_INCREMENT',
'BooleanField': 'bool',
'CharField': 'varchar(%(max_length)s)',
...
To change that, you should either monkey-patch this dict doing:
from django.db.backends.mysql.creation import DatabaseCreation
DatabaseCreation.data_types['AutoField'] = 'integer UNSIGNED AUTO_INCREMENT'
Or you create your own class, so you won't mess up with the other AutoFields:
from django.db.models.fields import AutoField
class UnsignedAutoField(AutoField):
def get_internal_type(self):
return 'UnsignedAutoField'
from django.db.backends.mysql.creation import DatabaseCreation
DatabaseCreation.data_types['UnsignedAutoField'] = 'integer UNSIGNED AUTO_INCREMENT'
And then create your own PKs:
id = UnsignedAutoField()
As it descends from AutoField, it will inherit all of its behavior.

Edit: Just to be clear, neither of the solutions written by myself or Simanas should be used in real world projects. I wrote this as an example in which direction should one go if they'd decided to avoid DBMS built-in way, and not as a completed model ready to be used.
I am sorry for writing an answer instead of a comment on the post made by Simanas, but I do not have high reputation to post one, and I feel it's needed as this question is pretty high ranked on 'django autofield unsigned integer' keywords.
Using his method is not reliable as it will produce an existing integer for new row if one of the previous objects gets deleted. Here's a modified one:
from django.db import IntegrityError
import re
class MyModel(models.Model):
def next_id():
try:
# Find the ID of the last object
last_row = MyModel.objects.order_by('-id')[0]
return last_row.id + 1
except IndexError:
# No objects exist in database so far
return 1
id = models.PositiveIntegerField(primary_key=True, default=next_id)
def save(self, *args, **kwargs):
while True:
try:
super(MyModel, self).save(*args, **kwargs)
break
except IntegrityError, e:
if e.args[0] == 1062:
if re.match("^Duplicate entry \'.*\' for key \'%s\'$"
% re.escape(self._meta.pk.name), e.args[1]):
self.id = next_id()
else:
raise
While this would work, it wouldn't know whether newly assigned ID was previously used for another objects (in case of deletion of newest objects?) and may lead to collisions in such cases; but it will work cross-database compared to Augusto's answer, which is MySQL specific.
Another caveat to this method is that if you have another application hooking to the same database, it'll have to provide the ID on INSERTs, as auto incremental is not done at database level.
You almost certainly don't want to do it this way.

Related

Django create one database entry and update those fields

For some reason there needs to be a database table with fields being updated. These fields are attempt, success and failure. I thought it'd be better to do using the Django ORM, but this needs to be the way ..
Problem hower is, that I get an array of array with data in it, that I will need to parse. How would I go about when there is not yet an entry in the database for this field?
models.py
class SomeData(models.Model):
attempt = models.PositiveIntegerField(null=True)
success = models.PositiveIntegerField(null=True)
failure = models.PositiveIntegerField(null=True)
views.py
class PutSomeData(CreateOrUpdateAPIView):
model = OtherModel
def post(self, *args, **kwargs):
data = self.request.data
for k in data:
entry = OtherModel(
field1=k[0],
field2=k[1],
field3=k[2]
)
entry.save()
count = SomeData.objects.all()
if not count:
attempt, success, failure = 0, 0, 0
data = SomeData(
attempt=attempt,
success=success,
failure=failure
)
data.save()
else:
data = SomeData.objects.last()
if 'attempt' in k[2]:
data.attempt + 1
elif 'success' in k[2]:
data.success + 1
else:
data.failure + 1
data.save()
I was thinking something like this for now, but this is ofcourse a stupid thing to do, and besides that it will always skip the first one in the array and is thus inaccurate. This is just something I have for now, but I do not know how to make this better and more elegant. Any thoughts?
Edit: to be a bit more clear: problem is that there are no initial fields for now, otherwise i could just increment the fields like i also do, but now I have to look if there is an entry already (it will otherwise complain about NoneType not having 'attempt' ofcourse)
You could create a migration with initial data, so that you'll be sure the entry exists.
Also, for a table with global values that is going to have only one row I would suggest using table with rows key and value (or similar), so that when you'll need to add new global values you won't need to do a schema migration.

Django get_next_by_FIELD using complex Q lookups

While creating a front end for a Django module I faced the following problem inside Django core:
In order to display a link to the next/previous object from a model query, we can use the extra-instance-methods of a model instance: get_next_by_FIELD() or get_previous_by_FIELD(). Where FIELD is a model field of type DateField or DateTimeField.
Lets explain it with an example
from django.db import models
class Shoe(models.Model):
created = models.DateTimeField(auto_now_add=True, null=False)
size = models.IntegerField()
A view to display a list of shoes, excluding those where size equals 4:
def list_shoes(request):
shoes = Shoe.objects.exclude(size=4)
return render_to_response(request, {
'shoes': shoes
})
And let the following be a view to display one shoe and the corresponding
link to the previous and next shoe.
def show_shoe(request, shoe_id):
shoe = Shoe.objects.get(pk=shoe_id)
prev_shoe = shoe.get_previous_by_created()
next_shoe = shoe.get_next_by_created()
return render_to_response('show_shoe.html', {
'shoe': shoe,
'prev_shoe': prev_shoe,
'next_shoe': next_shoe
})
Now I have the situation that the show_shoe view displays the link to the previous/next regardless of the shoes size. But I actually wanted just shoes whose size is not 4.
Therefore I tried to use the **kwargs argument of the get_(previous|next)_by_created() methods to filter out the unwanted shoes, as stated by the documentation:
Both of these methods will perform their queries using the default manager for the model. If you need to emulate filtering used by a custom manager, or want to perform one-off custom filtering, both methods also accept
optional keyword arguments, which should be in the format described in Field lookups.
Edit: Keep an eye on the word "should", because then also (size_ne=4) should work, but it doesn't.
The actual problem
Filtering using the lookup size__ne ...
def show_shoe(request, shoe_id):
...
prev_shoe = shoe.get_previous_by_created(size__ne=4)
next_shoe = shoe.get_next_by_created(size__ne=4)
...
... didn't work, it throws FieldError: Cannot resolve keyword 'size_ne' into field.
Then I tried to use a negated complex lookup using Q objects:
from django.db.models import Q
def show_shoe(request, shoe_id):
...
prev_shoe = shoe.get_previous_by_created(~Q(size=4))
next_shoe = shoe.get_next_by_created(~Q(size=4))
...
... didn't work either, throws TypeError: _get_next_or_previous_by_FIELD() got multiple values for argument 'field'
Because the get_(previous|next)_by_created methods only accept **kwargs.
The actual solution
Since these instance methods use the _get_next_or_previous_by_FIELD(self, field, is_next, **kwargs) I changed it to accept positional arguments using *args and passed them to the filter, like the **kwargs.
def my_get_next_or_previous_by_FIELD(self, field, is_next, *args, **kwargs):
"""
Workaround to call get_next_or_previous_by_FIELD by using complext lookup queries using
Djangos Q Class. The only difference between this version and original version is that
positional arguments are also passed to the filter function.
"""
if not self.pk:
raise ValueError("get_next/get_previous cannot be used on unsaved objects.")
op = 'gt' if is_next else 'lt'
order = '' if is_next else '-'
param = force_text(getattr(self, field.attname))
q = Q(**{'%s__%s' % (field.name, op): param})
q = q | Q(**{field.name: param, 'pk__%s' % op: self.pk})
qs = self.__class__._default_manager.using(self._state.db).filter(*args, **kwargs).filter(q).order_by('%s%s' % (order, field.name), '%spk' % order)
try:
return qs[0]
except IndexError:
raise self.DoesNotExist("%s matching query does not exist." % self.__class__._meta.object_name)
And calling it like:
...
prev_shoe = shoe.my_get_next_or_previous_by_FIELD(Shoe._meta.get_field('created'), False, ~Q(state=4))
next_shoe = shoe.my_get_next_or_previous_by_FIELD(Shoe._meta.get_field('created'), True, ~Q(state=4))
...
finally did it.
Now the question to you
Is there an easier way to handle this? Should shoe.get_previous_by_created(size__ne=4) work as expected or should I report this issue to the Django guys, in the hope they'll accept my _get_next_or_previous_by_FIELD() fix?
Environment: Django 1.7, haven't tested it on 1.9 yet, but the code for _get_next_or_previous_by_FIELD() stayed the same.
Edit: It is true that complex lookups using Q object is not part of "field lookups", it's more part of the filter() and exclude() functions instead. And I am probably wrong when I suppose that get_next_by_FIELD should accept Q objects too. But since the changes involved are minimal and the advantage to use Q object is high, I think these changes should get upstream.
tags: django, complex-lookup, query, get_next_by_FIELD, get_previous_by_FIELD
(listing tags here, because I don't have enough reputations.)
You can create custom lookup ne and use it:
.get_next_by_created(size__ne=4)
I suspect the method you've tried first only takes lookup arg for the field you're basing the get_next on. Meaning you won't be able to access the size field from the get_next_by_created() method, for example.
Edit : your method is by far more efficient, but to answer your question on the Django issue, I think everything is working the way it is supposed to. You could offer an additional method such as yours but the existing get_next_by_FIELD is working as described in the docs.
You've managed to work around this with a working method, which is OK I guess, but if you wanted to reduce the overhead, you could try a simple loop :
def get_next_by_field_filtered(obj, field=None, **kwargs):
next_obj = getattr(obj, 'get_next_by_{}'.format(field))()
for key in kwargs:
if not getattr(next_obj, str(key)) == kwargs[str(key)]:
return get_next_by_field_filtered(next_obj, field=field, **kwargs)
return next_obj
This isn't very efficient but it's one way to do what you want.
Hope this helps !
Regards,

Django aggregate multiple columns after arithmetic operation

I have a really strange problem with Django 1.4.4.
I have this model :
class LogQuarter(models.Model):
timestamp = models.DateTimeField()
domain = models.CharField(max_length=253)
attempts = models.IntegerField()
success = models.IntegerField()
queue = models.IntegerField()
...
I need to gather the first 20 domains with the higher sent property. The sent property is attempts - queue.
This is my request:
obj = LogQuarter.objects\
.aggregate(Sum(F('attempts')-F('queue')))\
.values('domain')\
.filter(**kwargs)\
.order_by('-sent')[:20]
I tried with extra too and it isn't working.
It's really basic SQL, I am surprised that Django can't do this.
Did someone has a solution ?
You can actually do this via subclassing some of the aggregation functionality. This requires digging in to the code to really understand, but here's what I coded up to do something similar for MAX and MIN. (Note: this code is based of Django 1.4 / MySQL).
Start by subclassing the underlying aggregation class and overriding the as_sql method. This method writes the actual SQL to the database query. We have to make sure to quote the field that gets passed in correctly and associate it with the proper table name.
from django.db.models.sql import aggregates
class SqlCalculatedSum(aggregates.Aggregate):
sql_function = 'SUM'
sql_template = '%(function)s(%(field)s - %(other_field)s)'
def as_sql(self, qn, connection):
# self.col is currently a tuple, where the first item is the table name and
# the second item is the primary column name. Assuming our calculation is
# on two fields in the same table, we can use that to our advantage. qn is
# underlying DB quoting object and quotes things appropriately. The column
# entry in the self.extra var is the actual database column name for the
# secondary column.
self.extra['other_field'] = '.'.join(
[qn(c) for c in (self.col[0], self.extra['column'])])
return super(SqlCalculatedSum, self).as_sql(qn, connection)
Next, subclass the general model aggregation class and override the add_to_query method. This method is what determines how the aggregate gets added to the underlying query object. We want to be able to pass in the field name (e.g. queue) but get the corresponding DB column name (in case it is something different).
from django.db import models
class CalculatedSum(models.Aggregate):
name = SqlCalculatedSum
def add_to_query(self, query, alias, col, source, is_summary):
# Utilize the fact that self.extra is set to all of the extra kwargs passed
# in on initialization. We want to get the corresponding database column
# name for whatever field we pass in to the "variable" kwarg.
self.extra['column'] = query.model._meta.get_field(
self.extra['variable']).db_column
query.aggregates[alias] = self.name(
col, source=source, is_summary=is_summary, **self.extra)
You can then use your new class in an annotation like this:
queryset.annotate(calc_attempts=CalculatedSum('attempts', variable='queue'))
Assuming your attempts and queue fields have those same db column names, this should generate SQL similar to the following:
SELECT SUM(`LogQuarter`.`attempts` - `LogQuarter`.`queue`) AS calc_attempts
And there you go.
I am not sure if you can do this Sum(F('attempts')-F('queue')). It should throw an error in the first place. I guess, easier approach would be to use extra.
result = LogQuarter.objects.extra(select={'sent':'(attempts-queue)'}, order_by=['-sent'])[:20]

Django db.connection.cursor.fetchone() returns unexpected result

I have this manager in my models.py
class ItemManager(models.Manager):
def get_fee(self):
from django.db import connection
cursor = connection.cursor()
cursor.execute("""
SELECT fee
FROM item
WHERE itemID = %d AND item.type = %d
""", [self.myItemID, self.myItemType])
fee = cursor.fetchone()
return fee
and class
Sample(models.Model):
sampleID = models.AutoField(primary_key=True)
itemID = models.ForeignKey(Item)
item.type = models.ForeignKey(Item)
...
def save(self, *args, **kwargs):
is_new = self.pk is None
super(Sample, self).save(*args, **kwargs)
if is_new:
cd.amount = MonthlyFeeManager()
cd.save()
Then it produces an error:
Cannot convert <myapp.models.ItemManager object at 0xa6f07ec> to Decimal
In general, i want to execute a RAW SQL query in a manager and use it to get the result from the query. I tried to search but most returns are tuples, not a value.
This is not how you use a manager. Even with a perfectly normal class instance, your attempt wouldn't give you what you wanted: you would need to instantiate it and call get_fee() on the instance.
With a manager, you don't need to instantiate it, because that's already done for you as the objects attribute on the model. But you still need to call the relevant method:
cd.amount = Sample.objects.get_fee()
However, this still won't work. That's because you've referred to self.myItemId and self.myItemType, which don't exist on the Manager object. Which leads me to the conclusion that you don't want a Manager object at all: you just want a standard model method. And there's no need for the raw SQL, either: your code is perfectly easily expressed using normal model query syntax.
(I can't show you exactly what it would look like, because the ForeignKeys in your example don't make any sense, and it's not clear where fee is supposed to be coming from.)

Django: Adding objects to a related set without saving to DB

I'm trying to write an internal API in my application without necessarily coupling it with the database.
class Product(models.Model):
name=models.CharField(max_length=4000)
price=models.IntegerField(default=-1)
currency=models.CharField(max_length=3, default='INR')
class Image(models.Model):
# NOTE -- Have changed the table name to products_images
width=models.IntegerField(default=-1)
height=models.IntegerField(default=-1)
url=models.URLField(max_length=1000, verify_exists=False)
product=models.ForeignKey(Product)
def create_product:
p=Product()
i=Image(height=100, widght=100, url='http://something/something')
p.image_set.add(i)
return p
Now, when I call create_product() Django throws up an error:
IntegrityError: products_images.product_id may not be NULL
However, if I call p.save() & i.save() before calling p.image_set.add(i) it works. Is there any way that I can add objects to a related object set without saving both to the DB first?
def create_product():
product_obj = Product.objects.create(name='Foobar')
image_obj = Image.objects.create(height=100, widght=100, url='http://something/something', product=product_obj)
return product_obj
Explanation:
Product object has to be created first and then assign it to the Image object because id and name here is required field.
I am wondering why wouldn't you not require to make a product entry in DB in first case? If there is any specific reason then i may suggest you some work around?
EDIT: Okay! i think i got you, you don't want to assign a product to an image object initially. How about creating a product field as null is equal to true.
product = models.ForeignKey(Product, null=True)
Now, your function becomes something like this:
def create_product():
image_obj = Image.objects.create(height=100, widght=100, url='http://something/something')
return image_obj
Hope it helps you?
I got same issue with #Saurabh Nanda
I am using Django 1.4.2. When I read in django, i see that
# file django/db/models/fields/related.py
def get_query_set(self):
try:
return self.instance._prefetched_objects_cache[rel_field.related_query_name()]
except (AttributeError, KeyError):
db = self._db or router.db_for_read(self.model, instance=self.instance)
return super(RelatedManager,self).get_query_set().using(db).filter(**self.core_filters)
# file django/db/models/query.py
qs = getattr(obj, attname).all()
qs._result_cache = vals
# We don't want the individual qs doing prefetch_related now, since we
# have merged this into the current work.
qs._prefetch_done = True
obj._prefetched_objects_cache[cache_name] = qs
That 's make sese, we only need to set property _prefetched_objects_cache for the object.
p = Product()
image_cached = []
for i in xrange(100):
image=Image(height=100, widght=100, url='http://something/something')
image_cached.append(image)
qs = p.images.all()
qs._result_cache = image_cached
qs._prefetch_done = True
p._prefetched_objects_cache = {'images': qs}
Your problem is that the id isn't set by django, but by the database (it's represented in the database by an auto-incremented field), so until it's saved there's no id. More about this in the documentation.
I can think of three possible solutions:
Set a different field of your Image model as the primary key (documented here).
Set a different field of your Production model as the foreign key (documented here).
Use django's database transactions API (documented here).