There are some constraints to this problem. We currently use a production database, with live Virtual Machine Statistics. We are trying to create a django interface, that interfaces with the tables we want our administrators to be able to edit. Thus, migrations are out of the question, for unless I have come to understand migrations wrong it will affect the current database structure and or data.
I matched the database structure exactly in my models.py file. However I have run into a few issues. One of the issues I have run into is when I try to add a new item under the admin control panel it will give me an integrity error as it is attempting to insert a null value for the field I have set as the primary key in the models.py file.
We are currently using an oracle database.
My Models.py not all of it but a sample of it.
class License(models.Model):
license_id = models.AutoField(primary_key = True, editable = False, db_column='license_id')
license_authority_id = models.ForeignKey(License_authoritie, on_delete = models.PROTECT, db_column='license_authority_id')
product = models.CharField(max_length = 20)
class Meta:
managed = False
db_table = 'licenses'
ordering = ['product']
def __unicode__(self): # Python 3: def __str__(self):
return self.product
class Vm_license(models.Model):
vm_license_id = models.AutoField(primary_key = True, db_column='vm_license_id')
vm_id = models.ForeignKey(Vm, on_delete = models.PROTECT, db_column='vm_id')
license = models.ManyToManyField(License)
class Meta:
managed = False
db_table = 'vm_licenses'
The error I get:
Request Method: POST
Request URL: http://127.0.0.1:8000/admin/portal/vm_license/add/
Django Version: 1.6.5
Exception Type: IntegrityError
Exception Value:
ORA-01400: cannot insert NULL into ("DEV"."VM_LICENSES"."VM_LICENSE_ID")
On top of that I have run into another problem.
For these two tables, under the vm_licenses section in the admin panel which is a table that holds all VM's and their assigned licenses. I need the ability to select multiple licenses at a time for each vm_id under the add section of the admin panel but i'm not quite sure how to do this.
admin.py code
class vm_license_admin(admin.ModelAdmin):
#list_display = ('vm_id', 'license_id')
list_display = ('vm_id',)
search_fields = ('vm_id__vm_name',)
ordering = ('vm_id',)
filter_horizontal = ('license',)
admin.site.register(Vm_license, vm_license_admin)
I also made an oracle trigger to auto increment a primary key if there is none, but im still getting the same error.
CREATE OR REPLACE TRIGGER license_trigger
BEFORE INSERT ON vm_licenses
FOR EACH ROW
BEGIN
SELECT vm_license_seq.nextval
INTO :new.vm_license_id
FROM dual;
END;
to be more percise I am using a manytomany field and it displays correctly when I goto add a new item before clicking save and getting the null error, however if I goto an existing item it will say table or view doesnt exist.
I was going to comment on your question, but I do not have the reputation yet...
but can I suggest you post your relevant admin.py code? Perhaps there is something within it relating to the Null PK error.
With regards to the second part, a ManyToManyField sounds more suitable.
Related
I was using django-hitcont to count the views on my Post model. I am trying to get the most viewed post in my ListView using this query objects.order_by('hit_count_generic__hits') and it is working fine on SQLite but on PostgreSQL, it is giving me this error :
django.db.utils.ProgrammingError: operator does not exist: integer = text LINE 1: ...R JOIN "hitcount_hit_count" ON ("posts_post"."id" = "hitcoun....
models.py
class Post(models.Model, HitCountMixin):
author = models.ForeignKey(User, related_name='authors', on_delete=models.CASCADE)
title = models.CharField('Post Title', max_length = 150)
description = models.TextField('Description', max_length=1000, blank = True)
date_posted = models.DateTimeField('Date posted', default = timezone.now)
date_modifed = models.DateTimeField('Date last modified', default = timezone.now)
document = models.FileField('Document of Post', upload_to='documents', \
validators=[FileExtensionValidator(allowed_extensions = ['pdf', 'docx']), validate_document_size] \
)
hit_count_generic = GenericRelation(
HitCount,
object_id_field='object_pk',
related_query_name='hit_count_generic_relation'
)
views.py
queryset = Post.objects.order_by('hit_count_generic__hits')
I found this issue on Github related to the problem, but I am still not able to figure out the mentioned workaround.
When comparing different types (in this example integer and text), equals operator throws this exception. To fix that, convert HitCount model pk field to integer and you are good to go. To do that, you need to create and apply migration operation. Django is a really good framework to handle this kind of operations. You just need to check values are not null and are "convertable" to integer. Just change the field type and run two commands below.
python manage.py makemigrations
python manage.py migrate
Before updating your model, I highly recommend you to take a backup in case of failure. This is not an easy operation but you can follow the these links to understand what is going on during this the process.
migrations dump and restore initial data
If you don't care the data on table, just drop table and create a brand new migration file and recreate table.
class PurchaseOrder(models.Model):
purchase_order_id = models.AutoField(primary_key=True)
purchase_order_number = models.CharField(unique=True)
vendor = models.ForeignKey(Vendor)
i am creating Purchase Order(po) table. when po created i have to update purchase_order_number as "PO0"+purchase_order_id ex PO0123 (123 is Primary key). so i am using def save in models to accomplish this
def save(self):
if self.purchase_order_id is not None:
self.purchase_order_number = "PO"+str(self.purchase_order_id)
return super(PurchaseOrder, self).save()
It is working fine with single creation but when i try to create bulk of data using locust(Testing tool) its giving an error duplicate entry for PurchseOrdernumber Can we modify field value in models itself some thing like this
purchase_order_number = models.CharField(unique=True,default=("PO"+self.purchase_order_id )
To be honest, I don't think it should work when you create multiple instances. Because as I can see from the code:
if self.purchase_order_id is not None:
self.purchase_order_number = "PO"+str(self.purchase_order_id)
Here purchase_order_id will be None when you are creating new instance. Also, until you call super(PurchaseOrder, self).save(), it will not generate purchase_order_id, meaning purchase_order_number will be empty.
So, what I would recommend is to not store this information in DB. Its basically the same as purchase_order_id with PO in front of it. Instead you can use a property method to get the same value. Like this:
class PurchaseOrder(models.Model):
purchase_order_id = models.AutoField(primary_key=True)
# need to remove `purchase_order_number = models.CharField(unique=True)`
...
#property
def purchase_order_number(self):
return "PO{}".format(self.purchase_order_id)
So, you can also see the purchase_order_number like this:
p = PurchaseOrder.objects.first()
p.purchase_order_number
Downside of this solution is that, you can't make any query on the property field. But I don't think it would be necessary anyway, because you can do the same query for the purchase_order_id, ie PurchaseOrder.objects.filter(purchase_order_id=1).
I am running into a database issue in my unit tests. I think it has something to do with the way I am using TestCase and setUpData.
When I try to set up my test data with certain values, the tests throw the following error:
django.db.utils.IntegrityError: duplicate key value violates unique constraint
...
psycopg2.IntegrityError: duplicate key value violates unique constraint "InventoryLogs_productgroup_product_name_48ec6f8d_uniq"
DETAIL: Key (product_name)=(Almonds) already exists.
I changed all of my primary keys and it seems to be running fine. It doesn't seem to affect any of the tests.
However, I'm concerned that I am doing something wrong. When it first happened, I reversed about an hour's worth of work on my app (not that much code for a noob), which corrected the problem.
Then when I wrote the changes back in, the same issue presented itself again. TestCase is pasted below. The issue seems to occur after I add the sortrecord items, but corresponds with the items above it.
I don't want to keep going through and changing primary keys and urls in my tests, so if anyone sees something wrong with the way I am using this, please help me out. Thanks!
TestCase
class DetailsPageTest(TestCase):
#classmethod
def setUpTestData(cls):
cls.product1 = ProductGroup.objects.create(
product_name="Almonds"
)
cls.variety1 = Variety.objects.create(
product_group = cls.product1,
variety_name = "non pareil",
husked = False,
finished = False,
)
cls.supplier1 = Supplier.objects.create(
company_name = "Acme",
company_location = "Acme Acres",
contact_info = "Call me!"
)
cls.shipment1 = Purchase.objects.create(
tag=9,
shipment_id=9999,
supplier_id = cls.supplier1,
purchase_date='2015-01-09',
purchase_price=9.99,
product_name=cls.variety1,
pieces=99,
kgs=999,
crackout_estimate=99.9
)
cls.shipment2 = Purchase.objects.create(
tag=8,
shipment_id=8888,
supplier_id=cls.supplier1,
purchase_date='2015-01-08',
purchase_price=8.88,
product_name=cls.variety1,
pieces=88,
kgs=888,
crackout_estimate=88.8
)
cls.shipment3 = Purchase.objects.create(
tag=7,
shipment_id=7777,
supplier_id=cls.supplier1,
purchase_date='2014-01-07',
purchase_price=7.77,
product_name=cls.variety1,
pieces=77,
kgs=777,
crackout_estimate=77.7
)
cls.sortrecord1 = SortingRecords.objects.create(
tag=cls.shipment1,
date="2015-02-05",
bags_sorted=20,
turnout=199,
)
cls.sortrecord2 = SortingRecords.objects.create(
tag=cls.shipment1,
date="2015-02-07",
bags_sorted=40,
turnout=399,
)
cls.sortrecord3 = SortingRecords.objects.create(
tag=cls.shipment1,
date='2015-02-09',
bags_sorted=30,
turnout=299,
)
Models
from datetime import datetime
from django.db import models
from django.db.models import Q
class ProductGroup(models.Model):
product_name = models.CharField(max_length=140, primary_key=True)
def __str__(self):
return self.product_name
class Meta:
verbose_name = "Product"
class Supplier(models.Model):
company_name = models.CharField(max_length=45)
company_location = models.CharField(max_length=45)
contact_info = models.CharField(max_length=256)
class Meta:
ordering = ["company_name"]
def __str__(self):
return self.company_name
class Variety(models.Model):
product_group = models.ForeignKey(ProductGroup)
variety_name = models.CharField(max_length=140)
husked = models.BooleanField()
finished = models.BooleanField()
description = models.CharField(max_length=500, blank=True)
class Meta:
ordering = ["product_group_id"]
verbose_name_plural = "Varieties"
def __str__(self):
return self.variety_name
class PurchaseYears(models.Manager):
def purchase_years_list(self):
unique_years = Purchase.objects.dates('purchase_date', 'year')
results_list = []
for p in unique_years:
results_list.append(p.year)
return results_list
class Purchase(models.Model):
tag = models.IntegerField(primary_key=True)
product_name = models.ForeignKey(Variety, related_name='purchases')
shipment_id = models.CharField(max_length=24)
supplier_id = models.ForeignKey(Supplier)
purchase_date = models.DateField()
estimated_delivery = models.DateField(null=True, blank=True)
purchase_price = models.DecimalField(max_digits=6, decimal_places=3)
pieces = models.IntegerField()
kgs = models.IntegerField()
crackout_estimate = models.DecimalField(max_digits=6,decimal_places=3, null=True)
crackout_actual = models.DecimalField(max_digits=6,decimal_places=3, null=True)
objects = models.Manager()
purchase_years = PurchaseYears()
# Keep manager as "objects" in case admin, etc. needs it. Filter can be called like so:
# Purchase.objects.purchase_years_list()
# Managers in docs: https://docs.djangoproject.com/en/1.8/intro/tutorial01/
class Meta:
ordering = ["purchase_date"]
def __str__(self):
return self.shipment_id
def _weight_conversion(self):
return round(self.kgs * 2.20462)
lbs = property(_weight_conversion)
class SortingModelsBagsCalulator(models.Manager):
def total_sorted(self, record_date, current_set):
sorted = [SortingRecords['bags_sorted'] for SortingRecords in current_set if
SortingRecords['date'] <= record_date]
return sum(sorted)
class SortingRecords(models.Model):
tag = models.ForeignKey(Purchase, related_name='sorting_record')
date = models.DateField()
bags_sorted = models.IntegerField()
turnout = models.IntegerField()
objects = models.Manager()
def __str__(self):
return "%s [%s]" % (self.date, self.tag.tag)
class Meta:
ordering = ["date"]
verbose_name_plural = "Sorting Records"
def _calculate_kgs_sorted(self):
kg_per_bag = self.tag.kgs / self.tag.pieces
kgs_sorted = kg_per_bag * self.bags_sorted
return (round(kgs_sorted, 2))
kgs_sorted = property(_calculate_kgs_sorted)
def _byproduct(self):
waste = self.kgs_sorted - self.turnout
return (round(waste, 2))
byproduct = property(_byproduct)
def _bags_remaining(self):
current_set = SortingRecords.objects.values().filter(~Q(id=self.id), tag=self.tag)
sorted = [SortingRecords['bags_sorted'] for SortingRecords in current_set if
SortingRecords['date'] <= self.date]
remaining = self.tag.pieces - sum(sorted) - self.bags_sorted
return remaining
bags_remaining = property(_bags_remaining)
EDIT
It also fails with integers, like so.
django.db.utils.IntegrityError: duplicate key value violates unique constraint "InventoryLogs_purchase_pkey"
DETAIL: Key (tag)=(9) already exists.
UDPATE
So I should have mentioned this earlier, but I completely forgot. I have two unit test files that use the same data. Just for kicks, I matched a primary key in both instances of setUpTestData() to a different value and sure enough, I got the same error.
These two setups were working fine side-by-side before I added more data to one of them. Now, it appears that they need different values. I guess you can only get away with using repeat data for so long.
I continued to get this error without having any duplicate data but I was able to resolve the issue by initializing the object and calling the save() method rather than creating the object via Model.objects.create()
In other words, I did this:
#classmethod
def setUpTestData(cls):
cls.person = Person(first_name="Jane", last_name="Doe")
cls.person.save()
Instead of this:
#classmethod
def setUpTestData(cls):
cls.person = Person.objects.create(first_name="Jane", last_name="Doe")
I've been running into this issue sporadically for months now. I believe I just figured out the root cause and a couple solutions.
Summary
For whatever reason, it seems like the Django test case base classes aren't removing the database records created by let's just call it TestCase1 before running TestCase2. Which, in TestCase2 when it tries to create records in the database using the same IDs as TestCase1 the database raises a DuplicateKey exception because those IDs already exists in the database. And even saying the magic word "please" won't help with database duplicate key errors.
Good news is, there are multiple ways to solve this problem! Here are a couple...
Solution 1
Make sure if you are overriding the class method tearDownClass that you call super().tearDownClass(). If you override tearDownClass() without calling its super, it will in turn never call TransactionTestCase._post_teardown() nor TransactionTestCase._fixture_teardown(). Quoting from the doc string in TransactionTestCase._post_teardown()`:
def _post_teardown(self):
"""
Perform post-test things:
* Flush the contents of the database to leave a clean slate. If the
class has an 'available_apps' attribute, don't fire post_migrate.
* Force-close the connection so the next test gets a clean cursor.
"""
If TestCase.tearDownClass() is not called via super() then the database is not reset in between test cases and you will get the dreaded duplicate key exception.
Solution 2
Override TransactionTestCase and set the class variable serialized_rollback = True, like this:
class MyTestCase(TransactionTestCase):
fixtures = ['test-data.json']
serialized_rollback = True
def test_name_goes_here(self):
pass
Quoting from the source:
class TransactionTestCase(SimpleTestCase):
...
# If transactions aren't available, Django will serialize the database
# contents into a fixture during setup and flush and reload them
# during teardown (as flush does not restore data from migrations).
# This can be slow; this flag allows enabling on a per-case basis.
serialized_rollback = False
When serialized_rollback is set to True, Django test runner rolls back any transactions inserted into the database beween test cases. And batta bing, batta bang... no more duplicate key errors!
Conclusion
There are probably many more ways to implement a solution for the OP's issue, but these two should work nicely. Would definitely love to have more solutions added by others for clarity sake and a deeper understanding of the underlying Django test case base classes. Phew, say that last line real fast three times and you could win a pony!
The log you provided states DETAIL: Key (product_name)=(Almonds) already exists. Did you verify in your db?
To prevent such errors in the future, you should prefix all your test data string by test_
I discovered the issue, as noted at the bottom of the question.
From what I can tell, the database didn't like me using duplicate data in the setUpTestData() methods of two different tests. Changing the primary key values in the second test corrected the problem.
I think the problem here is that you had a tearDownClass method in your TestCase without the call to super method.
In this way the django TestCase lost the transactional functionalities behind the setUpTestData so it doesn't clean your test db after a TestCase is finished.
Check warning in django docs here:
https://docs.djangoproject.com/en/1.10/topics/testing/tools/#django.test.SimpleTestCase.allow_database_queries
I had similar problem that had been caused by providing the primary key value to a test case explicitly.
As discussed in the Django documentation, manually assigning a value to an auto-incrementing field doesn’t update the field’s sequence, which might later cause a conflict.
I have solved it by altering the sequence manually:
from django.db import connection
class MyTestCase(TestCase):
#classmethod
def setUpTestData(cls):
Model.objects.create(id=1)
with connection.cursor() as c:
c.execute(
"""
ALTER SEQUENCE "app_model_id_seq" RESTART WITH 2;
"""
)
I need to skip validation of the OrderAmount field but for it to still save the invalidated data. Is there a way this can be done? I know django allows you to make your own validation, but I don't know how to make it completely skip just one field's validation.
model:
class LiquorOrder(models.Model):
pack_size = (
('1', '1'),
('2', '2'),
)
LiquorOrderID = models.AutoField(primary_key=True)
storeliquorID = models.ForeignKey(StoreLiquor)
orderID = models.ForeignKey(Order)
OrderAmount = models.PositiveSmallIntegerField('Order Amount', max_length=3, choices=pack_size)
TotalPrice = models.DecimalField('Total Price', max_digits=5, decimal_places=2)
Form:
class AddToOrderForm(forms.ModelForm):
class Meta:
model = LiquorOrder
fields = ('OrderAmount',)
For a PositiveSmallIntegerField the only validation Django does is ensure the value is indeed a positive integer within the appropriate range. If you were to skip this, you would run into problems when Django tries to write the value to your database. If you were to, say, try to write the value "marshmallows" to a DB column that's expecting an integer, the DB will throw errors and Django will turn around and throw you an IntegrityError.
If you really wanted to try, you could override the field to be CharField with required set to False (basically allowing any keyboard input):
class AddToOrderForm(forms.ModelForm):
OrderAmount = forms.CharField(required=False)
Django would then return True when you run is_valid(), but throw errors when you try to call save().
It sounds like your real issue is with your model not matching your current project requirements. If that is the case, look into migrating to a new model. The Python library South is brilliant tool for this purpose and is used heavily by the Django community. I would read up on DB migrations and see if you can come up with a solution that way.
Thanks for taking time to read my question.
I have a django app with the following model:
class UserProfile(models.Model):
user = models.OneToOneField(User)
...
class Visit(models.Model):
profile = models.ForeignKey(UserProfile)
date = models.DateField(auto_now_add=True, db_index=True)
ip = models.IPAddressField()
class Meta:
unique_together = ('profile', 'date', 'ip')
In a view:
profile = get_object_or_404(Profile, pk = ...)
get, create = Visit.objects.get_or_create(profile=profile, date=now.date(), ip=request.META['REMOTE_ADDR'])
if create: DO SOMETHING
Everything works fine, except that the Postgres Logs are full with duplicate key errors:
2012-02-15 14:13:44 CET ERROR: duplicate key value violates unique constraint "table_visit_profile_id_key"
2012-02-15 14:13:44 CET STATEMENT: INSERT INTO "table_visit" ("profile_id", "date", "ip") VALUES (1111, E'2012-02-15', E'xx.xx.xxx.xxx') RETURNING "table_visit"."id"
Tried different solution e.g.
from django.db import transaction
from django.db import IntegrityError
#transaction.commit_on_success
def my_get_or_create(prof, ip):
try:
object = Visit.objects.create(profile=prof, date=datetime.now().date(), ip=ip)
except IntegrityError:
transaction.commit()
object = Visit.objects.get(profile=prof, date=datetime.now().date(), ip=ip)
return object
....
created = my_get_or_create(prof, request.META['REMOTE_ADDR'])
if created: DO SOMETHING
This only helps for MySQL? Does anyone know how to avaid the duplicate key value errors for postgres?
Another possible reason for these errors in get_or_create() is data type mismatch in one of the search fields - for example passing False instead of None into a nullable field. The .get() inside .get_or_create() will not find it and Django will continue with new row creation - which will fail due to PostgreSQL constraints.
I had issues with get_or_create when using postgres. In the end I abandoned the boilerplate code for traditional:
try:
jobInvite = Invite.objects.get(sender=employer.user, job=job)
except Invite.DoesNotExist:
jobInvite = Invite(sender=employer.user, job=job)
jobInvite.save()
# end try
Have you at some point had unique=True set on Visit's profile field?
It looks like there's been a unique constraint generated for postgres that's still in effect. "table_visit_profile_id_key" is what it's auto generated name would be, and naturally it would cause those errors if you're recording multiple visits for a user.
If this is the case, are you using South to manage your database changes? If you aren't, grab it!
PostgreSQL behaves somewhat differently in some subtle queries, which results in IntegrityError errors, especially after you switch to Django 1.6. Here's the solution - you need to add select_on_save option to each failing model:
class MyModel(models.Model):
...
class Meta:
select_on_save = True
It's documented here: Options.select_on_save