I need to ensure that both Model1 and Model2 save or neither of them save and I'm pretty sure I should be able to do this in a transaction, however it doesn't seem to be working.
I'm not sure if I am understanding how transactions work properly. I thought that the transaction would be good for the entirety of the function call that was wrapped in the decorator.
I'm also not sure if this is just an Sqlite problem too.
models.py
from django.db import models
class Model1( models.Model ):
name = models.CharField( max_length = 20 )
number = models.IntegerField( )
class Model2( models.Model ):
name = models.CharField( max_length = 20 )
number = models.IntegerField( )
parent = models.ForeignKey( Model1 )
Sample Transaction Code
from django.db import transaction
#transaction.commit_manually
def somefunction( name ):
try:
a = Model1.objects.get( name = name )
except Model1.DoesNotExist:
a = Model1( name = name )
try:
b = Mode2.objects.get( name = name)
except Model2.DoesNotExist:
b = Model2( name = name )
a.number = 3
b.number = 'a'
try:
a.save()
b.parent = a
b.save()
transaction.commit()
except:
transaction.rollback()
I have not tried MySQL or Postgres as I have not actually found anything specifically saying Sqlite cannot do this, so just want to find out if I am doing something wrong or if it is indeed Sqlite3.
So the final answer was that Transactions were working fine it was just that my Unit Tests were not inheriting from
https://docs.djangoproject.com/en/1.2/topics/testing/#django.test.TransactionTestCase
Since I was inheriting from just TestCase, every test_ method was wrapped in a transaction and I could not control .commit() or .rollback() behaviour.
This was just an utter failure on my part to realize how test cases were run.
Related
I'm trying to create a high score statistic table/list for a quiz, where the table/list is supposed to be showing the percentage of (or total) correct guesses on a person which was to be guessed on. To elaborate further, these are the models which are used.
The Quiz model:
class Quiz(models.Model):
participants = models.ManyToManyField(
User,
through="Participant",
through_fields=("quiz", "correct_user"),
blank=True,
related_name="related_quiz",
)
fake_users = models.ManyToManyField(User, related_name="quiz_fakes")
user_quizzed = models.ForeignKey(
User, related_name="user_taking_quiz", on_delete=models.CASCADE, null=True
)
time_started = models.DateTimeField(default=timezone.now)
time_end = models.DateTimeField(blank=True, null=True)
final_score = models.IntegerField(blank=True, default=0)
This model does also have some properties; I deem them to be unrelated to the problem at hand.
The Participant model:
class Participant(models.Model): # QuizAnswer FK -> QUIZ
guessed_user = models.ForeignKey(
User, on_delete=models.CASCADE, related_name="clicked_in_quiz", null=True
)
correct_user = models.ForeignKey(
User, on_delete=models.CASCADE, related_name="solution_in_quiz", null=True
)
quiz = models.ForeignKey(
Quiz, on_delete=models.CASCADE, related_name="participants_in_quiz"
)
#property
def correct(self):
return self.guessed_user == self.correct_user
To iterate through what I am trying to do, I'll try to explain how I'm thinking this should work:
For a User in User.objects.all(), find all participant objects where the user.id equals correct_user(from participant model)
For each participantobject, evaluate if correct_user==guessed_user
Sum each participant object where the above comparison is True for the User, represented by a field sum_of_correct_guesses
Return a queryset including all users with parameters [User, sum_of_correct_guesses]
^Now ideally this should be percentage_of_correct_guesses, but that is an afterthought which should be easy enough to change by doing sum_of_correct_guesses / sum n times of that person being a guess.
Now I've even made some pseudocode for a single person to illustrate to myself roughly how it should work using python arithmetics
# PYTHON PSEUDO QUERY ---------------------
person = get_object_or_404(User, pk=3) # Example-person
y = Participant.objects.filter(
correct_user=person
) # Find participant-objects where person is used as guess
y_corr = [] # empty list to act as "queryset" in for-loop
for el in y: # for each participant object
if el.correct: # if correct_user == guessed_user
y_corr.append(el) # add to queryset
y_percentage_corr = len(y_corr) / len(y) # do arithmetic division
print("Percentage correct: ", y_percentage_corr) # debug-display
# ---------------------------------------------
What I've tried (with no success so far), is to use an ExtensionWrapper with Count() and Q object:
percentage_correct_guesses = ExpressionWrapper(
Count("pk", filter=Q(clicked_in_quiz=F("id")), distinct=True)
/ Count("solution_in_quiz"),
output_field=fields.DecimalField())
all_users = (
User.objects.all().annotate(score=percentage_correct_guesses).order_by("score"))
Any help or directions to resources on how to do this is greatly appreciated :))
I found an answer while looking around for related problems:
Django 1.11 Annotating a Subquery Aggregate
What I've done is:
Create a filter with an OuterRef() which points to a User and checks if Useris the same as correct_person and also a comparison between guessed_person and correct_person, outputs a value correct_user in a queryset for all elements which the filter accepts.
Do an annotated count for how many occurrences there are of a correct_user in the filtered queryset.
Annotate User based on the annotated-count, this is the annotation that really drives the whole operation. Notice how OuterRef() and Subquery are used to tell the filter which user is supposed to be correct_user.
Below is the code snippet which I made it work with, it looks very similar to the answer-post in the above linked question:
from django.db.models import Count, OuterRef, Subquery, F, Q
crit1 = Q(correct_user=OuterRef('pk'))
crit2 = Q(correct_user=F('guessed_user'))
compare_participants = Participant.objects.filter(crit1 & crit2).order_by().values('correct_user')
count_occurrences = compare_participants.annotate(c=Count('*')).values('c')
most_correctly_guessed_on = (
User.objects.annotate(correct_clicks=Subquery(count_occurrences))
.values('first_name', 'correct_clicks')
.order_by('-correct_clicks')
)
return most_correctly_guessed_on
This works wonderfully, thanks to Oli.
Is there a simple way, using on_delete, to delete the object containing a ForeignKey when that ForeignKey is deleted (like models.CASCADE would normally do) but just the ForeignKey relationship to None if some condition is met (like models.SET_NULL would normally do).
Here is some code:
class SomeThing(models.Model):
user = models.ForeignKey(
settings.AUTH_USER_MODEL,
related_name="things",
on_delete=models.CONDITIONAL_CASCADE, # obviously this is wrong
)
is_private = models.BooleanField(default=False)
>>> user = User(username="bob").save()
>>> public_thing = SomeThing(is_private=False)
>>> private_thing = SomeThing(is_private=True)
>>> user.things.add(public_thing)
>>> user.things.add(private_thing)
>>> user.delete()
When user is deleted I would like private_thing to be deleted along w/ user, but public_thing to not be deleted and for public_thing.user to be set to None.
Thanks.
edit:
Thanks to #Jarad's suggestion, I wound up writing the following generic function:
def CONDITIONAL_CASCADE(collector, field, sub_objs, using, **kwargs):
condition = kwargs.get("condition", {})
default_value = kwargs.get("default_value", None)
sub_objs_to_cascade = sub_objs.filter(**condition)
sub_objs_to_set = sub_objs.exclude(**condition)
models.CASCADE(collector, field, sub_objs_to_cascade, using)
collector.add_field_update(field, default_value, sub_objs_to_set)
and using it like this:
CASCADE_PRIVATE_THINGS_ONLY = partial(
CONDITIONAL_CASCADE,
condition={"is_private": True},
default_value=None
)
Maybe I misunderstand the purpose of Django's update_or_create Model method.
Here is my Model:
from django.db import models
import datetime
from vc.models import Cluster
class Vmt(models.Model):
added = models.DateField(default=datetime.date.today, blank=True, null=True)
creation_time = models.TextField(blank=True, null=True)
current_pm_active = models.TextField(blank=True, null=True)
current_pm_total = models.TextField(blank=True, null=True)
... more simple fields ...
cluster = models.ForeignKey(Cluster, null=True)
class Meta:
unique_together = (("cluster", "added"),)
Here is my test:
from django.test import TestCase
from .models import *
from vc.models import Cluster
from django.db import transaction
# Create your tests here.
class VmtModelTests(TestCase):
def test_insert_into_VmtModel(self):
count = Vmt.objects.count()
self.assertEqual(count, 0)
# create a Cluster
c = Cluster.objects.create(name='test-cluster')
Vmt.objects.create(
cluster=c,
creation_time='test creaetion time',
current_pm_active=5,
current_pm_total=5,
... more simple fields ...
)
count = Vmt.objects.count()
self.assertEqual(count, 1)
self.assertEqual('5', c.vmt_set.all()[0].current_pm_active)
# let's test that we cannot add that same record again
try:
with transaction.atomic():
Vmt.objects.create(
cluster=c,
creation_time='test creaetion time',
current_pm_active=5,
current_pm_total=5,
... more simple fields ...
)
self.fail(msg="Should violated integrity constraint!")
except Exception as ex:
template = "An exception of type {0} occurred. Arguments:\n{1!r}"
message = template.format(type(ex).__name__, ex.args)
self.assertEqual("An exception of type IntegrityError occurred.", message[:45])
Vmt.objects.update_or_create(
cluster=c,
creation_time='test creaetion time',
# notice we are updating current_pm_active to 6
current_pm_active=6,
current_pm_total=5,
... more simple fields ...
)
count = Vmt.objects.count()
self.assertEqual(count, 1)
On the last update_or_create call I get this error:
IntegrityError: duplicate key value violates unique constraint "vmt_vmt_cluster_id_added_c2052322_uniq"
DETAIL: Key (cluster_id, added)=(1, 2018-06-18) already exists.
Why didn't wasn't the model updated? Why did Django try to create a new record that violated the unique constraint?
The update_or_create(defaults=None, **kwargs) has basically two parts:
the **kwargs which specify the "filter" criteria to determine if such object is already present; and
the defaults which is a dictionary that contains the fields mapped to values that should be used when we create a new row (in case the filtering fails to find a row), or which values should be updated (in case we find such row).
The problem here is that you make your filters too restrictive: you add several filters, and as a result the database does not find such row. So what happens? The database then aims to create the row with these filter values (and since defaults is missing, no extra values are added). But then it turns out that we create a row, and that the combination of the cluster and added already exists. Hence the database refuses to add this row.
So this line:
Model.objects.update_or_create(field1=val1,
field2=val2,
defaults={
'field3': val3,
'field4': val4
})
Is to semantically approximately equal to:
try:
item = Model.objects.get(field1=val1, field2=val2)
except Model.DoesNotExist:
Model.objects.create(field1=val1, field2=val2, field3=val3, field4=val4)
else:
item = Model.objects.filter(
field1=val1,
field2=val2,
).update(
field3 = val3
field4 = val4
)
(but the original call is typically done in a single query).
You probably thus should write:
Vmt.objects.update_or_create(
cluster=c,
creation_time='test creaetion time',
defaults = {
'current_pm_active': 6,
'current_pm_total': 5,
}
)
(or something similar)
You should separate your field:
Fields that should be searched for
Fields that should be updated
for example:
If I have the model:
class User(models.Model):
username = models.CharField(max_length=200)
nickname = models.CharField(max_length=200)
And I want to search for username = 'Nikolas' and update this instance nickname to 'Nik'(if no User with username 'Nikolas' I need to create it) I should write this code:
User.objects.update_or_create(
username='Nikolas',
defaults={'nickname': 'Nik'},
)
see in https://docs.djangoproject.com/en/3.1/ref/models/querysets/
This is already answered well in the above.
To be more clear the update_or_create() method should have **kwargs as those parameters on which you want to check if that data already exists in DB by filtering.
select some_column from table_name where column1='' and column2='';
Filtering by **kwargs will give you objects. Now if you wish to update any data/column of those filtered objects, you should pass them in defaults param in update_or_create() method.
so lets say you found an object based on a filter now the default param values are expected to be picked and updated.
and if there's no matching object found based on the filter then it goes ahead and creates an entry with filters and the default param passed.
I am running into a database issue in my unit tests. I think it has something to do with the way I am using TestCase and setUpData.
When I try to set up my test data with certain values, the tests throw the following error:
django.db.utils.IntegrityError: duplicate key value violates unique constraint
...
psycopg2.IntegrityError: duplicate key value violates unique constraint "InventoryLogs_productgroup_product_name_48ec6f8d_uniq"
DETAIL: Key (product_name)=(Almonds) already exists.
I changed all of my primary keys and it seems to be running fine. It doesn't seem to affect any of the tests.
However, I'm concerned that I am doing something wrong. When it first happened, I reversed about an hour's worth of work on my app (not that much code for a noob), which corrected the problem.
Then when I wrote the changes back in, the same issue presented itself again. TestCase is pasted below. The issue seems to occur after I add the sortrecord items, but corresponds with the items above it.
I don't want to keep going through and changing primary keys and urls in my tests, so if anyone sees something wrong with the way I am using this, please help me out. Thanks!
TestCase
class DetailsPageTest(TestCase):
#classmethod
def setUpTestData(cls):
cls.product1 = ProductGroup.objects.create(
product_name="Almonds"
)
cls.variety1 = Variety.objects.create(
product_group = cls.product1,
variety_name = "non pareil",
husked = False,
finished = False,
)
cls.supplier1 = Supplier.objects.create(
company_name = "Acme",
company_location = "Acme Acres",
contact_info = "Call me!"
)
cls.shipment1 = Purchase.objects.create(
tag=9,
shipment_id=9999,
supplier_id = cls.supplier1,
purchase_date='2015-01-09',
purchase_price=9.99,
product_name=cls.variety1,
pieces=99,
kgs=999,
crackout_estimate=99.9
)
cls.shipment2 = Purchase.objects.create(
tag=8,
shipment_id=8888,
supplier_id=cls.supplier1,
purchase_date='2015-01-08',
purchase_price=8.88,
product_name=cls.variety1,
pieces=88,
kgs=888,
crackout_estimate=88.8
)
cls.shipment3 = Purchase.objects.create(
tag=7,
shipment_id=7777,
supplier_id=cls.supplier1,
purchase_date='2014-01-07',
purchase_price=7.77,
product_name=cls.variety1,
pieces=77,
kgs=777,
crackout_estimate=77.7
)
cls.sortrecord1 = SortingRecords.objects.create(
tag=cls.shipment1,
date="2015-02-05",
bags_sorted=20,
turnout=199,
)
cls.sortrecord2 = SortingRecords.objects.create(
tag=cls.shipment1,
date="2015-02-07",
bags_sorted=40,
turnout=399,
)
cls.sortrecord3 = SortingRecords.objects.create(
tag=cls.shipment1,
date='2015-02-09',
bags_sorted=30,
turnout=299,
)
Models
from datetime import datetime
from django.db import models
from django.db.models import Q
class ProductGroup(models.Model):
product_name = models.CharField(max_length=140, primary_key=True)
def __str__(self):
return self.product_name
class Meta:
verbose_name = "Product"
class Supplier(models.Model):
company_name = models.CharField(max_length=45)
company_location = models.CharField(max_length=45)
contact_info = models.CharField(max_length=256)
class Meta:
ordering = ["company_name"]
def __str__(self):
return self.company_name
class Variety(models.Model):
product_group = models.ForeignKey(ProductGroup)
variety_name = models.CharField(max_length=140)
husked = models.BooleanField()
finished = models.BooleanField()
description = models.CharField(max_length=500, blank=True)
class Meta:
ordering = ["product_group_id"]
verbose_name_plural = "Varieties"
def __str__(self):
return self.variety_name
class PurchaseYears(models.Manager):
def purchase_years_list(self):
unique_years = Purchase.objects.dates('purchase_date', 'year')
results_list = []
for p in unique_years:
results_list.append(p.year)
return results_list
class Purchase(models.Model):
tag = models.IntegerField(primary_key=True)
product_name = models.ForeignKey(Variety, related_name='purchases')
shipment_id = models.CharField(max_length=24)
supplier_id = models.ForeignKey(Supplier)
purchase_date = models.DateField()
estimated_delivery = models.DateField(null=True, blank=True)
purchase_price = models.DecimalField(max_digits=6, decimal_places=3)
pieces = models.IntegerField()
kgs = models.IntegerField()
crackout_estimate = models.DecimalField(max_digits=6,decimal_places=3, null=True)
crackout_actual = models.DecimalField(max_digits=6,decimal_places=3, null=True)
objects = models.Manager()
purchase_years = PurchaseYears()
# Keep manager as "objects" in case admin, etc. needs it. Filter can be called like so:
# Purchase.objects.purchase_years_list()
# Managers in docs: https://docs.djangoproject.com/en/1.8/intro/tutorial01/
class Meta:
ordering = ["purchase_date"]
def __str__(self):
return self.shipment_id
def _weight_conversion(self):
return round(self.kgs * 2.20462)
lbs = property(_weight_conversion)
class SortingModelsBagsCalulator(models.Manager):
def total_sorted(self, record_date, current_set):
sorted = [SortingRecords['bags_sorted'] for SortingRecords in current_set if
SortingRecords['date'] <= record_date]
return sum(sorted)
class SortingRecords(models.Model):
tag = models.ForeignKey(Purchase, related_name='sorting_record')
date = models.DateField()
bags_sorted = models.IntegerField()
turnout = models.IntegerField()
objects = models.Manager()
def __str__(self):
return "%s [%s]" % (self.date, self.tag.tag)
class Meta:
ordering = ["date"]
verbose_name_plural = "Sorting Records"
def _calculate_kgs_sorted(self):
kg_per_bag = self.tag.kgs / self.tag.pieces
kgs_sorted = kg_per_bag * self.bags_sorted
return (round(kgs_sorted, 2))
kgs_sorted = property(_calculate_kgs_sorted)
def _byproduct(self):
waste = self.kgs_sorted - self.turnout
return (round(waste, 2))
byproduct = property(_byproduct)
def _bags_remaining(self):
current_set = SortingRecords.objects.values().filter(~Q(id=self.id), tag=self.tag)
sorted = [SortingRecords['bags_sorted'] for SortingRecords in current_set if
SortingRecords['date'] <= self.date]
remaining = self.tag.pieces - sum(sorted) - self.bags_sorted
return remaining
bags_remaining = property(_bags_remaining)
EDIT
It also fails with integers, like so.
django.db.utils.IntegrityError: duplicate key value violates unique constraint "InventoryLogs_purchase_pkey"
DETAIL: Key (tag)=(9) already exists.
UDPATE
So I should have mentioned this earlier, but I completely forgot. I have two unit test files that use the same data. Just for kicks, I matched a primary key in both instances of setUpTestData() to a different value and sure enough, I got the same error.
These two setups were working fine side-by-side before I added more data to one of them. Now, it appears that they need different values. I guess you can only get away with using repeat data for so long.
I continued to get this error without having any duplicate data but I was able to resolve the issue by initializing the object and calling the save() method rather than creating the object via Model.objects.create()
In other words, I did this:
#classmethod
def setUpTestData(cls):
cls.person = Person(first_name="Jane", last_name="Doe")
cls.person.save()
Instead of this:
#classmethod
def setUpTestData(cls):
cls.person = Person.objects.create(first_name="Jane", last_name="Doe")
I've been running into this issue sporadically for months now. I believe I just figured out the root cause and a couple solutions.
Summary
For whatever reason, it seems like the Django test case base classes aren't removing the database records created by let's just call it TestCase1 before running TestCase2. Which, in TestCase2 when it tries to create records in the database using the same IDs as TestCase1 the database raises a DuplicateKey exception because those IDs already exists in the database. And even saying the magic word "please" won't help with database duplicate key errors.
Good news is, there are multiple ways to solve this problem! Here are a couple...
Solution 1
Make sure if you are overriding the class method tearDownClass that you call super().tearDownClass(). If you override tearDownClass() without calling its super, it will in turn never call TransactionTestCase._post_teardown() nor TransactionTestCase._fixture_teardown(). Quoting from the doc string in TransactionTestCase._post_teardown()`:
def _post_teardown(self):
"""
Perform post-test things:
* Flush the contents of the database to leave a clean slate. If the
class has an 'available_apps' attribute, don't fire post_migrate.
* Force-close the connection so the next test gets a clean cursor.
"""
If TestCase.tearDownClass() is not called via super() then the database is not reset in between test cases and you will get the dreaded duplicate key exception.
Solution 2
Override TransactionTestCase and set the class variable serialized_rollback = True, like this:
class MyTestCase(TransactionTestCase):
fixtures = ['test-data.json']
serialized_rollback = True
def test_name_goes_here(self):
pass
Quoting from the source:
class TransactionTestCase(SimpleTestCase):
...
# If transactions aren't available, Django will serialize the database
# contents into a fixture during setup and flush and reload them
# during teardown (as flush does not restore data from migrations).
# This can be slow; this flag allows enabling on a per-case basis.
serialized_rollback = False
When serialized_rollback is set to True, Django test runner rolls back any transactions inserted into the database beween test cases. And batta bing, batta bang... no more duplicate key errors!
Conclusion
There are probably many more ways to implement a solution for the OP's issue, but these two should work nicely. Would definitely love to have more solutions added by others for clarity sake and a deeper understanding of the underlying Django test case base classes. Phew, say that last line real fast three times and you could win a pony!
The log you provided states DETAIL: Key (product_name)=(Almonds) already exists. Did you verify in your db?
To prevent such errors in the future, you should prefix all your test data string by test_
I discovered the issue, as noted at the bottom of the question.
From what I can tell, the database didn't like me using duplicate data in the setUpTestData() methods of two different tests. Changing the primary key values in the second test corrected the problem.
I think the problem here is that you had a tearDownClass method in your TestCase without the call to super method.
In this way the django TestCase lost the transactional functionalities behind the setUpTestData so it doesn't clean your test db after a TestCase is finished.
Check warning in django docs here:
https://docs.djangoproject.com/en/1.10/topics/testing/tools/#django.test.SimpleTestCase.allow_database_queries
I had similar problem that had been caused by providing the primary key value to a test case explicitly.
As discussed in the Django documentation, manually assigning a value to an auto-incrementing field doesn’t update the field’s sequence, which might later cause a conflict.
I have solved it by altering the sequence manually:
from django.db import connection
class MyTestCase(TestCase):
#classmethod
def setUpTestData(cls):
Model.objects.create(id=1)
with connection.cursor() as c:
c.execute(
"""
ALTER SEQUENCE "app_model_id_seq" RESTART WITH 2;
"""
)
I am trying to build a messaging application with Django. The reason I don’t use postman is that I need messaging between other objects than users and I don’t need most of the postman’s features.
Here are my models:
class Recipient(models.Model):
...
def get_unread_threads():
see below...
class Thread(models.Model):
author = models.ForeignKey(Recipient, related_name='initiated_threads')
recipients = models.ManyToManyField(
Tribe,
related_name='received_thread',
through='ThreadReading')
subject = models.CharField(max_length=64)
class Meta:
app_label = 'game'
class Message(models.Model):
author = models.ForeignKey(Recipient, related_name='sent_messages')
date_add = models.DateTimeField(auto_now_add=True)
date_edit = models.DateTimeField(blank=True)
content = models.TextField(max_length=65535)
thread = models.ForeignKey(Thread)
class Meta:
get_latest_by = 'date'
class ThreadReading(models.Model):
thread = models.ForeignKey(Thread)
recipient = models.ForeignKey(Recipient)
date_last_reading = models.DateTimeField(auto_now=True)
My problem is about get_unread_threads. I can’t really find out how to do that. Here is a first try:
def get_unread_threads(self):
"""
return a queryset corresponding to the threads
which at least one message is unread by the recipient
"""
try:
query = self.received_thread.filter(
message__latest__date_add__gt=\
self.threadreading_set.get(thread_id=F('id')).date_last_reading)
except ObjectDoesNotExist:
query = None
return query
But obviously it doesn’t work because lookup can’t follow method latest.
Here you go:
# Get all the readings of the user
thread_readings = recipient.threadreading_set.all()
# Build a query object including all messages who's last reading is after the
# last edit that was made AND whose thread is the thread of the current
# iteration's thread reading
q = models.Q()
for thread_reading in thread_readings:
q = q | models.Q(
models.Q(
date_edit__lte=thread_reading.date_last_reading
& models.Q(
thread=thread_reading.thread
)
)
)
# Get a queryset of all the messages, including the threads (via a JOIN)
queryset = Message.objects.select_related('thread')
# Now, exclude from the queryset every message that matches the above query
# (edited prior to last reading) OR that is in the list of last readings
queryset = queryset.exclude(
q | models.Q(
thread__pk__in=[thread_reading.pk for thread_reading in thread_readings]
)
)
# Make an iterator (to pretend that this is efficient) and return a generator of it
iterator = queryset.iterator()
return (message.thread for message in iterator)
:)
Now, don't ever actually do this - rethink your models. I would read a book called "Object Oriented Analysis and Design with Applications". It'll teach you a great deal about how to think when you're data modelling.