django.db.transaction.TransactionManagementError: cannot perform saving of other object in model within transaction - django

Can't seem to find much info about this. This is NOT happening in a django test. I'm using DATABASES = { ATOMIC_REQUESTS: True }. Within a method (in mixin I created) called by the view, I'm trying to perform something like this:
def process_valid(self, view):
old_id = view.object.id
view.object.id = None # need a new instance in db
view.object.save()
old_fac = Entfac.objects.get(id=old_id)
new_fac = view.object
old_dets = Detfac.objects.filter(fk_ent__id__exact = old_fac.id)
new_formset = view.DetFormsetClass(view.request.POST, instance=view.object, save_as_new=True)
if new_formset.is_valid():
new_dets = new_formset.save()
new_fac.fk_cancel = old_fac # need a fk reference to initial fac in new one
old_fac.fk_cancel = new_fac # need a fk reference to new in old fac
# any save() action after this crashes with TransactionManagementError
new_fac.save()
I do not understand this error. I already created & saved a new object in db (when I set the object.id to None & saved that). Why would creating other objects create an issue for further saves?
I have tried not instantiating the new_dets objects with the Formset, but instead explicitely defining them:
new_det = Detfac(...)
new_det.save()
But then again, any further save after that raises the error.
Further details:
Essentially, I have an Entfac model, and a Detfac model that has a foreignkey to Entfac. I need to instantiate a new Enfac (distinct in db), as well as corresponding new Detfac for the new Entfac. Then I need to change some values in some of the fields for both new & old objects, and save all that to db.

Ah. The code above is fine.
But turns out, signals can be bad. I had forgotten that upon saving Detfac, there is a signal that goes to another class and that depending on the circumstances, adds a record to another table (sort of an history table).
Since that signal is just a single operation. Something like that:
#receiver(post_save, sender=Detfac)
def quantity_adjust_detfac(sender, **kwargs):
try:
detfac_qty = kwargs["instance"].qte
product = kwargs["instance"].fk_produit
if kwargs["created"]:
initial = {# bunch of values}
adjustment = HistoQuantity(**initial)
adjustment.save()
else:
except TypeError as ex:
logger.error(f"....")
except AttributeError as ex:
logger.error(f"....")
In itself, the fact that THIS wasn't marked as atomic isn't problematic. BUT if one of those exception throws, THEN I get the transactionmanagementerror. I am still not 100% sure why, tough the django docs do mention that when wrapping a whole view in atomic (or any chunk of code for that matter), then try/except within that block can yield unexpected result, because DJango does rely on exception to decide whether or not to commit the transaction as a whole. And the data I was testing with actually threw the exception (type error when creating the HistoQuantity object).
Wrapping the try/exception with a transaction.atomic manager worked however. Guessing that this... removed/handled the throw, thus the outer atomic could work.

Related

Django forms without widgets

I understand that Django want to generate forms automatically so you don't have to do so in your template, and I do understand that many people find it cool.
But I have specific requirements and I have to write my forms on my own. I just need something to parse the data, be it a form submitted using a user interface, or an API request, or whatever.
I tried to use ModelForm, but it doesn't seem to work as I want it to work.
I'd like to have something with the following behavior:
possibility to specify the model of the object I am going to create/update
possibility to specify an object in case of an update
possibility to provide new data in a dictionary
if I am creating a new object, missing fields in my data should be replaced by their default values as specified in my model definition
if I am updating an existing object, missing fields in my data should be replaced by the current values of the object I am updating. Another way of saying is, do not update values that are missing in my data dictionary.
data validation should be performed before calling save(), and it should throw a ValidationError with the list of erroneous fields and errors.
Currently, I prefer to do everything manually :
o = myapp.models.MyModel() # or o = myapp.Models.MyModel.objects.get(pk = data['pk'])
o.field1 = data['field1']
o.field2 = data['field2']
…
o.full_clean()
o.save()
It would be nice to have a shortcut :
o = SuperCoolForm(myapp.models.MyModel, data)
o.save()
Do you know if Django does provide a solution for this or am I asking too much?
Thank you!

Implementing unread/read checking for a message

I'm having a message model. To this model I want to add a read/unread field, which I did by using a boolean field. Now, if someone reads this message, I want this boolean field to be turned to true. I access these messages at different parts in my app, so updating the field manually is going to be tedious.
Is there any way I can get some messages according to some condition, and when the message is fetched from db, the field gets auto updated?
Why don't you create a read_message() method on a custom model manager. Have this method return the messages you want, whilst also updating the field on each message returned.
You new method allow you to replace Message.objects.get() with Message.objects.read_message()
class MessageManager(models.Manager):
def read_message(self, message_id):
# This won't fail quietly it'll raise an ObjectDoesNotExist exception
message = super(MessageManager, self).get(pk=message_id)
message.read = True
message.save()
return message
Then include the manager on your model -
class Message(models.Model):
objects = MessageManager()
Obviously you could write other methods that return querysets whilst marking all the messages returned as read.
If you don't want to update your code (places where you call Message.objects.get()), then you could always actually override get() so that it updates the read field. Just replace the read_message function name above with get.
Depending on your database management system, you may be able to install a trigger:
PostgreSQL: http://www.postgresql.org/docs/9.1/static/sql-createtrigger.html
MySQL: http://dev.mysql.com/doc/refman/5.0/en/triggers.html
SQLite: http://www.sqlite.org/lang_createtrigger.html
Of course, this will need to be done manually in the database - outside of the Django application.

Assigning values to a query result already set up with a foreign key

I have a database of exhibition listings related by foreign key to a database of venues where they take place. Django templates access the venue information in the query results through listing.venue.name, listing.venue.url, and so on.
However, some exhibitions take place in temporary venues, and that information is stored in the same database, in what would be listing.temp_venue_url and such. Because it seems wasteful and sad to put conditionals all over the templates, I want to move the info for temporary venues to where the templates are expecting info for regular venues. This didn't work:
def transfer_temp_values(listings):
for listing in listings:
if listing.temp_venue:
listing.venue = Venue
listing.venue.name = listing.temp_venue
listing.venue.url = listing.temp_venue_url
listing.venue.state = listing.temp_venue_state
listing.venue.location = listing.temp_venue_location
The error surprised me:
ValueError at /[...]/
Cannot assign "<class 'myproject.gsa.models.Venue'>": "Exhibition.venue" must be a "Venue" instance.
I rather thought it was. How do I go about accomplishing this?
The error message is because you have assigned the class Venue to the listing, rather than an instance of it. You need to call the class to get an instance:
listing.venue = Venue()

How do I deal with this race condition in django?

This code is supposed to get or create an object and update it if necessary. The code is in production use on a website.
In some cases - when the database is busy - it will throw the exception "DoesNotExist: MyObj matching query does not exist".
# Model:
class MyObj(models.Model):
thing = models.ForeignKey(Thing)
owner = models.ForeignKey(User)
state = models.BooleanField()
class Meta:
unique_together = (('thing', 'owner'),)
# Update or create myobj
#transaction.commit_on_success
def create_or_update_myobj(owner, thing, state)
try:
myobj, created = MyObj.objects.get_or_create(owner=user,thing=thing)
except IntegrityError:
myobj = MyObj.objects.get(owner=user,thing=thing)
# Will sometimes throw "DoesNotExist: MyObj matching query does not exist"
myobj.state = state
myobj.save()
I use an innodb mysql database on ubuntu.
How do I safely deal with this problem?
This could be an off-shoot of the same problem as here:
Why doesn't this loop display an updated object count every five seconds?
Basically get_or_create can fail - if you take a look at its source, there you'll see that it's: get, if-problem: save+some_trickery, if-still-problem: get again, if-still-problem: surrender and raise.
This means that if there are two simultaneous threads (or processes) running create_or_update_myobj, both trying to get_or_create the same object, then:
first thread tries to get it - but it doesn't yet exist,
so, the thread tries to create it, but before the object is created...
...second thread tries to get it - and this obviously fails
now, because of the default AUTOCOMMIT=OFF for MySQLdb database connection, and REPEATABLE READ serializable level, both threads have frozen their views of MyObj table.
subsequently, first thread creates its object and returns it gracefully, but...
...second thread cannot create anything as it would violate unique constraint
what's funny, subsequent get on the second thread doesn't see the object created in the first thread, due to the frozen view of MyObj table
So, if you want to safely get_or_create anything, try something like this:
#transaction.commit_on_success
def my_get_or_create(...):
try:
obj = MyObj.objects.create(...)
except IntegrityError:
transaction.commit()
obj = MyObj.objects.get(...)
return obj
Edited on 27/05/2010
There is also a second solution to the problem - using READ COMMITED isolation level, instead of REPEATABLE READ. But it's less tested (at least in MySQL), so there might be more bugs/problems with it - but at least it allows tying views to transactions, without committing in the middle.
Edited on 22/01/2012
Here are some good blog posts (not mine) about MySQL and Django, related to this question:
http://www.no-ack.org/2010/07/mysql-transactions-and-django.html
http://www.no-ack.org/2011/05/broken-transaction-management-in-mysql.html
Your exception handling is masking the error. You should pass a value for state in get_or_create(), or set a default in the model and database.
One (dumb) way might be to catch the error and simply retry once or twice after waiting a small amount of time. I'm not a DB expert, so there might be a signaling solution.
Since 2012 in Django we have select_for_update which lock rows until the end of the transaction.
To avoid race conditions in Django + MySQL
under default circumstances:
REPEATABLE_READ in the Mysql
READ_COMMITTED in the Django
you can use this:
with transaction.atomic():
instance = YourModel.objects.select_for_update().get(id=42)
instance.evolve()
instance.save()
The second thread will wait for the first thread (lock), and only if first is done, the second will read data saved by first, so it will work on updated data.
Then together with get_or_create:
def select_for_update_or_create(...):
instance = YourModel.objects.filter(
...
).select_for_update().first()
if order is None:
instnace = YouModel.objects.create(...)
return instance
The function must be inside transaction block, otherwise, you will get from Django:
TransactionManagementError: select_for_update cannot be used outside of a transaction
Also sometimes it's good to use refresh_from_db()
In case like:
instance = YourModel.objects.create(**kwargs)
response = do_request_which_lasts_few_seconds(instance)
instance.attr = response.something
you'd like to see:
instance = MyModel.objects.create(**kwargs)
response = do_request_which_lasts_few_seconds(instance)
instance.refresh_from_db() # 3
instance.attr = response.something
and that # 3 will reduce a lot a time window of possible race conditions, thus chance for that.

Django : Setting a generic (content_type) field with a real object sets it to None

Update 3 (Read This First) :
Yes, this was caused by the object "profile" not having been saved. For those getting the same symptoms, the moral is "If a ForeignKey field seems to be getting set to None when you assign a real object to it, it's probably because that other objects hasn't been saved."
Even if you are 100% sure that it was saved, check again ;-)
Hi,
I'm using content_type / generic foreign keys in a class in Django.
The line to create an instance of the class is roughly this :
tag = SecurityTag(name='name',agent=an_agent,resource=a_resource,interface=an_interface)
Where both agent and resource are content_type fields.
Most of the time, this works as I expect and creates the appropriate object. But I have one specific case where I call this line to create a SecurityTag but the value of the agent field seems to end up as None.
Now, in this particular case, I test, in the preceding line, that the value of an_agent does contain an existing, saved Django.model object of an agent type. And it does.
Nevertheless, the resulting SecurityTag record comes out with None for this field.
I'm quite baffled by this. I'm guessing that somewhere along the line, something is failing in the ORM's attempt to extract the id of the object in an_agent, but there's no error message nor exception being raised. I've checked that the an_agent object is saved and has a value in its id field.
Anyone seen something like this? Or have any ideas?
====
Update : 10 days later exactly the same bug has come to bite me again in a new context :
Here's some code which describes the "security tag" object, which is basically a mapping between
a) some kind of permission-role (known as "agent" in our system) which is a generic content_type,
b) a resource, which is also a generic content_type, (and in the current problem is being given a Pinax "Profile"),
and c) an "interface" (which is basically a type of access ... eg. "Viewable" or "Editable" that is just a string)
class SecurityTag(models.Model) :
name = models.CharField(max_length='50')
agent_content_type = models.ForeignKey(ContentType,related_name='security_tag_agent')
agent_object_id = models.PositiveIntegerField()
agent = generic.GenericForeignKey('agent_content_type', 'agent_object_id')
interface = models.CharField(max_length='50')
resource_content_type = models.ForeignKey(ContentType,related_name='security_tag_resource')
resource_object_id = models.PositiveIntegerField()
resource = generic.GenericForeignKey('resource_content_type', 'resource_object_id')
At a particular moment later, I do this :
print "before %s, %s" % (self.resource,self.agent)
t = SecurityTag(name=self.tag_name,agent=self.agent,resource=self.resource,interface=self.interface_id)
print "after %s, %s, %s, %s" % (t.resource,t.resource_content_type,type(t.resource),t.resource_object_id)
The result of which is that before, the "resource" variable does reference a Profile, but after ...
before phil, TgGroup object
after None, profile, <type 'NoneType'>, None
In other words, while the value of t.resource_content_type has been set to "profile", everything else is None. In my previous encounter with this problem, I "solved" it by reloading the thing I was trying to assign to the generic type. I'm starting to wonder if this is some kind of ORM cache issue ... is the variable "self.resource" holding some kind proxy object rather than the real thing?
One possibility is that the profile hasn't been saved. However, this code is being called as the result of an after_save signal for profile. (It's setting up default permissions), so could it be that the profile save hasn't been committed or something?
Update 2 : following Matthew's suggestion below, I added
print self.resource._get_pk_value() and self.resource.id
which has blown up saying Profile doesn't have _get_pk_value()
So here's what I noticed passing through the Django code: when you create a new instance of a model object via a constructor, a pre-init function called (via signals) for any generic object references.
Rather than directly storing the object you pass in, it stores the type and the primary key.
If your object is persisted and has an ID, this works fine, because when you get the field at a later date, it retrieves it from the database.
However -- if your object doesn't have an ID, the fetch code returns nothing, and the getter returns None!
You can see the code in django.contrib.contenttypes.generic.GenericForeignKey, in the instance_pre_init and __get__ functions.
This doesn't really answer my question or satisfy my curiosity but it does seem to work if I pull the an_agent object out of the database immediately before trying to use it in the SecurityTag constructor.
Previously I was passing a copy that had been made earlier with get_or_create. Did this old instance somehow go out of date or scope?