from django.db import connection
def executeQuery(query, params):
cur=connection.cursor()
cur.execute(query, params) // this is update query
cur.close()
I have series of queries and I call this method for each query, but it looks like it rollbacks entire operation if any query (let's say 3rd query) gets failed.
I thought, after execute(), it immediately commit it and it does not depend on the next query.
Shouldn't django has auto commit feature?
Database altering operations are automatically committed. However, if you are using the django.middleware.transaction.TransactionMiddleware or something similar, then they will be only committed if the page rendering finishes without any error, otherwise a rollback will happen.
For further details refer to the documentation for django 1.5 (the version used in the question). Check the latest documentation too.
Related
We have a system consisting of a Django server connected to a PostgreSQL database, and some AWS Lambda functions which need to access the database. When an admin saves a model (PremiumUser - contains premium plan information that needs to be read by the Lambdas), we want to set up a schedule of CloudWatch events based on the saved information. Those events then trigger other Lambdas which also need to directly access the database, as the database state may change at any time and they should work off the most recent state of the database.
The issue is that Django seems to think it has saved the values, but when the Lambdas read the database the expected values aren't there. We have tried using Django's post_save signal, calling the Lambdas inside the triggered function; we have tried overriding Django's default PremiumUser.save method to perform super(PremiumUser, self).save(*args, **kwargs), and only then call Lambdas (in case the post_save signal was getting triggered too early); and we have tried overriding the PremiumUser.save method and calling super(PremiumUser, self).save(*args, **kwargs) in the context of an atomic transaction (ie with transactions.atomic():).
When we call the Lambdas a few seconds after the admin dashboard has updated, they can find the values as expected and work properly, which suggests that somehow Django considers the model as having been 'saved' to the database, while the database has not yet been updated.
Is there a way to force Django to write to the database immediately? This would be the preferred solution, as it would keep Django's model and the database consistent.
An alternative solution we have considered but would prefer not to resort to would be to put a sleep in the Lambdas and calling them asynchronously so that Django's save is able to complete before the Lambda functions access the database. Obviously this would be a race condition, so we don't want to do this if it can be at all avoided.
Alright, after spending more time poring over Django's documentation we found a solution: using on_commit. It seems that post_save is triggered after Django's model is updated, but before the database has been written to. on_commit, on that other hand is triggered after the current database transaction has been committed (completed).
For us, the solution involved setting up some code like this:
from django.db import models, transaction
class PremiumUser(models.Model):
# Normal setup code...
def save(self, *args, **kwargs):
# Do some necessary things before the actual save occurs
super(PremiumUser, self).save(*args, **kwargs)
# Set up a callback for the on_commit
def create_new_schedule():
# Call lambda to create new schedule...
# Register the callback with the current transaction's on_commit
transaction.on_commit(create_new_schedule)
Project use Django and Postgresql 9.5. Sometimes I see the error in celery task.
When object need change specified column it uses celery task.
This task writes in separate table change history of an object and update column(not raw SQL, by Django ORM).
Task write history by FDW extension into the foreign table.
Thrown exception:
Remote SQL command: COMMIT TRANSACTION\nSQL statement "SELECT 1 FROM ONLY "public"."incident_incident" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x"\n',)
I can't take understand why it raises the exception. Task very simple
screen logs(maybe it help):
In celery, when you are doing transaction, then you can use transaction.atomic block to do that.
For example:
#app.task(bind=True)
def do_task(self)
try:
with transaction.atomic():
# Do DB OP
except (SomeException,Exception) as exc:
raise self.retry(exc=exc)
There are other approaches as well. You can add a new field regarding object change in Model and track it. You can read this article on medium regarding this approach. Hope it helps!!
Why Django-Simple history records get created on calling save method if I call update then it doesn't create history record ?
Django : 1.11.15
Django-simple-history : 1.9.0
Python : 3.6
As is written in the documentation this is a known issue:
Django Simple History functions by saving history using a post_save
signal every time that an object with history is saved. However, for
certain bulk operations, such as bulk_create and queryset updates, signals are not sent, and the history is not saved
automatically. However, Django Simple History provides utility
functions to work around this.
So basically the app makes use of the fact that you .save() the model, and this is circumvented by some ORM calls (because then you can not perform the actions in "bulk" at the database level anymore).
Instead of using
Entry.objects.filter(pub_date__year=2010).update(comments_on=False)
you thus need to perform:
for e in Entry.objects.filter(pub_date__year=2010):
e.comments_on = False
e.save()
For a bulk_create there is a variant: bulk_create_with_history, since then it simply makes two bulk creates: one for the objects, and one for the "histories".
I have two models
LongJob(models.Model):
pass
Report(models.Model):
job = models.OneToOneField('LongJob', related_name='report')
I first create a job in a view:
job = LongJob.objects.create()
Then I pass the job to a Celery task and I get or create a report:
#shared_task
def celery_task(job):
report, created = Report.objects.get_or_create(job=job)
Sometimes, the get_or_create call raises the following IntegrityError:
IntegrityError: insert or update on table "report" violates foreign key constraint "job_id_c7400e5ba78c00c_fk_longjob_id"
DETAIL: Key (job_id)=(120057) is not present in table "longjob".
After the error occurs, I check whether the job object with the supposedly non-existing ID exists in the LongJob table and I do find it. How is this possible ?
The database is Postgresql 9.4.
Preamble
Read Committed is the default isolation level in PostgreSQL. When a
transaction uses this isolation level, a SELECT query (without a FOR
UPDATE/SHARE clause) sees only data committed before the query began;
it never sees either uncommitted data or changes committed during
query execution by concurrent transactions. In effect, a SELECT query
sees a snapshot of the database as of the instant the query begins to
run. However, SELECT does see the effects of previous updates executed
within its own transaction, even though they are not yet committed.
Also note that two successive SELECT commands can see different data,
even though they are within a single transaction, if other
transactions commit changes after the first SELECT starts and before
the second SELECT starts.
https://www.postgresql.org/docs/9.4/static/transaction-iso.html
The red herring
get_or_create
This method is atomic assuming correct usage, correct database
configuration, and correct behavior of the underlying database.
However, if uniqueness is not enforced at the database level for the
kwargs used in a get_or_create call (see unique or unique_together),
this method is prone to a race-condition which can result in multiple
rows with the same parameters being inserted simultaneously.
https://docs.djangoproject.com/en/1.9/ref/models/querysets/#django.db.models.query.QuerySet.get_or_create
The cause of the error
You either have auto commit switched off for your database in your django settings or you are running the following code inside a transaction.
job = LongJob.objects.create()
You are inserting the Job record in one thread and the Report in another. But because of the transaction isolation level the celery worker cannot actually see the job record with the LongJob instance is created.
The atomicity of the get_or_create method only guarantees that a same report with the same unique keys cannot be created by a second thread. It doesn't effect the object referred by the foreign key.
The Solution
Commit the transaction before passing onto celery or make the celery task wait on the record being created (the transaction in main django thread to be committed).
I have a view which reads the excel sheet and saves the data. I need a way if there is any error happens(500) in that view the database transactions should not commit and hence they should rollback.
I use the following code but it saves the data before the errors comes. My tasks is if there is any error in views the database should rollback.
from django.db import transaction
#transaction.commit_on_success
def upload_data(request):
..... and so on .....
obj.save()
Error comes here on this line
Want rollback the database as it
was before this view was called
obj1.save()
If error is here noting should be saved.
Thanks
Per Django's documentation on transactions, if you're using Django 1.6, I'd wrap the whole view in #transaction.atomic:
from django.db import transaction
#transaction.atomic
If you want this behavior for your whole app, set ATOMIC_REQUESTS=True in your database configuration, as described in that documentation.
Otherwise, if you're on 1.5 and not getting the behavior you expect, you could switch to #transaction.commit_manually, wrap the whole view in a try block, and do a commit() or rollback() explicitly. It's not elegant, but it might work if you want fine-grained control of when exactly commits happen.
try to put #transaction.commit_on_success just top of your view. So, if you get an error within that view function it will rollback else commit your work.