When using Model.objects.bulk_create() if an exception occurs during the insertion does it roll back the entire operation or does it continue with the non-conflicting records, and is there any way to know which records were inserted and which threw an error?
If an exception occurs the entire operation will be rolled back. If you look at the source code you'll see that all database operations are wrapped in transaction.atomic().
There's no way of knowing which values caused the conflict. Such information may be available in the database-specific error message, but that's not part of the API.
Note that as of Django 2.2 there will be an ignore_conflicts parameter that will allow you to explicitly control whether the operation will roll back or whether the conflicts will be ignored.
Related
I had to use transaction.on_commit() for synchronous behaviour in one of the signals of my project. Though it works fine, I couldn't understand how does transaction.on_commit() decide which transaction to take. I mean there can be multiple transactions at the same time. But how does django know which transaction to take by using transaction.on_commit()
According to the docs
You can also wrap your function in a lambda:
transaction.on_commit(lambda: some_celery_task.delay('arg1'))
The function you pass in will be called immediately after a hypothetical database write made where on_commit() is called would be successfully committed.
If you call on_commit() while there isn’t an active transaction, the callback will be executed immediately.
If that hypothetical database write is instead rolled back (typically when an unhandled exception is raised in an atomic() block), your function will be discarded and never called.
If you are using it on post_save method with sender=SomeModel. Probably the on_commit is executed each time a SomeModel object is saved. Without the proper code we would not be able to tell the exact case.
If I understand the question correctly, I think the docs on Savepoints explains this.
Essentially, you can nest any number of transactions, but on_commit() is only called after the top most one commits. However, on_commit() that's nested within a savepoint will only be called if that savepoint was committed and all the ones above it are committed. So, it's tied to which ever one is currently open at the point it's called.
I am running a calculation in a PL/pgSQL function and I want to use the result of that calculation in my C++ code. What's the best way to do that?
I can insert that result into a table and use it from there but I'm not sure how well that fares with best practices. Also, I can send message to stderr with RAISE NOTICE but I don't know can I use that message in my code.
The details here are a bit thin on the ground, so it's hard to say for sure.
Strongly preferable whenever possible is to just get the function's return value directly. SELECT my_function(args) if it returns a single result, or SELECT * FROM my_function(args); if it returns a row or set of rows. Then process the result like any other query result. This is part of the basic use of simple SQL and PL/PgSQL functions.
Other options include:
Returning a refcursor. This can be useful in some circumstances where you want to return a dynamic result set or multiple result sets, though it's now mostly superseded by RETURN QUERY and RETURN QUERY EXECUTE. You then FETCH from the refcursorto get the result rows.
LISTENing for an event and having the function NOTIFY when the work is done, possibly with the result as a notify payload. This is useful when the function isn't necessarily called on the same connection as the program that wants to use its results.
Create a temporary table in the function, then SELECT from the table from the session that called the function.
Emitting log messages via RAISE and setting client_min_messages so you receive them, then processing them. This is a very ugly way to do it and should really be avoided at all costs.
INSERTing the results into an existing non-temporary table, then SELECTing them out once the transaction commits and the rows become visible to other transactions.
Which is better? It depends entirely on what you're trying to do. In almost all cases the correct thing to do is just call the function and process the return value, but there are exceptions in special cases.
I have a class that can perform many types of transformations to a given object.
All data is supplied by the users (via hand crafted command files) but for the most part there is no way to tell if the file commands are valid until we start transforming it (if we checked everything first we'd end up doing the same amount of work as the transformations themselves)
These transformation methods are called on the object and it returns a newly transformed object however if there is a problem we throw an exception since returning an object (even a blank object) could be confused with success whereas an exception sends a clear signal there was a problem and an object can not be returned. It also means we don't need to call a "get last error" type function to find out exactly what went wrong (error code, message, statement, etc.)
However from reading numerous answers on SO it seems this is incorrect since invalid user input is not an exceptional circumstance and due to the complexity of this thing its not uncommon for the user to submit incorrect command files.
Performance is not really an issue here because if we hit an error we have to stop and go back to the user.
If I use return codes I'd have to take the output object as a parameter and then to check the return codes I'd need long nested if blocks (ie. the way you check HRESULT from COM)
How would I go about avoiding exceptions in this case while still keeping the code reasonably tidy?
The design of your problem really lends itself for exceptions. The fact that program execution will halt or be invalid once the user supplies a bad input file, is a sign of "exceptional circumstance". Sure, a lot of program runs will end in an exception being thrown (and caught), but this is in one execution of the program.
The things you are reading about exceptions being slow when used for every other circumstance, is only valid if the program can recover, and has to recover often (e.g. a compiler that fails to find a header in a search directory, and has to look at the next directory in the list of search directories, which really happens a lot).
Use exceptions. When in doubt, measure if this is killing performance. Cleaner code >> following what people say on SO. But then again, that would be a reason to ignore what I just said.
Well, I wouldn't (avoid exceptions). If you really need an "error reporting" mechanism (and you do), there is not much besides return values and exceptions. I think the whole argument about what is exceptional enough (and therefor deserves exceptions) and what is not, is not that important to keep you from solving your problem. Especially if performace isn't an issue. But if you REALLY want avoid exceptions, you could use some kind of global queues to queue your error iformation. But that is utterly ugly, if you ask me.
So imagine you've got an exception you're catching and then in the catch you write to a log file that some exception occurred. Then you want your program to continue, so you have to make sure that certain invariants are still in a a good state. However what actually occurs in the system after the exception was "handled" by a catch?
The stack has been unwound at that point so how does it get to restore it's state?
"Stack unwinding" means that all scopes between throw and the matching catch clause are left, calling destructors for all automatic objects in those scopes, pretty much in the same way function scopes are left when you return from a function.
Nothing else "special" is done, the scope of a catch clause is a normal scope, and leaving it is no different from leaving the scope of an else clause.
If you need to make sure certain invariants still hold, you need to program the code changing them in a thread-safe manner. Dave Abrahams wrote a classic on the different levels of exception safety, you might want to read that. Basically, you will have to consequently employ RAII in order to be on the safe side when exceptions are thrown.
Only objects created inside the try will have been destroyed during unwinding. It's up to you to write a program in such way that if an exception occurs program state stays consistent - that's called exception safety.
C++ doesn't care - it unwinds stack, then passes control into an appropriate catch, then control flow continues normally.
It is up to you to ensure that the application is recovered into a stable state after catching the exception. Usually it is achieved by "forgetting" whatever operation or change(s) produced the exception, and starting afresh on a higher level.
This includes ensuring that any resources allocated during the chain of events leading to the exception gets properly released. In C++, the standard idiom to ensure this is RAII.
Update
For example, if an error occurs while processing a request in a web server, it generates an exception in some lower level function, which gets caught in a higher level class (possibly right in the top level request handler). Usually the best possible thing to do is to roll back any changes done and free any resources allocated so far related to the actual request, and return an appropriate error message to the client. Changes may include DB transactions, file writes, etc - one must implement all these in an exception safe manner. Databases typically have built in transactions to deal with this; with other resources it may be more tricky.
This is up to the application. There are several levels of exception-safety. The level you describe is hard to achieve for the whole application.
Certain pieces of code, however, can be made 'Failure transparent', by using techniques like RAII, and by smartly ordering the sequence of actions. I could imagine a piece of code querying several urls for data, for instance: when one url would 'throw', the rest of the urls can still be handled. Or it can be retried...
If you have exception handling in every function you can resume on the next higher level but its rather complicated, in fact I use exceptions mainly to detect errors as close to the source as possible but don't use them for resuming execution.
if on the other hand there are errors that are predictable one can devise schemes to handle that, but for me exceptions are considered exceptions so I tend to try and exit gracefully instead with a good hint in the log file where it happened. JM2CW
It can't. Exceptions aren't resumable in C++. Nor in most
modern languages; some of the first languages to support
exceptions did support resumable exceptions, and found that it
wasn't a good idea.
If you want to be able to resume from some specific point, you
have to put your try/catch block there. If you just want to log
and continue, don't throw the exception in the first place.
Should a web service return an exception or an empty result when data is not found?
If no data is a normal and acceptable result (as it normally is), it should return an empty result set. You should only return an exception if an error/exception has occurred.
A regular .net exception is a platform specific construct that should not be returned outside the webservice since the client calling upon the service might not implement such a thing, instead use a SoapException for exceptional circumstances
Is the empty result set something the client calling the webservice can recover from? If so I wouldn’t return the SoapException. If it’s something that’s so exceptional that it warrants a special handling logic then by all means do use the SoapException.
There resources might help you along
http://msdn.microsoft.com/en-us/library/ds492xtk(VS.71).aspx
http://www.developer.com/net/csharp/article.php/10918_3088231_1
How about an error message that describes the problem?
That depends entirely on the domain of the problem. If there should never be a time that no result is found them that is an error. If no results is an acceptible condition then an empty result set is fine. It just depends on the specific problem. As someone consuming the webservice that is what I would expect. An error is a problem.
Thank you all for your response. Your responses are all common sense and right where I was thinking. However, our application architect had other ideas. Now, the returned exception will be changed from a sql exception to a more informative one. We have agreed that an empty result is acceptable only where the parameter doesn't exist in a relationship table but an exception will result if the value isn't in the value table at all.
Thank you all for being the subject matter experts today!