I get all the other active record error but when there is a callback failure, it goes on and saves the record while actions to be performed in callbacks have malfunctioned.
I am looking for a better way to debug this. A simple configuration would also help. If I can force exceptions to raise instead of being ignored in callbacks.
Related
I have a step functions orchestration flow and I want to do error handling in some of the states using the catch field. However, the catch field requires a Next assignment and therefore I am unable to include a catch field in my last state if i want my step function to run.
I would like to have a catch field in the last state of the flow but I am wondering if it is good practise to have a catch statement in the last state. When i introduce an ending state e.g. a Type:Succeed state the stepfunction is able to run. But this solution feels a bit hacky.
I have tried to set the value of Next in the catch field to End. But was thrown this error in cloudformation when it tried to update the stack.
Resource handler returned message: "Invalid State Machine Definition: 'MISSING_TRANSITION_TARGET: Missing 'Next' target: EndState at /States/last_jobs/Branches[0]/States/last_state/Catch[0]/Next' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Proxy: null)" (HandlerErrorCode: InvalidRequest)
The purpose of Catch is so you can tell Step Functions to take a different action in response to a failure (after retries) as the default behavior will be to fail the execution. That action needs to be captured in the workflow, hence the need for this to point to another state where that action is described.
I'm not 100% sure what you are looking to accomplish with this catch block, but I suspect it's one of the following cases.
If you are looking to take further action to compensate, then you will need to add that to your workflow (e.g. another task or a wait state that re-enters into the existing flow).
If you are looking to provide a specific failure cause and / or error as opposed to the default you would get from the task failing, then you will need a Fail state with those specifics. And set that as Next for your Catch.
If you are looking to ignore this task failure and complete the workflow successfully, then you need a Succeed state that you can specify as Next for your Catch.
Imagine you have a User model in your web app, and that you need to keep this user in sync with an external service via an API. Thus, when you create a user locally, you need to create it remotely as well.
You have all your operations under transaction.atomic() and you try to keep all your 3rd-party API calls after the atomic block, which is reasonable.
But, a system being a system, it grows in complexity until the point you have some really hard to remove 3rd-party calls within an update call.
That said, is there a way to extend Django's transaction mechanism, kind of adding some callback functions, like rollback.add_callback(clean_3rdparty_user(user_id=134))?
That way I can guarantee that all necessary rollback actions are taken and my system is in sync?
The author of Django's transaction hook code has this to say about why there is on_commit() but not on_rollback():
A rollback hook is even harder to implement robustly than a commit hook, since a variety of things can cause an implicit rollback. For instance, your database connection was dropped because your process was killed without a chance to shutdown gracefully: your rollback hook will never run.
Since rollbacks are typically triggered by an exception, a simple approach is to just catch any exceptions and run your undo code there.
try:
with transaction.atomic():
# Do database stuff
# Do external stuff
except:
# We know the database stuff has rolled back, so...
# Undo external stuff
raise
This is not particularly elegant. I agree with the following from the same source:
The solution is simple: instead of doing something during the atomic block (transaction) and then undoing it if the transaction fails, use on_commit to delay doing it in the first place until after the transaction succeeds. It’s a lot easier to undo something you never did in the first place!
But it sounds like you already agree with that as well.
In the SmtpClient class, does SendAsyncCancel cancel the SendMailAsync method?
I see a few code examples on the www which imply it does.
However MSDN says,
Use the SendAsyncCancel method to cancel a pending SendAsync operation. If there is mail waiting to be sent, this method releases resources used to store the mail. If there is no mail waiting to be sent, this method does nothing.
... which implies that it cancels SendAsync but not SendMailAsync.
Is there a way to cancel SendMailAsync? If not, why not?
If you want to cancel an asynchronous send (therefore use the old SendAsync instead of the newer SendMailAsync), what are any other disadvantages of using SendAsync instead of SendMailAsync?
I tried to invoke SendMailAsync from a post-back handler of an ASP web page, and it threw an exception:
System.InvalidOperationException: Asynchronous operations are not allowed in this context. Page starting an asynchronous operation has to have the Async attribute set to true and an asynchronous operation can only be started on a page prior to PreRenderComplete event.
at System.Web.LegacyAspNetSynchronizationContext.OperationStarted()
at System.ComponentModel.AsyncOperation.CreateOperation(Object userSuppliedState, SynchronizationContext syncContext)
at System.Net.Mail.SmtpClient.SendAsync(MailMessage message, Object userToken)
--- End of inner exception stack trace ---
at System.Net.Mail.SmtpClient.SendAsync(MailMessage message, Object userToken)
at System.Net.Mail.SmtpClient.SendMailAsync(MailMessage message)
at MyWebSite.AdminTest.TestEmail.<sendAsynchronous>d__9.MoveNext()
From this I deduce two things:
SendMailAsync is indeed implemented using SendAsync (you can see it on the call-stack of the exception). SendAsyncCancel therefore will presumably work for SendMailAsync too.
You can't call SendMailAsync from an ASP unless you want to deal with the "Asynchronous operations are not allowed in this context" problem. That's discussed here and looks like it might be messy. Whereas I guess that calling SendAsync probably doesn't have this problem (because I'm using HttpClient.SendAsync and ContinueWith elsewhere, with a CountdownEvent.Wait at the end to wait for the operation to complete, without seeing this exception being thrown).
SendMailAsync can be used when it's invoked via HostingEnvironment.QueueBackgroundWorkItem.
I'm unclear on the exact behaviour of Django in the face of database serialization errors in transactions.
The docs transaction.atomic() docs don't specify this behaviour as far as I can tell.
If the DB hits a consistency error while committing a transaction (e.g. another transaction updated a value that was read in the current transaction), reading django.db.transaction.py, it looks like the transaction will rollback, and the DatabaseError will be raised to the calling code (e.g. the transaction.atomic() context manager). Is this correct?
And, more importantly, are there cases when the transaction could be rolled back without the transaction.atomic wrapper receiving an exception?
(Note that I'm not asking about DatabaseErrors that are raised inside the context manager, as the docs clearly explain what happens to them. I'm asking only about database errors which occur during the commit of the transaction, which occurs on exit of the context manager.)
If the DB hits a consistency error while committing a transaction ... it looks like the transaction will rollback, and the DatabaseError will be raised to the calling code (e.g. the transaction.atomic() context manager). Is this correct?
Yes, precisely.
Are there cases when the transaction could be rolled back without the transaction.atomic wrapper receiving an exception?
No. You can verify this from the code inside transaction.py where the only time a rollback is initiated is if DatabaseError is thrown. This is also confirmed in the documentation that you link to:
When exiting an atomic block, Django looks at whether it’s exited normally or with an exception to determine whether to commit or roll back.
I'm creating a few simple helper classes and methods for working with libpq, and am wondering if I receive an error from the database - (e.g. SQL error), how should I handle it?
At the moment, each method returns a bool depending on whether the operation was a success, and so is up to the user to check before continuing with new operations.
However, after reading the libpq docs, if an error occurs the best I can come up with is that I should log the error message / status and otherwise ignore. For example, if the application is in the middle of a transaction, then I believe it can still continue (Postgresql won't cancel the transaction as far as I know).
Is there something I can do with PostgreSQL / libpq to make the consequences of such errors safe regarding the database server, or is ignorance the better policy?
You should examine the SQLSTATE in the error and make handling decisions based on that and that alone. Never try to make decisions in code based on the error message text.
An application should simply retry transactions for certain kinds of errors:
Serialization failures
Deadlock detection transaction aborts
For connection errors, you should reconnect then re-try the transaction.
Of course you want to set a limit on the number of retries, so you don't loop forever if the issue doesn't clear up.
Other kinds of errors aren't going to be resolved by trying again, so the app should report an error to the client. Syntax error? Unique violation? Check constraint violation? Running the statement again won't help.
There is a list of error codes in the documentation but the docs don't explain much about each error, but the preamble is quite informative.
On a side note: One trap to avoid falling into is "testing" connections with a trivial query before using them, and assuming that means the real query can't fail. That's a race condition. Don't bother testing connections; simply run the real query and handle any error.
The details of what exactly to do depend on the error and on the application. If there was a single always-right answer, libpq would already do it for you.
My suggestions:
Always keep a record of the transaction until you've got a confirmed commit from the DB, in case you have to re-run. Don't just fire-and-forget SQL statements.
Retry the transaction without a disconnect and reconnect for SQLSTATEs 40001 (serialization_failure) and 40P01 (deadlock_detected), as these are transient conditions generally resolved by re-trying. You should log them, as they're opportunities to improve how the app interacts with the DB and if they happen a lot they're a performance problem.
Disconnect, reconnect, and retry the transaction at least once for error class 08 (connection exceptions).
Handle 53300 (too_many_connections) and 53400 (connection limit exceeded) with specific and informative errors to the user. Same with the other 53 class entries.
Handle class 57's entries with specific and informative errors to the user. Do not retry if you get a query_cancelled (57014), it'll make sysadmins very angry.
Handle 25006 (read_only_sql_transaction) by reporting a different error, telling the user you tried to write to a read-only database or using a read-only transaction.
Report a different error for 23505 (UNIQUE violation), indicating that there's a conflict in a unique constraint or primary key constraint. There's no point retrying.
Error class 01 should never produce an exception.
Treat other cases as errors and report them to the caller, with details from the problem - most importantly SQLSTATE. Log all the details if you return a simplified error.
Hope that's useful.