.NET exception handler causing stack overflow on Visual C++ 6.0 exceptions - c++

I have a plugin for an old application written in C++ 6.0. The files are connected in the following fashion:
Starts with: C++ 6.0 .exe (third party application)
loads: C++ 6.0 simple loader .dll (officially a plugin for application)
loads: C++ 10.0 simple loader .dll (managed C++/CLI)
loads one of: C# .NET 4.0 assembly which contains plugin
loads: C++ 6.0 .dll which provides API for C# plugin to talk to application
The problem is once .NET 4.0 is loaded into the C++ 6.0 application, the next time it throws a native exception, .NET uses a vectored exception handle to handle the exception and fails spectacularly. The part that makes it really bad is that the vectored exception handler throws an exception itself, which it then tries to handle, and that fails, and it gets stuck in an infinite loop until it gets a stack overflow exception.
Here's what the stack trace looks like:
// The next 7 lines repeat until the stack overflows
clr.dll!CreateHistoryReader()
clr.dll!CreateHistoryReader()
clr.dll!GetMetaDataInternalInterfaceFromPublic()
ntdll.dll!_RtlpCallVectoredHandlers#12()
ntdll.dll!_RtlCallVectoredExceptionHanders#8()
ntdll.dll!_RtlDispatchException#8()
ntdll.dll!_KiUserExceptionDispatcher#8()
// Below is an example exception that causes this:
KernelBase.dll!RaiseException()
rpcrt4.dll!RpcRaiseException()
rpcrt4.dll!I_RpcTransConnectionFreePacket()
rpcrt4.dll!I_RpcBindingInqCurrentModifiedId()
rpcrt4.dll!NdrConformantStringMemorySize()
rpcrt4.dll!NdrComplexStructMarshall()
rpcrt4.dll!SimpleTypeMemorySize()
rpcrt4.dll!NdrClientCall2()
ole32.dll!ServerRegisterClsid(void* hRpc, void* phProcess, _RegInput* pregin, _RegOutput** ppregout unligned long* prpcstat
ole32.dll!CRpcResolver::NotifyStarted(_RegInput* pRegIn, _RegOutput** ppRegOut)
ole32.dll!CClassCache::ResumeProcessClassObjects()
There are only really 2 ways to deal with this and neither are very great:
I found with a trivial program, if I completely isolate .NET on its own thread, the non-.NET threads never run into this issue. This doesn't work in practice because the plugin API needs to make synchronous callbacks to the .NET plugin.
The other I've come up with is to iterate over every single address in memory until a call to "RemoveVectoredExceptionHandler(HANDLE)" succeeds and removes .NET's vectored exception handler. (I can speed up the search by registering my own VEH temporarily and use its handle as a starting spot). This tends to break debugging of native code.
Is there any better way to deal with this?

The CLR seems to have changed behaviors since I reported this issue. Since the CLR is now open source, it's possible to see what's going on under the hood.
The CLR installs its own vectored exception handler. During vectored exception handling, it does a stack check to make sure there's enough space, unless it's a stack overflow exception. The stack space check goes wrong, and it thinks it's out of space when it's not, so it throws a stack overflow exception to unroll the stack enough to do actual work.
I was able to trick .NET into not crashing the app by installing 2 vectored exception handlers, one before and one after. If it's the exception type that's causing the crash, I change the exception code to STACKOVERFLOW in the first handler, and change it back in the second handler. This way the CLR thinks it's a stack overflow exception and doesn't try to do stack probing.

Related

When does Windows Error Reporting create a dump file? Is it configurable? Has this changed in Windows 7?

I'm relying on Windows Error Reporting to create full user-mode dumps for a large, multi-threaded application. I know that when I started using it (early 2012) these dumps contained all application memory, and full stacks for all threads that were accurate for the time the application crashed (threw the unhandled exception, etc). But at at some unknown point in the last year, crash dumps created by WER have changed. They still contain all memory, but only show one thread, and the stack appears to be from after the process is already shutting down:
ntdll.dll!_LdrpCallInitRoutine#16() + 0x14 bytes
ntdll.dll!_LdrShutdownProcess#0() + 0x141 bytes
ntdll.dll!_RtlExitUserProcess#4() + 0x74 bytes
kernel32.dll!_UnhandledExceptionFilter#4() + 0x18928 bytes
This is an unmanaged (unmanagable?) 32-bit C++ application compiled with VS2010 SP1, running on 64-bit Win7 SP1 (and kept updated). Does anyone know of any Windows updates that have changed WER behavior in the last year? Is there something configurable other than 'HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\AppName.exe'?
Also, killing the application by calling 'RaiseFailFastException' still results in a good dump with valid stacks for all threads.
Aha! A third-party library was calling SetUnhandledExceptionFilter, which prevented Windows Error Reporting from getting the original exception. Their handler did some internal cleanup, then called abort, at which point WER was finally able to create a dump.
For anyone encountering this sort of problem, I recommend checking that the installed handlers (the return values of SetUnhandledExceptionFilter, set_terminate, etc) are what you expect (null if you're relying on WER, or your own handlers or CrashRpt).
It is better to make your application self-dump-writer when it crashes. You just need to call SetUnhandledExceptionFilter function, specify the call back. In the call back, use MiniDumpWriteDump function with MiniDumpWithFullMemory dump type.
Your exception filter will be called whenever unhandled (by your code) exception occurs. In the call back, it will be better to enumerate and suspend all other threads of your process.
You may also need to install a hook for SetUnhandledExceptionFilter itself! Why? Well, CRT would always disable any installed exception filter, and by the hooked function, you may avoid that.

What are the reasons for SEHException in multi threaded DirectX code?

I am working with a fairly large and complicated codebase I am not familiar with, which uses DirectWrite for text rendering. I don't know whether if it matters, but this is a WinRT (Windows Store) application.
The app is multi threaded, and some DirectX calls are done from a background (not UI) thread. All the BeginDraw and EndDraw calls are made from the UI thread, and some of the DrawText calls are made from background threads. I made sure that none of these calls overlap, and DrawText is only called between BeginDraw and EndDraw pairs.
However, sometimes I get a SEHException when calling the DrawText method. It never happens when it is called from the UI thread, only when called from a background thread.
This does not happen on every computer. It very often happens on some, and can not be reproduced at all on others.
The exact message the VS Debugger says is this: "A first chance exception of type 'System.Runtime.InteropServices.SEHException' occurred in x.DLL", and it happens when I call the ID2D1DeviceContext.DrawText() function, and the debugger breaks in the d2d1.h header file at the DrawText call.
Is there any way to debug or diagnose what is going wrong when a SEHException is thrown? (It is clearly connected to calling the DrawText function from a background thread.)
And apart from this, are there any general rules of calling DirectX functions from background threads? (I made sure that the D2D1Factory gets created with D2D1_FACTORY_TYPE_MULTI_THREADED option.)
A SEH exception is simply a Win32 exception: the exception mechanism used by Windows internally, which is a different from C++ exceptions.
So all a SEH exception tells you is that "some Windows function threw an exception". As with any other exception, that can happen for a lot of reasons.
So asking what the "usual reasons" for such an exception is, is kind of futile.
What are the usual reasons for a std::exception? There are a lot of them.
The problem only occurred on a single computer, and couldn't be reproduced on others. And after a video card driver update it ceased on that computer as well, so it might have been some driver issue (there have already been some problems with the Nvidia drivers for Windows 8).
Make sure to turn on the DirectX debug runtime. Run in visual studio. It has very verbose error checking that is printed to the console. To turn it on run the DirectX control panel.

Capturing OpenMP exceptions with SetUnhandledExceptionFilter

I have an exception handler set up using SetUnhandledExceptionFilter, which works fine. However, if I throw an exception from within OpenMP code, I get the standard "the application crashed" window and the handler is not called -- however, I can attach a debugger just fine and see that the call stack is ending with _CxxThrowException and continues into KernelBase.dll!RaiseException. I know that an OpenMP program which throws exceptions inside the parallel regions is wrong, but I'd still like to get a crash dump. How can I get my exception handler get called in this case?
Should be possible, especially as the debugger manges to get an "Unhandled exception" window, when attached to the application after the crash (i.e. I can get a nice stack trace and stuff.) This is on Windows 7 with VC++ 2010.
(Eventually, each thread actually calls my exception handler. If it crashes, and I select 'Debug', and then continue on each unhandled exception, the handler eventually gets called and it also manages to write out a meaningful minidump. Wtf?)
Interesting. Going out on a limb, I'll wager that the OpenMP concurrency runtime doesn't honor the SetUnhandledExceptionFilter (which will work for "standard" threads), and isn't integrated into this feature of Structured Exception Handling.
Note this warning from the MSDN page on Exception Handling in the Concurrency Runtime
To prevent abnormal termination of your application, make sure that your code handles exceptions when it calls into the runtime. Also handle exceptions when you call into external code that uses the Concurrency Runtime, for example, a third-party library.
Perhaps you can try wrapping your OpenMP stuff in the style of exception handling outlined above, and then see if you can re-package and throw it (outside of OpenMP context) to get caught by the filter?

DUMP in unhandled C++ exception

In MSVC, how can I make any unhandled C++ exception (std::runtime_error, for instance) crash my release-compiled program so that it generates a dump with the full stack from the exception throw location?
I have installed NTSD in the AeDebug registry and can generate good dumps for things like memory access violation, so the matter here comes down to crashing the program correctly, I suppose.
Thanks in advance.
I finally cracked it down.
Use the set_terminate() function to register a handler for every thread
In you main function(), make it impossible for external DLLs (event Windows') to successfully call SetUnhandledExceptionFilter(). A great article on how to do that here: http://www.debuginfo.com/articles/debugfilters.html#overwrite .
As for the handle itself, it is quite straightforward:
void Terminate()
{
OutputDebugStringA("Terminate\r\n");
RaiseException(0xE0000010, EXCEPTION_NONCONTINUABLE, 0, 0);
}
Calling RaiseException() like the above example is enough to make the process crash and produce my mush desired dump.
Just so you know, the problem I was having was:
The IPHelper Windows API loads dynamically another Windows DLL
This DLL uses Windows own version of the C runtime (MSVCRT instead of MSVCRT90)
The new C++ runtime calls SetUnhandledExceptionFilter() on startup to catch C++ exceptions. Since the latest filter for C++ exceptions is the one who gets to call the handle set by set_terminate(), my handle wasn't called.
SetUnhandledExceptionFilter and DebugBreak should probably do the job.
Edit: oops, rereading, you want to deal with uncaught C++ exceptions. That would be a bit trickier to do well -- by the time you (normally) get wind of a C++ exception, it's already unwound the stack back to the level of the handler. You don't have any real way to know an exception has been thrown until a catch is invoked, but by then the stack has been unwound.
Look into using the Windows Debugger.
Windbg – wraps KD and NTSD with a decent UI.
Also check out ADPlus that comes with the Windows Debugger.
Here is a good place to start learning how to use it.

How do I avoid popping up an error dialog when my MSVS C++ app crashes

When my Visual Studio 2008 C++ command-line application crashes, it sometimes produces this dialog.
CommandProcessor.exe has encountered a problem and needs to close.
We are sorry for the inconvenience. If you were in the middle of something, the information you were working on might be lost. For more informaiton about this error, click here.
I tried this in Release and in Debug mode.
(By the way, the debugger shows that this is a divide by zero error.)
If it is going to crash, I don't want this dialog, which blocks the application. How do I compile my application so that crashes do not produce the dialog?
With /EHa option you can use catch(...) to catch all exceptions included structured exceptions and write a console message. You can also use VC++ - specific __try for structured exception handling instead, but that's a bit harder to code.
However this will not protect you against situations when terminate() is called by the C++ runtime - like when an exception escapes a destructor during stack unwinding - you will also have to change the terminate() handler by calling set_terminate().
Read series of articles Exception Handling and Crash Reporting. It is possible to catch exception and process it as you wish (you can save crash dump for instance).