I have a huge code-base I am unfamiliar with, and the program terminates abnormally because a thread somewhere is calling __fastfail. This is based on the message, which ends with
... Fatal program exit requested.
The call stack has no symbols because it's inside C++ 2015 runtime (ucrtbase.dll). The call appears to be made on a thread other than my main thread. This mysterious thread only starts just before the issue happens so I can't catch the act of it starting in the debugger - I don't know what's starting it, and what causes the whole process in the first place.
I have SEH in my main() using __try/__catch, so any unhandled exceptions should be getting trapped there. Instead I am guessing something somewhere bubbles up to the runtime and results in __fastfail.
I tried peppering all my threads with SEH just like main(), tried hooking abort(), exit() and terminate(), and can't find the problem. How do I debug this, any tips?
WinDbg
I would say this is a nice task for WinDbg. WinDbg is part of the Debugging Tools for Windows and it's free. Install both versions, x64 and x86 so that you can debug any kind of application.
Start WinDbg, use the correct bitness
Run your executable under WinDbg (File/Open executable). It will stop at the initial breakpoint.
Set up the symbols, at least .symfix c:\debug\symbols and .reload. As mentioned by #James McNellis, the symbols are available and this will download them when needed.
Continue running the application with g
Reproduce the issue
When WinDbg stops,
create a crash dump with .dump /ma c:\debug\mydump.dmp so you can analyze it later
get information about the exception with .exr -1
switch to the thread that caused the exception with ~#s
look at the callstack with k
Learning WinDbg is a hard task, since most things are done via cryptic commands and not via UI, but it can do almost everything.
For more specific problems when you have some more clues, ask additional questions using the windbg tag.
Visual Studio
Also Visual Studio can download symbols (PDB files; callstack information) from Microsoft Servers.
Go to Tools | Options ... in the main menu
Select Debugging | Symbols from the options menu
Enter a directory name if you want the file to be stored in a specific directory.
Here it is how it looks like in Visual Studio 2015 Community Edition:
Hans and James are right, getting a readable call stack was essential in solving this case.
The codebase installed a pure call handler which was performing a fast fail. Pure handlers fall through SEH. One of the 40-something threads in the program had a pure call error because some other thread partially cleaned up program state which the problem thread then tried to access. Once I had the symbols Visual Studio broke into C++ runtime library at a place where it was calling the purecall handler, and I traced it back into the program where they installed their own.
The way they did fast fail was convoluted and resolved into some RtlXXX function that apparently could not be trapped with signal(SIGABRT) because I did try that before.
Thanks again for the help everyone!
Related
I cannot get dump files created with Task Manager (32 or 64 bit) or Process Explorer to give useful results in WinDBG or Visual Studio 2015, whereas the dump written directly from VS works brilliantly in both. I need Task Manager dumps to work so that I can analyse dump files sent by my end users.
I have reduced the problem to the simplest possible Win32 application, created in VS 2015 C++, with a deliberate NULL pointer write to cause an exception. If I run the program in VS and use Save Dump As when the exception occurs, then the dump file can be used in VS and WinDBG to see the code causing the problem. This is as expected.
However, if I run the application outside of VS, then Windows shows the usual dialog:
“Win32Project.exe has stopped working … Debug / Close Program”.
Whilst this dialog is still active I go to Task Manager 32bit and select Create Dump file. But loading this dump file into VS or WinDBG gives no useful information. In particular VS shows Exception Code as “not found”. Clicking on “Debug with Native only” causes “The application is in break mode to be shown”. See below…
I am running a new Win 10 64bit PC. DMP, PDB and EXE files are in the same directory, and I have tried endlessly with symbol directories
Visual Studio 2015 output after loading .DMP file:
Dump Summary
------------
Dump File: Win32Project1 (4).DMP : C:\Users\Rob\AppData\Local\Temp\Win32Project1 (4).DMP
Last Write Time: 24/08/2017 16:38:27
Process Name: Win32Project1.exe : C:\Temp\ConsoleAp2\Win32Project2\Debug\Win32Project1.exe
Process Architecture: x86
Exception Code: not found
Exception Information:
Heap Information: Present
System Information
------------------
OS Version: 10.0.15063
CLR Version(s):
Modules
-------
Module Name Module Path Module Version
----------- ----------- --------------
Win32Project1.exe C:\Temp\ConsoleAp2\Win32Project2\Debug\Win32Project1.exe 0.0.0.0
ntdll.dll C:\Windows\System32\ntdll.dll 10.0.15063.447
kernel32.dll C:\Windows\System32\kernel32.dll 10.0.15063.296
...
Why does that happen what you see?
It works in Visual Studio because the debugger is already attached. The debugger is informed about the exception before the process terminates. The debugger will halt the process before the Windows Error Reporting Dialog occurs and create a crash dump when the original exception is still active.
To learn more about the process on how exceptions are passed from the program to the debugger (first chance), back to the program (catch block), to the debugger again (second chance) and finally to the OS, google for the term "exception dispatching".
It does not work with Task Manager, because exception dispatching is already in its last state, which is "get handled by the OS". This makes Windows halt the program by making use of a breakpoint. It then shows that dialog. When you create a crash dump now, it's too late and it's very hard to get useful information from that crash dump.
What options do you have?
a) Windows Error Reporting
The mechanism that applies here is called Windows Error Reporting. If you had an account at Microsoft, your customer could simply click the "submit" button. You would then get some information from Microsoft. The way you ask the question makes me assume that you don't have such an account.
Then, use a feature called LocalDumps (MSDN). It's a Registry key to configure Windows to save a crash dump on disk. Make sure you understand what you need from such a dump in order to configure it correctly. In doubt, have a look at How do I take a good crash dump for .NET and use the same settings (full memory user mode mini dump). It will be good for C++ as well.
It might even be possible to activate this Registry key while the dialog is shown but I have not confirmed this any more since 2014 and I can't recommend it.
Check if your settings work by using your null pointer dereference sample application. To do so, rename your executable to the same name as your actual program.
b) Attaching a debugger
Attach a debugger to the process, then let the application continue. Press "Debug" on the dialog and confirm the message that says "a debugger is already attached". The second chance exception is thrown again, the debugger will get it and you can take a crash dump.
If you need screenshots, see my article about it
Note that in approach b) you can make many mistake which will lead to improper results. The safe way is to activate LocalDumps as described in a)
I recommend giving ProcDump from Windows Sysinternals a shot. You can set it to capture a dump of your app when it crashes, or set it as the Just-in-Time debugger and have it write a dump for any app crash.
See the documentation and examples in the above website for more detail.
I set Qt Creator to break when a C++ exception is thrown:
I then tested it with this code:
try {
throw std::runtime_error("error");
} catch (std::exception &e) {
qDebug("%s", e.what());
}
But it didn't break on throw std::runtime_error("error");. I'm using CDB, not GDB, because I'm using the MSVC Kit.
Edit: There is another question where CDB is working for the OP, even if slowly. So it should work in principle. My configuration is: Qt Creator 3.3.0, compiling with Qt4/MSVC 9.0 (x86), the debugger is CDB 6.2.9200.16384.
Edit 2: This is what I'm getting in the CDB log window (I made a diff between the CDB log with and without the breakpoint):
<bu100400 CxxThrowException
<!qtcreatorcdbext.breakpoints -t 1 -v
<!qtcreatorcdbext.pid -t 2
dATTEMPT SYNC
d*** Bp expression 'CxxThrowException' contains symbols not qualified with module name.
1 breakpoint(s) pending...
*** Unable to resolve unqualified symbol in Bp expression 'CxxThrowException' from module 'C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_5.82.7601.18201_none_ec80f00e8593ece5\comctl32.dll'.
Full CDB log (in case needed): http://pastebin.com/jhNRy9bE
Edit 3: #HansPassant's explained why it fails in the comments:
Keep in mind that you are using a very old version of MSVC++, big changes at VS2012. The pastebin shows it being out of sync pretty badly, never getting to the DLL that contains __CxxThrowException#8 (MSVCR90D.dll) before the exception is thrown. It is simple with the sxe debugger command, automatic break when any exception is thrown.. Maybe you shouldn't be using QT's UI at all, it looks too gimpy. – Hans Passant 10 hours ago
Just look at the trace, the debugger shows what DLLs it is searching for "CxxThrowException". It never gets to msvcr90d.dll. And the exception is thrown while it is searching for the symbol after which it all ends. Completely out of sync. – Hans Passant 56 mins ago
I'll just write up why this is going wrong, a workaround is going to be difficult to find. The debugger trace in your pastebin tells the tale.
The basic issue is that the communication between the debugger and the QT front-end is rather poor. And in your case it gets out of sync, QT doesn't wait long enough for the debugger to complete the command. QT tries to set a breakpoint on the msvcr90d.dll!__CxxThrowException#8 function, the one that raises a C++ exception in the Microsoft CRT. This function can be present in more than one executable file if the program uses multiple CRTs. A pretty common mishap, caused by building with /MT. And sometimes intentional if you use a well-isolated DLL that you interface with by using COM.
This takes a while as you might imagine, the complaint in the linked question, the debugger has to plow through the symbol information for every DLL that's loaded. It will take especially long if the PDB for the DLL needs to be downloaded from the symbol server and doesn't otherwise get cached so it is available the next time you debug. Not your issue afaict, it does setup the cache location to C:\Users\sasho\AppData\Local\Temp\symbolcache. Go have a look-see to verify that you do see PDBs for the operating system DLLs there.
This operation is tricky, the debugger doesn't give a good signal that it is done searching the DLLs. What the QT should do is verify the debugger feedback against the list of DLLs it obtained. It does not do that, it issues the g command before the debugger is done searching. Could be a timeout that is too short but it actually looks like QT doesn't count on the debugger performing this command in the background. A convenience to a human, not exactly very helpful here :)
There ought to be a way to configure CDB to not perform this search in the background. This is well-hidden, I don't see anything in the debugger.chm help file but it probably wasn't updated in a while. Google doesn't help either. I'd recommend you ask a question about it. Most significantly perhaps is that you have a rather major mismatch in version numbers. The compiler you use is 2008 vintage, the debugger is quite new, SDK 8.0 version, I can't tell what QT version you use.
So a possible workaround is to intentionally use an older version of CDB, one that's more likely to have been tested with the QT front-end version you use. Download the corresponding SDK version, version 6.0 matches the VS2008 time frame. I think the "Debugging Tools for Windows" was still a separate download back then and not yet included in the SDK. Another workaround is to stop relying on the friendly QT front-end and learn to drive CDB from the command prompt. Moderately more useful is WINDBG, uses the same debugging engine but has a GUI interface. Just moderate, it is still mostly prompt driven. You do lose several days of your life learning the commands however. Getting the debugger to break when an exception is thrown is trivial, use the sxe command.
There are a number of errors going on here but I'm sure they are all linked, so first off I'm getting Access violation reading location exceptions at locations (generally near 0x00000000)
but I only get these exceptions in release and not in debug, even if i setup debugging to do so. Also I found where the error happens but when I step over that it goes in to a complete unrelated function that is not called in the line or in any of the functions in that line of code. But when similar code is called it still goes to this unrelated function but doesn't fail. This all only happens in release versions. I know which piece of code is causing the error as if I comment it out its all fine. Also in the watch window in release all the numbers are wrong most of the time, which has never been a problem but it could help also a added some code to it to output the values of variables to a file and the variables are all fine, the pointers are sensible, but when the exception happens it always says their 0x00000020 but when I look at the log they are fine. This is in 32 bit , but on 64 bit windows 7 in c++. Please any help would be good!
Visual Studio's debugger will cause the program to use a debug heap that can hide uninitialized memory if you use the option "Start with Debugging" to start your program, in release or debug. You can try to run your program built in the Debug configuration without the debugger attached, and then use the Attach to Process... option in the debug menu to attach to your executable.
If this doesn't help, I recommend using windbg to debug your program. For help getting started with windbg, take a look at the answers on this question.
Once windbg is configured and symbol paths are set up, you can use the "!analyze -v" option once the program crashes to get a lot of information about the crash automatically.
I left a program running last night, it worked fine for about 5 hours and then one of its built-in self-diagnostic tests detected a problem and brought up a dialog box telling me the issue. The program was built with debug information (/Zi). Is it possible to somehow get the debugger started so I can examine the value of some variables within the program? Or is it too late?
You can attach the debugger to the running process:
Debug > Attach to Process...
Just open up the program's solution first.
Assuming you've still got the error dialog on the screen you can break into the program and work back up the call stack examining variables etc.
For the future crashes ... if you have windbg or Visual Studio Professional, you can debug crash dumps, even when program isn't running. It is quite useful sometimes. See "MiniDumpWriteDump" on MSDN for more info.
Other than that it is "Attach to process".
Professional edition of Visual Studio have Just-in-Time debugger, that will kick in as soon as anything crashes, even if MSVC wasn't running. It will also locate source code (if debug info and source code are available) and open/create solution for you.
There is an option in the Debug menu to attach the debugger to a running process, IIRC.
I'm trying to generate a release build for a C++ application that I've written. The application runs fine (debug & release) when you run it from within VS2008; but when you run the executable it crashes nearly every single time.
Now, is there a hack so I can run this application as a standalone application without having to run through all of the code and finding the bug that is causing it?
Thanks in advance.
In short, no.
you will have to find the bug, if it works within VS, then I'd hazard a guess that it is a timing issue, possibly you're overwriting shared thread data, this would be less likely (though still possible to see) inside VS as its being run in a debug environment which slows it down a bit.
If you want help finding your bug, then tell us more. Otherwise, build your release with debug symbols (pdbs), install DrWatson as the system debugger and run it standalone. When it crashes DrWatson will create a minidump file, load this into WinDbg (my favourite) and you'll be able to see exactly where your bug is (it'll even tell you that the dump contains an exception and show you it by default. You need to add your source code path and path to your symbols in WinDbg to get it to do this correctly).
Then you will also know how to diagnose crashes when the app is run on-site too.
Are you loading external resources? If you are check that your relative paths are correct in the C++ program.
One possibility is that your program uses uninitialized heap data. Launching a program from the debugger enables the NT debug heap, which causes the heap allocator to fill new memory blocks with a fill pattern, and also enables some heap checking. Launching the same program from outside the debugger leaves the NT debug heap disabled, but if the program was linked against the debug version of the C runtime, then the CRT debug heap will still be enabled.
A much less likely possibility is that your program requires SeDebugPrivilege to be set in its process token. The debugger enables this privilege in its process token, which has the side effect that all programs launched from the debugger inherit this privilege. If your program tries to use OpenProcess()/ReadProcessMemory()/WriteProcessMemory() and doesn't handle errors correctly, it's conceivable that it could crash.
There are a few possibilities. Besides what has already been mentioned, running an app from Visual Studio will execute in the same security context as the Visual Studio instance. So if, for instance, you are working on Vista, you might be hitting an unhandled security violation if you're trying to access protected files, or the registry.
What if you build a debug version and run it standalone? Does it crash? If so, you can usually break into the debugger from there and get a call stack to see what the malfunction is.
From the details you've given, it sounds like there may be a library issue. Are you running the program on the same computer? If not then you'll also have to deploy the appropriate libraries for your application. If you are running on the same computer but outside of the dev environment, ensure that your application can see the appropriate libraries.
Best way i have found to debug in release is to create a crash dump when an crash happens and the dump then allows me to load debug symbols on my dev computer and find out whats going on. More info here: http://www.debuginfo.com/articles/effminidumps.html
You can also go to file => open in Visual Studio and open the .exe, so you are not starting it under the debugger per se. Not sure if it will help.
http://blogs.msdn.com/saraford/archive/2008/08/21/did-you-know-you-can-debug-an-executable-that-isn-t-a-part-of-a-visual-studio-project-without-using-tools-attach-to-process-296.aspx