I'm having some difficulties determining what is causing a process to exit. I have a breakpoint in some shutdown code that I'm debugging, but, after breaking in the debugger at the breakpoint and stepping once, the whole process exits immediately. Every thread reports an exit code of -1 in the output window. There are a large number of threads in the process at that time, and the code base is quite large, making searching for the culprit difficult.
I've tried installing a std::atexit function, but this doesn't get hit. I've also tried overriding SetUnhandledExceptionFilter, in case it is caused by a crash, and it also doesn't get hit. The project has exceptions disabled (#define _HAS_EXCEPTIONS=0), so I cannot call std::set_terminate or std::set_unexpected.
Is there some other way to determine what is causing the process to exit? Some option to break in the debugger when the process is about to terminate?
Run your app with debugger and read the debug output. If the app terminates because C++ exceptions, or SEH, you’ll read it in the output window.
If you’ll see nothing interesting there, it means your app called ExitProcess/ExitThread/exit, or worse, TerminateProcess/TerminateThread/_exit.
You can put breakpoints on these. Set a breakpoint at startup, launch debugger. Ensure you have debug symbols loaded for relevant DLLs, kernel32.dll for ExitProcess and friends, some other DLL for exit, e.g. ucrtbase.dll. Press “New / Function breakpoint” in the Breakpoints window, type e.g. “ExitProcess”, press OK.
You can also try using gflags tool from Windows SDK.
If you’ll find (by reading Windows Logs > Application) the reason was self exit, you can check “Enable dump collection” in gflags, then you’ll be able to load the dump in WinDBG and get the complete call stack telling you who called what.
Unfortunately, the latest version of the tool is broken beyond repair.
But you can install older Windows SDK. You only need “Debugging tools for Windows” from there, no need to install the complete SDK.
Related
Well, I am writing a small application using C++, Qt and some libraries. Some computation is executed when hitting a UI button.
Now I found that after around one minute the application exits reproducibly and reliably without any visible reason. The error seems not happen statistically but deterministically. I guess there is some bug triggering the crash but I can not find any hint where/why this is happening.
Honestly I have never seen such a crash during the last decade, normally I get even a small indication what happened.
I am using GCC 11.3, GDB, Qt Creator on Kubuntu 22.10 Linux. I built a debug executeable and run it. This is what I checked:
Run using debugger, Qt creator console says "...app has finished with exit code 0."
Run without debugger, Qt creator console says "...app crashed."
Run on bash it only says "Getötet" (which is german and means "killed")
No error related application console output or dialogs are generated
Logging component (which logs all signals, intercept-able C++ exit calls, Qt output and more) does not log anything unexpected
Application does not exit by main()-return (debugger break-point is not triggered and corresponding log trace isn't generated)
No signal dialog (like SIGTERM and SIGABRT) is shown
No assertion seems to be triggered (would generate output and would halt the debugger)
I enabled break-on-abort but nothing happens
I enabled break-on-C++-exception but nothing happens
I searched my code for exit calls (especially with a return code of 0) but there is no such call.
After all I have no idea how to debug the issue other than cluttering my entire application with debug messages and check for missing prints.
Any idea what could happening here and why I don't get even the slightest indication for the error reason?
Is there a way to track down the issue using GDB?
I will try the exit break-point...
You should set a breakpoint on the exit_group and exit system calls, like so:
(gdb) catch syscall exit_group
(gdb) catch syscall exit
But I have doubts since my logging component is using on_exit() to capture exits.
There are many ways for an application to exit without passing through exit and without executing any on_exit-registered functions.
Here are some:
_exit(0)
syscall(SYS_exit, 0)
syscall(SYS_exit_group, 0)
Ok, solution was pretty easy. It wasn't any leak or memory error. Also it wasn't an exit call of any kind.
Reason was basically an internal buffer that could grow beyond the size of physically installed memory under some circumstances. That triggered the OS' OOM-Killer process.
Problem didn't produce any immediate error indication but it could have been tracked down by monitoring system resources and by checking the kernel log (that contained corresponding "oom-kill" and "oom_reaper" messages).
I have a Qt GUI program, inside it I could click a button to load/unload many dock widgets.
I have the problem that when I click the button to load/unload dock widgets, the programm crash with saying that
Debug Assertion Failed, Expression: _BLOCK_TYPE_IS_VALID(pHead->nBlockUse)
It doesn't happen every time. (Actually very rare to happen.)
And when I check the Windows's event log, it says the application hang with a cross thread dead lock.
But most people online said that the _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) means a memory error.
I just don't what's going on...
It is a very big program by someone else and the bug happens very rarely...
What I could do now to locate the bug?
EDIT:
Hi, I have got the crash dump file, and I have seen that my program stop at a worker thread with the call stack: > ntdll.dll!_NtWaitForMultipleObjects#20() + 0x15 bytes
How could I trace back to the source code that the program actually stop?
That usually means you're trying to access an illegal memory block inside an std container.
To debug this properly, just look at the stack in the Call Stack window, look up the stack until you get to your code, and see why the value is invalid.
It's hard to describe it, but briefly, here is what you should do:
Install Application Verifier and run it.
Ctrl+A, select your executable.
Deselect all tests in the right pane, select only Basic->Heaps.
Ensure you have 'Full heap' enabled and 'Traces' enabled (properties via right click on 'Heaps' item).
Save. You may close Application Verifier now.
Launch WinDBG of proper architecture (the same as your app).
Ctrl+E, select your executable.
The program will be stopped on first instruction, run it using F5
The probability you'll hit the bug will be much higher. You'll also may found memory access issue you were not aware of before. When you hit one of them, the debugger will stop with one of 'Verifier stops' and you'll see the message in console telling you which command you can use to investigate further. Usually you'll be able to see detailed info about the heap using !heap -p -a <address>, including allocation and deallocation stacks.
Remember, that Application Verifier checks are enabled even when Application Verifier application is not started. You need to run Application Verifier, disable the checks and press 'Save' to actually disable them.
Hope this will help, at least a bit. Read more about Application Verifier techniques on the Internet.
I have an application for which I do not have the code, and a dll for which I do have the code. I need to be able to debug into the dll, but lacking the source for the exe, how do I do this?
The dll code is mfc c++; I believe that the main app is the same thing as well.
I've tried doing a 'set target application' deal, where I set the application that the dll will be called from, and the application crashes a horrible, horrible death when called that way. I don't know if the fault lies with this dll or with the executable for that behavior, and it's just one of the myriad things I'd like to solve.
I'm thinking that there should be some call to allow the dll to spin indefinitely until a debugger is attached to the process, at which point I should be able to debug the dll by attaching to the process. Does that make sense? Is there a better way to do this?
I used to use the DebugBreak function for this. You could have it called conditionally based on the presence of a particular file, perhaps.
#ifdef DEBUG
if (... file exists...) {
DebugBreak();
}
#endif
This will halt application execution until you attach a debugger or terminate the app.
If the application is linked against a non-debug DLL and does not have debug symbols itself, this isn't really likely to be fruitful. You might want to look here for information on using windows symbol packages to help you if you're curious about what's going in inside windows DLL's, but by and large, an application which does not have debug info and which you can't compile isn't debuggable in any meaningful way.
There is a registry setting called ImageFileExecutionOptions that can be set up to launch a debugger whenever your DLL is loaded. I used to use it to debug ISAPI extensions. Here is a link to a decent blog entry about it.
__asm int {3};
in your DLL main. Then attach a debugger to the process?
If this kills the process, then it probably has it's own int3 trap and is quitting. Are you trying to debug a copy protected game or anything like that? As they tend to do that kind of tricksy behaviour.
With a DLL project, you should be able to tell Visual Studio to start debugging and it will ask you for an executable name. Enter your exe there. I've done this a lot for when I've worked on DLL code that was called from another process. Works for straight DLLs as well as COM components.
It might also help to set some breakpoints in your code ahead of time if you have an idea of where the problem might be.
Update: Since that does not work for you, the only other thing I can think of would be to attach to the running exe, but that could be problematic if your code gets loaded before you have a chance to get in there.
Wait until a debugger is present:
while(!IsDebuggerPresent())
{
Sleep(0); // yield
}
MSDN Documentation: IsDebuggerPresent().
Ensure that the application is indeed using the DLL that you built, in debug mode, with symbols. You can verify this by using a program such as Process Explorer (in this application, enable the lower pane in the View menu and select DLLs).
Then, in Visual Studio's Debug menu, select "Attach to Process", and select the application that uses your DLL. Your debug breakpoints should become filled in, if and when your DLL is loaded.
I recently had to use one of the methods listed here:
http://blogs.msdn.com/greggm/archive/2004/07/08/177418.aspx
Does that help?
Here's a simple solution: add a Sleep(10000); in DllMain (or some other startup code) and then use Tools / Attach to Process to attach your debugger while the code is sleeping.
I'm thinking that there should be some
call to allow the dll to spin
indefinitely until a debugger is
attached to the process, at which
point I should be able to debug the
dll by attaching to the process. Does
that make sense? Is there a better way
to do this?
Why not do it the way you are describing it? Just start the application that you want to debug. Attach the debugger to it, either through Visual Studio or simply by right clicking on the application in the task manager and selecting Debug. Once the debugger is attached, set a break point with F9 at suitable location in your dll code.
I've tried doing a 'set target
application' deal, where I set the
application that the dll will be
called from, and the application
crashes a horrible, horrible death
when called that way. I don't know if
the fault lies with this dll or with
the executable for that behavior, and
it's just one of the myriad things I'd
like to solve.
Starting a process inside the debugger causes Windows to enable the NT debug heap. It sounds like the application or DLL has heap corruption or relies on the value of uninitialized heap memory.
You can disable the NT debug heap by setting the environment variable _NO_DEBUG_HEAP to 1 (on XP and later). This may make it possible to get the application to not die a horrible death when started from the debugger.
Starting the application outside the debugger will also result in the NT debug heap being disabled, and attaching a debugger later will not enable it.
If my C++ app crashes on Windows I want to send useful debugging information to our server.
On Linux I would use the GNU backtrace() function - is there an equivalent for Windows?
Is there a way to extract useful debugging information after a program has crashed? Or only from within the process?
(Advice along the lines of "test you app so it doesn't crash" is not helpful! - all non-trivial programs will have bugs)
The function Stackwalk64 can be used to snap a stack trace on Windows.
If you intend to use this function, you should be sure to compile your code with FPO disabled - without symbols, StackWalk64 won't be able to properly walk FPO'd frames.
You can get some code running in process at the time of the crash via a top-level __try/__except block by calling SetUnhandledExceptionFilter. This is a bit unreliable since it requires you to have code running inside a crashed process.
Alternatively, you can just the built-in Windows Error Reporting to collect crash data. This is more reliable, since it doesn't require you to add code running inside the compromised, crashed process. The only cost is to get a code-signing certificate, since you must submit a signed binary to the service. https://sysdev.microsoft.com/en-US/Hardware/signup/ has more details.
You can use the Windows API call MiniDumpWriteDump if you wish to roll your own code. Both Windows XP and Vist automate this process and you can sign up at https://winqual.microsoft.com to gain access to the error reports.
Also check out http://kb.mozillazine.org/Breakpad and http://www.codeproject.com/KB/debug/crash_report.aspx for other solutions.
This website provides quite a detailed overview of stack retrieval on Win32 after a C++ exception:
http://www.eptacom.net/pubblicazioni/pub_eng/except.html
Of course, this will only work from within the process, so if the process gets terminated or crashes to the point where it terminates before that code is run, it won't work.
Generate a minidump file. You can then load it up in windbg or Visual Studio and inspect the entire stack where the crash occurred.
Here's a good place to start reading.
Its quite simple to dump the current stackframe addresses into a log file. All you have to do is get such a function called on program faults (i.e. a interrupt handler in Windows) or asserts. This can be done at released versions as well. The log file then can be matched with a map file resulting in a call stack with function names.
I published a article about this some years ago.
See http://www.ddj.com/architect/185300443
Let me describe how I handle crashes in my C++/WTL application.
First, in the main function, I call _set_se_translator, and pass in a function that will throw a C++ exception instead of using structured windows exceptions. This function gets an error code, for which you can get a Windows error message via FormatMessage, and a PEXCEPTION_POINTERS argument, which you can use to write a minidump (code here). You can also check the exception code for certain "meltdown" errors that you should just bail from, like EXCEPTION_NONCONTINUABLE_EXCEPTION or EXCEPTION_STACK_OVERFLOW :) (If it's recoverable, I prompt the user to email me this minidump file.)
The minidump file itself can be opened in Visual Studio like a normal project, and providing you've created a .pdb file for your executable, you can run the project and it'll jump to the exact location of the crash, together with the call stack and registers, which can be examined from the debugger.
If you want to grab a callstack (plus other good info) for a runtime crash, on a release build even on site, then you need to set up Dr Watson (run DrWtsn32.exe). If you check the 'generate crash dumps' option, when an app crashes, it'll write a mini dump file to the path specified (called user.dmp).
You can take this, combine it with the symbols you created when you built your server (set this in your compiler/linker to generate pdb files - keep these safe at home, you use them to match the dump so they can work out the source where the crash occurred)
Get yourself windbg, open it and use the menu option to 'load crash dump'. Once it's loaded everything you can type '~#kp' to get a callstack for every thread (or click the button at the top for the current thread).
There's good articles to know how to do this all over the web, This one is my favourite, and you'll want to read this to get an understanding of how to helpyourself manage the symbols really easily.
You will have to set up a dump generation framework in your application, here is how you may do it.
You may then upload the dump file to the server for further analysis using dump analyzers like windbg.
You may want to use adplus to capture the crash callstack.
You can download and install Debugging tools for Windows.
Usage of adplus is mentioned here:
Adplus usage
This creates the complete crash or hang dump. Once you have the dump, Windbg comes to the rescue. Map the correct pdbs and symbols and you are all set to analyze the dump. To start with use the command "!analyze -v"
Sometimes my c++ program crashes in debug mode, and what I got is a message box saying that an assertion failed in some of the internal memory management routines (accessing unallocated memory etc.). But I don't know where that was called from, because I didn't get any stack trace. How do I get a stack trace or at least see where it fails in my code (instead of library/ built-in routines)?
If you have a crash, you can get information about where the crash happened whether you have a debug or a release build. And you can see the call stack even if you are on a computer that does not have the source code.
To do this you need to use the PDB file that was built with your EXE. Put the PDB file inside the same directory as the EXE that crashed. Note: Even if you have the same source code, building twice and using the first EXE and the second PDB won't work. You need to use the exact PDB that was built with your EXE.
Then attach a debugger to the process that crashed. Example: windbg or VS.
Then simply checkout your call stack, while also having your threads window open. You will have to select the thread that crashed and check on the callstack for that thread. Each thread has a different call stack.
If you already have your VS debugger attached, it will automatically go to the source code that is causing the crash for you.
If the crash is happening inside a library you are using that you don't have the PDB for. There is nothing you can do.
If you run the debug version on a machine with VS, it should offer to bring it up and let you see the stack trace.
The problem is that the real problem is not on the call stack any more. If you free a pointer twice, that can result in this problem somewhere else unrelated to the program (the next time anything accesses the heap datastructures)
I wrote this blog on some tips for getting the problem to show up in the call stack so you can figure out what is going on.
http://www.atalasoft.com/cs/blogs/loufranco/archive/2007/02/06/6-_2200_Pointers_2200_-on-Debugging-Unmanaged-Code.aspx
The best tip is to use the gflags utility to make pointer issues cause immediate problems.
You can trigger a mini-dump by setting a handler for uncaught exceptions. Here's an article that explains all about minidumps
Google actually implemented their own open source crash handler called BreakPad, which also mozilla use I think (that's if you want something more serious - a rich and robust crash handler).
If I remember correctly that message box should have a button which says 'retry'. This should then break the program (in the debugger) at the point where the assertion happened.
CrashFinder can help you locate the place of the exception given the DLL and the address of the exception reported.
You can take this code and integrate it into your application to have a stack trage automatically generated when there is an uncaught exception. This is generally performed using __try{} __except{} or with a call to SetUnhandledExceptionFilter which allows you to specify a callback to all unhandled exceptions.
You can also have a post-mortem debugger installed on the client system. This is a decent, general way to get information when you do not have dump creation built into your application (maybe for an older version for which you must still get information).
Dr. Watson on Windows can be installed by running: drwtsn32 -i Running drwtsn32 (without any options) will bring up the configuration dialog. This will allow the creation of crash dump files, which you can later analyze with WinDbg or something similar.
You can use Poppy for this. You just sprinkle some macros across your code and it will gather the stack trace, together with the actual parameter values, local variables, loop counters, etc. It is very lightweight so it can be left in the release build to gather this information from crashes on end-user machines