Well, I am writing a small application using C++, Qt and some libraries. Some computation is executed when hitting a UI button.
Now I found that after around one minute the application exits reproducibly and reliably without any visible reason. The error seems not happen statistically but deterministically. I guess there is some bug triggering the crash but I can not find any hint where/why this is happening.
Honestly I have never seen such a crash during the last decade, normally I get even a small indication what happened.
I am using GCC 11.3, GDB, Qt Creator on Kubuntu 22.10 Linux. I built a debug executeable and run it. This is what I checked:
Run using debugger, Qt creator console says "...app has finished with exit code 0."
Run without debugger, Qt creator console says "...app crashed."
Run on bash it only says "Getötet" (which is german and means "killed")
No error related application console output or dialogs are generated
Logging component (which logs all signals, intercept-able C++ exit calls, Qt output and more) does not log anything unexpected
Application does not exit by main()-return (debugger break-point is not triggered and corresponding log trace isn't generated)
No signal dialog (like SIGTERM and SIGABRT) is shown
No assertion seems to be triggered (would generate output and would halt the debugger)
I enabled break-on-abort but nothing happens
I enabled break-on-C++-exception but nothing happens
I searched my code for exit calls (especially with a return code of 0) but there is no such call.
After all I have no idea how to debug the issue other than cluttering my entire application with debug messages and check for missing prints.
Any idea what could happening here and why I don't get even the slightest indication for the error reason?
Is there a way to track down the issue using GDB?
I will try the exit break-point...
You should set a breakpoint on the exit_group and exit system calls, like so:
(gdb) catch syscall exit_group
(gdb) catch syscall exit
But I have doubts since my logging component is using on_exit() to capture exits.
There are many ways for an application to exit without passing through exit and without executing any on_exit-registered functions.
Here are some:
_exit(0)
syscall(SYS_exit, 0)
syscall(SYS_exit_group, 0)
Ok, solution was pretty easy. It wasn't any leak or memory error. Also it wasn't an exit call of any kind.
Reason was basically an internal buffer that could grow beyond the size of physically installed memory under some circumstances. That triggered the OS' OOM-Killer process.
Problem didn't produce any immediate error indication but it could have been tracked down by monitoring system resources and by checking the kernel log (that contained corresponding "oom-kill" and "oom_reaper" messages).
Related
I'm working on a qt5 application, using qtcreator 4.9.0. My application is crashing due to SIGILL, but that is not the main issue.
What I don't like is that if I launch it in debug mode, I can see it clearly displayed (I get a dialog telling me that the SIGILL signal was received).
If I just run it (NOT in debug mode) I can tell it's crashed, since the application stops working and the exit code is bad.
But i would also like to see the same dialog I see when in debug mode, or something similar.
Is it possible?
Edit:
Sorry, the question does lack details.
I am working in windows, using mingw. I am relying on SIGILL because I'm using the -fsanitize=undefined g++ switch, in order to trap undefined behaviors.
Since I don't have libubsan in my sistem, I also am using -fsanitize-undefined-trap-on-error: this allows me to use the sanitizer without libubsan, but causes SIGILL to be raised.
So the signal is actually desired in my case, and I'm not debating on how to show it to the user, rather than to find a more convenient way for myself: I like that when the application crashes, a dialog appears, since it is very explicit.
When I say "in debug mode" I mean launching the application from the Debug menu, or simply by pressing F5: in this case I can clearly see the signal has been raised, since the "crash" dialog appears. If instead I just run the application (CTRL+R) I surely can tell the application isn't working: it kinda freezes and exits with an abnormal exit code, but that's it.
I mean, I can live with that, but it would be nice to see the dialog.
Expanding my comment a bit.
First, my assumption is that the platform is some unix-like system, SIGILL suggests that. No clue about Windows.
Second, I assume "debug mode" is "debug build", not "execution under debugger's control".
Given the above assertions are true:
The most natural way seems to be signal or sigaction.
See: https://linux.die.net/man/2/sigaction
However, there is only a specific subset of functions one is allowed to call from a signal handler, so you might want to check on that first: https://pubs.opengroup.org/onlinepubs/000095399/functions/xsh_chap02_04.html#tag_02_04_01
Also, QT docs have some examples on how to deal with POSIX signals
https://doc.qt.io/qt-5/unix-signals.html
Last but not least, as it has already been mentioned in the comments:
be careful what you present to the user, from both the UX perspective as well as from the security standpoint.
In a regular case I'd probably either leave the signal unhadled, or abort in order to generate a core dump if this happens (don't recall if SIGILL does that by default).
Even better option might be to fail at application startup and request the user to do whatever it's needed (fail fast, basically).
But again, "context is king" here and maybe due to security it's favourable not to generate any dumps and just fail as silently as possible, OP has to answer for him/herself here.
I'm having some difficulties determining what is causing a process to exit. I have a breakpoint in some shutdown code that I'm debugging, but, after breaking in the debugger at the breakpoint and stepping once, the whole process exits immediately. Every thread reports an exit code of -1 in the output window. There are a large number of threads in the process at that time, and the code base is quite large, making searching for the culprit difficult.
I've tried installing a std::atexit function, but this doesn't get hit. I've also tried overriding SetUnhandledExceptionFilter, in case it is caused by a crash, and it also doesn't get hit. The project has exceptions disabled (#define _HAS_EXCEPTIONS=0), so I cannot call std::set_terminate or std::set_unexpected.
Is there some other way to determine what is causing the process to exit? Some option to break in the debugger when the process is about to terminate?
Run your app with debugger and read the debug output. If the app terminates because C++ exceptions, or SEH, you’ll read it in the output window.
If you’ll see nothing interesting there, it means your app called ExitProcess/ExitThread/exit, or worse, TerminateProcess/TerminateThread/_exit.
You can put breakpoints on these. Set a breakpoint at startup, launch debugger. Ensure you have debug symbols loaded for relevant DLLs, kernel32.dll for ExitProcess and friends, some other DLL for exit, e.g. ucrtbase.dll. Press “New / Function breakpoint” in the Breakpoints window, type e.g. “ExitProcess”, press OK.
You can also try using gflags tool from Windows SDK.
If you’ll find (by reading Windows Logs > Application) the reason was self exit, you can check “Enable dump collection” in gflags, then you’ll be able to load the dump in WinDBG and get the complete call stack telling you who called what.
Unfortunately, the latest version of the tool is broken beyond repair.
But you can install older Windows SDK. You only need “Debugging tools for Windows” from there, no need to install the complete SDK.
I have a Qt GUI program, inside it I could click a button to load/unload many dock widgets.
I have the problem that when I click the button to load/unload dock widgets, the programm crash with saying that
Debug Assertion Failed, Expression: _BLOCK_TYPE_IS_VALID(pHead->nBlockUse)
It doesn't happen every time. (Actually very rare to happen.)
And when I check the Windows's event log, it says the application hang with a cross thread dead lock.
But most people online said that the _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) means a memory error.
I just don't what's going on...
It is a very big program by someone else and the bug happens very rarely...
What I could do now to locate the bug?
EDIT:
Hi, I have got the crash dump file, and I have seen that my program stop at a worker thread with the call stack: > ntdll.dll!_NtWaitForMultipleObjects#20() + 0x15 bytes
How could I trace back to the source code that the program actually stop?
That usually means you're trying to access an illegal memory block inside an std container.
To debug this properly, just look at the stack in the Call Stack window, look up the stack until you get to your code, and see why the value is invalid.
It's hard to describe it, but briefly, here is what you should do:
Install Application Verifier and run it.
Ctrl+A, select your executable.
Deselect all tests in the right pane, select only Basic->Heaps.
Ensure you have 'Full heap' enabled and 'Traces' enabled (properties via right click on 'Heaps' item).
Save. You may close Application Verifier now.
Launch WinDBG of proper architecture (the same as your app).
Ctrl+E, select your executable.
The program will be stopped on first instruction, run it using F5
The probability you'll hit the bug will be much higher. You'll also may found memory access issue you were not aware of before. When you hit one of them, the debugger will stop with one of 'Verifier stops' and you'll see the message in console telling you which command you can use to investigate further. Usually you'll be able to see detailed info about the heap using !heap -p -a <address>, including allocation and deallocation stacks.
Remember, that Application Verifier checks are enabled even when Application Verifier application is not started. You need to run Application Verifier, disable the checks and press 'Save' to actually disable them.
Hope this will help, at least a bit. Read more about Application Verifier techniques on the Internet.
Well. that´s the question. Just that.
I got an app made with SDL and OpenGL. SDL opens an extra window (which is the console) additional to the graphical one. When i execute i´m getting a 3 output error, the console tells me. And it gets closed (the graphical one).
But i know this happens when a SIGSEGV signal is received (don´t know how to capture it) and it appears in my IDE (Code::blocks) while debugging. But this time nothing appears, and everything works all right. But when executing it crashes..
What the...
What kind of error can i expect?. Sometimes it gets closed, sometimes it doesn´t. How to know what kind of problem i got?.
SIGSEGV is a segmentation fault, you're trying to access memory that isn't accessible to your process.
Assuming you're on a UNIXy system, you should be able to get the program to core dump and then look at the core dump in a debugger; alternatively, use a memory debugger like Valgrind to pinpoint the memory management issue that's causing this problem.
I have a Windows application written in C++ that occasionally evaporates. I use the word evaporate because there is nothing left behind: no "we're sorry" message from Windows, no crash dump from the Dr. Watson facility...
On the one occasion the crash occurred under the debugger, the debugger did not break---it showed the application still running. When I manually paused execution, I found that my process no longer had any threads.
How can I capture the reason this process is terminating?
You could try using the adplus utility in the windows debugging tool package.
adplus -crash -p yourprocessid
The auto dump tool provides mini dumps for exceptions and a full dump if the application crashes.
If you are using Visual Studio 2003 or later, you should enable the debuggers "First Chance Exception" handler feature by turning on ALL the Debug Exception Break options found under the Debug Menu | Exceptions Dialog. Turn on EVERY option before starting the debug build of the process within the debugger.
By default most of these First Chance Exception handlers in the debugger are turned off, so if Windows or your code throws an exception, the debugger expects your application to handle it.
The First Chance Exception system allows debuggers to intercept EVERY possible exception thrown by the Process and/or System.
http://support.microsoft.com/kb/105675
All the other ideas posted are good.
But it also sounds like the application is calling abort() or terminate().
If you run it in the debugger set a breakpoint on both these methods and exit() just for good measure.
Here is a list of situations that will cause terminate to be called because of exceptions going wrong.
See also:
Why destructor is not called on exception?
This shows that an application will terminate() if an exceptions is not caught. So stick a catch block in main() that reports the error (to a log file) then re-throw.
int main()
{
try
{
// Do your code here.
}
catch(...)
{
// Log Error;
throw; // re-throw the error for the de-bugger.
}
}
Well, the problem is you are getting an access violation. You may want to attach with WinDBG and turn on all of the exception filters. It may still not help - my guess is you are getting memory corruption that isn't throwing an exception.
You may want to look at enabling full pageheap checking
You might also want to check out this older question about heap corruption for some ideas on tools.
The most common cause for this kind of sudden disappearance is a stack overflow, usually caused by some kind of infinite recursion (which may, of course, involve a chain of several functions calling each other).
Is that a possibility in your app?
You could check the Windows Logs in Event Viewer on Windows.
First of all I want to say that I've only a moderate experience on windows development.
After that I think this is a typical case where a log may help.
Normally debugging and logging supply orthogonal info. If your debugger is useless probably the log will help you.
This could be a call to _exit() or some Windows equivalent. Try setting a breakpoint on _exit...
Have you tried PC Lint etc and run it over your code?
Try compiling with maximum warnings
If this is a .NET app - use FX Cop.
Possible causes come to mind.
TerminateProcess()
Stack overflow exception
Exception while handling an exception
The last one in particular results in immediate failure of the application.
The stack overflow - you may get a notification of this, but unlikely.
Drop into the debugger, change all exception notifications to "stop always" rather than "stop if not handled" then do what you do to cause the program failure. The debugger will stop if you get an exception and you can decide if this is the exception you are looking for.
I spent a couple hours trying to dig into this on Visual Studio 2017 running a 64-bit application on Windows 7. I ended up having to set a breakpoint on the RtlReportSilentProcessExit function, which lives in the ntdll.dll file. Just the base function name was enough for Visual Studio to find it.
That said, after I let Visual Studio automatically download symbols for the C standard library, it also automatically stopped on the runtime exception that caused the problem.