My multithreaded application is crashing randomly and a core dump is not getting generated which is making my analysis very very difficult. I have the ulimit set to unlimited and I have the dumper running. I see a signal 11 generated on crash which should have ideally generated a coredump.
I have had coredumps generated before for prior issues on the same system.
Note:
- directory has appropriate permissions.
- exe has the required permissions.
OS used - QNX
Any suggestions are welcome.
I got the core-dump generated by manually adding a call to abort() in a function which i knew would be called when a signal to terminate the process is sent.
Related
I normally use GDB (in Linux, with the Qt Creator debugger GUI) to debug. But right now I have a crash that refuses to ever happen when running under the debugger, yet happens easily when running outside of it.
How do I get a stack trace of my crash, in these circumstances?
A linux-specific solution is OK.
Note: I'm talking about running a debug build only, even when it's run outside the debugger.
The easiest way to be sure you can obtain a stacktrace after a crash is to run
ulimit -c unlimited
In your shell before starting the program.
This will ensure that the kernel is allowed to produce a "core dump" of unlimited size (for many distros the default size is 0) when a program crashes.
That core file can then be loaded into gdb as gdb programfile corefile and then the command thread apply all bt will give you stack traces for all threads for that specific crash (use just bt if you only care about the crashing thread).
You can also use the pstack program to get a stacktrace from a running program.
I develop complex server program for Linux. It uses thousands of parallel SSL connections and implements custom protocol that serves client applications.
The problem is that this program sometimes crashes. It shows Terminated message in terminal and stops to work. I enabled automatic core dump to get crash report and analyze it with GDB. However, for any reason Linux (CentOS) doesn't create core dump. It looks like it creates core dump for segmentation fault only (I run ulimit -c unlimited, and when I provoke segmentation fault, dump is created).
Why Linux doesn't create core dump when software stops to work and shows Terminated message, and how to debug such issues if they exist in production enviroment only and never appear in test computer?
I have an C++ application, that crashes on a computer of some person on the other end of the world. There's no way for me to simulate it or get the same computer. The person is no developer, so I cannot really ask him to install Visual Studio or something. I have pretty deep debug logs, but they didn't reveal anything usable.
Is there a tool, that could generate the stack trace of the application at the moment of the crash? Such thing is available inside OSX, but seems that Windows doesn't have it.
You can use procdump. It can be setup as a debugger to automatically create dumps for crashing processes.
Procdump is part of Sysinternal tools and can be found at:
http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx
Relevant switches:
Create a dump for a hung application:
Write a mini dump for a process named 'hang.exe' when one of it's
Windows is unresponsive for more than 5 seconds:
C:\>procdump -h hang.exe hungwindow.dmp
Automatically create dumps for crashing apps:
Register as the Just-in-Time (AeDebug) debugger. Makes full dumps in
c:\dumps.
C:\>procdump -ma -i c:\dumps
Create a dump for a pid:
C:\>procdump <PID>
You can read the dump files using windbg: Getting started with dump file analysis
Use google's lib that makes minidump for msvc to debug.
CrashRpt
Since google seems to lead here for all "stack traces of windows" you can also get the stack trace of a running process using "sleepy" or "very sleepy" utilities.
another option: run it in gdb.exe, hit ctrl+c
c++ program terminated with no exceptions or stacktrace
I have a multi threaded application
If one of my threads has an access violation with reading out of bounds on an array (or any seg fault condition) my entire application immediately terminates.
If this happens on my windows counter part using visual studio I get a nice stacktrace of where the error was, and what the issue was.
I desperately need this type of debugging environment to be able to succeed at my project. I have too many threads and too many developers running different parts of the project to have one person not handle an exception properly and it destroys the entire project.
I am running Fedora Core 14
I am compiling with gcc 4.5.1
gdb is fedora 7.2-16.fc14
My IDE is eclipse Juno I am using the CDT builder
my toolchain is the cross GCC and my builder is the CDT Internal Builder
Is there ANY setting for the gdb or gcc or eclipse that will help me detect these type of situations?
That's what's supposed to happen. Under Unix, you get a full
core dump (which you can examine in the debugger), provided
you've authorized them. (ulimits -c—traditionally, they
were authorized by default, but Linux seems to have changed
this.)
Of course, to get any useful information from the core dump,
you'll need to have compiled the code with symbol information,
and not stripped it later. (On the other hand, you can copy the
core dump from your users machine onto your development machine,
and see what happened there.)
You're definitely looking for core dumps, as James Kanze wrote.
I would just add that core dumps will show you where the program crashed, which is not necessarily the same place as where the problem (memory corruption etc.) occurred. And of course some out-of-bounds reads/writes may not exhibit themselves by crashing immediately.
You can narrow the search by enabling check for incorrect memory allocations/deallocations in glibc. The simplest way is to set environmental variable MALLOC_CHECK_. Set it to 2 and glibc will check for heap corruption with every memory allocation/deallocation and abort the program when it founds any problem (producing a core dump, if enabled). This often helps to get closer to the real problem.
http://www.gnu.org/software/libc/manual/html_node/Heap-Consistency-Checking.html
I'm trying to generate a release build for a C++ application that I've written. The application runs fine (debug & release) when you run it from within VS2008; but when you run the executable it crashes nearly every single time.
Now, is there a hack so I can run this application as a standalone application without having to run through all of the code and finding the bug that is causing it?
Thanks in advance.
In short, no.
you will have to find the bug, if it works within VS, then I'd hazard a guess that it is a timing issue, possibly you're overwriting shared thread data, this would be less likely (though still possible to see) inside VS as its being run in a debug environment which slows it down a bit.
If you want help finding your bug, then tell us more. Otherwise, build your release with debug symbols (pdbs), install DrWatson as the system debugger and run it standalone. When it crashes DrWatson will create a minidump file, load this into WinDbg (my favourite) and you'll be able to see exactly where your bug is (it'll even tell you that the dump contains an exception and show you it by default. You need to add your source code path and path to your symbols in WinDbg to get it to do this correctly).
Then you will also know how to diagnose crashes when the app is run on-site too.
Are you loading external resources? If you are check that your relative paths are correct in the C++ program.
One possibility is that your program uses uninitialized heap data. Launching a program from the debugger enables the NT debug heap, which causes the heap allocator to fill new memory blocks with a fill pattern, and also enables some heap checking. Launching the same program from outside the debugger leaves the NT debug heap disabled, but if the program was linked against the debug version of the C runtime, then the CRT debug heap will still be enabled.
A much less likely possibility is that your program requires SeDebugPrivilege to be set in its process token. The debugger enables this privilege in its process token, which has the side effect that all programs launched from the debugger inherit this privilege. If your program tries to use OpenProcess()/ReadProcessMemory()/WriteProcessMemory() and doesn't handle errors correctly, it's conceivable that it could crash.
There are a few possibilities. Besides what has already been mentioned, running an app from Visual Studio will execute in the same security context as the Visual Studio instance. So if, for instance, you are working on Vista, you might be hitting an unhandled security violation if you're trying to access protected files, or the registry.
What if you build a debug version and run it standalone? Does it crash? If so, you can usually break into the debugger from there and get a call stack to see what the malfunction is.
From the details you've given, it sounds like there may be a library issue. Are you running the program on the same computer? If not then you'll also have to deploy the appropriate libraries for your application. If you are running on the same computer but outside of the dev environment, ensure that your application can see the appropriate libraries.
Best way i have found to debug in release is to create a crash dump when an crash happens and the dump then allows me to load debug symbols on my dev computer and find out whats going on. More info here: http://www.debuginfo.com/articles/effminidumps.html
You can also go to file => open in Visual Studio and open the .exe, so you are not starting it under the debugger per se. Not sure if it will help.
http://blogs.msdn.com/saraford/archive/2008/08/21/did-you-know-you-can-debug-an-executable-that-isn-t-a-part-of-a-visual-studio-project-without-using-tools-attach-to-process-296.aspx