I develop complex server program for Linux. It uses thousands of parallel SSL connections and implements custom protocol that serves client applications.
The problem is that this program sometimes crashes. It shows Terminated message in terminal and stops to work. I enabled automatic core dump to get crash report and analyze it with GDB. However, for any reason Linux (CentOS) doesn't create core dump. It looks like it creates core dump for segmentation fault only (I run ulimit -c unlimited, and when I provoke segmentation fault, dump is created).
Why Linux doesn't create core dump when software stops to work and shows Terminated message, and how to debug such issues if they exist in production enviroment only and never appear in test computer?
Related
My multithreaded application is crashing randomly and a core dump is not getting generated which is making my analysis very very difficult. I have the ulimit set to unlimited and I have the dumper running. I see a signal 11 generated on crash which should have ideally generated a coredump.
I have had coredumps generated before for prior issues on the same system.
Note:
- directory has appropriate permissions.
- exe has the required permissions.
OS used - QNX
Any suggestions are welcome.
I got the core-dump generated by manually adding a call to abort() in a function which i knew would be called when a signal to terminate the process is sent.
I normally use GDB (in Linux, with the Qt Creator debugger GUI) to debug. But right now I have a crash that refuses to ever happen when running under the debugger, yet happens easily when running outside of it.
How do I get a stack trace of my crash, in these circumstances?
A linux-specific solution is OK.
Note: I'm talking about running a debug build only, even when it's run outside the debugger.
The easiest way to be sure you can obtain a stacktrace after a crash is to run
ulimit -c unlimited
In your shell before starting the program.
This will ensure that the kernel is allowed to produce a "core dump" of unlimited size (for many distros the default size is 0) when a program crashes.
That core file can then be loaded into gdb as gdb programfile corefile and then the command thread apply all bt will give you stack traces for all threads for that specific crash (use just bt if you only care about the crashing thread).
You can also use the pstack program to get a stacktrace from a running program.
I have run my app for several hours, and it has crashed. (c++ app, on LINUX os)
I know that when running app via gdb, we can get the line code & reason of the crash.
But unfortunately I forgot to run it via gdb :(
Is there a way to get the reason of the crash ?
You can enable unlimited core dump sizes by
ulimit -c unlimited
This will write down a core file in case of a crash into the same directory from where you started the program. Afterwards you can load it with the gdb option --core.
I'm having a problem on my embedded linux board. My program runs, then somewhere in a particular function throws a segmentation fault. So i'm trying to track down why this is happening so I can fix it.
The problem is that my target board runs a filesystem image which does not include gdb or gdbserver. Is there any way I can debug this application as it runs on the embedded target without rebuilding the kernel or my application?
c++ program terminated with no exceptions or stacktrace
I have a multi threaded application
If one of my threads has an access violation with reading out of bounds on an array (or any seg fault condition) my entire application immediately terminates.
If this happens on my windows counter part using visual studio I get a nice stacktrace of where the error was, and what the issue was.
I desperately need this type of debugging environment to be able to succeed at my project. I have too many threads and too many developers running different parts of the project to have one person not handle an exception properly and it destroys the entire project.
I am running Fedora Core 14
I am compiling with gcc 4.5.1
gdb is fedora 7.2-16.fc14
My IDE is eclipse Juno I am using the CDT builder
my toolchain is the cross GCC and my builder is the CDT Internal Builder
Is there ANY setting for the gdb or gcc or eclipse that will help me detect these type of situations?
That's what's supposed to happen. Under Unix, you get a full
core dump (which you can examine in the debugger), provided
you've authorized them. (ulimits -c—traditionally, they
were authorized by default, but Linux seems to have changed
this.)
Of course, to get any useful information from the core dump,
you'll need to have compiled the code with symbol information,
and not stripped it later. (On the other hand, you can copy the
core dump from your users machine onto your development machine,
and see what happened there.)
You're definitely looking for core dumps, as James Kanze wrote.
I would just add that core dumps will show you where the program crashed, which is not necessarily the same place as where the problem (memory corruption etc.) occurred. And of course some out-of-bounds reads/writes may not exhibit themselves by crashing immediately.
You can narrow the search by enabling check for incorrect memory allocations/deallocations in glibc. The simplest way is to set environmental variable MALLOC_CHECK_. Set it to 2 and glibc will check for heap corruption with every memory allocation/deallocation and abort the program when it founds any problem (producing a core dump, if enabled). This often helps to get closer to the real problem.
http://www.gnu.org/software/libc/manual/html_node/Heap-Consistency-Checking.html