I have a multi threaded program written in c++ running on ARM.
I have enabled global core dump of applications by running:
echo "/var/log/core.%e.%t" > /proc/sys/kernel/core_pattern
ulimit -c unlimited
An exception is thrown in the program which isn't being caught.
I open the core dump that was generated using eclipse.
I expect to see the code that throw the exception in the callstack.
But I can't see it.
The rest of the callstack for the other threads look ok
I tried to manually change the $SP of the first line in the callstack to the address defined in the verbose termiate line: 0xb6eba0d8 but that cause the callstack to only show the syscall line
How can I see more lines in this thread's stack?
There were two problems:
The stack was unwound. Until GCC 8 - When using std::thread, libstdc++'s function that calls the user's thread function contains a try/catch all statement. Using pthread instead solves this issue. See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55917 for details
I was using libstdc++ without debug symbols while trying to debug. The function that was called by the thread was compiled with debug symbols but that wasn't enough. I used the file command to see that the library inside /usr/lib didn't have debug symbols, so I used the libraries from my Linaro distribution installation which did have.
Related
Is there any gcc option I can set that will give me the line number of the segmentation fault?
I know I can:
Debug line by line
Put printfs in the code to narrow down.
Edits:
bt / where on gdb give No stack.
Helpful suggestion
I don't know of a gcc option, but you should be able to run the application with gdb and then when it crashes, type where to take a look at the stack when it exited, which should get you close.
$ gdb blah
(gdb) run
(gdb) where
Edit for completeness:
You should also make sure to build the application with debug flags on using the -g gcc option to include line numbers in the executable.
Another option is to use the bt (backtrace) command.
Here's a complete shell/gdb session
$ gcc -ggdb myproj.c
$ gdb a.out
gdb> run --some-option=foo --other-option=bar
(gdb will say your program hit a segfault)
gdb> bt
(gdb prints a stack trace)
gdb> q
[are you sure, your program is still running]? y
$ emacs myproj.c # heh, I know what the error is now...
Happy hacking :-)
You can get gcc to print you a stacktrace when your program gets a SEGV signal, similar to how Java and other friendlier languages handle null pointer exceptions. See my answer here for more details:
how to generate a stacktace when my C++ app crashes ( using gcc compiler )
The nice thing about this is you can just leave it in your code; you don't need to run things through gdb to get the nice debug output.
If you compile with -g and follow the instructions there, you can use a command-line tool like addr2line to get file/line information from the output.
Run it under valgrind.
you also need to build with debug flags on -g
You can also open the core dump with gdb (you need -g though).
If all the preceding suggestions to compile with debugging (-g) and run under a debugger (gdb, run, bt) are not working for you, then:
Elementary: Maybe you're not running under the debugger, you're just trying to analyze the postmortem core dump. (If you start a debug session, but don't run the program, or if it exits, then when you ask for a backtrace, gdb will say "No stack" -- because there's no running program at all. Don't forget to type "run".) If it segfaulted, don't forget to add the third argument (core) when you run gdb, otherwise you start in the same state, not attached to any particular process or memory image.
Difficult: If your program is/was really running but your gdb is saying "No stack" perhaps your stack pointer is badly smashed. In which case, you may be a buffer overflow problem somewhere, severe enough to mash your runtime state entirely. GCC 4.1 supports the ProPolice "Stack Smashing Protector" that is enabled with -fstack-protector-all. It can be added to GCC 3.x with a patch.
There is no method for GCC to provide this information, you'll have to rely on an external program like GDB.
GDB can give you the line where a crash occurred with the "bt" (short for "backtrace") command after the program has seg faulted. This will give you not only the line of the crash, but the whole stack of the program (so you can see what called the function where the crash happened).
The No stack problem seems to happen when the program exit successfully.
For the record, I had this problem because I had forgotten a return in my code, which made my program exit with failure code.
When debugging a C++ program with the GNU GDB debugger, I can step over the next line of code with the GDB command:
next
However, when an exception is thrown within that next line, like e.g.
throw SomeException();
then GDB continues to run until the next breakpoint instead of stopping within the first line of the catch block.
Is this a bug within GDB, or am I just using the wrong command?
I'm using GDB version 7.7 on MinGW32 / Windows.
On Linux this works properly. In particular there is a special debugging marker (either a function or an "SDT probe", depending on how things were built) in the low-level unwinding code that is used when an exception is thrown. GDB puts a breakpoint at this spot. When this breakpoint is hit, GDB examines the target stack frame of the throw and, if it is above the nexting frame, puts a temporary breakpoint at the target of the throw.
This required some changes in GDB, but also some changes in the C++ runtime in order to inform GDB about throws. As far as I know, nobody has ever done the work to port this code to MinGW. Maybe it could be done by modifying the appropriate unwind-mumble.c file in the GCC sources.
Is there anyway to debug a link error or any kind of error that may occur before the execution of the main() function using GDB?
Is there anyway to debug a link error
Presumably you are asking about runtime link error (e.g. `error: libfoo.so: no such file or directory'), and not about (static) link step of your build process.
The trick is to set a breakpoint on exit or (exit_group on Linux) system call with e.g. catch syscall exit. You will then be stopped inside ld.so at the point where it gives up running your binary.
or any kind of error that may occur before the execution of the main() function using GDB?
Any other kind of error, e.g. SIGSEGV can be debugged "normally" -- for signal you don't need to do anything at all -- GDB will just stop. For other errors, just set a breakpoint as usual.
On way to debug initialization code (even if you don't have symbols) goes like this:
gdb somebinary
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
etc.
info file
Symbols from "somebinary".
Local exec file:
`/somebinary', file type elf64-x86-64.
Entry point: 0x4045a4, etc.
break *0x4045a4
run
...Breakpoint 1, 0x00000000004045a4 in ?? ()
From here on you can proceed as usual.
I have a program running in Linux and It's been mysteriously crashing. I already know one way to know where it crashes at is to use GDB. But I don't want to attach to it every time I restart it (do this a lot since I'm testing it). Is there an alternative way to do this?
First use ulimit -c unlimited to allow crashed programs to write core dumps.
After the program crashes, you'll find a core dump file, called core, or perhaps core.<pid> if your program is multithreaded.
You can load this into GDB to examine the state at the point of the crash with gdb program core.
First do a ulimit -c unlimited, so the program will leave a core dump.
Then, when it crashes, invoke gdb with the core dump, to read the
state of the program at the moment of the crash.
You can configure your OS to dump a core file any time a program crashes. You can then examine the core to determine the crash location.
-> compile the code with gdb flags enabled.
gcc -o < binary name > -g < file.c > (assuming it is a c/c++ program)
-> run the executable withing gdb.
gdb < binary name >
after this there are ways to find the crash location:
1. stepwise execution.
2. run the code, it crashes (as expected), type "where" within gdb (without quotes) it gives the backtrace. from that, you can find out the address.
here is a nice quick guide to gdb : http://www.cs.cmu.edu/~gilpin/tutorial/
I have a large C++ function which uses OpenCV library and running on Windows with cygwin g++ compiler. At the end it gives Aborted(core dumped) but the function runs completely before that. I have also tried to put the print statement in the end of the function. That also gets printed. So I think there is no logical bug in code which will generate the fault.
Please explain.
I am also using assert statements.But the aborted error is not due to assert statement. It does not say that assertion failed. It comes at end only without any message.
Also the file is a part of a large project so I cannot post the code also.
gdb results:
Program received signal SIGABRT, Aborted.
0x7c90e514 in ntdll!LdrAccessResource () from /c/WINDOWS/system32/ntdll.dll
It looks like a memory fault (write to freed memory, double-free, stack overflow,...). When the code can be compiled and run under Linux you can use valgrind to see if there are memory issues. Also you can try to disable parts of the application until the problem disappears, to get a clue where the error happens. But this method can also give false positives, since memory related bugs can cause modules to fail which are not the cause of the error. Also you can run the program in gdb. But also here the position the debugger points to may not be the position where the error happened.
You don't give us much to go on. However, this looks like you are running into some problem when freeing resources. Maybe a heap corruption. Have you tried running it under gdb and then looking where it crashes? Also, check if all your new/delete calls match.
Load the core dump together with the binary into gdb to get an idea at which location the problem list. Command line is:
gdb <path to the binary> <path to the core file>
For more details on gdb see GDB: The GNU Project Debugger.
Run it through AppVerifier and cdb.
E.g.
cdb -xd sov -xd av -xd ch <program> <args>