Is possible to recover the exact values of argv and argc parameters of a main after the application crashed?
I need to use only the application core-dump and gdb debugger on Linux.
Yes, if application was compiled with debug info. Open core dump in gdb and find frame containing main function. Then go to this frame and print values of argv and argc. Here is sample gdb session.
[root#localhost ~]# gdb ./a.out core.2020
GNU gdb (GDB) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/a.out...done.
[New Thread 2020]
warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Core was generated by `./a.out'.
Program terminated with signal 6, Aborted.
#0 0x0027b424 in __kernel_vsyscall ()
(gdb) bt
#0 0x0027b424 in __kernel_vsyscall ()
#1 0x00b28b91 in raise () from /lib/libc.so.6
#2 0x00b2a46a in abort () from /lib/libc.so.6
#3 0x007d3397 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#4 0x007d1226 in ?? () from /usr/lib/libstdc++.so.6
#5 0x007d1263 in std::terminate() () from /usr/lib/libstdc++.so.6
#6 0x007d13a2 in __cxa_throw () from /usr/lib/libstdc++.so.6
#7 0x08048940 in main (argv=1, argc=0xbfcf1754) at test.cpp:14
(gdb) f 7
#7 0x08048940 in main (argv=1, argc=0xbfcf1754) at test.cpp:14
14 throw std::runtime_error("123");
(gdb) p argv
$1 = 1
(gdb) p argc
$2 = (char **) 0xbfcf1754
(gdb)
Looks like you need to start from the basics..!!
compiler your application code with -g flag, make sure you dont strip it.
Say if I wanted to compile hello.c
gcc -c -g hello.c -o hello.o
gcc hello.o -o hello
now if you dont want to debug
ulimit -c unlimited
./hello
when the application crashes a core file wiil be generated.
To examine the core file
"gdb ./hello core.$$$" this will list you your stack.
you can also choose to debug the image
gdb hello
There are a lot of stuffs over the internet about GDB, do go through them.
Related
Code (m1.cpp):
#include <iostream>
using namespace std;
int main (int argc, char *argv[])
{
cout << "running m1" << endl;
return 0;
}
GDB Version: GNU gdb (GDB) 7.6.2
Built using: g++ -g m1.cpp
Command line history:
(gdb) b main
Breakpoint 1 at 0x40087b: file m1.cpp, line 6.
(gdb) r
Starting program: .../a.out
Program received signal SIGSEGV, Segmentation fault.
0x00002aaaaaac16a0 in strcmp () from /lib64/ld-linux-x86-64.so.2
(gdb) c
Continuing.
Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb)
When I run without setting any breakpoints, it runs without errors.
As requested:
(gdb) bt
#0 strcmp () from /lib64/ld-linux-x86-64.so.2
#1 in check_match.12104 () from /lib64/ld-linux-x86-64.so.2
#2 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#3 in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#4 in _dl_relocate_object () from /lib64/ld-linux-x86-64.so.2
#5 in dl_main () from /lib64/ld-linux-x86-64.so.2
#6 in _dl_sysdep_start () from /lib64/ld-linux-x86-64.so.2
#7 in _dl_start () from /lib64/ld-linux-x86-64.so.2
#8 in _start () from /lib64/ld-linux-x86-64.so.2
#9 in ?? ()
I was able to replicate the OP's observed behavior (using the same compile and getting the same backtrace). The behavior was persistent across a range GDBs and GCCs. I noticed that the symptom goes away when I unset SHELL. In my normal environment I use tcsh (version 1.15.00). If SHELL is set, then (I believe) gdb launches using tcsh. If I unset SHELL, gdb launches using sh. This is enough for me to make forward progress. I don't have a crisp explanation for what would be different in tcsh to manifest the issue but if others have the same behavior, it may shed more light on the issue.
I checked that in my GNU gdb version 7.11.1. It worked really fine in it.
I first compiled the same program and built it using:
g++ -g m1.cpp
Then, ran the executable in the gdb as follows:
gdb -q ./a.out
And did the same things you mentioned. It worked fine.
Update your gdb, and check that again and let know.
I have followed the guide here: debugging pintool guide, but I cannot get GDB to find the debugging symbols for my pintool.
First I compiled my pintool with debug information
lotus#c02-0:~/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools$ make DEBUG=1 obj-intel64/dst7.so
g++ -shared -Wl,--hash-style=sysv -Wl,-Bsymbolic -Wl,--version-script=../../../source/include/pin/pintool.ver -g -o obj-intel64/dst7.so obj-intel64/dst7.o -L../../../intel64/lib -L../../../intel64/lib-ext -L../../../intel64/runtime/glibc -L../../../extras/xed-intel64/lib -lpin -lxed -lpindwarf -ldl
In two different terminal windows I did the following:
In the first window:
lotus#c02-0:~/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools$ ~/PerforceArch/home/Shadi/HLS/pin/intel64/bin/pinbin -pause_tool 10 -t obj-intel64/dst7.so -- ls
Pausing for 10 seconds to attach to process with pid 2214
To load the tool's debug info to gdb use:
add-symbol-file /home/lotus/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools/obj-intel64/dst7.so 0x7fa8952e44c0 -s .data 0x7fa895c0f720 -s .bss 0x7fa895c10c40
In the the second window:
lotus#c02-0:~$ /usr/bin/gdb ~/PerforceArch/home/Shadi/HLS/pin/intel64/bin/pinbin
GNU gdb (GDB) SUSE (7.4.50.20120603-2.1.2)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/lotus/PerforceArch/home/Shadi/HLS/pin/intel64/bin/pinbin...(no debugging symbols found)...done.
(gdb) attach 2214
Attaching to program: /home/lotus/PerforceArch/home/Shadi/HLS/pin/intel64/bin/pinbin, process 2214
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /home/lotus/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools/obj-intel64/dst7.so...(no debugging symbols found)...done.
Loaded symbols for /home/lotus/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools/obj-intel64/dst7.so
0x00007fa895ce6c80 in __nanosleep_nocancel () from /lib64/libc.so.6
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.15-22.6.4.x86_64 libgcc47-debuginfo-4.7.1_20120723-1.1.1.x86_64 libstdc++47-debuginfo-4.7.1_20120723-1.1.1.x86_64
(gdb) add-symbol-file /home/lotus/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools/obj-intel64/dst7.so 0x7fa8952e44c0 -s .data 0x7fa895c0f720 -s .bss 0x7fa895c10c40
add symbol table from file "/home/lotus/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools/obj-intel64/dst7.so" at
.text_addr = 0x7fa8952e44c0
.data_addr = 0x7fa895c0f720
.bss_addr = 0x7fa895c10c40
(y or n) y
Reading symbols from /home/lotus/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools/obj-intel64/dst7.so...(no debugging symbols found)...done.
The problem is that on the last line of the second window, it says no debugging symbols were found.
Before I compiled with debug symbols on, I had compiled with debug symbols off. Before recompiling with debug symbols on, I deleted the .so file but not the .o file. Deleting the old .o file and then compiling again with debug symbols on, fixed the problem.
here is my CPP program
#include <iostream>
int main(int argc, char *argv[]){
return 0;
}
compiling and executing this, causes a seg fault:
[18:08 ~] > g++ test.cpp
[18:09 ~] > ./a.out
Segmentation fault
Does anybody have an idea about this issue please?
Thanks a lot for your help.
Here are some details that might be useful:
[18:13 ~] > ldd a.out
linux-vdso.so.1 => (0x00007fff3fdff000)
libstdc++.so.6 => /usr/lib/gcc/x86_64-redhat-linux/3.4.6/libstdc++.so.6 (0x00007fc8f4a53000)
libm.so.6 => /lib64/libm.so.6 (0x00007fc8f47ce000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc8f45b8000)
libc.so.6 => /lib64/libc.so.6 (0x00007fc8f4225000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc8f4ca5000)
and also gdb:
[18:15 ~] > gdb ./a.out
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from ...done.
(gdb) run
Starting program: a.out
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7bd48ec in std::ios_base::Init::Init() () from /usr/lib/gcc/x86_64-redhat-linux/3.4.6/libstdc++.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.107.el6.x86_64 libgcc-4.4.7-3.el6.x86_64
(gdb) bt
#0 0x00007ffff7bd48ec in std::ios_base::Init::Init() () from /usr/lib/gcc/x86_64-redhat-linux/3.4.6/libstdc++.so.6
#1 0x0000000000400831 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/iostream:77
#2 0x0000000000400873 in global constructors keyed to main() () at test.cpp:7
#3 0x0000000000400976 in __do_global_ctors_aux ()
#4 0x00000000004005a3 in _init ()
#5 0x00007fffffffe448 in ?? ()
#6 0x0000000000400905 in __libc_csu_init ()
#7 0x00007ffff7385c70 in __libc_start_main () from /lib64/libc.so.6
#8 0x0000000000400659 in _start ()
(gdb)
g++
[10:14 ~] > g++ -v
Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.6/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-languages=c,c++,f77 --disable-libgcj --host=x86_64-redhat-linux
Thread model: posix
gcc version 3.4.6 20060404 (Red Hat 3.4.6-19.el6)
and last thing
[10:25 ~] > ls /usr/lib/gcc/x86_64-redhat-linux
3.4.6
Thank you.
So apparently your problem is that your g++ installation is broken. I see from the messages that you have RHEL6, but somehow have version 3.4.6 of libstdc++.
My RHEL6 system has version 4.4.6 of gcc, libstdc++, etc. I assume that this older version of libstdc++ is related to your problem.
What versions do you have installed of gcc, g++, and libstdc++? From the other messages, it looks like you have 4.4.7, but do you happen to have multiple versions or mixed versions?
rpm -q gcc g++ libstdc++
Also, what other directories do you have under /usr/lib/gcc/x86_64-redhat-linux?
ls /usr/lib/gcc/x86_64-redhat-linux
i think that you miss something while installing g++
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.107.el6.x86_64 libgcc-4.4.7-3.el6.x86_64
I am trying to create a core dump and analyze it with gdb. This is the code I wrote to create a core dump.
#include <iostream>
void bar()
{
char *p = (char *) 123;
std::cout << "bar start\n";
std::cout << *p << "\n";
std::cout << "bar end\n";
}
void foo()
{
std::cout << "foo start\n";
bar();
std::cout << "foo end\n";
}
int main()
{
foo();
}
This is my Makefile.
all:
g++ -g foo.cc -o foo
objcopy --only-keep-debug foo foo.dbg
objcopy --strip-debug foo
clean:
rm -rf core* foo
After running make and ./foo, this is what my directory looks like.
# ls
core.28091 foo foo.cc foo.dbg Makefile
I am able to analyze the core dump like this. I launch gdb by specifying the executable and the core file as command line arguments. Then I load the symbols from foo.dbg with the symbol-file foo.dbg command.
[root#centos crash]# gdb foo core.28091
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/lab/crash/foo...(no debugging symbols found)...done.
[New Thread 28091]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/81/a81be2e44c93640adedb62adc93a47f4a09dd1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./foo'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000040076f in bar() ()
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64 libstdc++-4.4.7-4.el6.x86_64
(gdb) symbol-file foo.dbg
Reading symbols from /root/lab/crash/foo.dbg...done.
(gdb) bt
#0 0x000000000040076f in bar () at foo.cc:8
#1 0x00000000004007b7 in foo () at foo.cc:15
#2 0x00000000004007d1 in main () at foo.cc:21
(gdb) list
12 void foo()
13 {
14 std::cout << "foo start\n";
15 bar();
16 std::cout << "foo end\n";
17 }
18
19 int main()
20 {
21 foo();
(gdb)
However, I want to specify the symbol file name in the command line argument as well. But it doesn't seem to work. See the output below.
[root#centos crash]# gdb -s foo.dbg foo core.28091
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/lab/crash/foo...(no debugging symbols found)...done.
[New Thread 28091]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/81/a81be2e44c93640adedb62adc93a47f4a09dd1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./foo'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000040076f in bar() ()
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64 libstdc++-4.4.7-4.el6.x86_64
(gdb) bt
#0 0x000000000040076f in bar() ()
#1 0x00000000004007b7 in foo() ()
#2 0x00000000004007d1 in main ()
(gdb) list
No symbol table is loaded. Use the "file" command.
Why does it say no symbol table has been loaded even though I have specified it as an argument to the -s option?
It looks like a bug in gdb. gdb sets symarg to the argument that follows -s, but then later in the code, it unconditionally sets symarg to the executable's name. Proposed minimal diff follows:
$ diff -C 1 main.c.orig main.c
*** main.c.orig 2014-07-29 08:37:42.000000000 -0400
--- main.c 2014-09-02 16:27:54.079039046 -0400
***************
*** 864,866 ****
}
! symarg = argv[optind];
execarg = argv[optind];
--- 864,866 ----
}
! if (symarg == NULL) symarg = argv[optind];
execarg = argv[optind];
***************
*** 877,879 ****
{
! symarg = argv[optind];
execarg = argv[optind];
--- 877,879 ----
{
! if (symarg == NULL) symarg = argv[optind];
execarg = argv[optind];
Pursuant to the post, Standalone functions/data in C++, I proceeded to put my "common data" in an anonymous namespace as below and everything worked great on Windows (Vista 64 bit) on VS 2005/2008/2010
namespace {
...
static std::string mystrings[] = {
str1,
str2,
...,
strN
};
...
}
namespace mynamesp {
...
use mystrings[] here..
...
}
But on Linux (so far tested RHEL5 built with GCC-4.1.2) I promptly got a segmentation fault.
$>myprog
Segmentation fault
$>gdb myprog
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(gdb) r
Starting program: <path/to>/myprog
[Thread debugging using libthread_db enabled]
[New Thread 0x2b8901a9da60 (LWP 32710)]
Program received signal SIGSEGV, Segmentation fault.
0x0000003e4ce9c928 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string ()
from /usr/lib64/libstdc++.so.6
(gdb) bt
#0 0x0000003e4ce9c928 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string ()
from /usr/lib64/libstdc++.so.6
#1 0x00002b88ffde482b in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535)
at <path/to>/mysource.cpp:140
#2 0x00002b88ffde4d65 in global constructors keyed to _ZN91_GLOBAL__N__underscore_separated_path_to_mysource.cpp_00000000_6994A7DA2_1E () at <path/to>/mysource.cpp:12139
#3 0x00002b890011a296 in __do_global_ctors_aux ()
from <path/to/libs>/debug/libmylibd.so
#4 0x00002b88ffcd7f33 in _init () from <path/to/libs>/debug/libmylibd.so
#5 0x00002b8901672e40 in ?? ()
#6 0x000000326940d22b in call_init () from /lib64/ld-linux-x86-64.so.2
#7 0x000000326940d335 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#8 0x0000003269400aaa in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#9 0x0000000000000001 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)
Line 140 in the backtrace call stack item #1 basically points to the end of my array of strings definition. I've seen some others get this error; but no obvious fixes. Appreciate any thoughts/ideas/corrections as always. Thanks!
Your problem could be releated to a static initialization order fiasco.
This happens when you initialize a static variable using another static variable. When the latter one has not been initialized yet, then the first one is using a non-initialized variable for its initialization.
The root cause is that the order, in which static variables are initialized, is undefined.
Further reading:
https://isocpp.org/wiki/faq/ctors#static-init-order
A typical workaround would be to wrap the static variables inside a function. Example:
T& GetStaticA() {
T static_var_A; // <--initialization here
return A;
}
T static_var_B = GetStaticA(); // <-- static_var_A is guaranteed to be initialized
I had this problem and it turned out that in my compiling line I had missed the final output file in the linking.
g++ main.o logger.o timer.o keyboard.o -o main -lSDL -lSDL_image -lSDL_ttf -Wall
should have been
g++ main.o logger.o timer.o keyboard.o drawer.o -o main -lSDL -lSDL_image -lSDL_ttf -Wall
(Notice the now inclusion of drawer.o?)
It was easy to miss because my actual bash compilation script had many more lines to it.