My binary uses a number of different shared libraries. While attaching the process with gdb, it takes around 5 minutes to load and read symbols from all of these libraries.
Is there a way to read and load symbols selectively while attaching a process with gdb?
You can set auto-solib-add off and then use the sharedlibrary command to selectively load symbols. Example:
$ gdb -e ./a.out
(gdb) set auto-solib-add off
(gdb) file a.out
Reading symbols from a.out...done.
(gdb) start
Temporary breakpoint 1 at 0x4004ba: file sig.c, line 4.
Starting program: /home/a3f/a.out
Temporary breakpoint 1, main () at sig.c:4
4 do();
(gdb) print environ
No symbol "environ" in current context.
(gdb) info shared
From To Syms Read Shared Object Library
0x00007ffff7ddcae0 0x00007ffff7df5130 No /lib64/ld-linux-x86-64.so.2
No linux-vdso.so.1
0x00007ffff7a504a0 0x00007ffff7b7cc73 No /lib/x86_64-linux-gnu/libc.so.6
(gdb) sharedlibrary libc
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
(gdb) print environ
$1 = (char **) 0x7fffffffe3b8
(gdb) print _dl_open
No symbol "_dl_open" in current context.
(gdb) sharedlibrary
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.19.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
(gdb) print _dl_open
$1 = {void *(const char *, int, const void *, Lmid_t, int, char **, char **)} 0x7ffff7dee350 <_dl_open>
Related
Code (m1.cpp):
#include <iostream>
using namespace std;
int main (int argc, char *argv[])
{
cout << "running m1" << endl;
return 0;
}
GDB Version: GNU gdb (GDB) 7.6.2
Built using: g++ -g m1.cpp
Command line history:
(gdb) b main
Breakpoint 1 at 0x40087b: file m1.cpp, line 6.
(gdb) r
Starting program: .../a.out
Program received signal SIGSEGV, Segmentation fault.
0x00002aaaaaac16a0 in strcmp () from /lib64/ld-linux-x86-64.so.2
(gdb) c
Continuing.
Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb)
When I run without setting any breakpoints, it runs without errors.
As requested:
(gdb) bt
#0 strcmp () from /lib64/ld-linux-x86-64.so.2
#1 in check_match.12104 () from /lib64/ld-linux-x86-64.so.2
#2 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#3 in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#4 in _dl_relocate_object () from /lib64/ld-linux-x86-64.so.2
#5 in dl_main () from /lib64/ld-linux-x86-64.so.2
#6 in _dl_sysdep_start () from /lib64/ld-linux-x86-64.so.2
#7 in _dl_start () from /lib64/ld-linux-x86-64.so.2
#8 in _start () from /lib64/ld-linux-x86-64.so.2
#9 in ?? ()
I was able to replicate the OP's observed behavior (using the same compile and getting the same backtrace). The behavior was persistent across a range GDBs and GCCs. I noticed that the symptom goes away when I unset SHELL. In my normal environment I use tcsh (version 1.15.00). If SHELL is set, then (I believe) gdb launches using tcsh. If I unset SHELL, gdb launches using sh. This is enough for me to make forward progress. I don't have a crisp explanation for what would be different in tcsh to manifest the issue but if others have the same behavior, it may shed more light on the issue.
I checked that in my GNU gdb version 7.11.1. It worked really fine in it.
I first compiled the same program and built it using:
g++ -g m1.cpp
Then, ran the executable in the gdb as follows:
gdb -q ./a.out
And did the same things you mentioned. It worked fine.
Update your gdb, and check that again and let know.
I have followed the guide here: debugging pintool guide, but I cannot get GDB to find the debugging symbols for my pintool.
First I compiled my pintool with debug information
lotus#c02-0:~/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools$ make DEBUG=1 obj-intel64/dst7.so
g++ -shared -Wl,--hash-style=sysv -Wl,-Bsymbolic -Wl,--version-script=../../../source/include/pin/pintool.ver -g -o obj-intel64/dst7.so obj-intel64/dst7.o -L../../../intel64/lib -L../../../intel64/lib-ext -L../../../intel64/runtime/glibc -L../../../extras/xed-intel64/lib -lpin -lxed -lpindwarf -ldl
In two different terminal windows I did the following:
In the first window:
lotus#c02-0:~/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools$ ~/PerforceArch/home/Shadi/HLS/pin/intel64/bin/pinbin -pause_tool 10 -t obj-intel64/dst7.so -- ls
Pausing for 10 seconds to attach to process with pid 2214
To load the tool's debug info to gdb use:
add-symbol-file /home/lotus/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools/obj-intel64/dst7.so 0x7fa8952e44c0 -s .data 0x7fa895c0f720 -s .bss 0x7fa895c10c40
In the the second window:
lotus#c02-0:~$ /usr/bin/gdb ~/PerforceArch/home/Shadi/HLS/pin/intel64/bin/pinbin
GNU gdb (GDB) SUSE (7.4.50.20120603-2.1.2)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/lotus/PerforceArch/home/Shadi/HLS/pin/intel64/bin/pinbin...(no debugging symbols found)...done.
(gdb) attach 2214
Attaching to program: /home/lotus/PerforceArch/home/Shadi/HLS/pin/intel64/bin/pinbin, process 2214
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /home/lotus/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools/obj-intel64/dst7.so...(no debugging symbols found)...done.
Loaded symbols for /home/lotus/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools/obj-intel64/dst7.so
0x00007fa895ce6c80 in __nanosleep_nocancel () from /lib64/libc.so.6
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.15-22.6.4.x86_64 libgcc47-debuginfo-4.7.1_20120723-1.1.1.x86_64 libstdc++47-debuginfo-4.7.1_20120723-1.1.1.x86_64
(gdb) add-symbol-file /home/lotus/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools/obj-intel64/dst7.so 0x7fa8952e44c0 -s .data 0x7fa895c0f720 -s .bss 0x7fa895c10c40
add symbol table from file "/home/lotus/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools/obj-intel64/dst7.so" at
.text_addr = 0x7fa8952e44c0
.data_addr = 0x7fa895c0f720
.bss_addr = 0x7fa895c10c40
(y or n) y
Reading symbols from /home/lotus/PerforceArch/home/Shadi/HLS/pin/source/tools/lotusTools/obj-intel64/dst7.so...(no debugging symbols found)...done.
The problem is that on the last line of the second window, it says no debugging symbols were found.
Before I compiled with debug symbols on, I had compiled with debug symbols off. Before recompiling with debug symbols on, I deleted the .so file but not the .o file. Deleting the old .o file and then compiling again with debug symbols on, fixed the problem.
I want to debug libgcc_s.so.1 in GDB on the source level on Ubuntu 14.04.1. So I downloaded libgcc1-dbg and got the source of gcc.
In GDB, I use directory to specify the gcc source directory. But when I GDB some problem, and break in libgcc_s.so.1, I cannot get the source code.
Reading symbols from /home/love/pri...(no debugging symbols found)...done.
(gdb) directory ~/source/gcc-4.9/
Source directories searched: /home/lovelydream/source/gcc-4.9:$cdir:$cwd
(gdb) c
The program is not being run.
(gdb) run
Starting program: /home/lovelydream/prison
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
>
Program received signal SIGINT, Interrupt.
0x00007ffff75e6350 in __read_nocancel () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: 没有那个文件或目录.
(gdb) b _Unwind_Find_FDE
Breakpoint 1 at 0x7ffff78d20c0
(gdb) c
Continuing.
x
Breakpoint 1, 0x00007ffff78d20c0 in _Unwind_Find_FDE ()
from /lib/x86_64-linux-gnu/libgcc_s.so.1
(gdb) info source
Current source file is ../sysdeps/unix/syscall-template.S
Compilation directory is /build/buildd/eglibc-2.19/io
Source language is asm.
Compiled with DWARF 2 debugging format.
Does not include preprocessor macro info.
(gdb) ni
0x00007ffff78d20c2 in _Unwind_Find_FDE () from /lib/x86_64-linux-gnu/libgcc_s.so.1
(gdb) directory ~/source/gcc-4.9/
Source directories searched: /home/lovelydream/source/gcc-4.9:$cdir:$cwd
(gdb) ni
0x00007ffff78d20c4 in _Unwind_Find_FDE () from /lib/x86_64-linux-gnu/libgcc_s.so.1
(gdb) list
76 ../sysdeps/unix/syscall-template.S: 没有那个文件或目录.
(gdb) list
76 in ../sysdeps/unix/syscall-template.S
(gdb) list
76 in ../sysdeps/unix/syscall-template.S
(gdb) info source
I am trying to create a core dump and analyze it with gdb. This is the code I wrote to create a core dump.
#include <iostream>
void bar()
{
char *p = (char *) 123;
std::cout << "bar start\n";
std::cout << *p << "\n";
std::cout << "bar end\n";
}
void foo()
{
std::cout << "foo start\n";
bar();
std::cout << "foo end\n";
}
int main()
{
foo();
}
This is my Makefile.
all:
g++ -g foo.cc -o foo
objcopy --only-keep-debug foo foo.dbg
objcopy --strip-debug foo
clean:
rm -rf core* foo
After running make and ./foo, this is what my directory looks like.
# ls
core.28091 foo foo.cc foo.dbg Makefile
I am able to analyze the core dump like this. I launch gdb by specifying the executable and the core file as command line arguments. Then I load the symbols from foo.dbg with the symbol-file foo.dbg command.
[root#centos crash]# gdb foo core.28091
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/lab/crash/foo...(no debugging symbols found)...done.
[New Thread 28091]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/81/a81be2e44c93640adedb62adc93a47f4a09dd1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./foo'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000040076f in bar() ()
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64 libstdc++-4.4.7-4.el6.x86_64
(gdb) symbol-file foo.dbg
Reading symbols from /root/lab/crash/foo.dbg...done.
(gdb) bt
#0 0x000000000040076f in bar () at foo.cc:8
#1 0x00000000004007b7 in foo () at foo.cc:15
#2 0x00000000004007d1 in main () at foo.cc:21
(gdb) list
12 void foo()
13 {
14 std::cout << "foo start\n";
15 bar();
16 std::cout << "foo end\n";
17 }
18
19 int main()
20 {
21 foo();
(gdb)
However, I want to specify the symbol file name in the command line argument as well. But it doesn't seem to work. See the output below.
[root#centos crash]# gdb -s foo.dbg foo core.28091
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/lab/crash/foo...(no debugging symbols found)...done.
[New Thread 28091]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/81/a81be2e44c93640adedb62adc93a47f4a09dd1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./foo'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000040076f in bar() ()
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64 libstdc++-4.4.7-4.el6.x86_64
(gdb) bt
#0 0x000000000040076f in bar() ()
#1 0x00000000004007b7 in foo() ()
#2 0x00000000004007d1 in main ()
(gdb) list
No symbol table is loaded. Use the "file" command.
Why does it say no symbol table has been loaded even though I have specified it as an argument to the -s option?
It looks like a bug in gdb. gdb sets symarg to the argument that follows -s, but then later in the code, it unconditionally sets symarg to the executable's name. Proposed minimal diff follows:
$ diff -C 1 main.c.orig main.c
*** main.c.orig 2014-07-29 08:37:42.000000000 -0400
--- main.c 2014-09-02 16:27:54.079039046 -0400
***************
*** 864,866 ****
}
! symarg = argv[optind];
execarg = argv[optind];
--- 864,866 ----
}
! if (symarg == NULL) symarg = argv[optind];
execarg = argv[optind];
***************
*** 877,879 ****
{
! symarg = argv[optind];
execarg = argv[optind];
--- 877,879 ----
{
! if (symarg == NULL) symarg = argv[optind];
execarg = argv[optind];
Is possible to recover the exact values of argv and argc parameters of a main after the application crashed?
I need to use only the application core-dump and gdb debugger on Linux.
Yes, if application was compiled with debug info. Open core dump in gdb and find frame containing main function. Then go to this frame and print values of argv and argc. Here is sample gdb session.
[root#localhost ~]# gdb ./a.out core.2020
GNU gdb (GDB) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/a.out...done.
[New Thread 2020]
warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Core was generated by `./a.out'.
Program terminated with signal 6, Aborted.
#0 0x0027b424 in __kernel_vsyscall ()
(gdb) bt
#0 0x0027b424 in __kernel_vsyscall ()
#1 0x00b28b91 in raise () from /lib/libc.so.6
#2 0x00b2a46a in abort () from /lib/libc.so.6
#3 0x007d3397 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#4 0x007d1226 in ?? () from /usr/lib/libstdc++.so.6
#5 0x007d1263 in std::terminate() () from /usr/lib/libstdc++.so.6
#6 0x007d13a2 in __cxa_throw () from /usr/lib/libstdc++.so.6
#7 0x08048940 in main (argv=1, argc=0xbfcf1754) at test.cpp:14
(gdb) f 7
#7 0x08048940 in main (argv=1, argc=0xbfcf1754) at test.cpp:14
14 throw std::runtime_error("123");
(gdb) p argv
$1 = 1
(gdb) p argc
$2 = (char **) 0xbfcf1754
(gdb)
Looks like you need to start from the basics..!!
compiler your application code with -g flag, make sure you dont strip it.
Say if I wanted to compile hello.c
gcc -c -g hello.c -o hello.o
gcc hello.o -o hello
now if you dont want to debug
ulimit -c unlimited
./hello
when the application crashes a core file wiil be generated.
To examine the core file
"gdb ./hello core.$$$" this will list you your stack.
you can also choose to debug the image
gdb hello
There are a lot of stuffs over the internet about GDB, do go through them.