I am debugging a C++ application, info shared tells me this library I want to break in has been read and has debugging symbols,
0x00007fffedc1f530 0x00007ffff18e4e60 Yes /home/me/WPEWebKit/WebKitBuild/Debug/lib/libWPEWebKit.so
I then ran nm on this loaded shared library and found a symbol I would like to break on (just to make sure the name was right):
$ nm /home/me/WPEWebKit/WebKitBuild/Debug/lib/libWPEWebKit.so | c++filt | grep -iw handleSyncmessage
0000000008a03e14 T WebCore::MediaPlayerPrivateGStreamer::handleSyncMessage(_GstMessage*)
It might be worth me saying that libWPEWebKit.so is just over 650MB, so quite a heavyweight.
Now in GDB, I say this,
(gdb) break WebCore::MediaPlayerPrivateGStreamer::handleSyncMessage
Function "WebCore::MediaPlayerPrivateGStreamer::handleSyncMessage" not defined.
Make breakpoint pending on future shared library load? (y or [n])
But we know at this point the symbol is loaded. If I rerun with the breakpoint set, it never gets fired, despite having proved the function is indeed running.
Am I doing something wrong here?
Edit: Adding info functions output
(gdb) info functions handleSyncMessage
All functions matching regular expression "handleSyncMessage":
.. lots of template instantiations I am ommitting because they are from unrelated things that just ref. this type ..
File ../../Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
bool (anonymous namespace)::MediaPlayerPrivateGStreamer::handleSyncMessage(GstMessage*);
File ../../Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.cpp:
bool (anonymous namespace)::MediaPlayerPrivateGStreamerBase::handleSyncMessage(GstMessage*);
..
Non-debugging symbols:
0x00007fffed84bcf0 WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)#plt
0x00007fffed84d810 WebCore::MediaPlayerPrivateGStreamer::handleSyncMessage(_GstMessage*)#plt
I now notice that I missed some output from nm above that I add for completeness
0000000008a03e14 T WebCore::MediaPlayerPrivateGStreamer::handleSyncMessage(_GstMessage*)
0000000008a11868 T WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)
0000000008a16fa8 t std::remove_reference<WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}&>::type&& std::move<WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}&>(std::remove_reference&&)
0000000008a16f78 t WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}& std::forward<WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}&>(std::remove_reference<WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}&>::type&)
0000000008a17959 t WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}&& std::forward<WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}>(std::remove_reference<WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}>::type&)
000000000bdac7d0 r WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::__FUNCTION__
000000000bdac800 r WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::__PRETTY_FUNCTION__
0000000008a11834 t WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#2}::operator()() const
0000000008a115a4 t WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}::operator()() const
0000000008a18128 t WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}::_GstMessage({lambda()#1}&&)
0000000008a18128 t WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}::_GstMessage({lambda()#1}&&)
0000000008a117f8 t WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}::~_GstMessage()
0000000008a117f8 t WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}::~_GstMessage()
000000000bdaddc8 r WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)::{lambda()#1}::operator()() const::__FUNCTION__
I tried breaking on the MediaPlayerPrivateGStreamerBase symbol as well as the MediaMediaPlayerPrivateGStreamer symbol to no success.
Edit 2
Another attempt,
(gdb) rbreak ^WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage
Breakpoint 1 at 0x7fffed84bcf0
<function, no debug info> WebCore::MediaPlayerPrivateGStreamerBase::handleSyncMessage(_GstMessage*)#plt;
So I guess the underlying question is why does GDB only set breaks on the procedure linkage table and not the real library function? How people even debug on Linux platforms? This seems utterly insane. Needless to say, even the breakpoint on the plt entry is not firing..
Related
Background: I'm trying to simplify my answer to the question c++ - Calling a lambda function in GDB - Stack Overflow.
Consider this simple program
int main(){
auto a=[&](int x){return x+1;};
}
When it's compiled (for example to a.out), I can see
$ nm --demangle a.out |grep lambda
000000000000113a t main::{lambda(int)#1}::operator()(int) const
$ nm a.out |grep 113a
000000000000113a t _ZZ4mainENKUliE_clEi
So there's a symbol _ZZ4mainENKUliE_clEi for the lambda operator() method in the text (code) section.
Given the exact name, I can print its value in gdb, and tab-completion also lists the symbol:
$ gdb -q ./a.out
(gdb) print _ZZ4mainENKUliE_clEi
$1 = {int (const struct {...} * const, int)} 0x55555555513a <operator()(int) const>
(gdb) print 'main::{lambda(int)#1}::operator()(int) const'
$2 = {int (const struct {...} * const, int)} 0x55555555513a <operator()(int) const>
But how can I find the symbol without using the external executable nm (inside gdb itself)?
Answers using the Python API is preferred.
Failed attempts
As suggested in c - Ask GDB to list all functions in a program - Stack Overflow:
(gdb) info functions .*lambda.*
All functions matching regular expression ".*lambda.*":
File /build/gcc/src/gcc/libstdc++-v3/src/c++11/compatibility-thread-c++0x.cc:
void std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<void (std::thread::*)(), std::
thread*>(std::once_flag&, void (std::thread::*&&)(), std::thread*&&)::{lambda()#1}>(void (std::thread::*&)())::{l
ambda()#1}::_FUN();
which is not the one I want, and
(gdb) info functions .*ZZ4.*
All functions matching regular expression ".*ZZ4.*":
doesn't match anything at all.
Also tried other things in Debugging with GDB - Examining the Symbol Table,
maintenance print symbols doesn't have it either.
In Symbols In Python (Debugging with GDB),
gdb.lookup_global_symbol("_ZZ4mainENKUliE_clEi") and gdb.lookup_static_symbol("_ZZ4mainENKUliE_clEi") both returns None.
file ./a.out in gdb does not help (suggested by GDB does not see symbols )
This solution finds the address of the .text section by parsing the textual output of info files, then find the symbol correspond to each address by parsing the output of info symbol gdb command.
Because it parses the textual output, in some special cases it might break. If anyone have any better solution/improvement please suggest/post a new answer.
I took a look at https://sourceware.org/gdb/current/onlinedocs/gdb/Progspaces-In-Python.html#Progspaces-In-Python / https://sourceware.org/gdb/current/onlinedocs/gdb/Objfiles-In-Python.html but can't find any.
I can't use block_for_pc, explained in this comment.
Run the Python code. The result is collected into the list symbols.
Read the comments in the code for more details how it works.
# (setup measure time)
import time
start=time.time()
# step 1. Find the limit of `.text`. Store in (a, b).
import re
[[a, b]]=[
(int(match[1], 16), int(match[2], 16))
for line in gdb.execute("info files", to_string=True).splitlines()
for match in [re.fullmatch(r"\s*(0x[0-9a-f]+) - (0x[0-9a-f]+) is \.text", line)]
if match
]
# Step 2. Iterate through symbols.
symbols=[]
mismatch=0
i=b-1
while i>=a:
line=gdb.execute("info symbol "+hex(i), to_string=True)
if line=='No symbol matches '+hex(i)+'.\n':
# There's no symbol here. Unfortunately I can't find any way to quickly skip through it.
i-=1
mismatch+=1
continue
# Determine the symbol and the offset.
match=re.fullmatch(r"(.+) \+ (\d+) in section \.text of .*\n?", line)
if match:
symbol=match[1]
offset=int(match[2])
else:
match=re.fullmatch(r"(.+) in section \.text of .*\n?", line)
symbol=match[1]
offset=0
# collect the symbol into the list.
i-=offset
symbols.append((symbol, hex(i)))
i-=1
# (report time taken)
print(time.time()-start, len(symbols), mismatch)
For a reasonably large executable, nm takes 0.042s, while the script takes 0.778s and report 5769 symbols and 44171 misses. (the misses takes most of the time)
I'm writing a little BASIC compiler using LLVM, everything works fine. I'm trying to get debugging to work, and it's going fine, but I have a weird issue with variable names and GDB's print statement.
A little background on BASIC for those not familiar:
In BASIC, variable names use a symbol on the end to determine their type. So a variable x% would be integer, and variable x$ would be a string, and both can co-exist.
The statement DEFINT A-Z means that any variable starting with any letter from A to Z will be type INT, even without the trailing %. But, internally, my compiler will store the name mangled with the trailing % to avoid overlap with another variable with a trailing $.
Here is my little test program:
defint a-z
x = 5
y = 100
z = x + y
if x = 5 then
print "z=";z
else
print "z!=";z
end if
So 'x' is actually stored internally and put into the symbol table as x%.
OK, so I load my compiled EXE into GDB...
D:\dev\BasicParser>gdb t.exe
GNU gdb (GDB) 7.4
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-mingw32".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from D:\dev\BasicParser\t.exe...done.
(gdb) b _jbc_main
Breakpoint 1 at 0x401000: file t.bas, line 1.
(gdb) r
Starting program: D:\dev\BasicParser\t.exe
[New Thread 1532.0x25b8]
Breakpoint 1, _jbc_main () at t.bas:1
1 defint a-z
(gdb) n
4 x = 5
(gdb) info address x%
Symbol "x%" is static storage at address 0x4263c4.
At this point, you can see that GDB recognizes the symbol x% because it recognizes and display it's address. But the following "print" fails:
(gdb) print x%
No symbol "x" in current context.
Hmmm... that's interesting ... it dropped the % symbol.
But I can look at the memory location where x% is stored and this looks correct:
(gdb) x 0x4263c4
0x4263c4 <x%>: 0x00000000
If I show all symbols I get this as expected:
(gdb) info variables
All defined variables:
File t.bas:
static i32 x%;
static i32 y%;
static i32 z%;
Continuing on, to show that x% does get modified by the code:
(gdb) n
5 y = 100
(gdb) print x%
No symbol "x" in current context.
(gdb) x 0x4263c4
0x4263c4 <x%>: 0x00000005
(gdb)
And as you can see, the memory that GDB thinks x% is at is definitely updated.
For some reason, the "print" command doesn't like the % symbol.
Here is the "info source" results on this module, you can see I didn't specify C as the language:
(gdb) info source
Current source file is t.bas
Compilation directory is D:\dev\BasicParser
Located in D:\dev\BasicParser\t.bas
Contains 18 lines.
Source language is minimal.
Compiled with DWARF 2 debugging format.
Does not include preprocessor macro info.
Any ideas for workarounds? I've been searching and searching and can't find a solution.
Thanks!
gdb simply interprets your command as modulo operation. It is not an error in gdb.
See, info source shows minimal for your file. This is from gdb's doc: http://sourceware.org/gdb/current/onlinedocs/gdb/Unsupported-Languages.html#Unsupported-Languages
In addition to the other fully-supported programming languages, GDB
also provides a pseudo-language, called minimal. It does not represent
a real programming language, but provides a set of capabilities close
to what the C or assembly languages provide.
So, what does this warning in your debugging session mean?
(gdb) print x%
No symbol "x" in current context.
Since minimal is a set of capabilities close C then gdb must iterpret this as trying to get remainder of division (http://en.wikipedia.org/wiki/Modulo_operation) - according to C programming language rules. So x is left arg, % is operation and the right arg is missing.
Let me show you example. This is a test C++ program that shows the minimal debugging session:
D:\src-c++\tests\test.vars>cat minimal.cpp
int main()
{
int k;
k = 1;
k = 2;
return 0;
}
And this is a debugging session:
D:\src-c++\tests\test.vars>gdb -q minimal.exe
Reading symbols from D:\src-c++\tests\test.vars\minimal.exe...done.
(gdb) start
Temporary breakpoint 1 at 0x4016be: file minimal.cpp, line 5.
Starting program: D:\src-c++\tests\test.vars/minimal.exe
[New Thread 2872.0x8c0]
Temporary breakpoint 1, main () at minimal.cpp:5
5 k = 1;
(gdb) show language
The current source language is "auto; currently c++".
(gdb) set language minimal
Warning: the current language does not match this frame.
(gdb) show language
The current source language is "minimal".
Warning: the current language does not match this frame.
(gdb) n
6 k = 2;
(gdb) print k
$1 = 1
(gdb) print k%
A syntax error in expression, near `'.
(gdb) print kk%
No symbol "kk" in current context.
Look -- No symbol "kk" in current context, the same error that you have.
Just a cross-reference, in case anybody else like me that might need help with this: I found this answer
In GDB, how to print the content of a symbol which has special characters?
where they indicate that you can quote names by starting with ampersand (&) and quote the name with single ticks ('), that is use something like
p &'my#complex.symbol%name'
x &'my#complex.symbol%name'
to inspect symbols with names beyond the usual C-style naming convention. I'm not completely clear what characters you can use at that point, but it seems like a good improvement.
As the other answer also points out, the GDB manual often refers to escaping symbol names just using single ticks (say something like p 'foo.c'). After some further reading I found in the GDB manual that this use of & seems to be an extension added for Windows support (see section titled "Features for Debugging MS Windows PE Executables", numbered 21.1.4 in the current edition), so I'm not sure if this may work outside of the Windows environment
In my application I have setup signal handler to catch Segfaults, and print bactraces.
My application loads some plugins libraries, when process starts.
If my application crashes with a segfault, due to an error in the main executable binary, I can analyze the backtrace with:
addr2line -Cif -e ./myapplication 0x4...
It accurately displays the function and the source_file:line_no
However how do analyze if the crash occurs due to an error in the plugin as in the backtrace below?
/opt/myapplication(_Z7sigsegvv+0x15)[0x504245]
/lib64/libpthread.so.0[0x3f1c40f500]
/opt/myapplication/modules/myplugin.so(_ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi+0x6af)[0x7f5588fe4bbf]
/opt/myapplication/modules/myplugin.so(_Z11myplugin_reqmodP12CONNECTION_TP7Filebuf+0x68)[0x7f5588fe51e8]
/opt/myapplication(_ZN10Processors7ExecuteEiP12CONNECTION_TP7Filebuf+0x5b)[0x4e584b]
/opt/myapplication(_Z15process_requestP12CONNECTION_TP7Filebuf+0x462)[0x4efa92]
/opt/myapplication(_Z14handle_requestP12CONNECTION_T+0x1c6d)[0x4d4ded]
/opt/myapplication(_Z13process_entryP12CONNECTION_T+0x240)[0x4d79c0]
/lib64/libpthread.so.0[0x3f1c407851]
/lib64/libc.so.6(clone+0x6d)[0x3f1bce890d]
Both my application and plugin libraries have been compiled with gcc and are unstripped.
My application when executed, loads the plugin.so with dlopen
Unfortunately, the crash is occurring at a site where I cannot run the application under gdb.
Googled around frantically for an answer but all sites discussing backtrace and addr2line exclude scenarios where analysis of faulty plugins may be required.
I hope some kind-hearted hack knows solution to this dilemma, and can share some insights. It would be so invaluable for fellow programmers.
Tons of thanks in advance.
Here are some hints that may help you debug this:
The address in your backtrace is an address in the address space of the process at the time it crashed. That means that, if you want to translate it into a 'physical' address relative to the start of the .text section of your library, you have to subtract the start address of the relevant section of pmap from the address in your backtrace.
Unfortunately, this means that you need a pmap of the process before it crashed. I admittedly have no idea whether loading addresses for libraries on a single system are constant if you close and rerun it (imaginably there are security features which randomize this), but it certainly isn't portable across systems, as you have noticed.
In your position, I would try:
demangling the symbol names with c++filt -n or manually. I don't have a shell right now, so here is my manual attempt: _ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi is ICAPSection::process(CONNECTION_T *, Filebuf *, int). This may already be helpful. If not:
use objdump or nm (I'm pretty sure they can do that) to find the address corresponding to the mangled name, then add the offset (+0x6af as per your stacktrace) to this, then look up the resulting address with addr2line.
us2012's answer was quite the trick required to solve the problem. I am just trying to restate it here just to help any other newbie struggling with the same problem, or if somebody wishes to offer improvements.
In the backtrace it is clearly visible that the flaw exists in the code for myplugin.so. And the backtrace indicates that it exists at:
/opt/myapplication/modules/myplugin.so(_ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi+0x6af)[0x7f5588fe4bbf]
The problem of locating the line corresponding to this fault cannot be determined as simplistically as:
addr2line -Cif -e /opt/myapplication/modules/myplugin.so 0x7f5588fe4bbf
The correct procedure here would be to use nm or objdump to determine the address pointing to the mangled name. (Demangling as done by us2012 is not really necessary at this point). So using:
nm -Dlan /opt/myapplication/modules/myplugin.so | grep "_ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi"
I get:
0000000000008510 T _ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi /usr/local/src/unstable/myapplication/sources/modules/myplugin/myplugin.cpp:518
Interesting to note here is that myplugin.cpp:518 actually points to the line where the opening "{" of the function ICAPSection::process(CONNECTION_T *, Filebuf *, int)
Next we add 0x6af to the address (revealed by the nm output above) 0000000000008510 using linux shell command
printf '0x%x\n' $(( 0x0000000000008510 + 0x6af ))
And that results in 0x8bbf
And this is the actual source_file:line_no of the faulty code, and can be precisely determined with addr2line as:
addr2line -Cif -e /opt/myapplication/modules/myplugin.so 0x8bbf
Which displays:
std::char_traits<char>::length(char const*)
/usr/include/c++/4.4/bits/char_traits.h:263
std::string::assign(char const*)
/usr/include/c++/4.4/bits/basic_string.h:970
std::string::operator=(char const*)
/usr/include/c++/4.4/bits/basic_string.h:514
??
/usr/local/src/unstable/myapplication/sources/modules/myplugin/myplugin.cpp:622
I am not too sure why the function name was not displayed here, but myplugin.cpp:622 was quite precisely where the fault was.
I would like to access the frames stored in a core dump of a program that doesn't has debug symbols (I want to do this in C). When I open up the program and the core dump inside GDB I get a stack trace including the names of the functions. For example:
(gdb) bt
#0 0x08048443 in layer3 ()
#1 0x08048489 in layer2 ()
#2 0x080484c9 in layer1 ()
#3 0x0804854e in main ()
The names of all functions are stored in the executable in the .strtab section. How can I build up the stack trace with the different frames? Running GDB in batch mode is not an option. And also just "copy the parts from gdb the are needed" is also a bad idea because the code is not independently written.
So to make my question more precisely: Where do I find the point inside a core dump where I can start reading the stack information? Is there a library of some sort for accessing those information? A struct I can use? Or even better, a documentation how those informations are structured inside a core dump?
(I already seen the question "how to generate a stack trace from a core dump file in C, without invoking an external tool such as gdb", but since there is no valid answer, I thought I would ask it again)
[Edit] I'm doing this under Linux x86
Coredump contains stack information as well. If you can use this stack information along with the EBP and EIP register values in the coredump file, you can print the stack trace. I had written a program to do this. You can find the program in the following link.
http://www.emntech.com/programs/corestrace.c
Usage: Compile the above program and give the corefile when you execute it.
$corestrace core
If you want symbols also to be printed, you do like this: Let's assume the program that generated the core is 'test'.
$ nm -n test > symbols
$ corestrace core symbols
Sample output looks like this:
$ ./coretrace core symbols
0x80483cd foo+0x9
0x8048401 func+0x1f
0x8048430 main+0x2d
I'd like to go through a binary file my teacher gave me line by line to check addresses on the stack and the contents of different registers, but I'm not extremely familiar with using gdb. Although I have the C code, we're supposed to work entirely from a binary file. Here are the commands I've used so far:
(gdb) file SomeCode
Which gives me this message:
Reading symbols from ../overflow/SomeCode ...(no debugging symbols found)...done.
Then I use :
(gdb) disas main
which gives me all of the assembly. I wanted to set up a break point and use the "next" command, but none of the commands I tried work. Does anyone know the syntax I would use?
try using ni which is nexti. equivalent is si which is step instruction
nexti if you want to jump over function calls.
stepi if you want to enter a function call.
The following documentation is very helpful; it has a list of all the important commands you could use on gdb.
X86-64: http://csapp.cs.cmu.edu/public/docs/gdbnotes-x86-64.pdf
IA32: http://csapp.cs.cmu.edu/public/docs/gdbnotes-ia32.pdf