There was a core dump produced at the customer end for my application and while looking at the backtrace I don't have the symbols loaded...
(gdb) where
#0 0x000000364c032885 in ?? ()
#1 0x000000364c034065 in ?? ()
#2 0x0000000000000000 in ?? ()
(gdb) bt full
#0 0x000000364c032885 in ?? ()
No symbol table info available.
#1 0x000000364c034065 in ?? ()
No symbol table info available.
#2 0x0000000000000000 in ?? ()
No symbol table info available.
One think I want to mention in here is that the application being used is build with -g option.
To me it seems that the required libraries are not being loaded. I tried to load the libraries manually using the "symbol-file", but this doesn't help.
What could be the possible issue?
No symbol table info available.
Chances are you invoked GDB incorrectly. Don't do this:
gdb core
gdb -c core
Do this instead:
gdb exename core
Also see this answer for what you'll likely have to do to get meaningful crash stack trace for a core from customer's machine.
I was facing a similar issue and later found out that I am missing -g option, Make sure you have compiled the binary with -g.
This happens when you run gdb with path to executable that does not correspond to the one that produced the core dump.
Make sure that you provide gdb with the correct path.
<put an example of correct code or commands here>
Related
So I am experiencing this really weird behavior of gdb on Linux (KDE Neon 5.20.2):
I start gdb and load my executable using the file command:
(gdb) file mumble
Reading symbols from mumble...
As you can see it did find debug symbols. Then I start my program (using start) which causes gdb to pause at the entry to the main function. At this point I can also print out the back trace using bt and it works as expected.
If I now continue my program and interrupt it at any point during startup, I can still display the backtrace without issues. However if I do something in my application that happens in another thread than the startup (which all happens in thread 1) and interrupt my program there, gdb will no longer be able to display the stacktrace properly. Instead it gives
(gdb) bt
#0 0x00007ffff5bedaff in ?? ()
#1 0x0000555556a863f0 in ?? ()
#2 0x0000555556a863f0 in ?? ()
#3 0x0000000000000004 in ?? ()
#4 0x0000000100000001 in ?? ()
#5 0x00007fffec005000 in ?? ()
#6 0x00007ffff58a81ae in ?? ()
#7 0x0000000000000000 in ?? ()
which shows that it can't find the respective debug symbols.
I compiled my application with cmake (gcc) using -DCMAKE_BUILD_TYPE=Debug. I also ensured that a bunch of debug symbols are present in the binary using objdump --debug mumble (Which also printed a few lines of objdump: Error: LEB value too large, but I'm not sure if this is related to the problem I am seeing).
While playing around with gdb, I also encountered the error
Cannot find user-level thread for LWP <SomeNumber>: generic error
a few times, which lets me suspect that maybe there is indeed some issue invloving threads here...
Finally I tried starting gdb and before loading my binary using set verbose on which yields
(gdb) set verbose on
(gdb) file mumble
Reading symbols from mumble...
Reading in symbols for /home/user/Documents/Git/mumble/src/mumble/main.cpp...done.
This does also look suspicious to me as only main.cpp is explicitly listed here (even though the project has much, much more source files). I should also note that all successful backtraces that I am able to produce (as described above) all originate from main.cpp.
I am honestly a bit clueless as to what might be the root issue here. Does someone have an idea what could be going on? Or how I could investigate further?
Note: I also tried using clang as a compiler but the result was the same.
Used program versions:
cmake: 3.18.4
gcc: 9.3.0
clang: 10.0.0
make: 4.2.1
I'm debugging a SIGSEGV error on a huge application running on Yocto/ARM64 (iMX8QM).
If I run the application in GDB, I can get the backtrace:
Thread 1 "HmiAppCentral" received signal SIGSEGV, Segmentation fault.
0x0000000000b0a0d0 in kanzi::Node3D::~Node3D() ()
(gdb) bt
#0 0x0000000000b0a0d0 in kanzi::Node3D::~Node3D() ()
#1 0x0000000000cd4e44 in kanzi::Model3D::~Model3D() ()
#2 0x0000000000b09c38 in kanzi::Node3D::removeChild(unsigned long) ()
[...]
Then I export the core dump, quit GDB and restart it:
(gdb) generate-core-file
warning: target file /proc/2279/cmdline contained unexpected null characters
[...]
gdb -c core.2279
Then GDB is not able to print the backtrace anymore:
(gdb) bt full
#0 0x0000000000b0a0d0 in ?? ()
No symbol table info available.
#1 0x0000000000000001 in ?? ()
No symbol table info available.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
The address of the first frame is correct (0x0000000000b0a0d0), however GDB is not able to find the function name when reloading the core dump. Any hint?
Just like when the OS creates a core file, the original program executable is not included in the core file itself, and it is this executable that contains the debug information (or allows GDB to find the debug information).
What this means is, if you want to debug with the debug information then you need to provide both the executable and the core file, so something like:
gdb my_program.exe -c core.pid
I am trying to analyze segfault on a core file on linux. I am not sure if the following behavior is correct, thus i deliberately caused a segfault using
#include <signal.h>
int main() {
raise(SIGSEGV);
}
the binary is build with debug info i.e.
file mainTestFile
mainTestFile: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, with debug_info, not stripped
notice how it does say "with debug_info, not stripped" at the end
when i execute the binary i get a core file generated which is called core-mainTestFile.20474
(In order to generate the core file i hat to set my ulimit to unlimited i.e.
ulimit -c unlimited
)
if i run only the binary under GDB and do backtrace "bt" then i get the segfault and i get all names of the functions involved
printed nicely i.e. notice how the gdb says when starting "reading symbols from ./mainTestFile...done."
gdb ./mainTestFile
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
....
reading symbols from ./mainTestFile...done.
(gdb) run
Starting program: /src/exe/mainTestFile
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault.
__GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x0000000000402dad in main (argc=1, argv=0x7fffffffda38) at /src/exe/main.cpp:53
(gdb)
however if i try to anaylise only the core file with gdb like that
gdb -c core-mainTestFile.20474
then i get only question marks
the when i execute "bt" then i do not see the names of the methods, instead i get question marks
(gdb) bt
#0 0x00007f34d8842e97 in ?? ()
#1 0x0000000000000000 in ?? ()
they only workaround i found is to supply the binary directly at the command line then it all gets printed nicely.
even if i try to tell GDB to use the symbols file and point that to the binary file which does have the symbols
i.e.
symbol-file /src/exe/mainTestFile
then GDB says
Reading symbols from /src/exe/mainTestFile...done
and when i execute bt i see the question marks again? Why is that. Is GDB not able to get the symbols out of the binary?
it only works if supply the binary directly on the command like like:
gdb /src/exe/mainTestFile -c core-mainTestFile.20474
my question is should the GDB be able to read symbols of the binary when directly supplying him the binary over the "symbol-file" command or not. Why is this working when supping him directly the binary over the command line, what is the difference?
should the GDB be able to read symbols of the binary when directly supplying him the binary over the "symbol-file" command or not.
In theory, using symbol-file and core-file commands in either order in GDB should be equivalent.
But there is a bug: symbol-file followed by core-file works, and the opposite order doesn't.
Since generally the end-user can always rearrange his commands into the order that works, this has never propagated to the top of any GDB developer's queue of things to fix.
Related bug (but not an exact duplicate).
When I compile same code with g++ with -o2 flag I can backtrace successfully without any Bogus adresses. Like;
0x08156079 in CItem::GetValue (this=0x3adb0f00, idx=0) at item.cpp:957
0x081b123c in quest::item_has_flag (L=0x3af9bdc0) at questlua_item.cpp:155
0x08363cba in luaD_precall (L=0x3af9bdc0, func=0x3b1cedd8) at ldo.c:249
0x0836ba86 in luaV_execute (L=0x3af9bdc0) at lvm.c:637
0x08363fad in resume (L=0x3af9bdc0, ud=0xffffa164) at ldo.c:337
0x0836393b in luaD_protectedparser (L=0x3af9bdc0, z=0x8363f80, bin=-24220)
....
But I need to use g++48 (with c++11) for better performance and other reasons... So, when I do same thing with -o3 optimize flag and g++48 I can't get any file names etc. Like;
#0 0x28a56f3c in ?? ()
No symbol table info available.
#1 0x00000032 in ?? ()
No symbol table info available.
#2 0xbfbf9838 in ?? ()
No symbol table info available.
#3 0x28a4ea3a in ?? ()
No symbol table info available.
#4 0x00000032 in ?? ()
No symbol table info available.
#5 0x00000004 in ?? ()
No symbol table info available.
#6 0x00000001 in ?? ()
No symbol table info available.
#7 0x28a70694 in ?? ()
No symbol table info available.
#8 0xbfbf969c in ?? ()
No symbol table info available.
#9 0x28a6b06c in ?? ()
No symbol table info available.
Which flags I must not use for debugging? (-fno-omit-frame-pointer) Which flags should I use for debugging? And reason... I'm not a SO expert.
With gcc 4.8 you can use -Og switch, to enable all optimizations that do not interfere with debugging.
Also make sure you enabled debug info (-g switch). If you updated your compiler to newer release, you should also update the debugger. Another thing you may try is to make sure gcc generates debug info in compatible format (try -gdwarf-2 or similar).
Im receving many cores from different programs on my redhat server those cores happens without any specific pattern it can happen with Tuxedo servers as well as ordinary programs the only common thing between all program is that all of them have the same top error with this 8629 number check_match.8629 ()
how can I identify what this number refering to?
Thanks in advance
data from core dump file
#0 0x005546b1 in check_match.8629 () from /lib/ld-linux.so.2
No symbol table info available.
#1 0x00554e17 in do_lookup_x () from /lib/ld-linux.so.2
No symbol table info available.
#2 0x005550da in _dl_lookup_symbol_x () from /lib/ld-linux.so.2
No symbol table info available.
#3 0x00559a05 in _dl_fixup () from /lib/ld-linux.so.2
No symbol table info available.
#4 0x0055fc90 in _dl_runtime_resolve () from /lib/ld-linux.so.2
You need a library with debugging symbols to debug the core file. Once you have that, you can get a backtrace from core which will give you leads.
The number with the core file might be the PID. Check this to confirm that - How to generate core dump file in Ubuntu
or cat /proc/sys/kernel/core_pattern