I have seen gdb reading and loading symbols automatically. (like the below code reading from ld-linux-x86-64.so.2)
----------
gdb 86exec 86core
...
Reading symbols from 86exec...done.
Illegal process-id: 86core.
[New Thread 14983]
[New Thread 14978]
[New Thread 14980]
[New Thread 14981]
[New Thread 14982]
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./86exec'.
Program terminated with signal 11, Segmentation fault.
----------
But in my mips platform I am not getting that automatic reading done.
---------
Reading symbols from /mipsexec...done.
[New LWP 1333]
[New LWP 1328]
[New LWP 1330]
[New LWP 1331]
[New LWP 1332]
Core was generated by `./mipsexec'.
Program terminated with signal 11, Segmentation fault.
----------
I wanted to know, from where does this information come.
If it is from core file, how can I analyze whether my mipscore is having the specific info.
I doubt that mips gdb doesn't support PIE well. As I know, in PIC, mips use rld_map but not DE_DEBUG for dynamic information but in PIE mode, mips likes other architectures, using DT_DEBUG.
In dynamic linker, there are statements like this:
#if !defined __mips__
/* we disable for mips because normally this page is readonly
* and modifying the value here needlessly dirties a page.
* see this post for more info:
* http://uclibc.org/lists/uclibc/2006-April/015224.html */
if (dpnt->d_tag == DT_DEBUG)
dpnt->d_un.d_val = (unsigned long)debug_addr;
#endif
this statement in PIC is ok, but not work in PIE. If you remove this #ifdef, you will get a segment fault because .dynamic section in mips is readonly. .dynamic section in arm and x86 etc is writeable.
if you change .dynamic section of mips to writeable, and open DT_DEBUG handling in dynamic linker, it works, but still has some problems. For example, the readonly VMA is writeable, and need more page copy when fork a subprocess.
I suffered the problem weeks ago and I don't know whether mips gdb works well in Android 4.2 or not. In Android 4.2, all c/c++ files are built with PIE.
Related
Environment
OS: CentOS 7.9_x64
Memory, CPU, current Disk Space:Memory 96G, Disk 1T
TDengine Version:TDengine-server-2.0.20.13-Linux-x64
TDengine taosd daemon coredump.
gdb output:
[New LWP 5461]
[New LWP 5499]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/taosd'.
Program terminated with signal 11, Segmentation fault.
#0 0x000056308db735cf in gcBuildQueryJson (pContext=0x7fdfdc0008c0, cmd=0x7fdfe00014a0, result=0x7fdfcc048ab0, numOfRows=682) at /home/ubuntu/workroom/jenkins/TDinternal/community/src/plugins/http/src/httpGcJson.c:154
154 /home/ubuntu/workroom/jenkins/TDinternal/community/src/plugins/http/src/httpGcJson.c: No such file or directory.
Missing separate debuginfos, use: debuginfo-install glibc-2.17-324.el7_9.x86_64
how to resolve it?
how to resolve it?
It's a bug in TDengine-server. You don't "resolve" bugs.
You can try to figure out what the bug is (via debugging), or you can try newer version of TDengine-server (current appears to be 2.2.0.2) and hope that the particular bug you've hit has been fixed.
I want to restore memory in GDB when I debug core file.
I checked restore function in GDB when process is running. (https://sourceware.org/gdb/onlinedocs/gdb/Dump_002fRestore-Files.html)
It was success.
But I want to restore memory when I debug core file.
I can use core_filter and I can select segment. but It is out very big size core file.
so I use a way that dump and restoring memory.
but it is possible when process is running.
I want to restore memory when process is not running.
I need the way when I debug core dump file.
Can I restore memory in this way?
Do you know another ways?
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
...
Reading symbols from main...done.
[New LWP 19274]
Core was generated by `./main'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000000000040053a in main () at main.c:6
6 *a = 3;
(gdb) restore memory_dump binary &array 0 40
You can't do that without a process to debug.
I have a core dump from a C++ application running on an embedded imx6 board (yocto linux). I can put gdb on the box and run it in a terminal to examine the core file like so just fine:
gdb myApplication core.udpsrc256:src.1520419431.5526
I get extremely limited information, and really need to know more about what caused the core dump. All I have is a printout from the application:
(myApplication:5526): GLib-ERROR **: ../../glib-2.46.2/glib/gmem.c:100: failed to allocate 65611 bytes
./run-app.sh: line 8: 5526 Trace/breakpoint trap (core dumped) XDG_RUNTIME_DIR=/run/user/root ./myApplication
Also the core dump backtrace gives some useless stuff. I need to know more stuff up the stack that led to this frame:
#0 0x75ff1910 in raise () from /lib/libc.so.6
[Current thread is 1 (LWP 5533)]
(gdb)
(gdb)
(gdb) bt
#0 0x75ff1910 in raise () from /lib/libc.so.6
#1 0x6b169558 in g_logv () from /usr/lib/libglib-2.0.so.0
#2 0x6b169610 in g_log () from /usr/lib/libglib-2.0.so.0
#3 0x6b1681c4 in g_malloc () from /usr/lib/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Sidenote -- there is some warnings when I startup gdb:
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-poky-linux-gnueabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from qt5qmlvideo...done.
warning: exec file is newer than core file.
[New LWP 5533]
[New LWP 5526]
[New LWP 5531]
[New LWP 5528]
[New LWP 5534]
[New LWP 21064]
[New LWP 5536]
[New LWP 21065]
[New LWP 5532]
[New LWP 5527]
[New LWP 5530]
[New LWP 5537]
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `./qt5qmlvideo -platform wayland'.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0 0x75ff1910 in raise () from /lib/libc.so.6
[Current thread is 1 (LWP 5533)]
(gdb)
Can anyone help? Do I need some of the stuff gdb warns about... or can i rebuild my application and its dependencies in some other configuration that would give more output? Thank you!
Some more notes that may matter -
This is a multithreaded application running a gstreamer pipeline. Many gstreamer plugins generate their own threads, one of which in this pipeline is 'udpsrc'. I'm wondering if it's because this failure happens in one of those threads is the reason why I can't get details, but I want to know how to get it to show the details if possible!
(1)
The
Do you need "set solib-search-path" or "set sysroot"?
is a problem. Check the path (on your device) where linux-vdso.so.1 resides, and include that in the solib-search-path. Similarly for the other shared-object libraries that your program uses. E.g. if some shared-object libraries are in /lib, some are in /usr/adowdy/lib and some are in /usr/adowdy/arm/lib, you can say:
(gdb) set solib-search-path /lib:/usr/adowdy/lib:/usr/adowdy/arm/lib
(2) The
warning: Unable to find libthread_db matching inferior's thread
library, thread debugging will not be available.
is also a problem. See the answer to this question
(3) The
failed to allocate 65611 bytes
is a clue. Are you, by any chance, trying to allocate a negative number of bytes (maybe 65536 - 65611 = -75 bytes)?
Also the core dump backtrace gives some useless stuff.
It's not entirely useless. The stack trace, and the message from the application say the same thing: your application ran out of memory (malloc failed to allocate 65611 bytes).
While a more complete stack would tell you which particular call to g_malloc failed, it's very likely to not matter in practice -- if this g_malloc didn't fail, the next one would.
You likely have a memory leak, or are simply allocating too much memory for what your system allows.
You should look into many debugging tools built for solving this exact problem.
I'm fairly new to debugging core dumps on Linux, and I'm running into a weird issue. Hoping to get some suggestions.
We're getting occasional crashes on our game servers running on AWS Linux boxes. I set up the boxes to generate core dumps. Often, the dumps are around a few hundred MB -- roughly the size of the program in memory. These I'm able to load in gdb and seemingly get a valid backtrace.
But frequently, we're getting dumps that are multiple GB in size. Usually, when I load these core dumps in gdb, there's no usable info in the backtrace.
Here's an example output:
> gdb AAPGOrbis core.3871
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from AAPGOrbis...Reading symbols from <path to>/AAPGOrbis.dbg...done.
done.
[New LWP 3871]
[New LWP 3877]
[New LWP 6557]
[New LWP 3876]
[New LWP 6558]
[New LWP 6559]
warning: Error reading shared library list entry at 0x302e6f732e646165
warning: Error reading shared library list entry at 0x74756d5f64616572
Core was generated by `/opt/aapg/Binaries/Linux/AAPGOrbis aaentry?game=AAGame.AAGamePreGameLobbyDedica'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fed61d001f7 in ?? ()
(gdb) bt full
#0 0x00007fed61d001f7 in ?? ()
No symbol table info available.
#1 0x00007fed61d018e8 in ?? ()
No symbol table info available.
#2 0x0000000000000020 in ?? ()
No symbol table info available.
#3 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)
Any ideas as to what might be causing this? I'm wondering if the size of the core dumps coupled with the lack of valid data is indicative of some really bad memory corruption.
Any insight would be greatly appreciated!
warning: Error reading shared library list entry at 0x302e6f732e646165
warning: Error reading shared library list entry at 0x74756d5f64616572
GDB is attempting to read the list of loaded shared libraries from a clearly bogus address (both of these addresses are ASCII strings ead.so.0read_mut).
The most frequent cause is that you have given GDB the wrong binary: the AAPGOrbis that you give GDB must be exactly the same binary as the one that crashed.
Another possibility is that the shared library list (which is in heap) has indeed been corrupted by the program running amok.
I have a coredump file from a colleague's machine.
We both have the same sources for the program and the same third party *.so files (like libmysqlclient18 and several in-house ones).
The problem is that we both compile the software (from the same sources) independently and I want to use his core dump files for inspection with GDB on my machine.
When I try to load the core file and my executable into gdb I get:
user#ubuntu:/mnt/hgfs/share/dir$ gdb prog core
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /mnt/hgfs/share/dir/prog...done.
warning: exec file is newer than core file.
[New LWP 4465]
[New LWP 4462]
[New LWP 4464]
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `./prog'.
Program terminated with signal 11, Segmentation fault.
#0 0x002d3706 in ?? () from /lib/i386-linux-gnu/libc.so.6
(gdb) bt full
#0 0x002d3706 in ?? () from /lib/i386-linux-gnu/libc.so.6
No symbol table info available.
#1 0x00000000 in ?? ()
No symbol table info available.
(gdb)
Is this possible for this scenario to work? (I compile the software with debugging symbols enabled, of course)
If not, what are the technical details I'm missing?
I know for sure that it wouldn't be possible if he or I would make modifications to the source, since then the executable would be different, but this is not the case, the third party *.so files and the sources, they all match.
UPDATE:
After installing libc6-dbg as user mkfs suggested in the comments, I get this in gdb:
user#ubuntu:/mnt/hgfs/share/dir$ gdb prog core
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /mnt/hgfs/share/dir/prog...done.
warning: exec file is newer than core file.
[New LWP 4465]
[New LWP 4462]
[New LWP 4464]
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `./prog'.
Program terminated with signal 11, Segmentation fault.
#0 0x002d3706 in _IO_helper_overflow (s=0x0, c=0) at vfprintf.c:2188
2188 vfprintf.c: No such file or directory.
(gdb) bt
#0 0x002d3706 in _IO_helper_overflow (s=0x0, c=0) at vfprintf.c:2188
#1 0x00000000 in ?? ()
(gdb) bt full
#0 0x002d3706 in _IO_helper_overflow (s=0x0, c=0) at vfprintf.c:2188
written = 47
target = <optimized out>
used = -1226838776
#1 0x00000000 in ?? ()
No symbol table info available.
(gdb)
Is this possible for this scenario to work?
Yes, but you need to ensure that the symbol layout of the binary you build on both machines is identical (or at least close enough). This isn't necessarily trivial: things like local username, pathname for sources or installation directory, and hostname sometimes leak into the built object files, and may cause symbol mismatch.
To check whether the binaries are close, run diff <(nm a.out) <(nm b.out) -- there should only be few differences. If you see a lot of differences, your binaries aren't close enough.
I compile the software with debugging symbols enabled, of course
This may be your first mistake: if your coworker builds with -O2, and you build with -g (and implied -O0), the binary is guaranteed to not match.
You need to build with exactly the flags your coworker builds with (but you may add debugging symbols; e.g. if your coworker builds with -O2, you should build with -O2 -g together).
P.S. Note that you also need identical versions of system libraries on the two machines.