I have a following problem. I runned a programm which has created several threads.
Gdb outputs the following threads:
(gdb) info thread
Id Target Id Frame
3 LWP 23941 0x00007f5fc327c2d4 in ?? ()
2 LWP 22925 0x00007f5fc327e420 in ?? ()
* 1 LWP 23934 0x00007f5fc2779475 in ?? ()
Backtrace of the 2nd thread looks as follows:
(gdb) thread 2
[Switching to thread 2 (LWP 22925)]
#0 0x00007f5fc327e420 in ?? ()
(gdb) bt
#0 0x00007f5fc327e420 in ?? ()
#1 0x00007f5fb406f300 in ?? ()
#2 0x0000000000439021 in AaSemWait (semPtr=0x7f5fb406f300) at ../bm/testing/unit/src/ccs/AaSem.cpp:97
#3 0x00000000004197d5 in bm_semaphore_base::wait (this=0x7fff75b01db8) at /home/michaelfrysztacki/bm_local_build2/BTS_SC_BM/bm/interface/framework/bm_semaphore.h:174
#4 0x0000000000500e1c in radio::thread_synchroniser::wait (this=0x7fff75b01c00) at /home/michaelfrysztacki/bm_local_build2/BTS_SC_BM/bm/radio/testing/stubs/radio_module_receiver.h:107
#5 0x00000000005034a5 in radio::mt_radio_service::send_messages (this=0x7f5fb406d140) at ../bm/radio/testing/unit/mt_radio_service.cpp:236
#6 0x0000000000503010 in radio::mt_radio_service::simulate_radio_modules (this=0x7f5fb406d140, set_timeoff_test=false) at ../bm/radio/testing/unit/mt_radio_service.cpp:189
#7 0x00000000004fde77 in radio::mt_radio_service_module_test_sending_message_and_receive_from_radio_service_Test::TestBody (this=0x7f5fb406d140) at ../bm/radio/testing/unit/mt_radio_service.cpp:347
#8 0x000000000069cab1 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ()
....
This backtrace satisfies me because I know where the programm stopped.
I am confused about the 1st backtrace:
(gdb) thread 1
[Switching to thread 1 (LWP 23934)]
#0 0x00007f5fc2779475 in ?? ()
(gdb) bt
#0 0x00007f5fc2779475 in ?? ()
#1 0x00007f5fc277c675 in ?? ()
#2 0x0000000000000000 in ?? ()
Perhaps this thread runned into a shared library where I don't have debug symbols and sources and I am ok with it. I want to know where from my programm invoked the shared library method. I cannot determine it with this backtrace. I can download some shared libraries with debuging symbols but how can I know which shared library entered my thread ? Or maybe this is the entry point of starting thread using pthread ? Why is this backtrace so poor ? Lastly I am not entirely sure if this is a shared library how can I get it ?
(gdb) info sharedlibrary
warning: Can't read pathname for load map: Input/output error.
From To Syms Read Shared Object Library
No /lib/x86_64-linux-gnu/librt.so.1
No /lib/x86_64-linux-gnu/libexpat.so.1
No /usr/lib/libboost_thread.so.1.49.0
No /usr/lib/libboost_system.so.1.49.0
No /usr/local/lib/libprotobuf.so.8
No /lib/x86_64-linux-gnu/libpthread.so.0
No /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No /lib/x86_64-linux-gnu/libm.so.6
No /lib/x86_64-linux-gnu/libgcc_s.so.1
No /lib/x86_64-linux-gnu/libc.so.6
No /lib64/ld-linux-x86-64.so.2
the 'info sharedlibrary' command lists the shared libraries where I have no debugging symbols. How can I get the library e.g. /usr/lib/libboost_thread.so.1.49.0 with debuging symbols ( and sources ) ?
Your info shared output shows that GDB did not load symbols for any shared library.
The most likely cause: you have set auto-solib-add off somewhere in your ~/.gdbinit file.
You should be able to fix this with either
(gdb) shared . -- force GDB to read all shared libraries, or
just take out the set auto-solib-add off from wherever it's coming from.
Related
I am trying to debug my c++ programming assignment application using gdb on an Ubuntu server, because it produces segmentation fault.
But the file produces ?? symbols that are unreadable to me when I try bt it gives me.
(gdb) bt
#0 0x00007f141956d277 in ?? ()
#1 0x00007ffc1a866bd0 in ?? ()
#2 0x000055e1f101d5e0 in ?? ()
#3 0x00007ffc1a866db0 in ?? ()
#4 0x000055e1f1433e70 in ?? ()
#5 0x00007ffc1a866bd0 in ?? ()
#6 0x000055e1f10224a9 in ?? ()
#7 0x000055e1f14341f8 in ?? ()
#8 0x00000001f14344d0 in ?? ()
#9 0x0000000000000000 in ?? ()
I was following this link, and it told me to load these symbols
symbol-file /path/to/my/binary
sharedlibrary
The sharedlibrary was found, but the symbol-file path is not there. So,it did change bt command output somehow
(gdb) bt
#0 tcache_get (tc_idx=0) at malloc.c:2943
#1 _GI__libc_malloc (bytes=19) at malloc.c:3050
#2 0x000055e1f10224a9 in ?? ()
#3 0x000055e1f14341f8 in ?? ()
#4 0x00000001f14344d0 in ?? ()
#5 0x0000000000000000 in ?? ()
I still don't understand the bug.
Now, I don't know it's a problem from the GDB for not having this symbol-file or its a compilation problem which I don't know how or that's enough for me to debug, but I was following Debugging a Segmentation Fault and it was much clearer to troubleshoot.
When I search for similar cases, all of them were answered only for their case, not a general solution how to deal with these kinds of error. I also thought of installing or locating that symbol-file but I didn't understand how.
If someone could help me, I need to understand what is my problem and how should I fix it.
Note: core dump is produced in the /tmp not in current application directory
I was following this link, and it told me to load these symbols
Don't follow this link (it's unnecessarily complicated for your use case).
Instead, do this:
gdb /path/to/my/binary
(gdb) run
... GDB will stop when your program encounters SIGSEGV
(gdb) bt # should produce meaningful output now.
I'm trying to do a post mortem analysis using gdb on a C++ application inside a docker container. I have increased ulimit, added SYS_PTRACE capability, and used seccomp=unconfined. But when I use gdb and the main application with debug symbols I see this error:
warning: Unexpected size of section `.reg-xstate/10734' in core file.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Unexpected size of section `.reg-xstate/10734' in core file.
#0 0x00007f05ce755350 in ?? ()
[Current thread is 1 (Thread 0x7f04d3ffe700 (LWP 10734))]
(gdb)
(gdb) bt
#0 0x00007f05ce755350 in ?? ()
#1 0x00007f05cea2382f in ?? ()
#2 0x00007f0000000008 in ?? ()
#3 0x00007f05ced76c40 in ?? ()
#4 0x00007f04d3ffdd50 in ?? ()
#5 0x00007f05ce897d69 in ?? ()
#6 0x00007f05ced76c40 in ?? ()
#7 0x00007f04d3ffdd70 in ?? ()
#8 0x00007f04d3ffdd90 in ?? ()
#9 0x00007f05ce89796c in ?? ()
#10 0x00007f04d3ffdd90 in ?? ()
#11 0x0000000000000000 in ?? ()
Why is this happening when the ulimit is unlimited?
root#57a1bc01c676:/workspace/builddir# ulimit
unlimited
Recently I upgraded my OS from Ubuntu Precise Saucy (13.10) to Trusty (14.04). After this upgrade, cv::VideoCapture became not working properly. The program aborts when reading a video file. For example,
int main(int argc, char**argv)
{
cv::VideoCapture vin("sample/vout2l.avi");
...
Executing this program, it aborts with a message:
*** Error in `./cv2-videoread.out': malloc(): memory corruption: 0x0000000000e3eff0 ***
Abort (core dumped)
The backtrace looks like:
[New LWP 15586]
[New LWP 15587]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./cv2-videoread.out'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007ff953e61c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ff953e61c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff953e65028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ff953e9e2a4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ff953eabe26 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007ff953eac1ab in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007ff953eadba4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007ff953eaf7d2 in posix_memalign () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x00007ff94fa640fe in av_malloc () from /usr/lib/x86_64-linux-gnu/libavutil.so.52
#8 0x00007ff94fa641b1 in av_strdup () from /usr/lib/x86_64-linux-gnu/libavutil.so.52
#9 0x00007ff94fa5e5db in av_dict_set ()
from /usr/lib/x86_64-linux-gnu/libavutil.so.52
#10 0x00007ff954738574 in CvCapture_FFMPEG::open(char const*) ()
from /usr/lib/libopencv_highgui.so.2.4
#11 0x00007ff954738719 in cvCreateFileCapture_FFMPEG ()
from /usr/lib/libopencv_highgui.so.2.4
#12 0x00007ff95473aac9 in cvCreateFileCapture_FFMPEG_proxy(char const*) ()
from /usr/lib/libopencv_highgui.so.2.4
---Type <return> to continue, or q <return> to quit---
#13 0x00007ff954722d89 in cvCreateFileCapture ()
from /usr/lib/libopencv_highgui.so.2.4
#14 0x00007ff954723045 in cv::VideoCapture::open(std::string const&) ()
from /usr/lib/libopencv_highgui.so.2.4
#15 0x00007ff95472315c in cv::VideoCapture::VideoCapture(std::string const&) ()
from /usr/lib/libopencv_highgui.so.2.4
#16 0x0000000000401281 in main (argc=1, argv=0x7fff1f938388) at cv2-videoread.cpp:30
(gdb)
NOTE: cv::VideoCapture vin(... is 30th line.
Before upgrading the OS, this code was working with the same input file.
From the backtrace, it seems that the trouble happens at CvCapture_FFMPEG and libavutil. I upgraded the packages ffmpeg libavutil-dev libavutil51 libavutil52 but they were already up-to-date.
Also, OpenCV packages are up-to-date (I checked libopencv-core-dev libopencv-core2.4 libopencv-dev libopencv-highgui-dev libopencv-highgui2.4).
I also tested OpenCV built from source, but got the same results.
Do you have ideas to figure this out?
So, I've solved this issue.
By analyzing the program with ldd, I found that it was linked to, for example, /usr/lib/libopencv_highgui.so. However, in x86_64 system, it should be /usr/lib/x86_64-linux-gnu/libopencv_highgui.so. In my system, both files were installed.
The issue was caused by /usr/lib/libopencv_*.so (I'm not sure how I installed them. Maybe from source code...?). I removed these files, and compiled the above program again. Then it worked without errors.
I'm using the Point Cloud Library with cmake for compilation, and I've got it building in debug mode, but my program doesn't seg fault or abort in the way I'd expect it to.
Specifically, I get messages like this:
(gdb) run bunny
Starting program: debug/our_cvfh bunny
libc++abi.dylib: terminating
[New Thread 0x170b of process 80178]
Program received signal SIGABRT, Aborted.
0x00007fff88c6f866 in ?? ()
(gdb) bt
#0 0x00007fff88c6f866 in ?? ()
#1 0x00007fff8bb5235c in ?? ()
#2 0x0000000000000000 in ?? ()
(gdb) break rec/registered_views_source.h:305
Cannot access memory at address 0x961d60
In this case, I know where the error is, but I'd like to be able to backtrace it and see what called the function in this case.
Is PCL creating another thread, and that's why I can't backtrace? I'm not doing any visualizations right now, so I can't figure out why it'd be using threading.
I've also tried running the program in the debug directory instead of from my source root directory. Here's another example of it not working:
$ gdb our_cvfh
GNU gdb (GDB) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin13.1.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from our_cvfh...done.
run (gdb) run
Starting program: /Users/jwoods/Projects/lidargl/fpfh/debug/our_cvfh
[New Thread 0x170b of process 33571]
Program received signal SIGSEGV, Segmentation fault.
0x000000010016cdec in ?? ()
(gdb) bt
#0 0x000000010016cdec in ?? ()
#1 0x00007fff5fbfbd08 in ?? ()
#2 0x00007fff5fbfbcc0 in ?? ()
#3 0x00007fff5fbfbcc8 in ?? ()
#4 0x00007fff5fbfbcc8 in ?? ()
#5 0x00007fff5fbfbcc8 in ?? ()
#6 0xffffffffffffffff in ?? ()
#7 0x00007fff5fbfbcc8 in ?? ()
#8 0x00007fff5fbfbcc8 in ?? ()
#9 0x00007fff5fbfbcc0 in ?? ()
#10 0x00007fff5fbfbcc0 in ?? ()
#11 0x00007fff5fbfbcc8 in ?? ()
#12 0x00007fff5fbfbcc8 in ?? ()
#13 0x00007fff5fbfbcc8 in ?? ()
#14 0x00007fff5fbff4a8 in ?? ()
#15 0x00007fff5fbff4d8 in ?? ()
#16 0x00007fff5fbff420 in ?? ()
#17 0x00007fff5fbff4d8 in ?? ()
#18 0x0000000000000000 in ?? ()
(gdb)
gdb works fine when I'm not using CMake, so my guess is it has something to do with CMake. This doesn't seem to present a problem for anyone else, which tells me it may also have to do with the fact that I'm using CMake with Mac OS X.
How do I get my normal GDB behavior?
Update
I can run dsymutil my_output_binary in order to generate the debugging symbols (following make). This is a workaround. I'd like it to be done automatically, and amn't sure why it's not. The dsymutil strategy works for most segfaults, but doesn't work for some cases of SIGABRT. Here is the output:
Calling compute <--- normal std::cerr output of my program, single-threaded
Assertion failed: (index >= 0 && index < size()), function operator[], file /usr/local/Cellar/eigen/3.2.1/include/eigen3/Eigen/src/Core/DenseCoeffsBase.h, line 378.
[New Thread 0x170b of process 64108]
Program received signal SIGABRT, Aborted.
0x00007fff84999866 in ?? ()
(gdb) bt
#0 0x00007fff84999866 in ?? ()
#1 0x00007fff862c335c in ?? ()
#2 0x0000000000000000 in ?? ()
(gdb) info threads
Id Target Id Frame
2 Thread 0x170b of process 64108 0x00007fff8499a662 in ?? ()
* 1 Thread 0x1503 of process 64108 0x00007fff84999866 in ?? ()
(gdb)
Note that my program is not itself multi-threaded, but it appears to be making use of a library which is creating threads — or at least that's what I gather from the gdb output.
I've tried disabling threading with Eigen, which is used by PCL.
Interestingly, lldb is able to generate the backtraces, but I'm curious as to why GDB can't.
Other than simply using LLDB in lieu of GDB, I haven't figured out how to debug threads.
However, I have figured out how to automatically produce debugging symbols! Hooray.
You need three files:
UseCompVer.cmake
AddOptions.cmake
UseDebugSymbols.cmake
Put these in your project's cmake/Modules/ directory.
In CMakeLists.txt, you'll need the following lines after your project declaration:
set(CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/Modules/")
if (APPLE) # this if statement is optional, but you definitely need the include()
include(UseDebugSymbols)
endif (APPLE)
That will look in the appropriate location.
I should note that I made another change to my project simultaneously, which may also be useful to you. In my add_executable line, right after the name of the target, I added MACOSX_BUNDLE. This flag will cause it to be compiled as a .app instead of a regular binary. Here's an example from my project:
add_executable(pose MACOSX_BUNDLE pose.cpp rec/global_nn_recognizer_cvfh.cpp rec/global_nn_recognizer_cvfh.hpp rec/render_views_tesselated_sphere.cpp)
C++ application gets crashed with the core file showing error
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff79e54000
Core was generated by `./server'.
Program terminated with signal 6, Aborted.
#0 0x0000003b67230265 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x0000003b67230265 in raise () from /lib64/libc.so.6
#1 0x0000003b67231d10 in abort () from /lib64/libc.so.6
#2 0x0000003b6726a9bb in __libc_message () from /lib64/libc.so.6
#3 0x0000003b6727247f in _int_free () from /lib64/libc.so.6
#4 0x0000003b672728db in free () from /lib64/libc.so.6
#5 0x00000000004060df in operator delete (p=0x20030190) at ../lib/m_string.cpp:43
#6 0x0000000000403892 in TStr::~TStr (this=0x2102c980, __in_chrg=<value optimized out>) at ../lib/m_string.cpp:175 –
could able to understand about this issue. Here is the link that i have verified https://bugzilla.redhat.com/show_bug.cgi?id=959013
shows that the size of vdso file is not enough. which is in path /proc/self/maps.
please let me know what kind of issue is this and please suggest a fix for this.
what kind of issue is this
Any crash inside malloc or free is a sure sign of previous heap corruption.
Use Valgrind or AddressSanitizer (incorporated into GCC-4.8 as well) to find the root cause.
Ignore vdso -- as Tom Tromey said, it has nothing to do with the problem.