I have a failing Cargo test:
$ cargo test
[snip]
Running target/gunzip-c62d8688496249d8
running 2 tests
test test_extract_failure ... FAILED
test test_extract_success ... ok
failures:
---- test_extract_failure stdout ----
task 'test_extract_failure' panicked at 'assertion failed: result.is_err()', /home/dhardy/other/flate2-rs/tests/gunzip.rs:19
failures:
test_extract_failure
test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured
task '<main>' panicked at 'Some tests failed', /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/src/libtest/lib.rs:250
How do I launch the failing test in a debugger like GDB?
This should be a general question, but for those wanting to retrace my steps, install a recent nightly Rust build and:
git clone https://github.com/dhardy/flate2-rs.git
git checkout 24979640a880
cd flate2-rs
cargo test
You can get a test binary to filter the tests it runs by passing additional arguments to it; Cargo exposes this directly, too. Thus, cargo test test_extract_failure will just run that specific case. (This is convenient if you have other tests that panic and are expected to fail, so that they won’t call the rust_panic function I am about to mention, leaving only the offending call there.)
In order to use gdb, you’ll need to run the test binary directly (if you use Cargo it runs in a subprocess and thus gdb won’t catch panics inside it). Cargo helpfully tells you the file name, target/gunzip-c62d8688496249d8. You can run this directly with --test to make it a test run:
$ target/gunzip-c62d8688496249d8 --test test_extract_failure
running 1 test
test test_extract_failure ... FAILED
failures:
---- test_extract_failure stdout ----
task 'test_extract_failure' panicked at 'assertion failed: result.is_err()', /home/dhardy/other/flate2-rs/tests/gunzip.rs:19
failures:
test_extract_failure
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
task '<main>' panicked at 'Some tests failed', /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/src/libtest/lib.rs:250
Now to hook it up with gdb. There is a convenient function for which you can insert a breakpoint, rust_panic. Once in gdb, break rust_panic means it will pause whenever something triggers a panic, before actually doing the unwinding.
Here’s what a session might end up looking like:
$ gdb target/demo-92d91e26f6ebc557
…
Reading symbols from target/demo-92d91e26f6ebc557...done.
(gdb) break rust_panic
Breakpoint 1 at 0xccb60
(gdb) run --test test_extract_failure
Starting program: /tmp/demo/target/demo-92d91e26f6ebc557 --test test_extract_failure
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
running 1 test
[New Thread 0x7ffff6ef4700 (LWP 14254)]
[New Thread 0x7ffff5fff700 (LWP 14255)]
[Switching to Thread 0x7ffff5fff700 (LWP 14255)]
Breakpoint 1, 0x0000555555620b60 in rust_panic ()
(gdb) bt
#0 0x0000555555620b60 in rust_panic ()
#1 0x0000555555621274 in unwind::begin_unwind_inner::hb821324209c8ed246Qc ()
#2 0x000055555556bb6d in unwind::begin_unwind::h7834652822578025936 ()
#3 0x000055555556b9fd in demo::do_something () at <std macros>:8
#4 0x000055555556b98e in demo::test_extract_failure () at src/lib.rs:3
#5 0x000055555559aa4b in task::TaskBuilder::try_future::closure.8077 ()
#6 0x000055555560fd03 in task::TaskBuilder::spawn_internal::closure.30919 ()
#7 0x000055555561f672 in task::Task::spawn::closure.5759 ()
#8 0x0000555555621cac in rust_try_inner ()
#9 0x0000555555621c96 in rust_try ()
#10 0x000055555561f713 in unwind::try::ha8078a6ae9b50ccepFc ()
#11 0x000055555561f51c in task::Task::run::hdb5fabf381084abafOb ()
#12 0x000055555561f168 in task::Task::spawn::closure.5735 ()
#13 0x0000555555620595 in thread::thread_start::h4d73784c295273b3i6b ()
#14 0x00007ffff79c2314 in start_thread () from /usr/lib/libpthread.so.0
#15 0x00007ffff72e25bd in clone () from /usr/lib/libc.so.6
(gdb)
In that particular case, #0–#2 and #5–#15 are noise, #3 and #4 are the signal we want.
cargo build now has a parameter
--test [<NAME>] Build only the specified test target
which builds a binary with only the set of tests provided as a parameter.
Related
It seems that gdb fails finding the code position of an assertion failure, after I recompile my code. More precisely, I expect the position of a signal raise, relative to an assertion failure, to be
0x00007ffff7a5ff00 in raise () from /lib64/libc.so.`6
while instead I obtain
0x00007ffff7a5ff00 in ?? ()
For instance, consider the following code
#include <assert.h>
int main()
{
assert(0);
return 0;
}
compiled with debug symbols and debugged with gdb.
> gcc -g main.c
> gdb a.out
On the first run of gdb, the position is found, and the backtrace is reported correctly:
GNU gdb (Gentoo 8.0.1 p1) 8.0.1
...
(gdb) r
Starting program: /home/myself/a.out
a.out: main.c:5: main: Assertion `0' failed.
Program received signal SIGABRT, Aborted.
0x00007ffff7a5ff00 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x00007ffff7a5ff00 in raise () from /lib64/libc.so.6
#1 0x00007ffff7a61baa in abort () from /lib64/libc.so.6
#2 0x00007ffff7a57cb7 in ?? () from /lib64/libc.so.6
#3 0x00007ffff7a57d72 in __assert_fail () from /lib64/libc.so.6
#4 0x00005555555546b3 in main () at main.c:5
(gdb)
The problem comes when I recompile the code. After recompiling, I issue the run command in the same gdb instance. Gdb re-reads the symbols, starts the program from the beginning, but does not find the right position:
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
`/home/myself/a.out' has changed; re-reading symbols.
Starting program: /home/myself/a.out
a.out: main.c:5: main: Assertion `0' failed.
Program received signal SIGABRT, Aborted.
0x00007ffff7a5ff00 in ?? ()
(gdb) bt
#0 0x00007ffff7a5ff00 in ?? ()
#1 0x0000000000000000 in ?? ()
(gdb) up
Initial frame selected; you cannot go up.
(gdb) n
Cannot find bounds of current function
At this point the debugger is unusable. One cannot go up, step forward.
As a workaround, I can manually reload the file, and positions are found again.
(gdb) file a.out
Load new symbol table from "a.out"? (y or n) y
Reading symbols from a.out...done.
(gdb) r
Starting program: /home/myself/a.out
a.out: main.c:5: main: Assertion `0' failed.
Program received signal SIGABRT, Aborted.
0x00007ffff7a5ff00 in raise () from /lib64/libc.so.6
(gdb)
Unfortunately, after reloading the file this way, gdb fails resetting the breakpoints.
ERRATA CORRIGE: I was experiencing failure in resetting the breakpoints using gdb 7.12.1. After upgrading to 8.0.1 the problem vanished. Supposedly, this was related to the bugfix https://sourceware.org/bugzilla/show_bug.cgi?id=21555. However, code positions where assertions fail still cannot be found correctly.
Does anybody have any idea about what is going on here?
This has started happening after a system update. The system update recompiled all system libraries, including the glibc, as position independent code, i.e., compiled with -fPIC.
Also, the version of the gcc I am using is 6.4.0
Here is a workaround. Since file re-reads the symbols correctly, while run does not, we can define a hook for the command run so to execute file before:
define hook-run
pi gdb.execute("file %s" % gdb.current_progspace().filename)
end
after you change the source file and recompile u are generating a different file from the one loaded to GDB.
you need to stop the running debug cession and reload the file.
you cant save the previously defined breakpoints and watch points in the file to a changed source, since gdb is actually inserting additional code to your source to support breakpoints and registrar handlers.
if you change the source the the behavior is undefined and you need to reset those breakpoints.
you can refer to gdb manual regarding saving breakpoints in a file as
Mark Plotnick suggested, but it wont work if you change the file(from my experience)
https://sourceware.org/gdb/onlinedocs/gdb/Save-Breakpoints.html
http://www.webtoolkit.eu/wt/doc/tutorial/dbo.html says
The complete source code for the examples used in this tutorial are available as ready-to-run programs in the examples/feature/dbo/ folder of Wt.
I'm trying to run tutorial1.C from that directory, and I get the following output:
(gdb) run
Starting program: /home/lawsa/sources/memory/dist/flashcard --docroot . --http-address 0.0.0.0 --http-port 9090
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
begin transaction
create table "user" (
"id" integer primary key autoincrement,
"version" integer not null,
"name" text not null,
"password" text not null,
"role" integer not null,
"karma" integer not null
)
commit transaction
created tables
ending transaction
Program received signal SIGSEGV, Segmentation fault.
0x000000000041a4a8 in void std::vector<Wt::Dbo::ptr_base*, std::allocator<Wt::Dbo::ptr_base*> >::emplace_back<Wt::Dbo::ptr_base*>(Wt::Dbo::ptr_base*&&) ()
(gdb) bt
#0 0x000000000041a4a8 in void std::vector<Wt::Dbo::ptr_base*, std::allocator<Wt::Dbo::ptr_base*> >::emplace_back<Wt::Dbo::ptr_base*>(Wt::Dbo::ptr_base*&&) ()
#1 0x0000000000419c8e in std::vector<Wt::Dbo::ptr_base*, std::allocator<Wt::Dbo::ptr_base*> >::push_back(Wt::Dbo::ptr_base*&&) ()
#2 0x0000000000419682 in void Wt::Dbo::Session::implSave<User>(Wt::Dbo::MetaDbo<User>&) ()
#3 0x0000000000418c4e in Wt::Dbo::MetaDbo<User>::flush() ()
#4 0x00007ffff6c8eae2 in Wt::Dbo::Session::flush() ()
from /usr/lib/libwtdbo.so.38
#5 0x00007ffff6c9d14d in Wt::Dbo::Transaction::Impl::commit() ()
from /usr/lib/libwtdbo.so.38
#6 0x00007ffff6c9d1a9 in Wt::Dbo::Transaction::commit() ()
from /usr/lib/libwtdbo.so.38
#7 0x00000000004063d2 in run() ()
#8 0x0000000000407066 in main ()
(gdb)
For your reference, here's my code: http://sprunge.us/PYSO (I hope that lasts for a while, but let me know if it stops working). And my Makefile: http://sprunge.us/UCge and I ran gdb using $ gdb --args ./flashcard --docroot . --http-address 0.0.0.0 --http-port 9090
You can see the output from line 80 but not from 83, and the backtrace from gdb suggests that line 81 (commit) is the problem. If I remove line 81 so that the transaction commits because of going out of scope, the same problem exists, but it comes from the destructor of transaction.
I'm running archlinux with Wt 3.3.4-4, gcc 5.1.0-5, compiling with -std=c++0x.
The only thing I can imagine is if there is some binary incompatibility with std::vector?
Recompiling Wt should fix the problem. Instead of using the Wt as packaged through archlinux (pacman), compile Wt from source.
There is perhaps an ABI discrepancy somewhere that should be solved by compiling everything on your own machine with the same standard and boost libraries.
I'm using the Point Cloud Library with cmake for compilation, and I've got it building in debug mode, but my program doesn't seg fault or abort in the way I'd expect it to.
Specifically, I get messages like this:
(gdb) run bunny
Starting program: debug/our_cvfh bunny
libc++abi.dylib: terminating
[New Thread 0x170b of process 80178]
Program received signal SIGABRT, Aborted.
0x00007fff88c6f866 in ?? ()
(gdb) bt
#0 0x00007fff88c6f866 in ?? ()
#1 0x00007fff8bb5235c in ?? ()
#2 0x0000000000000000 in ?? ()
(gdb) break rec/registered_views_source.h:305
Cannot access memory at address 0x961d60
In this case, I know where the error is, but I'd like to be able to backtrace it and see what called the function in this case.
Is PCL creating another thread, and that's why I can't backtrace? I'm not doing any visualizations right now, so I can't figure out why it'd be using threading.
I've also tried running the program in the debug directory instead of from my source root directory. Here's another example of it not working:
$ gdb our_cvfh
GNU gdb (GDB) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin13.1.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from our_cvfh...done.
run (gdb) run
Starting program: /Users/jwoods/Projects/lidargl/fpfh/debug/our_cvfh
[New Thread 0x170b of process 33571]
Program received signal SIGSEGV, Segmentation fault.
0x000000010016cdec in ?? ()
(gdb) bt
#0 0x000000010016cdec in ?? ()
#1 0x00007fff5fbfbd08 in ?? ()
#2 0x00007fff5fbfbcc0 in ?? ()
#3 0x00007fff5fbfbcc8 in ?? ()
#4 0x00007fff5fbfbcc8 in ?? ()
#5 0x00007fff5fbfbcc8 in ?? ()
#6 0xffffffffffffffff in ?? ()
#7 0x00007fff5fbfbcc8 in ?? ()
#8 0x00007fff5fbfbcc8 in ?? ()
#9 0x00007fff5fbfbcc0 in ?? ()
#10 0x00007fff5fbfbcc0 in ?? ()
#11 0x00007fff5fbfbcc8 in ?? ()
#12 0x00007fff5fbfbcc8 in ?? ()
#13 0x00007fff5fbfbcc8 in ?? ()
#14 0x00007fff5fbff4a8 in ?? ()
#15 0x00007fff5fbff4d8 in ?? ()
#16 0x00007fff5fbff420 in ?? ()
#17 0x00007fff5fbff4d8 in ?? ()
#18 0x0000000000000000 in ?? ()
(gdb)
gdb works fine when I'm not using CMake, so my guess is it has something to do with CMake. This doesn't seem to present a problem for anyone else, which tells me it may also have to do with the fact that I'm using CMake with Mac OS X.
How do I get my normal GDB behavior?
Update
I can run dsymutil my_output_binary in order to generate the debugging symbols (following make). This is a workaround. I'd like it to be done automatically, and amn't sure why it's not. The dsymutil strategy works for most segfaults, but doesn't work for some cases of SIGABRT. Here is the output:
Calling compute <--- normal std::cerr output of my program, single-threaded
Assertion failed: (index >= 0 && index < size()), function operator[], file /usr/local/Cellar/eigen/3.2.1/include/eigen3/Eigen/src/Core/DenseCoeffsBase.h, line 378.
[New Thread 0x170b of process 64108]
Program received signal SIGABRT, Aborted.
0x00007fff84999866 in ?? ()
(gdb) bt
#0 0x00007fff84999866 in ?? ()
#1 0x00007fff862c335c in ?? ()
#2 0x0000000000000000 in ?? ()
(gdb) info threads
Id Target Id Frame
2 Thread 0x170b of process 64108 0x00007fff8499a662 in ?? ()
* 1 Thread 0x1503 of process 64108 0x00007fff84999866 in ?? ()
(gdb)
Note that my program is not itself multi-threaded, but it appears to be making use of a library which is creating threads — or at least that's what I gather from the gdb output.
I've tried disabling threading with Eigen, which is used by PCL.
Interestingly, lldb is able to generate the backtraces, but I'm curious as to why GDB can't.
Other than simply using LLDB in lieu of GDB, I haven't figured out how to debug threads.
However, I have figured out how to automatically produce debugging symbols! Hooray.
You need three files:
UseCompVer.cmake
AddOptions.cmake
UseDebugSymbols.cmake
Put these in your project's cmake/Modules/ directory.
In CMakeLists.txt, you'll need the following lines after your project declaration:
set(CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/Modules/")
if (APPLE) # this if statement is optional, but you definitely need the include()
include(UseDebugSymbols)
endif (APPLE)
That will look in the appropriate location.
I should note that I made another change to my project simultaneously, which may also be useful to you. In my add_executable line, right after the name of the target, I added MACOSX_BUNDLE. This flag will cause it to be compiled as a .app instead of a regular binary. Here's an example from my project:
add_executable(pose MACOSX_BUNDLE pose.cpp rec/global_nn_recognizer_cvfh.cpp rec/global_nn_recognizer_cvfh.hpp rec/render_views_tesselated_sphere.cpp)
I'm trying to stop right after the module is loaded in gdb. Let's assume that the binary is completely stripped out of all symbol informations, so there's no main.
Ideally I'd set the breakpoint on the entry point, but that idea breaks down due to relocations:
(gdb) info target
Symbols from "./application".
Local exec file:
`./application', file type elf64-x86-64.
Entry point: 0xc154
...
(gdb) break *0xc154
Breakpoint 1 at 0xc154
(gdb) r
Starting program: ./application
Warning:
Cannot insert breakpoint 1.
Error accessing memory address 0xc154: Input/output error.
(gdb) info target
Symbols from "./application".
Unix child process:
Using the running image of child process 22835.
While running this, GDB does not access memory from...
Local exec file:
`./application', file type elf64-x86-64.
Entry point: 0x555555560154
Even though that kind-of works (I could set a new breakpoint on the new address and disable the original), it cannot be easily executed via gdb script / batch mode, because it has a failing instruction in the middle.
Is there a way to do that? Ideally something like "run single instruction", rather than "run" would be useful.
Update:
GDB-8.1 implemented starti command, which makes this very easy.
Entry point: 0xc154
This is a dynamically-linked, position-independent (PIE) binary.
You want to stop in the dynamic linker after that binary is loaded and relocated, but before it executed anything.
(gdb) set stop-on-solib-events 1
(gdb) run
Starting program: /tmp/a.out
Stopped due to shared library event (no libraries added or removed)
(gdb) info target
Symbols from "/tmp/a.out".
Unix child process:
Using the running image of child process 13746.
While running this, GDB does not access memory from...
Local exec file:
`/tmp/a.out', file type elf64-x86-64.
Entry point: 0x5555555545f0
...
(gdb) bt
#0 __GI__dl_debug_state () at dl-debug.c:77
#1 0x00007ffff7ddd488 in dl_main (phdr=<optimized out>, phnum=<optimized out>, user_entry=<optimized out>, auxv=0x7ffff7ffe870) at rtld.c:1678
#2 0x00007ffff7defb24 in _dl_sysdep_start (start_argptr=<optimized out>, dl_main=0x7ffff7ddc6e0 <dl_main>) at ../elf/dl-sysdep.c:244
#3 0x00007ffff7ddf365 in _dl_start_final (arg=0x7fffffffe440) at rtld.c:338
#4 _dl_start (arg=0x7fffffffe440) at rtld.c:564
#5 0x00007ffff7ddb6b8 in _start () from /lib64/ld-linux-x86-64.so.2
I have a core dump for c/c++ application. I am new to programming so this question may sound silly. Is there some how I can write scripts to use gdb to analyze the core dump?
Yes, just script whatever you want. For example:
gdb \
-ex "set pagination 0" \
-ex "thread apply all bt" \
-batch ${EXECUTABLE_FILE} ${CORE_FILE}
Use -x option as below
$ cat gdb.cmds
set confirm off
set height 0
cd /homes/syrajendra
file dump
core dump.core
set solib-search-path "/lib"
bt
printf "\n"
quit
$ gdb -x gdb.cmds
. . .
[Thread debugging using libthread_db enabled]
Core was generated by `dump'.
Program terminated with signal SIGABRT, Aborted.
#0 0x000000080149f6ca in thr_kill () from /lib/libc.so.7
#0 0x000000080149f6ca in thr_kill () from /lib/libc.so.7
#1 0x0000000801574149 in abort () from /lib/libc.so.7
#2 0x0000000801556011 in __assert () from /lib/libc.so.7
#3 0x000000000040130a in fun2 (num=100) at ./dump.cpp:10
#4 0x0000000000401343 in fun1 (num=100) at ./dump.cpp:20
#5 0x000000000040137e in main () at ./dump.cpp:27