I'm trying to use gdb to record the execution in valgrind's gdbserver like so:
valgrind --vgdb-error=0 path/to/binary args
$ gdb path/to/binary
(gdb) target remote | vgdb
0x0000000004001090 in _start () from /lib64/ld-linux-x86-64.so.2
(gdb) target record-full
(gdb) c
Continuing.
/build/gdb-GT4MLW/gdb-8.1/gdb/target.c:3132: internal-error: gdbarch* default_thread_architecture(target_ops*, ptid_t): Assertion `inf != NULL' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) n
This is a bug, please report it. For instructions, see:
<http://www.gnu.org/software/gdb/bugs/>.
/build/gdb-GT4MLW/gdb-8.1/gdb/target.c:3132: internal-error: gdbarch* default_thread_architecture(target_ops*, ptid_t): Assertion `inf != NULL' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Create a core file of GDB? (y or n) n
Command aborted.
(gdb)
I was hoping to use the recorded execution to set a read/write watchpoint and reverse execute to see where it's used. I've been running with valgrind's gdbserver to break on the SIGTRAP emitted when an invalid read/write occurs.
Is this genuinely a bug, or am I doing something wrong/unsupported?
The valgrind gdbserver does not support record.
See http://www.valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.gdbserver-limitations
Related
gdb -quiet -iex 'set pagination off' -ex run -ex 'thread apply all bt' --batch --args <your prog>
The above is my default way or running my programs in CI. It is very convenient to have a stacktrace printed if the binary crashes, without having to hunt for coredump files.
edit: my default way of running CI has changed, because I also need to propagate the return code from the tested program in some circumstances
gdb -quiet -iex 'set pagination off' -iex 'set debuginfod enabled on' -ex run -ex 'thread apply all bt' -ex 'quit $_exitcode' --batch --args <your prog>
The problem is that I cannot do this when I compile with -fsanitize=address,leak,undefined. I get an error message when the program runs to the end and lsan is triggered (in an atexit handler, according to its docs).
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
==2861213==LeakSanitizer has encountered a fatal error.
==2861213==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==2861213==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
[Inferior 1 (process 2861213) exited with code 01]
Q1: Can I maintain the convenience of my gdb -quiet setup, while still getting lsan leak reports when my program leaks?
Currently the solution I am implementing is to hunt for the coredump files and execute gdb on them in a subsequent CI step (that runs upon test failure). For this, I had to configure sanitizers to permit coredump generation with disable_coredump=0, as described in How to generate core dump when use Address Sanitizer and gcc7.1.0.
Q2: Is it possible to use AddressSanitizer to do the job gdb used to do for me? That is, to run the equivalent of bt or thread apply all bt when my program crashes?
export DEBUGINFOD_URLS=https://debuginfod.elfutils.org
gdb -quiet -iex 'set pagination off' -iex 'set debuginfod enabled on' -iex 'set detach-on-fork off' -iex 'set breakpoint pending on' -x gha_gdb_commandfile.txt --args <your prog> <args>
The gdb_commandfile.txt being
break __lsan::CheckForLeaks
commands
detach
end
run
thread apply all bt
thread apply all py-bt
The gdb command file breaks when leak sanitizer is starting and detaches the debugger, so that leak sanitizer is free to do its own attach.
I tried this to debug this main.c
#include <assert.h>
#include <stdlib.h>
int main(int argc, char** argv) {
void* x = malloc(42);
if (argc > 1) assert(1 == 2);
return 0;
}
compiled as
gcc -g3 -fsanitize=address,leak main.c
and it gives both leak report and backtrace, depending on whether I run with argument or not.
The command works even if binary is compiled without leak sanitizer. Then gdb simply never sets the breakpoint.
I wanted to have gdb watch both parent and child if my process starts a subprocess. I tried to use 'set detach-on-fork off' for it. I need it because there might be subprocesses that should be leak-checked (and stacktrace should be dumped if they crash). But what I tried is not the way. The parent process should instead start the child within new instance of GDB. By default, GDB continues attached to parent, ignoring the child.
The rr debugger is able to record entire process subtree. Maybe there can be a way. But that is for another question.
I am attaching a running process in linux platform using gdb.
gdb -p <pid>
Setting the breakpoint, say
b test.cpp : 23
Now continuing
c
After this when i perform some operation destined to attached process, getting the below error in gdb console:
(gdb) c
Continuing.
[New Thread 0x7409fb70 (LWP 8530)]
Cannot get thread event message: debugger service failed
Please help me how to proceed further.
Additional details:
Its icc compiler, compiling with -g flag enabled
We're running squid from with gdb - that way we can automatically generate backtraces for debugging.
backtrace=`mktemp`
gdb -q -x /etc/service/squid3/gdbcommands /usr/sbin/squid 2>&1 >$backtrace
/usr/bin/mail -s "`hostname`: Squid was restarted (backtrace)" someaddress#charite.de < backtracetrace
rm $backtrace
/etc/service/squid3/gdbcommands contains:
set args -NsYC
handle SIGPIPE pass nostop noprint
handle SIGTERM pass nostop noprint
handle SIGUSR1 pass nostop noprint
handle SIGHUP pass nostop noprint
handle SIGSEGV stop
handle SIGABRT stop
run
set print pretty
backtrace full
generate-core-file
quit
But, every now and then, squid is "just" being stopped & restarted, with no crash being involved at all. In that case I'm still getting an email containing:
Reading symbols from /usr/sbin/squid...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Inferior 1 (process 57867) exited normally]
/etc/service/squid3/gdbcommands:10: Error in sourced command file:
No stack.
(gdb) quit
And of course there's no stack, since the program exited ok.
How can I change my gdbcommands file to avoid this?
This can be done using either Python or the gdb CLI. Since the CLI is a bit simpler, when possible, I'll sketch that approach.
First, you might as well only create a core file on a bad exit. And, we'll use the gdb exit code later, so let's arrange for that to tell the calling script what happened.
Where your current script says:
backtrace full
generate-core-file
quit
... instead use:
if !$_isvoid($_exitsignal) || (!$_isvoid($_exitcode) && $_exitcode != 0))
backtrace full
generate-core-file
quit 0
end
quit 1
Then your calling script can check the exit code of gdb:
if gdb your args here; then
mail results
fi
I am writing a plugin for a application, occasionally a SIGSEGV would be throw out. However, the application catches the signal SIGSEGV. In other word, The plugin is a dynamical library. The error occurs in my plugin and dynamical library. But the applcation handle the sSIGSEGV and exit normally. So, it is quite difficult for me to debug and get the backtrace of all stack frames. Any idea?
Currently I am using gdb as debug tool.
GDB will catch SIGSEGV before the application does.
What you described in comment to Logan's answer makes no sense.
I suspect what's really happening is that the application creates a new process, and only gets SIGSEGV in that other process, not the one you attached GDB to.
The following commands may be useful if my guess is correct:
(gdb) catch fork
(gdb) catch vfork
(gdb) set follow-fork-mode child
You might also want to edit and expand your question:
how do you know there is a SIGSEGV to begin with?
Posting a log of your interaction with GDB may also prove useful.
Even if the program traps SIGSEGV, gdb should still get it first and give you an opportunity to debug the program. Have you done something like
handle SIGSEGV nostop
in GDB? If so that could be why it is not stopping.
Are you sure that a segfault is actually occurring? Can you duplicate this behavior with another program, or by intentionally causing a segmentation violation?
For example:
$ cat sig.c
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
void handle(int n)
{
puts("Bail");
exit(1);
}
int main()
{
signal(SIGSEGV, handle);
int *pi = 0;
*pi = 10;
return 0;
}
$ gcc -g sig.c
$ ./a.out
Bail
$ gdb ./a.out
GNU gdb 6.6-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i486-linux-gnu"...
Using host libthread_db library "/lib/tls/i686/cmov/libthread_db.so.1".
(gdb) run
Starting program: /home/elcapaldo/a.out
Program received signal SIGSEGV, Segmentation fault.
0x08048421 in main () at sig.c:15
15 *pi = 10;
(gdb) where
#0 0x08048421 in main () at sig.c:15
(gdb) c
Continuing.
Bail
Program exited with code 01.
(gdb) q
I have this output when trying to debug
Program received signal SIGSEGV, Segmentation fault 0x43989029 in
std::string::compare (this=0x88fd430, __str=#0xbfff9060) at
/home/devsw/tmp/objdir/i686-pc-linux-gnu/libstdc++-v3/include/bits/char_traits.h:253
253 { return memcmp(__s1, __s2, __n); }
Current language: auto; currently c++
Using valgrind I getting this output
==12485== Process terminating with default action of signal 11 (SIGSEGV)
==12485== Bad permissions for mapped region at address 0x0
==12485== at 0x1: (within path_to_my_executable_file/executable_file)
You don't need to use Valgrind, in fact you want to use the GNU DeBugger (GDB).
If you run the application via gdb (gdb path_to_my_executable_file/executable_file) and you've compiled the application with debugging enabled (-g or -ggdb for GNU C/C++ compilers), you can start the application (via run command at the gdb prompt) and once you arrive at the SegFault, do a backtrace (bt) to see what part of your program called std::string::compare which died.
Example (C):
mctaylor#mpc:~/stackoverflow$ gcc -ggdb crash.c -o crash
mctaylor#mpc:~/stackoverflow$ gdb -q ./crash
(gdb) run
Starting program: /home/mctaylor/stackoverflow/crash
Program received signal SIGSEGV, Segmentation fault.
0x00007f78521bdeb1 in memcpy () from /lib/libc.so.6
(gdb) bt
#0 0x00007f78521bdeb1 in memcpy () from /lib/libc.so.6
#1 0x00000000004004ef in main (argc=1, argv=0x7fff3ef4d848) at crash.c:5
(gdb)
So the error I'm interested in is located on crash.c line 5.
Good luck.
Just run the app in the debugger. At one point it will die and you will have a stack trace with the information you want.