We're running squid from with gdb - that way we can automatically generate backtraces for debugging.
backtrace=`mktemp`
gdb -q -x /etc/service/squid3/gdbcommands /usr/sbin/squid 2>&1 >$backtrace
/usr/bin/mail -s "`hostname`: Squid was restarted (backtrace)" someaddress#charite.de < backtracetrace
rm $backtrace
/etc/service/squid3/gdbcommands contains:
set args -NsYC
handle SIGPIPE pass nostop noprint
handle SIGTERM pass nostop noprint
handle SIGUSR1 pass nostop noprint
handle SIGHUP pass nostop noprint
handle SIGSEGV stop
handle SIGABRT stop
run
set print pretty
backtrace full
generate-core-file
quit
But, every now and then, squid is "just" being stopped & restarted, with no crash being involved at all. In that case I'm still getting an email containing:
Reading symbols from /usr/sbin/squid...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Inferior 1 (process 57867) exited normally]
/etc/service/squid3/gdbcommands:10: Error in sourced command file:
No stack.
(gdb) quit
And of course there's no stack, since the program exited ok.
How can I change my gdbcommands file to avoid this?
This can be done using either Python or the gdb CLI. Since the CLI is a bit simpler, when possible, I'll sketch that approach.
First, you might as well only create a core file on a bad exit. And, we'll use the gdb exit code later, so let's arrange for that to tell the calling script what happened.
Where your current script says:
backtrace full
generate-core-file
quit
... instead use:
if !$_isvoid($_exitsignal) || (!$_isvoid($_exitcode) && $_exitcode != 0))
backtrace full
generate-core-file
quit 0
end
quit 1
Then your calling script can check the exit code of gdb:
if gdb your args here; then
mail results
fi
Related
gdb -quiet -iex 'set pagination off' -ex run -ex 'thread apply all bt' --batch --args <your prog>
The above is my default way or running my programs in CI. It is very convenient to have a stacktrace printed if the binary crashes, without having to hunt for coredump files.
edit: my default way of running CI has changed, because I also need to propagate the return code from the tested program in some circumstances
gdb -quiet -iex 'set pagination off' -iex 'set debuginfod enabled on' -ex run -ex 'thread apply all bt' -ex 'quit $_exitcode' --batch --args <your prog>
The problem is that I cannot do this when I compile with -fsanitize=address,leak,undefined. I get an error message when the program runs to the end and lsan is triggered (in an atexit handler, according to its docs).
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
==2861213==LeakSanitizer has encountered a fatal error.
==2861213==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==2861213==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
[Inferior 1 (process 2861213) exited with code 01]
Q1: Can I maintain the convenience of my gdb -quiet setup, while still getting lsan leak reports when my program leaks?
Currently the solution I am implementing is to hunt for the coredump files and execute gdb on them in a subsequent CI step (that runs upon test failure). For this, I had to configure sanitizers to permit coredump generation with disable_coredump=0, as described in How to generate core dump when use Address Sanitizer and gcc7.1.0.
Q2: Is it possible to use AddressSanitizer to do the job gdb used to do for me? That is, to run the equivalent of bt or thread apply all bt when my program crashes?
export DEBUGINFOD_URLS=https://debuginfod.elfutils.org
gdb -quiet -iex 'set pagination off' -iex 'set debuginfod enabled on' -iex 'set detach-on-fork off' -iex 'set breakpoint pending on' -x gha_gdb_commandfile.txt --args <your prog> <args>
The gdb_commandfile.txt being
break __lsan::CheckForLeaks
commands
detach
end
run
thread apply all bt
thread apply all py-bt
The gdb command file breaks when leak sanitizer is starting and detaches the debugger, so that leak sanitizer is free to do its own attach.
I tried this to debug this main.c
#include <assert.h>
#include <stdlib.h>
int main(int argc, char** argv) {
void* x = malloc(42);
if (argc > 1) assert(1 == 2);
return 0;
}
compiled as
gcc -g3 -fsanitize=address,leak main.c
and it gives both leak report and backtrace, depending on whether I run with argument or not.
The command works even if binary is compiled without leak sanitizer. Then gdb simply never sets the breakpoint.
I wanted to have gdb watch both parent and child if my process starts a subprocess. I tried to use 'set detach-on-fork off' for it. I need it because there might be subprocesses that should be leak-checked (and stacktrace should be dumped if they crash). But what I tried is not the way. The parent process should instead start the child within new instance of GDB. By default, GDB continues attached to parent, ignoring the child.
The rr debugger is able to record entire process subtree. Maybe there can be a way. But that is for another question.
Ubuntu 16.04.4 LTS, GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
I am trying to call a function in a compiled C program and get the following:
"(gdb) call getVarName(someParam)
You can't do that without a process to debug."
There are no other codes or messages.
I can run the program from the shell prompt
jef#ubuntu$ ./program.
I can run the program within gdb after designating the file. Permissions are 777 (just to cover all bases).
Based on research, I set the SHELL with "export SHELL=/bin/bash"
and
set kernal.yama.ptrace_scope = 0 in /etc/sysctl.d/10-ptrace.conf
I still get the same behavior.
I still get the same behavior.
Naturally.
The error you are getting means: you can't do this, unless you are debugging a live process.
This will work:
(gdb) break main
(gdb) run
... GDB is now stopped, *and* you have a live process.
... you *can* call getVarName(...) now
(gdb) call getVarName(...)
(gdb) continue # causes the process to run to end and exit
[Inferior 1 (process 195969) exited normally]
(gdb) # Now you no longer have a live process, so you *again* can't
# call functions in it.
In Linux, GDB doesn't allow set new-console on and in stead uses something called tty. With
set inferior-tty /dev/pts/[number of an active console],
in a .gdbinit file (requires editing the number every time) it redirects std::cout, but std::cin isn't working properly. It just interprets my input as if I'm sending a bash command and reports an error, and my program continues to wait for input. I can no longer type in the console after that, so I assume std::cin is being redirected, but doesn't work properly.
I tried looking up how to launch a terminal from the application itself. I could only find this answer, which also mentions a bug that it doesn't redirect input.
Is there any way to fix this issue and redirect std::cin (and std::cout) to a Linux terminal properly when debugging?
Background info: What I'm trying to do should be simple. Print a > in front of user input before using std::cin. I have simple code in place that prints the >, flushes cout and then calls getline(). It works when just running the program normally. But sadly, GDB refuses to flush the stream when there isn't a newline, so it doesn't print the >, ignores the first character of the user input and then does prints the >, immediately followed by the error message that my program sends because of the mutilated input string.
In Windows, I've solved it by making a .gdbinit file with set new-console on. This causes GDB to use a Windows console in stead of its own and that works as intended.
If you need to start the debugging after the program has a chance to pause, just run the program in a terminal, and then gdb - your-program-pid in another terminal. Otherwise there's a sequence of steps which is faster to do than to describe.
Start two terminal windows.
In one terminal, figure out the PID of the shell. ps should tell you.
In the other terminal, use these commands
$ gdb - pid-of-the-other-shell
(gdb) br main
(gdb) c
Now you have gdb attached to your shell in another terminal. Not very useful.
In the first terminal, type at the shell prompt
$ exec your-program
Now you have one terminal running gdb and another terminal running your program under gdb, stopped at main. Bear in mind that when the program exits, its terminal window will close. If you don't want this, start a second level shell in the first terminal and attach gdb to it.
You can also use set inferior-tty command, but you must make sure the other tty exists and no other program attempts to read from it. The easiest way is to run a shell in another terminal and give it a while true; do sleep 1000; done command. Note that you may get warning: GDB: Failed to set controlling terminal: Operation not permitted messages. They are harmless.
You can use gdb's multiprocess debugging feature and a terminal emulator like xterm to do this. (gnome-terminal won't work so well as explained later)
prompt.c
#include <stdio.h>
int main()
{
enum {
N = 100,
};
char name[N];
printf("> ");
scanf("%s", name);
printf("Hi, %s\n", name);
return 0;
}
xterm.gdb
set detach-on-fork off
set target-async on
set pagination off
set non-stop on
file /usr/bin/xterm
# -ut tells xterm not to write a utmp log record
# which involves root privileges
set args -ut -e ./prompt
catch exec
run
Sample Session
$ gdb -q -x xterm.gdb
Catchpoint 1 (exec)
<...>
[New process 7073]
<...>
Thread 0x7ffff7fb2780 (LWP 7073) is executing new program: /home/scottt/Dropbox/stackoverflow/gdb-stdin-new-terminal/prompt
Reading symbols from /home/scottt/Dropbox/stackoverflow/gdb-stdin-new-terminal/prompt...done.
Catchpoint 1 (exec'd /home/scottt/Dropbox/stackoverflow/gdb-stdin-new-terminal/prompt), 0x0000003d832011f0 in _start () from /lib64/ld-linux-x86-64.so.2
(gdb) inferior 2
[Switching to inferior 2 [process 7073] (/home/scottt/Dropbox/stackoverflow/gdb-stdin-new-terminal/prompt)]
[Switching to thread 2 (Thread 0x7ffff7fb2780 (LWP 7073))]
#0 0x0000003d832011f0 in _start () from /lib64/ld-linux-x86-64.so.2
(gdb) break prompt.c:11
Breakpoint 2 at 0x4004b4: file prompt.c, line 11.
(gdb) continue &
Continuing.
(gdb)
Breakpoint 2, main () at prompt.c:11
11 printf("> ");
next
12 scanf("%s", name);
(gdb) next
13 printf("Hi, %s\n", name);
A new xterm window will appear after executing gdb -q -x xterm.gdb and you can interact with prompt inside it without interfering with GDB.
Related GDB commands:
inferior 2 switches the process GDB's debugging much like the thread command switches threads. (GDB: Debugging Multiple Inferiors and Programs)
continue & means to continue executing in the background. (GDB: Background Execution)
I used xterm because gnome-terminal uses dbus-launch to ask another process to launch the new terminal outside of the parent-child process relationship that GDB relies on.
Is it possible to run a process with gdb, modify some memory and then detach from the process afterwards?
I can't start the process from outside of gdb as I need to modify the memory, before the first instruction is executed.
When you detach from a process started with gdb, gdb will hang, but killing gdb from another process makes the debugged process still running.
I currently use the following script to launch the process:
echo '# custom gdb function that finds the entry_point an assigns it to $entry_point_address
entry_point
b *$entry_point_address
run
set *((char *)0x100004147) = 0xEB
set *((char *)0x100004148) = 0xE2
detach # gdb hangs here
quit # quit never gets executed
' | gdb -quiet "$file"
This happens in both of my gdb versions:
GNU gdb 6.3.50-20050815 (Apple version gdb-1824)
GNU gdb 6.3.50-20050815 (Apple version gdb-1822 + reverse.put.as patches v0.4)
I'm pretty sure that you can't detach from an inferior processes that was started directly under gdb, however, something like the following might work for you, this is based on a recent gdb, I don't know how much of this will work on version 6.3.
Create a small shell script, like this:
#! /bin/sh
echo $$
sleep 10
exec /path/to/your/program arg1 arg2 arg3
Now start this up, spot the pid from echo $$, and attach to the shell script like this gdb -p PID. Once attached you can:
(gdb) set follow-fork-mode child
(gdb) catch exec
(gdb) continue
Continuing.
[New process NEW-PID]
process NEW-PID is executing new program: /path/to/your/program
[Switching to process NEW-PID]
Catchpoint 1 (exec'd /path/to/your/program), 0x00007f40d8e9fc80 in _start ()
(gdb)
You can now modify the child process as required. Once you're finished just do:
(gdb) detach
And /path/to/your/program should resume (or start in this case) running.
I seem to have some kind of multithreading bug in my code that makes it crash once every 30 runs of its test suite. The test suite is non-interactive. I want to run my test suite in gdb, and have gdb exit normally if the program exits normally, or break (and show a debugging prompt) if it crashes. This way I can let the test suite run repeatedly, go grab a cup of coffee, come back, and be presented with a nice debugging prompt. How can I do this with gdb?
This is a little hacky but you could do:
gdb -ex='set confirm on' -ex=run -ex=quit --args ./a.out
If a.out terminates normally, it will just drop you out of GDB. But if you crash, the program will still be active, so GDB will typically prompt if you really want to quit with an active inferior:
Program received signal SIGABRT, Aborted.
0x00007ffff72dad05 in raise (sig=...) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
in ../nptl/sysdeps/unix/sysv/linux/raise.c
A debugging session is active.
Inferior 1 [process 15126] will be killed.
Quit anyway? (y or n)
Like I said, not pretty, but it works, as long as you haven't toggled off the prompt to quit with an active process. There is probably a way to use gdb's quit command too: it takes a numeric argument which is the exit code for the debugging session. So maybe you can use --eval-command="quit stuff", where stuff is some GDB expression that reflects whether the inferior is running or not.
This program can be used to test it out:
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
int main() {
if (time(NULL) % 2) {
raise(SIGINT);
}
puts("no crash");
return EXIT_SUCCESS;
}
You can also trigger a backtrace when the program crashes and let gdb exit with the return code of the child process:
gdb -return-child-result -ex run -ex "thread apply all bt" -ex "quit" --args myProgram -myProgramArg
The easiest way is to use the Python API offered by gdb:
def exit_handler(event):
gdb.execute("quit")
gdb.events.exited.connect(exit_handler)
You can even do it with one line:
(gdb) python gdb.events.exited.connect(lambda x : gdb.execute("quit"))
You can also examine the return code to ensure it's the "normal" code you expected with event.exit_code.
You can use it in conjuction with --eval-command or --command as mentioned by #acm to register the event handler from the command line, or with a .gdbinit file.
Create a file named .gdbinit and it will be used when gdb is launched.
run
quit
Run with no options:
gdb --args prog arg1...
You are telling gdb to run and quit, but it should stop processing the file if an error occurs.
Make it dump core when it crashes. If you're on linux, read the man core man page and also the ulimit builtin if you're running bash.
This way when it crashes you'll find a nice corefile that you can feed to gdb:
$ ulimit -c unlimited
$ ... run program ..., gopher coffee (or reddit ;)
$ gdb progname corefile
If you put the following lines in your ~/.gdbinit file, gdb will exit when your program exits with a status code of 0.
python
def exit_handler ( event ):
if event .exit_code == 0:
gdb .execute ( "quit" )
gdb .events .exited .connect ( exit_handler )
end
The above is a refinement of Kevin's answer.
Are you not getting a core file when it crashes? Start gdb like this 'gdb -c core' and do a stack traceback.
More likely you will want to be using Valgrind.