I'm wondering if it's possible to launch an application via GDB, on a SegFault write the backtrace to a file (to look at later), and then exit GDB all without any user input.
I'm running an application from a shell script in an infinite loop (so if it crashes it reloads) on OS boot from a non-interactive session. The application is crashing in a non-reproducible way so I need a backtrace from the crash to debug the issue. Ideally, I'd just modify the shell script to include the GDB debugging + backtracing functionality and preserve the automatic restarting of the application following a crash.
Is this possible to do?
Thanks to Aditya Kumar; acceptable solution:
gdb -batch -ex "run" -ex "bt" ${my_program} 2>&1 | grep -v ^"No stack."$
If the program needs arguments:
gdb -batch -ex "run" -ex "bt" --args ${my_program} param1 param2 \
param3 ... 2>&1 | grep -v ^"No stack."$
gdb --batch -q <debuged_executable> <core_file> -ex bt
This works with gdb 7.6:
My test program that causes a core dump if it is given a command line parameter:
int a(int argc)
{
if (argc > 1) {
int *p = 0;
*p = *p +1;
return *p;
}
else {
return 0;
}
}
int b(int argc)
{
return a(argc);
}
int main(int argc, char *argv[])
{
int res = b(argc);
return res;
}
My python script my_check.py:
def my_signal_handler (event):
if (isinstance(event, gdb.SignalEvent)):
log_file_name = "a.out.crash." + str(gdb.selected_inferior().pid) + ".log"
gdb.execute("set logging file " + log_file_name )
gdb.execute("set logging on")
gdb.execute("set logging redirect on")
gdb.execute("thread apply all bt")
gdb.execute("q")
gdb.events.stop.connect(my_signal_handler)
gdb.execute("set confirm off")
gdb.execute("set pagination off")
gdb.execute("r")
gdb.execute("q")
So, first I run a.out and there is no crash. No log files are created:
gdb -q -x my_check.py --args ./a.out >/dev/null
Next I run a.out and give it one parameter:
>gdb -q -x my_check.py --args ./a.out 1 >/dev/null
And this is a crash report:
>cat a.out.crash.13554.log
Thread 1 (process 13554):
#0 0x0000000000400555 in a (argc=2) at main.cpp:5
#1 0x000000000040058a in b (argc=2) at main.cpp:15
#2 0x00000000004005a3 in main (argc=2, argv=0x7fffffffe198) at main.cpp:20
Alternatively to just storing the backtrace, you could put ulimit -c unlimited in front of your infinite loop in your shell script. The result will be that every time your program segfaults, it will write a core dump into a file which on my system is just called core but on other systems might include the process id. If the program segfaults (you see this from its exit status being equal to 139) then just move the core file to a safe location using a unique name (for example using timestamps). With these core files and gdb you can then do even more than just look at the backtrace. Thus I guess using them might even be more useful to you.
Related
gdb -quiet -iex 'set pagination off' -ex run -ex 'thread apply all bt' --batch --args <your prog>
The above is my default way or running my programs in CI. It is very convenient to have a stacktrace printed if the binary crashes, without having to hunt for coredump files.
edit: my default way of running CI has changed, because I also need to propagate the return code from the tested program in some circumstances
gdb -quiet -iex 'set pagination off' -iex 'set debuginfod enabled on' -ex run -ex 'thread apply all bt' -ex 'quit $_exitcode' --batch --args <your prog>
The problem is that I cannot do this when I compile with -fsanitize=address,leak,undefined. I get an error message when the program runs to the end and lsan is triggered (in an atexit handler, according to its docs).
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
==2861213==LeakSanitizer has encountered a fatal error.
==2861213==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==2861213==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
[Inferior 1 (process 2861213) exited with code 01]
Q1: Can I maintain the convenience of my gdb -quiet setup, while still getting lsan leak reports when my program leaks?
Currently the solution I am implementing is to hunt for the coredump files and execute gdb on them in a subsequent CI step (that runs upon test failure). For this, I had to configure sanitizers to permit coredump generation with disable_coredump=0, as described in How to generate core dump when use Address Sanitizer and gcc7.1.0.
Q2: Is it possible to use AddressSanitizer to do the job gdb used to do for me? That is, to run the equivalent of bt or thread apply all bt when my program crashes?
export DEBUGINFOD_URLS=https://debuginfod.elfutils.org
gdb -quiet -iex 'set pagination off' -iex 'set debuginfod enabled on' -iex 'set detach-on-fork off' -iex 'set breakpoint pending on' -x gha_gdb_commandfile.txt --args <your prog> <args>
The gdb_commandfile.txt being
break __lsan::CheckForLeaks
commands
detach
end
run
thread apply all bt
thread apply all py-bt
The gdb command file breaks when leak sanitizer is starting and detaches the debugger, so that leak sanitizer is free to do its own attach.
I tried this to debug this main.c
#include <assert.h>
#include <stdlib.h>
int main(int argc, char** argv) {
void* x = malloc(42);
if (argc > 1) assert(1 == 2);
return 0;
}
compiled as
gcc -g3 -fsanitize=address,leak main.c
and it gives both leak report and backtrace, depending on whether I run with argument or not.
The command works even if binary is compiled without leak sanitizer. Then gdb simply never sets the breakpoint.
I wanted to have gdb watch both parent and child if my process starts a subprocess. I tried to use 'set detach-on-fork off' for it. I need it because there might be subprocesses that should be leak-checked (and stacktrace should be dumped if they crash). But what I tried is not the way. The parent process should instead start the child within new instance of GDB. By default, GDB continues attached to parent, ignoring the child.
The rr debugger is able to record entire process subtree. Maybe there can be a way. But that is for another question.
I seem to have some kind of multithreading bug in my code that makes it crash once every 30 runs of its test suite. The test suite is non-interactive. I want to run my test suite in gdb, and have gdb exit normally if the program exits normally, or break (and show a debugging prompt) if it crashes. This way I can let the test suite run repeatedly, go grab a cup of coffee, come back, and be presented with a nice debugging prompt. How can I do this with gdb?
This is a little hacky but you could do:
gdb -ex='set confirm on' -ex=run -ex=quit --args ./a.out
If a.out terminates normally, it will just drop you out of GDB. But if you crash, the program will still be active, so GDB will typically prompt if you really want to quit with an active inferior:
Program received signal SIGABRT, Aborted.
0x00007ffff72dad05 in raise (sig=...) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
in ../nptl/sysdeps/unix/sysv/linux/raise.c
A debugging session is active.
Inferior 1 [process 15126] will be killed.
Quit anyway? (y or n)
Like I said, not pretty, but it works, as long as you haven't toggled off the prompt to quit with an active process. There is probably a way to use gdb's quit command too: it takes a numeric argument which is the exit code for the debugging session. So maybe you can use --eval-command="quit stuff", where stuff is some GDB expression that reflects whether the inferior is running or not.
This program can be used to test it out:
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
int main() {
if (time(NULL) % 2) {
raise(SIGINT);
}
puts("no crash");
return EXIT_SUCCESS;
}
You can also trigger a backtrace when the program crashes and let gdb exit with the return code of the child process:
gdb -return-child-result -ex run -ex "thread apply all bt" -ex "quit" --args myProgram -myProgramArg
The easiest way is to use the Python API offered by gdb:
def exit_handler(event):
gdb.execute("quit")
gdb.events.exited.connect(exit_handler)
You can even do it with one line:
(gdb) python gdb.events.exited.connect(lambda x : gdb.execute("quit"))
You can also examine the return code to ensure it's the "normal" code you expected with event.exit_code.
You can use it in conjuction with --eval-command or --command as mentioned by #acm to register the event handler from the command line, or with a .gdbinit file.
Create a file named .gdbinit and it will be used when gdb is launched.
run
quit
Run with no options:
gdb --args prog arg1...
You are telling gdb to run and quit, but it should stop processing the file if an error occurs.
Make it dump core when it crashes. If you're on linux, read the man core man page and also the ulimit builtin if you're running bash.
This way when it crashes you'll find a nice corefile that you can feed to gdb:
$ ulimit -c unlimited
$ ... run program ..., gopher coffee (or reddit ;)
$ gdb progname corefile
If you put the following lines in your ~/.gdbinit file, gdb will exit when your program exits with a status code of 0.
python
def exit_handler ( event ):
if event .exit_code == 0:
gdb .execute ( "quit" )
gdb .events .exited .connect ( exit_handler )
end
The above is a refinement of Kevin's answer.
Are you not getting a core file when it crashes? Start gdb like this 'gdb -c core' and do a stack traceback.
More likely you will want to be using Valgrind.
I usually run a program as :
./a.out arg1 arg2 <file
I would like to debug it using gdb.
I am aware of the set args functionality, but that only works from the gdb prompt.
You can do this:
gdb --args path/to/executable -every -arg you can=think < of
The magic bit being --args.
Just type run in the gdb command console to start debugging.
Pass the arguments to the run command from within gdb.
$ gdb ./a.out
(gdb) r < t
Starting program: /dir/a.out < t
If you want to have bare run command in gdb to execute your program with redirections and arguments, you can use set args:
% gdb ./a.out
(gdb) set args arg1 arg2 <file
(gdb) run
I was unable to achieve the same behaviour with --args parameter, gdb fiercely escapes the redirections, i.e.
% gdb --args echo 1 2 "<file"
(gdb) show args
Argument list to give program being debugged when it is started is "1 2 \<file".
(gdb) run
...
1 2 <file
...
This one actually redirects the input of gdb itself, not what we really want here
% gdb --args echo 1 2 <file
zsh: no such file or directory: file
Start GDB on your project.
Go to project directory, where you've already compiled the project executable. Issue the command gdb and the name of the executable as below:
gdb projectExecutablename
This starts up gdb, prints the following:
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
.................................................
Type "apropos word" to search for commands related to "word"...
Reading symbols from projectExecutablename...done.
(gdb)
Before you start your program running, you want to set up your breakpoints. The break command allows you to do so. To set a breakpoint at the beginning of the function named main:
(gdb) b main
Once you've have the (gdb) prompt, the run command starts the executable running. If the program you are debugging requires any command-line arguments, you specify them to the run command. If you wanted to run my program on the "xfiles" file (which is in a folder "mulder" in the project directory), you'd do the following:
(gdb) r mulder/xfiles
Hope this helps.
Disclaimer: This solution is not mine, it is adapted from https://web.stanford.edu/class/cs107/guide_gdb.html
This short guide to gdb was, most probably, developed at Stanford University.
Wouldn't it be nice to just type debug in front of any command to be able to debug it with gdb on shell level?
Below it this function. It even works with following:
"$program" "$#" < <(in) 1> >(out) 2> >(two) 3> >(three)
This is a call where you cannot control anything, everything is variable, can contain spaces, linefeeds and shell metacharacters. In this example, in, out, two, and three are arbitrary other commands which consume or produce data which must not be harmed.
Following bash function invokes gdb nearly cleanly in such an environment [Gist]:
debug()
{
1000<&0 1001>&1 1002>&2 \
0</dev/tty 1>/dev/tty 2>&0 \
/usr/bin/gdb -q -nx -nw \
-ex 'set exec-wrapper /bin/bash -c "exec 0<&1000 1>&1001 2>&1002 \"\$#\"" exec' \
-ex r \
--args "$#";
}
Example on how to apply this: Just type debug in front:
Before:
p=($'\n' $'I\'am\'evil' " yay ")
"b u g" "${p[#]}" < <(in) 1> >(out) 2> >(two) 3> >(three)
After:
p=($'\n' $'I\'am\'evil' " yay ")
debug "b u g" "${p[#]}" < <(in) 1> >(out) 2> >(two) 3> >(three)
That's it. Now it's an absolute no-brainer to debug with gdb. Except for a few details or more:
gdb does not quit automatically and hence keeps the IO redirection open until you exit gdb. But I call this a feature.
You cannot easily pass argv0 to the program like with exec -a arg0 command args. Following should do this trick: After exec-wrapper change "exec to "exec -a \"\${DEBUG_ARG0:-\$1}\".
There are FDs above 1000 open, which are normally closed. If this is a problem, change 0<&1000 1>&1001 2>&1002 to read 0<&1000 1>&1001 2>&1002 1000<&- 1001>&- 1002>&-
You cannot run two debuggers in parallel. There also might be issues, if some other command consumes /dev/tty (or STDIN). To fix that, replace /dev/tty with "${DEBUGTTY:-/dev/tty}". In some other TTY type tty; sleep inf and then use the printed TTY (i. E. /dev/pts/60) for debugging, as in DEBUGTTY=/dev/pts/60 debug command arg... That's the Power of Shell, get used to it!
Function explained:
1000<&0 1001>&1 1002>&2 moves away the first 3 FDs
This assumes, that FDs 1000, 1001 and 1002 are free
0</dev/tty 1>/dev/tty 2>&0 restores the first 3 FDs to point to your current TTY. So you can control gdb.
/usr/bin/gdb -q -nx -nw runs gdb invokes gdb on shell
-ex 'set exec-wrapper /bin/bash -c "exec 0<&1000 1>&1001 2>&1002 \"\$#\"" creates a startup wrapper, which restores the first 3 FDs which were saved to 1000 and above
-ex r starts the program using the exec-wrapper
--args "$#" passes the arguments as given
Wasn't that easy?
I'm trying to create a little unit test with gdb,
for a embedded mcu that is controlled by OpenOCD (that gives me control over my target via a gdb server).
So I would like to automate this with some scripting of gdb.
I would like to write some kind of script for gdb that more or less does this:
Add a couple of breakpoints
Start the program
When we stop, where did it stop (get the frame info)
Quit.
Any ideas?
A example on how to do this in python gdb scripting would be nice.
Thanks
Johan
Note:
Let's say that we have this basic structure,
that more or less goes into test_failed() or test_success()
depending on what the function start_test() returns.
void test_failed() {
while(1);
}
void test_success() {
while(1);
}
int main(void) {
int status = start_test();
if( status > 0 ) {
test_failed();
}
test_success();
while(1);
}
To do this manually in gdb is very strait forward,
(gdb) break test_success
Breakpoint 1 at 0x20: file src/main.c, line 9.
(gdb) break test_failed
Breakpoint 2 at 0x18: file src/main.c, line 5.
(gdb) cont
Continuing.
Breakpoint 1, test_success () at src/main.c:9
9 while(1);
(gdb) frame
#0 test_success () at src/main.c:9
9 while(1);
(gdb)
So the next step I tried was to add those gdb commands into a gdb startup script that more or less just looked like this.
break test_success
break test_failed
target remote localhost:3333
cont
frame
and start it with
arm-none-eabi-gdb --batch --command=commands.gdb main.elf
And this kind of works, but it is not very nice.
How do I do this with the "new and cool" python scripts,
that gdb seem to support.
FYI recent gdb versions are scriptable in Python. You can call python code from the gdb command line. This opens a whole new world, check the relevant documentation. From the command line run:
dnf/yum/apt-get install gdb-doc
info gdb extending python
If you do not like the text-based info browser, here is one (among many?) alternative, graphical browser:
yelp 'info:gdb' # , go to "Extending"
Here is a sample gdb-python script. It attaches gdb to the first "your_program" found running.
#!/usr/bin/python
import subprocess
import string
def backquotes(cmdwords):
output = subprocess.Popen(cmdwords, stdout=subprocess.PIPE).communicate()[0]
return output.strip()
pid = backquotes(['pgrep', 'your_program'])
gdb.execute("attach " + str(pid))
A reduced example that I'm currently using:
class DebugPrintingBreakpoint(gdb.Breakpoint):
debugging_IDs = frozenset({37, 153, 420})
def stop(self):
top = gdb.newest_frame()
someVector = top.read_var('aVectorVar')
# Access the begin() & end() pointer of std::vector in GNU Standard C++ lib
first = someVector['_M_impl']['_M_start']
last = someVector['_M_impl']['_M_finish']
values = []
while first != last:
values.append(int(first.dereference()['intID']))
first = first + 1
if not set(values) & debugging_IDs:
return False # skip: none of the items we're looking for can be found by ID in the vector on the stack
print("Found other accompanying IDs: {}".format(values))
return True # drop to gdb's prompt
# Ensure shared libraries are loaded already
gdb.execute("start")
# Set our breakpoint, which happens to reside in some shared lib, hence the "start" previously
DebugPrintingBreakpoint("source.cpp:42")
gdb.execute("continue")
You can execute this script from gdb's prompt like this:
(gdb) source script.py
Or from the command-line:
$ gdb --command script.py ./executable.elf
See the complete GDB Python API docs for further info.
OK, I found the answer while asking the question... and it and it was a really simple thing.
You should not use both the "--command" and the "--eval" at the same time if you expect them to be executed in a specific order!
A more predicable way is to put everything in the commands.gdb file and ignore --eval.
So it becomes something like this:
arm-none-eabi-gdb --batch --command=commands.gdb main.elf
Where commands.gdb looks like this:
break test_success
break test_failed
target remote localhost:3333
cont
frame
But it would probably be so much nicer to do this with something like python instead.
Just wanted to note something that I find confusing whenever I come back to this topic (Note, I'm currently on Ubuntu 14.04, GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1):
First, there are references about it being "possible to invoke gdb as an interpreter":
9. Scripting gdb — The Cliffs of Inanity (http://tromey.com/blog/?p=548)
http://blog.scottt.tw/2012/01/exploring-gdb-python-api-with-ipython_31.html
... meaning, one would write a script text file with the shebang line #!/usr/bin/gbd -P or #!/usr/bin/gbd --python, then write Python code in it, then make it executable chmod +x pygdbscript, then run ./pygdbscript; ... but as in this post:
GDB - Missing "--python" Parameter? / Newbie Corner / Arch Linux Forums
..., I get gdb: unrecognized option '--python' if I try anything like that. Apparently this option is/was a feature in some "archer" branch of gdb?!
So, in order to run a Python script in gdb, there are actually two ways:
Name your script file with extension .py; say test.py here:
def Something():
print("hello from python")
Something()
gdb.execute("quit");
Note, in this case, you just write plain Python code; and you do not need to import gdb in order to access the gdb object. This you can run with either of:
gdb -x test.py
gdb -x=test.py
gdb --command test.py
gdb --command=test.py
gdb -command test.py
gdb -command=test.py
... which seem to be equivalent, as the result for any of these is the same printout, before gdb is instructed to exit by the script:
$ gdb -x=test.py
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
...
For help, type "help".
Type "apropos word" to search for commands related to "word".
hello from python
NOTE that for this case, also names like test.gdb.py will be interpreted as pure Python scripts, since then end in .py.
Name your script anything else - as long as it does not end with .py extension; say test.pygdb here:
python
def Something():
print("hello from python")
Something()
gdb.execute("quit");
end
In this case, gdb interprets the script as being a gdb script, i.e. with gdb commands - and that means, that whatever Python code you may want to write in here, must be wrapped in "python" as a starting line and "end" at end of the Python code. Again, it would be called with any of these equivalent calls:
gdb -x test.pygdb
gdb -x=test.pygdb
gdb --command test.pygdb
gdb --command=test.pygdb
gdb -command test.pygdb
gdb -command=test.pygdb
... and then the output is the same as in the previous case (since it is the same Python script running):
$ gdb -x test.pygdb
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
...
hello from python
And in response to OP: if the C code in OP is in /tmp/myprog.c - with an added int start_test() { return rand() % 50; } on top, otherwise it won't compile - , and is compiled with with gcc -g myprog.c -o myprog.exe into /tmp/myprog.exe; then you can use a myprog.gdb.py script like this:
# need to specify the executable file which we debug (in this case, not from command line)
# here `gdb` command `file` is used - it does not have a Python equivalent (https://sourceware.org/gdb/onlinedocs/gdb/Objfiles-In-Python.html#index-Objfile_002eframe_005ffilters)
# so we must use gdb.execute:
myexefile="/tmp/myprog.exe"
print("""
### myprog.gdb.py is running: """ + myexefile + """ - and adding breakpoints:
""")
gdb.execute("file " + myexefile)
gdb.execute("set pagination off")
ax = gdb.Breakpoint("test_success")
bx = gdb.Breakpoint("test_failed")
gdb.execute("run")
# here the program will break, so we can do:
print("""
### myprog.gdb.py after the break - current stack frame:
""")
current_frame_at_break = gdb.selected_frame()
print(current_frame_at_break) # instead of gdb.execute("frame")
print("""
### myprog.gdb.py - backtrace:
""")
gdb.execute("backtrace 2")
print("""
### myprog.gdb.py - go to frame that called current frame:
""")
parent_frame = current_frame_at_break.older()
print(parent_frame)
status_var = parent_frame.read_var("status")
print("status_var is: ", status_var)
... then run this script with:
$ gdb -x myprog.gdb.py
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
....
For help, type "help".
Type "apropos word" to search for commands related to "word".
### myprog.gdb.py is running: /tmp/myprog.exe - and adding breakpoints:
Breakpoint 1 at 0x400565: file myprog.c, line 8.
Breakpoint 2 at 0x40055f: file myprog.c, line 4.
Breakpoint 2, test_failed () at myprog.c:4
4 while(1);
### myprog.gdb.py after the break - current stack frame:
{stack=0x7fffffffdc70,code=0x40055b,!special}
### myprog.gdb.py - backtrace:
#0 test_failed () at myprog.c:4
#1 0x000000000040058c in main () at myprog.c:15
### myprog.gdb.py - go to frame that called current frame:
{stack=0x7fffffffdc90,code=0x400567,!special}
status_var is: 33
(gdb)
Note that at the end of this script, the (gdb) interactive prompt remains, and you can use it normally here; if you don't need the interactive prompt, you can do gdb.execute("quit"); as in the above scripts to force gdb to exit instead at end of script execution.
Also, for an example of subclassing breakpoint class in gdb Python, see How to print the current line of source at breakpoint in GDB and nothing else?
I'd like to have gdb immediately run the executable, as if I'd typed "run"
(motivation: I dislike typing "run").
One way is to pipe the command to gdb like this:
$ echo run | gdb myApp
But the problem with this approach is that you lose interactivity with gdb,
eg. if a breakpoint triggers or myApp crashes, gdb quits.
This method is discussed here.
Looking at the options in --help, I don't see a way to do this, but perhaps I'm missing something.
gdb -ex run ./a.out
If you need to pass arguments to a.out:
gdb -ex run --args ./a.out arg1 arg2 ...
EDIT:
Orion says this doesn't work on Mac OSX.
The -ex flag has been available since GDB-6.4 (released in 2005), but OSX uses Apple's fork of GDB, and the latest XCode for Leopard contains GDB 6.3.50-20050815 (Apple version gdb-967), so you are out of luck.
Building current GDB-7.0.1 release is one possible solution. Just be sure to read this.
I would use a gdb-script:
gdb -x your-script
where your-script contains something like:
file a.out
b main
r
afterwards you have the normal interactive gdb prompt
EDIT:
here is an optimization for the truly lazy:
save the script as .gdbinit in the working directory.
Afterwards you simply run gdb as
gdb
... and gdb automatically loads and executes the content of .gdbinit.
(echo r ; cat) | gdb a.out
The cat allows you keep typing after gdb breaks.
start command
This command is another good option:
gdb -ex start --args ./a.out arg1 arg2
It is like run, but also sets a temporary breakpoint at main and stops there.
This temporary breakpoint is deactivated once it is hit.
starti
There is also a related starti which starts the program and stops at the very first instruction instead, see also: Stopping at the first machine code instruction in GDB
Great when you are doing some low level stuff.
gdb -x <(echo run) --args $program $args