I am trying to debug an application in gdb running on Ubuntu 18.04.
In some parts of the code, I can set breakpoints and successfully debug problems.
But in other parts, triggering a breakpoint causes the process to exit.
Is there some way I can get debut statements to appear in the gdb console?
I currently use gdb to attach to the process and then debug from that point.
I have code (std::cout) that sends statements to standard out but they are not showing up in the gdb console.
Nor should you expect them to.
When an application is started, its std::cout messages are going to file descriptor 1 (stdout). This can be the terminal window in which the app was started, or a file if the output was redirected. It could also be a pipe or /dev/null.
GDB does not "steal" that output (if it did, it would be harder to debug a program that is the source of input for another program going through a pipe).
Your first task should be to determine where the output is going. On Linux, this is usually as easy as ls -l /proc/$pid/fds/1 (replace $pid with the actual process id of the process you are debugging).
An additional complication is that the stdout can be fully buffered (if it goes into a file, pipe or socket), and may not be flushed by the time your breakpoint is hit.
P.S. In theory, you can "steal" the output from wherever it's going to your current terminal by running the following GDB commands:
(gdb) print open("/dev/tty", 2, 0) # open new fd in the inferior process
# going to current terminal.
# This will print something, e.g. 5
# Now make stdout go to that newly-opened fd
(gdb) call dup2($whatever_last_command_printed, 1)
but I wouldn't recommend this, as it can interfere with the program in unexpected ways.
Related
I'm using the command-line to do my stm32 development. CubeIDE and Atom are too heavyweight for the specs of my machine.
I compile an elf and bin file with debug support and upload the bin to the stm32. It is a simple LED blink program, and it works.
I start stlink-server, and it reports port 7184. In another terminal I type:
$ arm-none-eabi-gdb
file app.elf
target remote localhost:7184
I do not get a response for about 30 seconds, then arm-non-eabi-gdb reports:
Ignoring packet error, continuing...
warning: unrecognized item "timeout" in "qSupported" response
Ignoring packet error, continuing...
Remote replied unexpectedly to 'vMustReplyEmpty': timeout
stlink-server reports:
Error: recv returned 0. Remote side has closed gracefully. Good.
But not good!
So, what do I do? I can't seem to halt the stm32, set breakpoints, run, etc..
I'm running a mish-mash of stlink-server, arm-none-eabi-gcc, and arm-none-eabi-gdb from various sources, which might not be helping.
I'm using a Chinese ST-LINK v2, which I hear might not have all the pins wired up for debugging, and that I have to short some pins. It uploads the bin OK, though.
Update 1 OK, perhaps a little progress (??)
I start st-util, which reports:
2020-07-06T14:50:03 INFO common.c: F1xx Medium-density: 20 KiB SRAM, 64 KiB flash in at least 1 KiB pages.
2020-07-06T14:50:03 INFO gdb-server.c: Listening at *:4242...
So then in a separate console I type:
$ arm-none-eabi-gdb
(gdb) target remote localhost:4242
(gdb) file app.elf
(gdb) load app.elf
You can't do that when your target is `exec'
Oh. Also:
(gdb) r
Don't know how to run. Try "help target".
So I think I'm getting closer, It appears that I can set breakpoints. And maybe I've run the commands in the wrong order.
I think maybe I have to do:
exec app.elf
but that doesn't seem to respect the breakpoints.
Hmmm.
Update 2 The saga continues.
This seems better:
$ $arm-none-eabi-gdb
(gdb) target remote localhost:4242
(gdb) file app.elf
(gdb) b 26
continue
That seems to respect breakpoints; but the debugger reports:
Continuing.
Note: automatically using hardware breakpoints for read-only addresses.
Program received signal SIGTRAP, Trace/breakpoint trap.
0x0800000c in _reset ()
(gdb) print i
No symbol "i" in current context
Hmmm. It seems that the program is now no longer in main(), but in a signal trap, and hence i is not in current context (even though I defined it it main).
So reaching a breakpoint basically causes the machine to reset?? Which kinda defeats the point of debugging. So I think I must be doing something wrong (?) and there's a better way of doing it?
Update 3
I switched to the Arduino IDE and uploaded a sketch. Using the procedure above, I didn't get the signal trap problem. I was able to debug my program, set breakpoints, and inspect variables. Nice. The Arduino is obviously incorporating some "secret sauce" that I had not added to my own non-Arduino code.
So, it mostly works. Mostly.
Try reset mcu before load:
target remote localhost:4242
file app.elf
monitor reset halt
load app.elf
got some big real time project to deal with (multiple processes (IPCs), multi Everything in short).
My working on process is started as service on Linux. I have the root access.
Here is the problem:
I'm trying to attach to a running proc, tried starting it through/with gdb but the result is the same: it stops the executable once I "touched" it with gdb or sometimes it throws:
Program received signal SIGUSR1, User defined signal 1. [Switching to Thread 0x7f9fe869f700 (LWP 2638)]
of course from there nothing can be done.
Tried:
handle all nostop
attach to launched as service (daemon) or launched as regular proc
started from gdb
thought maybe forking/multi-threaded problem - implemented in the very beginning sleep for 10 seconds - attached to it with "continue"
Guys, all I want it is to debug, hit the breakpoints, etc.
Please help! Share ideas.
Editing actual commands:
1) gdb attach myProcId. Then after reading symbols, I hit "c" which results:
Program received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7f9fe869f700 (LWP 2638)]
0x00007f9fec09bf73 in select () from /lib64/libc.so.6
2) If I make the first line 10 seconds sleep in the code, attaching to the process, hit "c", result: it runs, shows info threads, backtrace of main, but never hits the breakpoint (for sure the code runs there - I get logs and different behaviour if I change code there), meaning the process is stuck.
3) All other combinations like gdb path/to/my/proc args list, then start. Where arg list played with different related options gdb gives us.
Maybe worth to mention: process network packets related, timers driven also.
But for me the important thing is a current snapshot on break, i don't care what will happen to the system after timers expired.
Since you mentioned that you are debugging a multiprocessing program, I think the underlying program you have is to set the breakpoint in the correct subprocess.
Try break fork and set follow-fork-mode child/parent. What you want to achieve is have gdb attached to the process that is running the code you want to debug.
Refer to this link.
Another thought is to generate a crash, since you can compile the programe. For example add a int i = *(int*)NULL and that will generate a core dump. You can then debug the core dump with gdb <program> <core dump>. You can refer to this page for how to configure core dump.
If I run a program under gdb, how do I break at an arbitrary point? Pressing Ctrl+C kills the program. Ctrl+Break does nothing.
I can't enter gdb commands because my program is itself sitting in a REPL loop so anything I enter goes to my program, not to gdb.
My program uses linenoise to implement the REPL; I assume that this is hiding Ctrl+C, etc., from gdb.
Ctrl+\ results in a 001C square blob thingy in my program, rather than SIGUSR1.
Pressing Ctrl+C kills the program.
That is not the default GDB behavior.
Did you set handle SIGINT nostop pass?
You can examine current signal disposition with:
(gdb) handle SIGINT
Signal Stop Print Pass to program Description
SIGINT Yes Yes No Interrupt
Update:
My program is using linenoise for console input. I assume that it has done something to Ctrl+C
If your program is modifying terminal settings, you are going to have a very bad time debugging it from the same terminal.
For example, suppose the program sets no echo, and then hits a breakpoint. I think you would get a (gdb) prompt, but would not see any commands you are typing into GDB.
It seems that you would be much better off debugging this program from a different terminal. Use gdb -p $PID to attach to it from "outside".
I'm having a tough time figuring this one out; I have a program, iverilog that executes a system() call to invoke another program, ivl. I want to debug the second program, ivl in gdb, but I can't get gdb to set any breakpoints in the child process when I invoke gdb with the parent process. Here's what the programs look like:
//iverilog-main.cpp (Parent process)
int main(){
//...
system("ivl arg1 arg2");
//...
return 0;
}
.
//ivl-main.cpp (child process)
int main(){
//...
//stuff I want to debug
//...
return 0;
}
.
The gdb commands I'm running are: gdb iverilog -x cmds.gdb
# cmds.gdb
set args hello.v
set follow-fork-mode child
set breakpoint pending on
break ivl-main.cpp:main
run
Unfortunately, gdb doesn't break at ivl-main.cpp:main,it just completes without ever breaking; the output I get is:
Starting program: /usr/local/bin/iverilog hello.v
[New process 18117]
process 18117 is executing new program: /bin/dash
[Inferior 2 (process 18117) exited normally]
I'm certain ivl-main.cpp:main is being called because when I run the ivl program in gdb it successfully breaks there.
My thinking is that gdb doesn't recognize ivl-main.cpp as a source file when its running gdb iverilog, and it's not setting that breakpoint when it enters the child process which does contain ivl-main.cpp as a source file. So I think if I set the breakpoint for ivl-main.cpp when gdb enters the child process, it should work. The only way I can think of doing this is to manually break at the system() call and step into the child process, then set the breakpoint. Is there a more elegant approach that would force gdb to break whenever entering a child process?
Normally GDB only debugs one process at a time- if your program forks then you will debug the parent or the child, but not both simultaneously. By default, GDB continues debugging the parent after a fork, but you can change this behavior if you so desire with the following command:
set follow-fork-mode child
Alternately, you can tell GDB to keep both the parent and the child under its control. By default GDB only follows one process, but you can tell it to follow all child processes with this command:
set detach-on-fork off
GDB refers to each debugged process as an "inferior". When debugging multiple processes you can examine and interact each process with the "inferiors" command similar to how you would use "threads" to examine or interact with multiple threads.
See more documentation here:
https://sourceware.org/gdb/onlinedocs/gdb/Forks.html
This answer provides one way to achieve what you want.
In theory, set follow-fork-mode child should work.
In practice, the iverilog is likely itself a shell script that runs (forks) multiple commands, so at every fork you will need to decide whether you want to continue debugging the parent or the child. One wrong decision and you've lost control of the process that will eventually execute your program. This very likely explains why it didn't work for you.
Without getting into to to much detail, I'm working on a program that consists of several separate processes all running on embedded QNX RTOS. They don't have a parent-child relationship, they are all spawned using spawnlp(P_NOWAIT, ...) and they all communicate with each other using the IPC mechanism provided by the OS.
When I'm debugging with GDB and I hit a breakpoint in the process I'm working in, all of my threads are paused, which is great. But is there a way to also have it pause execution of my other processes? Right now what's happening is all the other processes keep on truckin' while my process is paused and so all the IPC queues get full etc. etc.
Thanks in advance,
HF
You can associate a list of GDB commands with each breakpoint. So when you hit a breakpoint in process A, you can for example send a SIGTRAP to process B, which should drop it into the debugger:
(gdb) b main
Breakpoint 1 at 0x804834a: file testA.c, line 40.
(gdb) command
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just "end".
>shell kill -s TRAP `pidof testB`
>end
(gdb)
More info at Breakpoint Command Lists