What is the difference between gdb mi and gdb server? Can both be used for local debugging?
I have facing problem in troubleshooting the debugging with gdb server on Red Hat. The debugging launcher stops at 72 %.
For local debugging you need neither MI or gdb server. You only need gdb.
MI is used by various GUIs/IDEs to communicate with gdb. This basically changes commands so they can be better parsed by machines (hence the name Machine Interface).
Gdbserver is used to debug remote systems. You only run gdbserver on remote (target) machine (this can for example be some embedded system running linux) and then you connect to gdbserver from GDB on your workstation (serial, ethernet or similar). This allows you to debug on your workstation and only requires target to run simpler gdbserver.
Related
Remote Setup
I need to debug a complex C++ program which is installed in a docker container, controlled by Kubernetes.
The docker container also provides a gdbserver and exposes the container port 44444.
Host Setup
The gdb part to control and examine the program is set in a another docker container.
This is due to the case, that the SUSE environment is only available in this container,
not on my Ubuntu 18.04 machine in a VM Box.
Local debugging works well
Debugging the program locally in the SUSE docker container works well. The program halts
at the specified breakpoints and these breakpoints are also specified in remote debugging.
All breakpoints are solely defined in the program's basic source code files, not in any libs.
It has been verified, that the executable in the remote docker container is identical
to the one in the host container; it has been compiled with debug symbols and non-optimized code (-ggdb -O0).
Problem
Debugging the program remotely only lacks in stopping at the defined breakpoints on the host.
The program in the container is started in background. When gdbserver attaches its process_id
the program halts until 'continue' is issued within the gdb host session and forwarded to gdbserver in the remote container.
The program is deployed with basic C++ class files and shared program libraries together with shared project libraries.
It is started with parameters and exits after the job is done.
When the program is started it reads configuration files, connects to a database,
reads database entries, prepares and formats the data to XML formated entries and
writes them into an output file.
HelloWorld remote debugging test works well
To verify that the remote debugging setup and connection via gdbserver port works well,
I created a simple HelloWorld C++ program and copied this into the same remote docker container
and tested the breakpoint behaviour therein.
The remote debug test scenario is working successfully when the HelloWorld program is run in the container:
the internal container port 44444 is mapped to the same external port id 44444:
$ kubectl port-forward eric-bss-eb-tools-65c4955565-xdqtx 44444:44444
Forwarding from 127.0.0.1:44444 -> 44444
Forwarding from [::1]:44444 -> 44444
HelloWorld in the remote container is started in background and sleeps a few seconds
bash-4.4$ ./HelloWorld &
[1] 1068
gdbserver attaches to the HelloWorld process_id and waits to forward the gdb commands
bash-4.4$ ./gdbserver :44444 --attach 1068 // gdbserver uses the exposed port
Attached; pid = 1068
Listening on port 44444
gdb in the host container is started in the HelloWorld source code folder with TUI mode
$ gdb -tui HelloWorld
reading symbols from HelloWorld...done.
(gdb) b 13
Breakpoint 1 at 0x400b2d: file HelloWorld.cpp, line 13.
(gdb) b 15
Breakpoint 2 at 0x400b37: file HelloWorld.cpp, line 15.
gdb connects to the gdbserver via localhost and (external) port id 44444
(gdb) target remote :44444
(gdb) c
Continuing.
the remote HelloWorld stops at breakpoint 2; variables can be examined;
further gdb commands like 'next' and 'step' can be issued; everything works smart
Target program remote debugging doesn't stop at breakpoints
When the target C++ program in the container is debugged with the same scenario it does not stop at the defined breakpoints:
the workflow is identical to the HelloWorld test scenario with the exception,
that the breakpoints are defined after gdb has made the connection to gdbserver
(target remote :44444).
This has been done as per the advice in the 2nd comment from this answer: (Remote gdb debugging does not stop at breakpoints).
Nevertheless, the breakpoints are still ignored even when they are defined after
the connection to the remote target has been established.
the program in the remote docker container is halted by the gdbserver and continues
its execution when gdb issues the 'continue' command, but does not stop at any of the breakpoints.
I tried several hints as per other analogous problem descriptions, but breakpoints are still ignored.
E.g. using hardware break points as been advised in the answer of the same request here:
(Remote gdb debugging does not stop at breakpoints)
Running the remote docker container with securityContext: privileged=true is forbidden in my environment, hence this could not be tested. See proposal here:
(gdb does not hit any breakpoints when I run it from inside Docker container)
What am I missing to get remote debugging in a docker container halted at defined breakpoints?
Due to a security enhancement in Ubuntu (versions >= 10.10), users are not allowed to ptrace processes that are not a descendant of the debugger.
By default, process A cannot trace a running process B unless B is a direct child of A
(or A runs as root).
Direct debugging is still always allowed, e.g. gdb EXE and strace EXE.
The restriction can be loosen by changing the value of /proc/sys/kernel/yama/ptrace_scope from 1 (=default) to 0 (=tracing allowed for all processes). The security setting can be changed with:
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
HelloWorld remote debugging test works well
How did it happen, that remote debugging in the HelloWorld container worked well?
The HelloWorld container was created with USER userName in the Dockerfile
which is the same user name as been logged in to Ubuntu.
The Dockerfile for to deploy the development container (with the C++ program to be debugged) defines both a different user name and group name than being used in my Ubuntu login.
All credits for the description of ptrace scope belong to the following post,
see 2nd answer by Eliah Kagan - thank you for the thorough explanation! - here:
https://askubuntu.com/questions/143561/why-wont-strace-gdb-attach-to-a-process-even-though-im-root
Target program remote debugging doesn't stop at breakpoints
A guess: target program fork()s and executes most of the code in a child process (and your gdbserver attaches the parent).
To verify this, insert some printf("%s:%d: pid=%d\n", __FILE__, __LINE__, getpid()); calls into strategic places in the target program. If my guess is correct, you should see that the pid changes between main() and connect_to_database().
I've been setting up remote debugging for an embedded Linux target. I've tested that I have the correct combination of gdbserver running on the target and gdb client running on my workstation. I can start the server and on the workstation side run:
(gdb) target remote 10.28.22.226:2345
and I can list source and step just fine.
In Qt Creator I have configured a device. In that window I specify the GDB server executable: gdbserver (see pic).
I also configure the cross debugger in the 'Build & Run' Debuggers tab as shown below:
In this case, I have manually started the gdbserver with my executable on the target with port 2345. To start debugging with Qt Creator, I choose the Debug->Start Debugging->Attach to running Debug server pulldown menu. It gives me the prompt shown below, where I enter the server port (2345). I give it the appropriate kit and local executable (copy of the executable on the remote target).
After I hit OK, it seems to run the gdb client locally with a few commands, then times out trying to communicate with the gdbserver. I have debug logging turned on with the server, so I know the server never sees anything from the client run from Qt Creator. Additionally, I did Window->Views->Debugger Log to help me get a better idea of what was going on. The log shows the correct IP address and port for my device, but I don't see anything that looks like a 'target remote IP:port' call.
I'm looking for a hint or reference here. I seem to be a bit stuck at the moment. Thanks!
EDIT: additional information - I've verified that my gdb client has support for python compiled in. I see that Qt Creator requires python support. I also have further support that nothing ever goes out on the wire from Qt Creator. I did a wireshark capture and don't see anything going to port 2345 on the target.
Alternately, if anyone would share their Qt Creator debugger-log output with me so I can compare, that would be helpful.
Avoid Qt Creator 4.5-beta1. It seems 4.5RC1 doesn't have this problem. I was able to do remote debugging using the method described above just fine.
Even QtCreator 4.5.0 seems to have the same bug.
The only workaround I've found is to start the GDB server manually on the remote target: with "top" I get the pid of the remote process to debug then I run the following command: gdbserver --attach remote ip:remote port pid.
After the server is listening, I finally choose: "Attach to Running Debug" server".
Maybe you can write a server start script on target which include the above statement...
I followed the steps in this link
and I manage to debug a binary which resides in linux host from my windows machine from command prompt.
I have gdbserver in linux and I installed gdb with the help of mingw in windows. As I told I can prompt "target remote x.x.x.x:10000 test" to command in windows and debug my test application.
My problem is I can't do the same with eclipse gui, it seems to me it has tones of buttons, options but they make no sense to me.
I am choosing debug_configurations-->C/C++ Remote Application(the only one which allows me to input linux machine ip/port), in "Main" tab to connection I am inputting my linux ip. In same menu under Debugger tab I am inputting my window's gdb path and gdbserver port.
After doing all those I believe I gave enough info to eclipse for connect gdb server but it never enough for eclipse. I am checking gdbserver logs by starting gdbserver with --debug, gdbserver never gets triggered, it does not writes a single line of log. Eclipse even does not starts a connection. But instead it gives me a error like "Error during file upload." which makes no sense to me.
I am using "Eclipse Version: Juno Service Release 2" . Any help will be appreciated .
I believe that the "C/C++ Remote Application" option uses Eclipse's RDT (Remote Development Tools) and RSE (Remote System Explorer) to connect, upload, execute, and debug the application itself. It
If all you want to do is connect to a gdbserver, then create a "C/C++ Attach to Application" debug configuration, and under the Debugger tab, set Debugger to gdbserver.
I also encountered this error message for other reason on Eclipse version 2019-09 R (4.13.0).
I very recommend to read paragraph "How do I debug a remote application?" from this great guide.
It explains which of three remote debugging options you should be using - Automatic Remote Launcher, Manual Remote Launcher and Remote Attach Launcher and how to use each one.
I was using the wrong launcher and hence got the error, while on the remote side gdbserver was listening and waiting for a connection. Switching to Manual Remote Launcher solved it.
for example it says:
If you don't have RSE installed, you cannot use the "Automatic Remote
Launch"
RSE = Remote System Explorer End-User Runtime
Launcher setting is configured from the bottom of the Remote Debug Configuration window - "Using GDB (DSF) Automatic/Manual Remote Debugging Launcher".
I'm trying to debug a C++ application which is running on a Linux box. This application will not build on OS X. I've got the app running on the Linux box, and I can start gdbserver for remote debugging. But since the application doesn't build on OS X, is there a way to use Eclipse on OS X to still debug the app?
Another way, assuming the remote Linux box is an SSH server and has gdb, is to run gdb on it thru ssh, e.g.
ssh remotelinuxbox gdb -tui prog
I am on windows XP want to build the programs on linux remote pc
i have eclipse Ganymede, CDT, RSE installed on remote machine... but how to configure all this?
am I doing correct? could anybody suggest
You could access your remote machine using VNC or similar remote desktop infrastructures. That would allow you to work with Eclipse (edit, build, run, debug, etc) as you would in your local machine.
If bandwidth is too narrow or you don't manage the Linux box, you could access via SSH or telnet and work in console mode (with Emacs/Vim, gdb and all that stuff).
Take a look at https://github.com/ericwoodruff/rmake you would configure your IDE and edit code locally but it uses rsync to build on the remote machine. I've used it at HP to build C++ programs across multiple platforms, linux->windows, windows->linux. It works in the command line and I've used an eclipse builder to invoke it as well. If you enable --no-decorate Eclipse can even parse the build output into the problems view.