I am trying to debug a device driver which is crashing the kernel on a Mac using a remote machine running gdb (trying to follow the instructions here). Both machines are connected to the same network by Ethernet (same router even, and both can access the network). I have also set nvram boot-args="debug=0x144" on the target and restarted.
I then load the kernel extension on the target as usual. On the host machine I start gdb like this:
$ gdb -arch i386 /Volumes/KernelDebugKit/mach_kernel
Once in gdb, I load the kernel macros and set up for remote attachment
(gdb) source /Volumes/KernelDebugKit/kgmacros
(gdb) target remote-kdp
(gdb) kdp-reattach 11.22.33.44
However, the last command then does not make a connection and I get an endless spool of
kdp_reply_wait: error from kdp_receive: receive timeout exceeded
kdp_transaction (remote_connect): transaction timed out
kdp_transaction (remote_connect): re-sending transaction
What is the correct way to get gdb connected to the target machine?
There are a number of ways to break into the target, including:
Kernel panic, as stated in your answer above.
Non-maskable interrupt, which is triggered by the cmd-option-ctrl-shift-esc key combination.
Code a break in your kernel extension using PE_enter_debugger(), which is declared in pexpert/pexpert.h
Halt at boot by setting DB_HALT (0x01) in the NVRAM boot-args value.
Additionally, you may need to set a persistent ARP table entry, as the target is unable to respond to ARP requests while stopped in the debugger. I use the following in my debugger-launch shell script to set the ARP entry if it doesn't already exist:
if !(arp -a -n -i en0 | grep '10\.211\.55\.10[)] at 0:1c:42:d7:29:47 on en0 permanent' > /dev/null) ; then
echo "Adding arp entry"
sudo arp -s 10.211.55.10 00:1c:42:d7:29:47
fi
Someone more expert could probably improve on my bit of shell script.
All of the above is documented in http://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/KernelProgramming.pdf.
The answer is simply to make sure the target has a kernel panic before you try to attach gdb from the host.
Related
I trying to follow the instruction on connecting to valgrind using gdb.
Valgrind memcheck is starts properly and asks to connect using following gdb command:
target remote | vgdb --pid=53181
but when I run this command, I get an error
Remote communication error. Target disconnected.: Connection reset by
peer
what is my mistake?
It appears that error
Remote communication error. Target disconnected.: Connection reset by peer.
is general and may indicate invalid command as well. If you run in gdb
target remote | something
it will give you the same error message.
It appeared for me, that extra space after | symbol was excess.
Correct command was actually
target remote |vgdb --pid=53181
My GDB version is GNU gdb (Ubuntu 10.2-0ubuntu1~18.04~2) 10.2
I am new to valgrind. I need to run the valgrind for a server written in C++. The server listens to a port. When the run the server inside the Valgrind, I couldn't communicate with the server. The Port is not listening.
valgrind --tool=memcheck --leak-check=yes --log-file=valgrind_log.txt /binary_path-c
I need the server should listen to the port when i run with valgrind.
If you have already confirmed that the exact same binary is doing that desired network socket open() and it doesn’t work in Valgrind, then read on.
Valgrind only works with binary file and cannot attach to an already running process (as explained here).
Valgrind is also sensitive to change of effective UID, particularly when running from root UID. You cannot use sudo with valgrind (detailed here).
You cannot Valgrind on an executable binary that has Linux capability bit enabled (details here).
Valgrind cannot handle root setuid on NFS filesystem (even when mounted to allow this). Workaround is to move your build or binary to non-NFS partition.
Having said all that above, it is a timing problem where Valgrind is taking things SLOWER and that the control flow of your code is “missing its mark” to performing that open to a network socket. Only way is to put in debug print statements throughout your code and nail that timing logic.
Alternatively...
To see what a production grade daemon is doing from the very beginning of startup, execute:
valgrind --trace-children=yes /usr/skin/<your-server-binary>
There’s another way to monitor network socket in action, read on ...
Tracing from start of execution
You can perform strace from the start and find out what network socket got opened (and described later, show its buffer content) by:
strace -eopen <your-server-binary> <server-arguments>
make a note of the desired fd (file descriptor) number.
As with any strace command in starting a process, pressing Ctrl-C will stop that process. But using strace on a live process, you safely detach using Ctrl-C from its targeted process (and let that process continue running) and return to your command shell prompt.
Attaching to already running server
But you could monitor an already running production daemon server using strace but it’s harder to find that opened fd number for your network socket. Do previous step briefly to get that fd.
Find out your PID using ps auxw.
Then plug in your server/daemon’s PID here:
strace -f -p <your-server-PID -fnetwork
to find out its fd number.
Exact socket monitoring
With the identified fd on hand, rerun strace to attach to that production server with:
strace -f -eread=<fd> -ewrite=<fd> -p<your-daemon-PID>
network troubleshooting checklist
lsof -i -n a list of open ports
strace
netstat -lt
tcpdump/wireshark
A list of network troubleshooting tools for Linux is given here, here and most comprehensively here.
I can find instructions online to break on accesses to memory addresses using gdb (Watch a memory range in gdb?) but I can't figure out how to do so for memory addresses on the guest machine when I use qemu.
You start qemu with gdb server listening on port 1234 by supplying -s to the qemu comman line. From qemu man page:
-s Shorthand for -gdb tcp::1234, i.e. open a gdbserver on TCP port
1234.
In additon to this, you can also use option -S which will stop Qemu from progressing until you connect gdb to it and issue continue command.
-S Do not start CPU at startup (you must type 'c' in the monitor).
From gdb, you connect to the gdb server running on qemu, by starting gdb (version of gdb that fits you guest architecture). Then connect to the gdb server by command (if qemu is running on the same machine):
(gdb) target remote :1234
References:
http://wiki.qemu.org/Documentation/Debugging
How to debug the Linux kernel with GDB and QEMU?
I am having issues with serial and usb connection between host and target. Below is my setup. Both host and target do NOT have any serial (DB9) ports.
Host : Running windows + VMshare + Ubuntu
Target: Running linux kernel 3.19 . Has a MINI usb port which acts as a serial port, i think its ( CP210x uart to usb )
Connection 1 : Host ( USB to DB9 male-PL2303) + DB9 female to female + (DB9 male to USB) target.
Connection 2 : Host ( USB ) --cable-- (USB mini) Target
Host ( ubuntu VM ), can recognize the USB device (both connections types ) as /dev/ttyUSB0. The device does not show up on the windows device manager since VM takes over the device control.
Target boots into UEFI shell. I modify the syslinux.cfg file to append "kgdbwait kgdboc =ttyS0, 115200" to APPEND flag. Save the change ( press F2) then exit ( press F3 ). Boot into the image. Target now enters the kdb prompt with the following message
kgdb: Waiting for connection from remote gdb...
Entering kdb ( current= <64bit address>, pid 1) on processor 0 due to Keyboard Entry
Kgdb > _
on the host side, i do the following commands and below is error
root#ubuntu: cd /images
root#ubuntu: sudo gdb ./vmlinux
Reading symbols from ./vmlinux done.
(gdb)
(gdb) target remote /dev/ttyUSB0
Remote debugging using /dev/ttyUSB0
Ignoring packet error, continuing...
warning: unrecognized item "timeout" in "qSupported" response
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Bogus trace status reply from target: timeout
Experiments i tried
on Host i used " target remote /dev/ttyS0 " , still same issue
Tried different cables in each connection ( 1 and 2 ) mentioned above
on Target removed the edit in the syslinux.cfg file in UEFI shell, booted the image and entered kgdb using "echo g > /proc/sysrq-trigger"
All kernel configuration related to KGDB* , KGDB_SERIAL*, KGDB_USB* are enabled
all available baud rates
Questions
If i use "kgdbwait kgdboc=ttyUSB0, 115200" instead of " kgdbwait kgdboc=ttyS0, 115200" the target does NOT halt. When the target boots up fully to login prompt, i can see the device is recognized as ttyUSB0 when using connection 1. However as it does not stop does that mean, KGDB using USB does not work ? or for USB debug , i need to use direct USB--USB wire ( connection 3 ) ?
does syslinux.cfg support USB debug? becuase there is a SERIAL flag which has value" 0, 115200 " where 0 refers to ttyS0. syslinux documentation does not have any values for USB type device.
using connection 2, why am i seeing the timeout and packet error issues
occasionally with connection 2, when i execute " target remote /dev/ttyUSB0 " on the host, i notice junk characters on the target. So there is some communication happening, so tried different baud rates still same issue. Does this indicate anything inherently wrong with my setup?
In many online forums/ documents i do not see the "entering kdb due to keyboard entry" when the kernel enters kdb prompt. is this unusual?
The setup of remote kgdb debug is a bit tedious. There are several prerequisites/limitations for the kgdb to work. I will try to break it down.
You have to prepare two machines in this setup.
HOST: Where the agent-proxy & GDB installed.
TARGET: The Linux system being debugged.
Connection Setup
[Host /dev/ttyUSB0] USB to Serial --------- COM port [Target /dev/ttyS0]
On the TARGET side, it's not possible to use the USB interface with kgdb. This is because all the USB-Serial driver (CP210x, PL2303, ...etc) did not implement the polling hook. You have to connect the COM port with a serial cable directly. It is ok to use the USB interface on the HOST side. Since it is a serial connection, you have to use a USB-to-Serial converter and install the proper driver on HOST.
Set the proper baud rate on both sides:
[Target] stty -F /dev/ttyS0 115200
[Host] stty -F /dev/ttyUSB0 115200
Make sure the serial connection works on both direction. You can use:
[Host] cat /dev/ttyUSB0
[Target] echo 'from TARGET to HOST' > /dev/ttyS0
[Target] cat /dev/ttyS0
[Host] echo 'from HOST to TARGET' > /dev/ttyUSB0
You should see the messages on both side of machine. If not, there might be some problems on the cable or driver.
Compile Kernel
Enable KGDB* , KGDB_SERIAL*, KGDB_USB*, DEBUG_INFO, DEBUG_INFO_DWARF4, MAGIC_SYSRQ in the kernel config. Compile and install on the TARGET.
The main purpose here is to enable KGDB feature & preserve debug information in vmlinux.
agent-proxy Setup
agent-proxy acts as a proxy for the TARGET's serial port. It splits up the serial port for multiplexing. One for primary console I/O, the other for GDB session. Thus, we can work on both simultaneously. You should run the agent-proxy on HOST machine.
git clone http://git.kernel.org/pub/scm/utils/kernel/kgdb/agent-proxy.git
cd agent-proxy ; make
./agent-proxy 5550^5551 0 /dev/ttyUSB0,115200
This will redirect:
TARGET's console to HOST:5550
TARGET's kgdb listening port to HOST:5551
Start To Debug
First, open the primary console:
[Host] telnet localhost 5550
Entering the kdb mode, either by:
[Target] echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc
[Target] dmesg | tail
(you should see KGDB: Registered I/O driver kgdboc, otherwise it failed)
[Target] echo g >/proc/sysrq-trigger
Or, by adding the following kernel parameters in TARGET's bootloader (for early kernel debug):
console=tty0 console=ttyS0,115200 kgdbwait kgdboc=ttyS0,115200
The TARGET machine will halt immediately once it breaks into the kdb.
At the same time, you will see a kdb prompt on the primary console:
....
Entering kdb (current=0xcb846c80, pid 2301) on processor 3 due to Keyboard Entry
[3]kdb>
Type kgdb then enter. The TARGET is now pending for remote GDB's connection. We will connect it from the HOST.
Host> gdb vmlinux
(gdb) target remote localhost:5551
Remote debugging using localhost:5551
kgdb_breakpoint () at kernel/debug/debug_core.c:1072
1072 wmb(); /* Sync point after breakpoint */
(gdb)
Enjoy your kernel debugging!
I used to the St-write to burn .bin to the STM32F4 and saw the message which I expected. Now, I hope to understand how GPIO init. Hence, I use OpenOCD and arm-none-eabi-gdb to do that. Here, it is my process.
$ minicom
$ openocd -f /opt/openocd/share/openocd/scripts/board/stm32f4discovery.cfg
$ arm-none-eabi-gdb main.elf
(gdb) target remote localhost:3333
(gdb) localhost:3333: Connection timed out.
How do I check the port of OpenOCD? Why does it occur timeout?
That certainly means that openocd did not start or that the port is busy.
Usually, you use :
openocd -f board/stm32f4discovery.cfg
You should check that your session is running.
Are you running a virtual linux machine on a windows host?
If so, you probably need to replace localhost with 10.0.0.2 (or whatever your windows IP is).
A good way to know, is to telnet to the openOCD address and port 4444 and see if you get the openOCD prompt, and can type a few commands.