If I have some multithreaded process and want to trace it with gdb using attach command, to which thread it will connect (e.g. current running or main)? I know that I can discover it with info threads but I want to know which thread it will choose by default.
For Linux, all of the threads are stopped by the ptrace command when gdb attaches.
It has been my experience that gdb defaults to the main thread for C/C++ applications. If you attach to a process and do a 'bt' it will list the stack for 'main'.
Information is available for all threads however. gdb can look at the thread(s) information in the /proc filesystem. The proc contains information about each thread in the tasks area. Details about the stack address is located in the stat file as well as the maps file. Details are also available regarding the register values for each thread.
Along the lines of your question, I've often wondered why stepping through a multithreaded application will cause gdb to jump from thread to thread. I think that gdb is still at the mercy of the kernel scheduler so that a step on a thread may lead to a different thread getting the CPU resource and a breakpoint being triggered.
On Linux, where thread ids exist in the same space as process ids, it appears you can run gdb -p tid to attach to the thread with given tid and its owning process, without knowing the pid. Because the main thread of a process has tid == pid, it makes sense that running gdb -p pid connects to the main thread.
Example code that connects gdb to the currently executing thread, e.g. for generating a pretty stack trace: https://github.com/facebook/rocksdb/pull/11150
Related
got some big real time project to deal with (multiple processes (IPCs), multi Everything in short).
My working on process is started as service on Linux. I have the root access.
Here is the problem:
I'm trying to attach to a running proc, tried starting it through/with gdb but the result is the same: it stops the executable once I "touched" it with gdb or sometimes it throws:
Program received signal SIGUSR1, User defined signal 1. [Switching to Thread 0x7f9fe869f700 (LWP 2638)]
of course from there nothing can be done.
Tried:
handle all nostop
attach to launched as service (daemon) or launched as regular proc
started from gdb
thought maybe forking/multi-threaded problem - implemented in the very beginning sleep for 10 seconds - attached to it with "continue"
Guys, all I want it is to debug, hit the breakpoints, etc.
Please help! Share ideas.
Editing actual commands:
1) gdb attach myProcId. Then after reading symbols, I hit "c" which results:
Program received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7f9fe869f700 (LWP 2638)]
0x00007f9fec09bf73 in select () from /lib64/libc.so.6
2) If I make the first line 10 seconds sleep in the code, attaching to the process, hit "c", result: it runs, shows info threads, backtrace of main, but never hits the breakpoint (for sure the code runs there - I get logs and different behaviour if I change code there), meaning the process is stuck.
3) All other combinations like gdb path/to/my/proc args list, then start. Where arg list played with different related options gdb gives us.
Maybe worth to mention: process network packets related, timers driven also.
But for me the important thing is a current snapshot on break, i don't care what will happen to the system after timers expired.
Since you mentioned that you are debugging a multiprocessing program, I think the underlying program you have is to set the breakpoint in the correct subprocess.
Try break fork and set follow-fork-mode child/parent. What you want to achieve is have gdb attached to the process that is running the code you want to debug.
Refer to this link.
Another thought is to generate a crash, since you can compile the programe. For example add a int i = *(int*)NULL and that will generate a core dump. You can then debug the core dump with gdb <program> <core dump>. You can refer to this page for how to configure core dump.
I'm debugging a C++ application which creates trees of forks. Using GDB defaults, the child processes will be detached on the fork and as a result I see only one inferior shown afterwards.
I tried to attach to one of the child processes and despite it not being listed as an inferior for the other GDB process, in the new GDB session I get an error that the process is already being traced (by the first GDB session).
Is this expected behavior? What steps can I take to debug the forked process in a separate GDB session? What steps can I take to debug the problem further?
When we use gdb attach to debug a running process, we could use gdb attach pid ,if the process have two or more threads, the pid is the main thread tid.
Now I want to implement a simple debugger to debug multi-thread process, but when I use my debugger to attach a multi-thread process, only the main thread suspended.
I want to know why only use the main thread tid, the gdb can attach all thread of this process, how does gdb suspend all the threads? we assume that when we use gdb attach, all the thread have been created.
I want to know why only use the main thread tid, the gdb can attach all thread of this process, how does gdb suspend all the threads?
When you do attach PROCESS_PID gdb internally calls ptrace (PTRACE_ATTACH) for each thread. on Linux you can check it yourself with:
$ strace -e ptrace -p GDB_PROCESS_PID
Just run a program with s few threads, run gdb and before running attach PROCESS_PID run strace in another console. You must see ptrace (PTRACE_ATTACH) for each thread.
ptrace PTRACE_ATTACH sends SIGSTOP to the process which suspends the whole process i.e. all threads.
The main thread TID having the same numerical value as the process PID is a historical accident of Linux systems; it isn't the case on other Unix systems.
When gdb (or any debugger) attaches to a process using ptrace, all the threads of that process are suspended.
I'm a little new to GDB. I'm hoping someone can help me with something that should be quite simple, I've used Google/docs but I'm just missing something.
What is the 'normal' way folks debug threaded apps with GDB? I'm using pthreads. I'm wanting to watch only one thread - the two options I see are
a) tell the debugger somehow to attach to a particular thread, such that stepping wont result in jumping threads on each context switch
b) tell the debugger to suspend/free any 'uninteresting' threads
I'd prefer to go route b) - reading the help for GDB I dont see a command for this, tips?
See documentation for set scheduler-locking on.
Beware: if you suspend other threads, and if one of them holds a lock, and if your interesting thread needs that lock at some point while stepping, you'll deadlock.
What is the 'normal' way folks debug threaded apps
You can never debug thread correctness, you can only design it in. In my experience, most of debugging of threaded apps is putting in assertions, and examining state of the world when one of the assertions is violated.
First, you need to enable comfortable for multi-threading debugger behavior with the following commands. No idea why it's disabled by default.
set target-async 1
set non-stop on
I personally put those commands into .gdbinit file. They make your every command to be applied only to the currently focused thread. Note: the thread might be running, so you have to pause it.
To see the focused thread execute the thread.
To switch to another thread append the number of the thread, e.g. thread 2.
To see all threads with their numbers issue info thread.
To apply a command to a particular thread issue something like thread apply threadnum command. E.g. thread apply 4 bt will apply backtrace command to a thread number 4. thread apply all continue continues all paused threads.
There is a small problem though — many commands needs the thread to be paused. I know a few ways of doing that:
interrupt command: interrupts the thread execution, accepts a number of a thread to pause, without an argument breaks the focused one.
Setting a breakpoint somewhere. Note that you may set a breakpoint to a particular thread, so that other threads will ignore it, like break linenum thread threadnum. E.g. break 25 thread 4.
You may also find very useful that you can set a list of commands to be executed when a breakpoint hit through the command commands — so e.g. you may quickly print interesting values, then continue execution.
I have a program which crashes due to a segmentation fault. The core file is produced.
running the core in gdb gives me the following:
HP gdb 6.1 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
Core was generated by `gcpf1fwcApp'.
Program terminated with signal 6, Aborted.
I used the command
thread apply all bt
When I check the stack trace I get error in the main thread which is in waiting state.
However when I run the same program in GDB I get a completely different error in stack trace. Which seems more correct than the core dump.
The program has 31 threads.
Why do I get this kind of difference?
It is possible that you are simply looking at the wrong thread.
Try thread apply all where, and see if one of the threads is in fact abort()ing.
When debugging a live process, GDB will stop when a thread receives SIGABRT, and so will likely show you the relevant thread.
When debugging a core (post-mortem), GDB doesn't know which thread is relevant, and so shows them to you in whichever order the OS saved them into the core. Linux kernels save the thread which caused the process to die first, so GDB on Linux shows relevant thread from core. I am guessing that HP-UX does not do that, and so GDB shows you a "random" thread instead.