This question already has answers here:
gdb how to get thread name displayed
(4 answers)
Closed 5 years ago.
During use gdb to debug multithread code, I need to find the thread name from gdb to locate error more faster.
for e.g. below gdb command only print thread 1,2,3..., etc, but I wish I can not thread A,B,C... that means I need to thread name list out.
Is it possible from gdb command.
(gdb) info threads
15 Thread 8725 __ioctl () at bionic/libc/arch-arm/syscalls/__ioctl.S:13
14 Thread 8726 __ioctl () at bionic/libc/arch-arm/syscalls/__ioctl.S:13
13 Thread 8730 __ioctl () at bionic/libc/arch-arm/syscalls/__ioctl.S:13
12 Thread 13328 __futex_wait () at bionic/libc/arch-arm/bionic/futex_arm.S:51
11 Thread 13330 __futex_wait () at bionic/libc/arch-arm/bionic/futex_arm.S:51
10 Thread 13331 __futex_wait () at bionic/libc/arch-arm/bionic/futex_arm.S:51
9 Thread 8711 __futex_wait () at bionic/libc/arch-arm/bionic/futex_arm.S:51
8 Thread 13334 nanosleep () at bionic/libc/arch-arm/syscalls/nanosleep.S:13
7 Thread 8722 nanosleep () at bionic/libc/arch-arm/syscalls/nanosleep.S:13
6 Thread 8724 nanosleep () at bionic/libc/arch-arm/syscalls/nanosleep.S:13
5 Thread 8710 __futex_wait () at bionic/libc/arch-arm/bionic/futex_arm.S:51
4 Thread 8712 __futex_wait () at bionic/libc/arch-arm/bionic/futex_arm.S:51
3 Thread 8723 __ioctl () at bionic/libc/arch-arm/syscalls/__ioctl.S:13
2 Thread 8721 read () at bionic/libc/arch-arm/syscalls/read.S:14
* 1 Thread 8709 __futex_wait () at bionic/libc/arch-arm/bionic/futex_arm.S:51
You don't say what version of gdb you are using.
For native Linux (that is, not using gdbserver), printing of thread names was added in gdb 7.3. So, upgrade to at least that version and you should see it work.
Support for this for gdbserver is planned, but not yet implemented.
Support for other platforms depends on volunteers.
Related
gdb switches a thread when the appropriate breakpoint fires.
I need to remember the thread number to come back with thread n.
Is there a special command to return to the thread where step-by-step execution was performed?
(gdb) n
57 m_app = create_app();
(gdb) n
[Switching to Thread 0x7fffdfffd700 (LWP 32059)]
Thread 9 "foo" hit Breakpoint 1, 0x0000000000421ef0 in App::App()#plt ()
(gdb) # command to return to the interrupted thread
You can setup custom GDB prompt to show current thread number with set extended-prompt:
(gdb) set extended-prompt <\t>
Every time you use any of GDB commands current thread number will be printed in GDB prompt, for example:
<1> thread
[Current thread is 1 (Thread 0x7ffff7dcd740 (LWP 148411))]
<1> info threads
Id Target Id Frame
* 1 Thread 0x7ffff7dcd740 (LWP 148411) "a.out" main () at /home/ks/t.c:7
<1>
With that information in GDB prompt you can return to interrupted thread with:
thread <id_taken_from_prompt_before_interrupt>
This question already has an answer here:
How to find which thread caused SEGFAULT in a post-mortem gdb session?
(1 answer)
Closed 8 years ago.
My application uses more than 8 threads. When I run info threads in gdb I see the threads and the last function they were executing. It does not seem obvious to me exactly which thread caused the SIGSEGV. Is it possible to tell it? Is it thread 1? How are the threads numbered?
When you use gdb to analyze the core dump file, the gdb will stop at the function which causes program core dump. And the current thread will be the murder. Take the following program as an example:
#include <stdio.h>
#include <pthread.h>
void *thread_func(void *p_arg)
{
while (1)
{
printf("%s\n", (char*)p_arg);
sleep(10);
}
}
int main(void)
{
pthread_t t1, t2;
pthread_create(&t1, NULL, thread_func, "Thread 1");
pthread_create(&t2, NULL, thread_func, NULL);
sleep(1000);
return;
}
The t2 thread will cause program down because it refers a NULL pointer. After the program down, use gdb to analyze the core dump file:
[root#localhost nan]# gdb -q a core.32794
Reading symbols from a...done.
[New LWP 32796]
[New LWP 32795]
[New LWP 32794]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./a'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000034e4281451 in __strlen_sse2 () from /lib64/libc.so.6
(gdb)
The gdb stops at __strlen_sse2 function, this means this function causes the program down. Then use bt command to see it is called by which thread:
(gdb) bt
#0 0x00000034e4281451 in __strlen_sse2 () from /lib64/libc.so.6
#1 0x00000034e4268cdb in puts () from /lib64/libc.so.6
#2 0x00000000004005cc in thread_func (p_arg=0x0) at a.c:7
#3 0x00000034e4a079d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00000034e42e8b6d in clone () from /lib64/libc.so.6
(gdb) i threads
Id Target Id Frame
3 Thread 0x7ff6104c1700 (LWP 32794) 0x00000034e42accdd in nanosleep () from /lib64/libc.so.6
2 Thread 0x7ff6104bf700 (LWP 32795) 0x00000034e42accdd in nanosleep () from /lib64/libc.so.6
* 1 Thread 0x7ff60fabe700 (LWP 32796) 0x00000034e4281451 in __strlen_sse2 () from /lib64/libc.so.6
The bt command shows the stack frame of the current thread(which is the murder). "i threads" commands shows all the threads, the thread number which begins with * is the current thread.
As for "How are the threads numbered?", it depends on the OS. you can refer the gdb manual for more information.
I am attaching with gdb to the running process (my multithread server). Then I request info thread and see that there is always +1 (or 2? ) additional thread(s), that I does not create at my code.
I created only:
4 workers (must be at cond_wait() )
1 signal thread ( always at sigwait() )
1 maintenance thread (executing each N seconds, then cond_wait() state)
1 thread that use popen() (executing each N seconds, then cond_wait() state)
1 main() thread (at accept() )
So, = 8 thread created by me. But why reported 9 or 10 ???
System is FreeBSD 6.4
Also, always have problem with this additional thread, it crash my program and it always at pthread_testcancel () state!
c++ pthreads - crash while trying to lock mutex for reading
Seems, marked by * thread number 10 is currently executed thread? and its the same as 8 thread??? or I have 2 additional thread? Is it normal? thanks.. and sorry for my bad english.
(gdb) info thread
* 10 LWP 100108 0x4865a79b in pthread_testcancel () from /lib/libpthread.so.2 ( WHAT IS ??? (1) )
9 Thread 0x80d4000 (runnable) 0x486d7bd3 in accept () from /lib/libc.so.6
8 Thread 0x80d4a00 (LWP 100090) 0x4865a79b in pthread_testcancel ()
from /lib/libpthread.so.2 ( WHAT IS??? (2) )
7 Thread 0x80d4c00 (sleeping) 0x48651cb6 in pthread_mutexattr_init ()
from /lib/libpthread.so.2
6 Thread 0x80d4e00 (sleeping) 0x48651cb6 in pthread_mutexattr_init ()
from /lib/libpthread.so.2
5 Thread 0x868b000 (sleeping) 0x48651cb6 in pthread_mutexattr_init ()
from /lib/libpthread.so.2
4 Thread 0x868b200 (sleeping) 0x48651cb6 in pthread_mutexattr_init ()
from /lib/libpthread.so.2
3 Thread 0x868b400 (sleeping) 0x48651cb6 in pthread_mutexattr_init ()
from /lib/libpthread.so.2
2 Thread 0x868b600 (sleeping) 0x48651cb6 in pthread_mutexattr_init ()
from /lib/libpthread.so.2
1 Thread 0x868b800 (sleeping) 0x48651cb6 in pthread_mutexattr_init ()
from /lib/libpthread.so.2
The additional threads are the result of 3rd party libraries. A quick search through curl, ImageMagick, tinyxml2, and pcre's source code shows that curl and ImageMagick have pthread_create() calls.
With regards to debugging in gdb:
In info threads, the * indicating the current thread being examined. It does not indicate the current thread running.
On the backtrace, the in ?? () can indicate that either the libraries are not built with debugging information (-g with gcc) or the stack is corrupted. Generally, if the stack is corrupted, gdb will give an explicit indication.
Also, be certain to check ImageMagick's thread of execution documentation.
UPDATE Well, it looks like adding PyEval_InitThreads() before the call to PyGILState_Ensure() does the trick. In my haste to figure things out I incorrectly attributed my "hanging" to PyEval_InitThreads().
However, after reading some Python documentation I am wondering if this is the correct solution.
It is not safe to call this function when it is unknown which thread (if any) currently has the global interpreter lock.
First of all, I am working on some modified GNU Radio code - particularly a modified gr_bin_statistics_f block. Now, there is a bug report (albeit an old one) which pretty much describes my exact situation.
http://gnuradio.org/redmine/issues/show/199
Now, usrp_spectrum_sense.py which is mentioned in the bug report calls gr_bin_statistics_f (C++) which then calls back to Python periodically to re-tune the USRP (radio).
Here is what happens when the Python code is called:
PyGILState_STATE d_gstate;
d_gstate = PyGILState_Ensure();
// call python code
PyGILState_Release(d_gstate);
So, once we return from the Python code a segmentation fault occurs when PyGILState_Release(d_gstate) is called. While there are differences between my code and the original gr_bin_statistics_f, nothing seems to be remotely related to this.
I read that calling PyEval_InitThreads() before PyGILState_Ensure() has solved the problem for some people, but it just causes my program to hang.
Can anyone shed light on this for me? Or is it simply time to send a message to the GNU Radio mailing list?
Using Python2.7 on Fedora 14 x86_64.
Here is the GDB backtrace:
(gdb) c
Continuing.
[New Thread 0x7fabd3a8d700 (LWP 23969)]
[New Thread 0x7fabd328c700 (LWP 23970)]
[New Thread 0x7fabd2a8b700 (LWP 23971)]
[New Thread 0x7fabd228a700 (LWP 23972)]
[New Thread 0x7fabd1a89700 (LWP 23973)]
[New Thread 0x7fabd1288700 (LWP 23974)]
[New Thread 0x7fabd0a87700 (LWP 23975)]
[New Thread 0x7fabbbfff700 (LWP 23976)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fabbbfff700 (LWP 23976)]
0x00000036b3e0db00 in sem_post () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00000036b3e0db00 in sem_post () from /lib64/libpthread.so.0
#1 0x00000036c1317679 in PyThread_release_lock () from /usr/lib64/libpython2.7.so.1.0
#2 0x00007fabd6159c1f in ~ensure_py_gil_state (this=0x2dc6fc0, x=887000000)
at gnuradio_swig_py_general.cc:5593
#3 gr_py_feval_dd::calleval (this=0x2dc6fc0, x=887000000) at gnuradio_swig_py_general.cc:5605
#4 0x00007fabd77c4b6e in gr_noise_level_f::tune_window (this=0x2db3ca0,
target_freq=) at gr_noise_level_f.cc:97
#5 0x00007fabd77c554b in gr_noise_level_f::work (this=0x2db3ca0, noutput_items=7,
input_items=, output_items=)
at gr_noise_level_f.cc:115
#6 0x00007fabd7860714 in gr_sync_block::general_work (this=0x2db3ca0,
noutput_items=, ninput_items=,
input_items=, output_items=) at gr_sync_block.cc:64
#7 0x00007fabd7846ce4 in gr_block_executor::run_one_iteration (this=0x7fabbbffed90)
at gr_block_executor.cc:299
#8 0x00007fabd7864332 in gr_tpb_thread_body::gr_tpb_thread_body (this=0x7fabbbffed90, block=...)
at gr_tpb_thread_body.cc:49
#9 0x00007fabd785cce7 in operator() (function_obj_ptr=...) at gr_scheduler_tpb.cc:42
#10 operator() (function_obj_ptr=...)
at /home/tja/Research/energy/detector/gnuradio-3.3.0/gruel/src/include/gruel/thread_body_wrapper.h:49
#11 boost::detail::function::void_function_obj_invoker0, void>::invoke (function_obj_ptr=...) at /usr/include/boost/function/function_template.hpp:153
---Type to continue, or q to quit---
#12 0x00007fabd74914ef in operator() (this=)
at /usr/include/boost/function/function_template.hpp:1013
#13 boost::detail::thread_data >::run (this=)
at /usr/include/boost/thread/detail/thread.hpp:61
#14 0x00007fabd725ca55 in thread_proxy () from /usr/lib64/libboost_thread-mt.so.1.44.0
#15 0x00000036b3e06d5b in start_thread () from /lib64/libpthread.so.0
#16 0x00000036b3ae4a7d in clone () from /lib64/libc.so.6
(gdb)
Thanks for looking!
Python expects a certain amount of initialization to be done by the main thread before anything attempts to call back in from a subthread.
If the main thread is an application that is embedding Python, then it should call PyEval_InitThreads() immediately after calling Py_Initialize().
If the main thread is instead the Python interpreter itself (as seems to be the case here), then the module using the multithreaded extension module should include an "import threading" early to ensure that PyEval_InitThreads() is called correctly before any subthreads are spawned.
I ran into this exact problem as well. The documentation for anything relating to threads in CPython is unfortunately patchy at best.
Essentially, you need to do the following:
In your main thread, BEFORE any other threads are spawned, you need to call PyEval_InitThreads(). A good place to do this is right after you call PyInitialize().
Now, PyEval_InitThreads() not only initializes the Python interpreter thread-state, it also implicitly acquires the Global Interpreter Lock. This means, you need to release the lock before you call PyGILEnsure_State() in some other thread, otherwise your program will hang. You can do this with the function PyEval_ReleaseLock().
So basically, in your main thread, before any other threads are launched, you want to say:
PyInitialize();
PyEval_InitThreads();
PyEval_ReleaseLock();
Then, in any additional thread, anytime you use the Python API you need to say:
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
/* ... some code that does things with Python ... */
PyGILState_Release(gstate);
I've run into some problems debugging a multi-threaded process using GDB. I have a multi-threaded process that splinters off into several (8 or 9) different threads, and I am trying to determine what the contents of variables are when the constructor for a class called XML_File_Data is called. However, I've run into a problem where, after I apply the correct function breakpoint to all threads and it's apparent one of the thread's break point is getting hit (the program temporarily halts execution), I'm not able to determine which thread hit the breakpoint. The command
(gdb) thread apply all where
is giving me shockingly useless information in the form:
#0 0x004ab410 in __kernel_vsyscall ()
#1 0x05268996 in nanosleep () from /lib/libc.so.6
#2 0x052a215c in usleep () from /lib/libc.so.6
#3 0x082ee313 in frame_clock_frame_end (clock=0xb4bfd2f8)
at frame_clock.c:143
#4 0x003a349a in ?? ()
#5 0x00b5cfde in thread_proxy ()
from /cets_development_libraries/install/lib/libboost_thread-gcc41-mt-1_38.so.1.38.0
#6 0x02c1f5ab in start_thread () from /lib/libpthread.so.0
#7 0x052a8cfe in clone () from /lib/libc.so.6
Of the 9 processes, 7 or so are giving me almost exactly that output, and the information about the last 2 isn't really much more helpful (functions far down the call stack have recognizable names, but any recent #0-#4 functions aren't recognizable).
This is what I have so far:
(gdb) gdb
(gdb) gdb attach <processid>
(gdb) thread apply all 'XML_File_Data::XML_File_Data()'
and (after the breakpoint is hit)
(gdb) thread apply all where
Could any experienced debuggers offer me some hints on what I am doing wrong or what is normally done in this situation?
Cheers,
Charlie
EDIT: Fortunately, I was able to find out that the cause of the ??'s was optimized code being run through the debugger, in addition to not running the debugger in the directory of the executable file. Still not much success with the debugging though.
I find myself doing this all the time:
> t a a f
Short for:
> thread apply all frame
Of course, other variants are possible:
> t a a bt 3
Which prints the bottom 3 frames of each thread's stack. (You can also use negative numbers to get the top N frames of the stack)
You can use command thread or info threads to find out the current thread number after breakpoint hit
(gdb) thread
[Current thread is 1 (Thread 0xb790d6c0 (LWP 2519))]
(gdb)
(gdb) info threads
17 Thread 0xb789cb90 (LWP 2536) 0xb7fc6402 in __kernel_vsyscall ()
16 Thread 0xb769bb90 (LWP 2537) 0xb7fc6402 in __kernel_vsyscall ()
15 Thread 0xb749ab90 (LWP 2543) 0xb7fc6402 in __kernel_vsyscall ()
14 Thread 0xb7282b90 (LWP 2544) 0xb7fc6402 in __kernel_vsyscall ()
13 Thread 0xb5827b90 (LWP 2707) 0xb7fc6402 in __kernel_vsyscall ()
12 Thread 0xb5626b90 (LWP 2708) 0xb7fc6402 in __kernel_vsyscall ()
11 Thread 0xb5425b90 (LWP 2709) 0xb7fc6402 in __kernel_vsyscall ()
10 Thread 0xb5161b90 (LWP 2713) 0xb7fc6402 in __kernel_vsyscall ()
9 Thread 0xb4ef9b90 (LWP 2715) 0xb7fc6402 in __kernel_vsyscall ()
8 Thread 0xb4af7b90 (LWP 2717) 0xb7fc6402 in __kernel_vsyscall ()
7 Thread 0xb46ffb90 (LWP 2718) 0xb7fc6402 in __kernel_vsyscall ()
6 Thread 0xb44feb90 (LWP 2726) 0xb7fc6402 in __kernel_vsyscall ()
5 Thread 0xb42fdb90 (LWP 2847) 0xb7fc6402 in __kernel_vsyscall ()
4 Thread 0xb40fcb90 (LWP 2848) 0xb7fc6402 in __kernel_vsyscall ()
3 Thread 0xb3efbb90 (LWP 2849) 0xb7fc6402 in __kernel_vsyscall ()
2 Thread 0xb3cfab90 (LWP 2850) 0xb7fc6402 in __kernel_vsyscall ()
* 1 Thread 0xb790d6c0 (LWP 2519) 0xb7fc6402 in __kernel_vsyscall ()
(gdb)
An asterisk `*' to the left of the gdb thread number indicates the current thread. See here.