I am trying to implement a thread pool using ACE Semaphore library. It does not provide any API like sem_getvalue which is in Posix semaphore. I need to debug some flow which is not behaving as expected. Can I examine the semaphore in GDB. I am using Centos as OS.
I initialized two semaphores using the default constructor providing count 0 and 10. I have declared them as static in the class and initialized it in the cpp file as
DP_Semaphore ThreadPool::availableThreads(10);
DP_Semaphore ThreadPool::availableWork(0);
But when I am printing the semaphore in GDB using the print command, I am getting the similar output
(gdb) p this->availableWork
$7 = {
sema = {
semaphore_ = {
sema_ = 0x6fe5a0,
name_ = 0x0
},
removed_ = false
}
}
(gdb) p this->availableThreads
$8 = {
sema = {
semaphore_ = {
sema_ = 0x6fe570,
name_ = 0x0
},
removed_ = false
}
}
Is there a tool which can help me here, or shall I switch to Posix thread and re-write all my code.
EDIT: As requested by #timrau the output of call this->availableWork->dump()
(gdb) p this->availableWork.dump()
[Switching to Thread 0x2aaaae97e940 (LWP 28609)]
The program stopped in another thread while making a function call from GDB.
Evaluation of the expression containing the function
(DP_Semaphore::dump()) will be abandoned.
When the function is done executing, GDB will silently stop.
(gdb) call this->availableWork.dump()
[Switching to Thread 0x2aaaaf37f940 (LWP 28612)]
The program stopped in another thread while making a function call from GDB.
Evaluation of the expression containing the function
(DP_Semaphore::dump()) will be abandoned.
When the function is done executing, GDB will silently stop.
(gdb) info threads
[New Thread 0x2aaaafd80940 (LWP 28613)]
6 Thread 0x2aaaafd80940 (LWP 28613) 0x00002aaaac10a61e in __lll_lock_wait_private ()
from /lib64/libpthread.so.0
* 5 Thread 0x2aaaaf37f940 (LWP 28612) ThreadPool::fetchWork (this=0x78fef0, worker=0x2aaaaf37f038)
at ../../CallManager/src/DP_CallControlTask.cpp:1043
4 Thread 0x2aaaae97e940 (LWP 28609) DP_Semaphore::dump (this=0x6e1460) at ../../Common/src/DP_Semaphore.cpp:21
2 Thread 0x2aaaad57c940 (LWP 28607) 0x00002aaaabe01ff3 in __find_specmb () from /lib64/libc.so.6
1 Thread 0x2aaaacb7b070 (LWP 28604) 0x00002aaaac1027c0 in __nptl_create_event () from /lib64/libpthread.so.0
(gdb)
sema.semaphore_.sema_ in your code looks like a pointer. Try to find it's type in the ACE headers, then convert it to a type and print:
(gdb) p *((sem_t)0x6fe570)
Update: try to convert the address within the structure you posted to sem_t. If you use linux, ACE should be using posix semaphores, so type sem_t must be visible to gdb.
Related
My binary (generated from C++) has two threads. When I care about exceptions, I care about exceptions thrown in one of them (the worker) but not the other.
Is there a way to tell gdb only to pay attention to one of the threads when using catch throw? The gdb manual (texinfo document) and googling suggest to me that this isn't possible, although I think I could request catching a specific type of exception, which hopefully only one of the threads would throw, using catch throw REGEXP.
Is there a way to tell gdb only to pay attention to one of the threads
The catch throw is really just a fancy way to set a breakpoint on __cxxabiv1::__cxa_throw (or similar), and you can make a breakpoint conditional on thread number, achieving the equivalent result.
Example:
#include <pthread.h>
#include <unistd.h>
void *fn(void *)
{
while(true) {
try {
throw 1;
} catch (...) {}
sleep(1);
}
return nullptr;
}
int main() {
pthread_t tid;
pthread_create(&tid, nullptr, fn, nullptr);
fn(nullptr);
return 0;
}
g++ -g -pthread t.cc
Using catch throw, you would get a breakpoint firing on the main and the second thread. But using break ... thread 2 you would only get the one breakpoint you care about:
gdb -q a.out
Reading symbols from a.out...
(gdb) start
Temporary breakpoint 1 at 0x11d6: file t.cc, line 17.
Starting program: /tmp/a.out
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Temporary breakpoint 1, main () at t.cc:17
17 pthread_create(&tid, nullptr, fn, nullptr);
(gdb) n
[New Thread 0x7ffff7a4c640 (LWP 1225199)]
## Note: GDB refuses to set thread-specific breakpoint until the thread actually exists.
18 fn(nullptr);
(gdb) break __cxxabiv1::__cxa_throw thread 2
Breakpoint 2 at 0x7ffff7e40370: file ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc, line 77.
(gdb) c
Continuing.
[Switching to Thread 0x7ffff7a4c640 (LWP 1225199)]
Thread 2 "a.out" hit Breakpoint 2, __cxxabiv1::__cxa_throw (obj=0x7ffff0000be0, tinfo=0x555555557dc8 <typeinfo for int#CXXABI_1.3>, dest=0x0) at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:77
77 ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc: No such file or directory.
(gdb) bt
#0 __cxxabiv1::__cxa_throw (obj=0x7ffff0000be0, tinfo=0x555555557dc8 <typeinfo for int#CXXABI_1.3>, dest=0x0) at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:77
#1 0x00005555555551b5 in fn () at t.cc:8
#2 0x00007ffff7d7eeae in start_thread (arg=0x7ffff7a4c640) at pthread_create.c:463
#3 0x00007ffff7caea5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) c
Continuing.
Thread 2 "a.out" hit Breakpoint 2, __cxxabiv1::__cxa_throw (obj=0x7ffff0000be0, tinfo=0x555555557dc8 <typeinfo for int#CXXABI_1.3>, dest=0x0) at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:77
77 in ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc
(gdb) bt
#0 __cxxabiv1::__cxa_throw (obj=0x7ffff0000be0, tinfo=0x555555557dc8 <typeinfo for int#CXXABI_1.3>, dest=0x0) at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:77
#1 0x00005555555551b5 in fn () at t.cc:8
#2 0x00007ffff7d7eeae in start_thread (arg=0x7ffff7a4c640) at pthread_create.c:463
#3 0x00007ffff7caea5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
VoilĂ -- thread-specific catch throw equivalent.
I am wrote on C++ multithread TCP server, for synchronization using boost:scoped_lock
After connecting to server client freezes.
in gdb i saw more threads in pthread_kill after call boost::mutex::lock
(gdb) info thread
277 Thread 808779c00 (LWP 245289330/xgps) 0x0000000802579d5c in poll () at poll.S:3
276 Thread 808779800 (LWP 245289329/xgps) 0x00000008019799bc in pthread_kill () from /lib/libthr.so.3
275 Thread 808779400 (LWP 245289328/xgps) 0x00000008019799bc in pthread_kill () from /lib/libthr.so.3
.....
246 Thread 808c92800 (LWP 245289296/xgps) 0x00000008019799bc in pthread_kill () from /lib/libthr.so.3
245 Thread 808643800 (LWP 245289295/xgps) 0x00000008019799bc in pthread_kill () from /lib/libthr.so.3
244 Thread 808643400 (LWP 245289294/xgps) 0x00000008019799bc in pthread_kill () from /lib/libthr.so.3
243 Thread 806c8f400 (LWP 245289292/xgps) 0x00000008019799bc in pthread_kill () from /lib/libthr.so.3
242 Thread 808643000 (LWP 245286262/xgps) 0x00000008019799bc in pthread_kill () from /lib/libthr.so.3
241 Thread 808c92400 (LWP 245289288/xgps) 0x00000008019799bc in pthread_kill () from /lib/libthr.so.3
[Switching to thread 205 (Thread 80863a000 (LWP 245289251/xgps))]#0 0x00000008019799bc in pthread_kill () from /lib/libthr.so.3
(gdb) where
#0 0x00000008019799bc in pthread_kill () from /lib/libthr.so.3
#1 0x0000000801973cfc in pthread_getschedparam () from /lib/libthr.so.3
#2 0x00000008019782fc in pthread_mutex_getprioceiling () from /lib/libthr.so.3
#3 0x000000080197838b in pthread_mutex_lock () from /lib/libthr.so.3
#4 0x0000000000442b2e in boost::mutex::lock (this=0x803835f10) at mutex.hpp:62
#5 0x0000000000442c36 in boost::unique_lock<boost::mutex>::lock (this=0x7fffe7334270) at lock_types.hpp:346
#6 0x0000000000442c7c in unique_lock (this=0x7fffe7334270, m_=#0x803835f10) at lock_types.hpp:124
#7 0x0000000000466e31 in XDevice::getDeviceIMEI (this=0x803835e20) at /home/xgps_app/device.cpp:639
#8 0x000000000049071f in XDevicePool::get_device (this=0x7fffffffd9c0, device_imei=868683024674230) at /home/xgps_app/pool_devices.cpp:351
Code at line device.cpp:639
IMEI
XDevice::getDeviceIMEI()
{
try {
boost::mutex::scoped_lock lock(cn_mutex);
return device_imei;
}
catch (std::exception &e )
{
cout << " ERROR in getDeviceIMEI " << e.what() << "\n";
}
return 0;
}
Code in pool_device
XDevicePtr
XDevicePool::get_device(IMEI device_imei)
{
XDevicePtr device;
unsigned int i = 0;
while(i < this->devices.size())
{
device = devices[i];
if (device->getDeviceIMEI() == device_imei) {
LOG4CPLUS_DEBUG(logger, "XDevicePool::get_device found!");
return device;
}
i++;
}
device.reset();
return device;
}
XDevicePtr
XDevicePool::get_device_mt(IMEI device_imei)
{
try
{
boost::mutex::scoped_lock lock(pool_mutex);
}
catch (std::exception & e)
{
LOG4CPLUS_ERROR(logger, "XDevicePool::get_device error! " << e.what());
}
// boost::mutex::scoped_lock lock(pool_mutex);
return get_device(device_imei);
}
Why after call to mutex lock thread terminating?
I think dead lock not reason for that behavior
Please help!
You have multiple locks.
Whenever you have multiple locks that can be required simultaneously you need to obtain them in a fixed order, to avoid dead-locking.
It seems likely that you have such a deadlock occurring. See Boost Thread's free function boost::lock http://www.boost.org/doc/libs/1_63_0/doc/html/thread/synchronization.html#thread.synchronization.lock_functions.lock_multiple for help acquiring multiple lock in reliable order.
You will also want to know about std::defer_lock.
Other than this, there might be interference from fork in multi-threaded programs. I think it's beyond the scope now to explain, unless you are indeed using fork in your process
tl;dr pthread_kill is likely a red herring.
Why after call to mutex lock thread terminating?
It doesn't. Your threads have not been terminated (as evidenced by them still appearing on info thread).
You seem to assume that pthread_kill kills the current thread. In fact, what pthread_kill does is send a signal to another thread. And even the sending is optional (if sig=0).
See the man page for further details.
In some of the answers to related questions I could see that gdb 7.3 should support displaying thread names atleast with 'info threads' command .
But I am not even getting that luxury. please help me to understand what I am doing wrong.
My sample code used for testing:
#include <stdio.h>
#include <pthread.h>
#include <sys/prctl.h>
static pthread_t ta, tb;
void *
fx (void *param)
{
int i = 0;
prctl (PR_SET_NAME, "Mythread1", 0, 0, 0);
while (i < 1000)
{
i++;
printf ("T1%d ", i);
}
}
void *
fy (void *param)
{
int i = 0;
prctl (PR_SET_NAME, "Mythread2", 0, 0, 0);
while (i < 100)
{
i++;
printf ("T2%d ", i);
}
sleep (10);
/* generating segmentation fault */
int *p;
p = NULL;
printf ("%d\n", *p);
}
int
main ()
{
pthread_create (&ta, NULL, fx, 0);
pthread_create (&tb, NULL, fy, 0);
void *retval;
pthread_join (ta, &retval);
pthread_join (tb, &retval);
return 0;
}
Output( using core dump generated by segmentation fault)
(gdb) core-file core.14001
[New LWP 14003]
[New LWP 14001]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `./thread_Ex'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x08048614 in fy (param=0x0) at thread_Ex.c:30
30 printf("%d\n",*p);
(gdb) info threads
Id Target Id Frame
2 Thread 0xb77d76c0 (LWP 14001) 0x00b95424 in __kernel_vsyscall ()
* 1 Thread 0xb6dd5b70 (LWP 14003) 0x08048614 in fy (param=0x0) at thread_Ex.c:30
(gdb) bt
#0 0x08048614 in fy (param=0x0) at thread_Ex.c:30
#1 0x006919e9 in start_thread () from /lib/libpthread.so.0
#2 0x005d3f3e in clone () from /lib/libc.so.6
(gdb) thread apply all bt
Thread 2 (Thread 0xb77d76c0 (LWP 14001)):
#0 0x00b95424 in __kernel_vsyscall ()
#1 0x006920ad in pthread_join () from /lib/libpthread.so.0
#2 0x080486a4 in main () at thread_Ex.c:50
Thread 1 (Thread 0xb6dd5b70 (LWP 14003)):
#0 0x08048614 in fy (param=0x0) at thread_Ex.c:30
#1 0x006919e9 in start_thread () from /lib/libpthread.so.0
#2 0x005d3f3e in clone () from /lib/libc.so.6
(gdb) q
As you can see I cant see any thread names that I have set. what could be wrong?
Note:
I am using gdb version 7.7 (Downloaded and compiled using no special options)
commands used to compile & install gdb : ./configure && make && make install
As far as I am aware, thread names are not present in the core dump.
If they are available somehow, please file a gdb bug.
I get thread name displayed on CentOS6.5, but not displayed on CentOS6.4 .
I am writing a ThreadPool Class in C++ using Boost ASIO. The following is the code that I have written so far:
The ThreadPool Class
using namespace std;
using namespace boost;
class ThreadPoolClass {
private:
/* The limit to the maximum number of threads to be
* instantiated within this pool
*/
int maxThreads;
/* Group of threads in the Pool */
thread_group threadPool;
asio::io_service asyncIOService;
void _Init()
{
maxThreads = 0;
}
public:
ThreadPoolClass();
ThreadPoolClass(int maxNumThreads);
ThreadPoolClass(const ThreadPoolClass& orig);
void CreateThreadPool();
void RunTask(JobClass * aJob);
virtual ~ThreadPoolClass();
};
ThreadPoolClass::ThreadPoolClass() {
_Init();
}
ThreadPoolClass::ThreadPoolClass(int maxNumThreads) {
_Init();
maxThreads = maxNumThreads;
}
void ThreadPoolClass::CreateThreadPool() {
asio::io_service::work work(asyncIOService);
for (int i = 0; i < maxThreads; i++) {
cout<<"Pushed"<<endl;
threadPool.create_thread(bind(&asio::io_service::run, &asyncIOService));
}
}
void ThreadPoolClass::RunTask(JobClass * aJob) {
cout<<"RunTask"<<endl;
asyncIOService.post(bind(&JobClass::Run,aJob));
}
ThreadPoolClass::ThreadPoolClass(const ThreadPoolClass& orig) {
}
ThreadPoolClass::~ThreadPoolClass() {
cout<<"Kill ye all"<<endl;
asyncIOService.stop();
threadPool.join_all();
}
The Job Class
using namespace std;
class JobClass {
private:
int a;
int b;
int c;
public:
JobClass() {
//Empty Constructor
}
JobClass(int val) {
a = val;
b = val - 1;
c = val + 1;
}
void Run()
{
cout<<"a: "<<a<<endl;
cout<<"b: "<<b<<endl;
cout<<"c: "<<c<<endl;
}
};
Main
using namespace std;
int main(int argc, char** argv) {
ThreadPoolClass ccThrPool(20);
ccThrPool.CreateThreadPool();
JobClass ccJob(10);
cout << "Starting..." << endl;
while(1)
{
ccThrPool.RunTask(&ccJob);
}
return 0;
}
So, basically I am creating 20 threads, but as of now just posting only one (same) task to be run by ioservice (just to keep things simple here and get to the root cause). The following is the output when I run this program in GDB:
Pushed
[New Thread 0xb7cd2b40 (LWP 15809)]
Pushed
[New Thread 0xb74d1b40 (LWP 15810)]
Pushed
[New Thread 0xb68ffb40 (LWP 15811)]
Pushed
[New Thread 0xb60feb40 (LWP 15812)]
Pushed
[New Thread 0xb56fdb40 (LWP 15813)]
Pushed
[New Thread 0xb4efcb40 (LWP 15814)]
Pushed
[New Thread 0xb44ffb40 (LWP 15815)]
Pushed
[New Thread 0xb3affb40 (LWP 15816)]
Pushed
[New Thread 0xb30ffb40 (LWP 15817)]
Pushed
[New Thread 0xb28feb40 (LWP 15818)]
Pushed
[New Thread 0xb20fdb40 (LWP 15819)]
Pushed
[New Thread 0xb18fcb40 (LWP 15820)]
Pushed
[New Thread 0xb10fbb40 (LWP 15821)]
Pushed
[New Thread 0xb08fab40 (LWP 15822)]
Pushed
[New Thread 0xb00f9b40 (LWP 15823)]
Pushed
[New Thread 0xaf8f8b40 (LWP 15824)]
Pushed
[New Thread 0xaf0f7b40 (LWP 15825)]
Pushed
[New Thread 0xae8f6b40 (LWP 15826)]
Pushed
[New Thread 0xae0f5b40 (LWP 15827)]
Pushed
[New Thread 0xad8f4b40 (LWP 15828)]
Starting...
RunTask
Kill ye all
[Thread 0xb4efcb40 (LWP 15814) exited]
[Thread 0xb30ffb40 (LWP 15817) exited]
[Thread 0xaf8f8b40 (LWP 15824) exited]
[Thread 0xae8f6b40 (LWP 15826) exited]
[Thread 0xae0f5b40 (LWP 15827) exited]
[Thread 0xaf0f7b40 (LWP 15825) exited]
[Thread 0xb56fdb40 (LWP 15813) exited]
[Thread 0xb18fcb40 (LWP 15820) exited]
[Thread 0xb10fbb40 (LWP 15821) exited]
[Thread 0xb20fdb40 (LWP 15819) exited]
[Thread 0xad8f4b40 (LWP 15828) exited]
[Thread 0xb3affb40 (LWP 15816) exited]
[Thread 0xb7cd2b40 (LWP 15809) exited]
[Thread 0xb60feb40 (LWP 15812) exited]
[Thread 0xb08fab40 (LWP 15822) exited]
[Thread 0xb68ffb40 (LWP 15811) exited]
[Thread 0xb74d1b40 (LWP 15810) exited]
[Thread 0xb28feb40 (LWP 15818) exited]
[Thread 0xb00f9b40 (LWP 15823) exited]
[Thread 0xb44ffb40 (LWP 15815) exited]
[Inferior 1 (process 15808) exited normally]
I have two questions:
Why is it so that my threads are exiting, even when I am posting
tasks in a while loop?
Why is the output from JobClass i.e. the values of the variables a,b
and c not getting printed?
I think this happens because you create work object in the CreateThreadPool method, which is automatically destroyed when goes out of scope -> in this case io_service has no active work and does not process your tasks.
Try to make 'work' instance variable of your ThreadPool class, not local one in the method.
class ThreadPoolClass {
private:
thread_group threadPool;
asio::io_service asyncIOService;
std::auto_ptr<asio::io_service::work> work_;
public:
};
ThreadPoolClass::ThreadPoolClass(int maxNumThreads) {
_Init();
maxThreads = maxNumThreads;
}
void ThreadPoolClass::CreateThreadPool() {
work_.reset(new asio::io_service::work(asyncIOService));
for (int i = 0; i < maxThreads; i++) {
cout<<"Pushed"<<endl;
threadPool.create_thread(bind(&asio::io_service::run, &asyncIOService));
}
}
OK, i'll be the first to admit I don't know boost, and more specifically boost::asio from a hole in the ground, but I know a hella-lot about thread pools and work crews.
The threads a supposed to sleep until notified of new work, but if they are not configured to do so they will likely just finish their thread proc and exit, A tell-tale sign that this is the case is to start up a pool, sleep for a reasonable amount of time before posting any work, and if the pool threads are all terminating, they're not properly waiting. A quick perusal of boost docs yielded this and it may be related to your problem.
On that note, is it possible that the destructor of your pool from the main() entry point is, in fact, prematurely killing your work crew? I see the join_all, but that stop() gives me the willies. if it does what its name implies that would explain a lot. According to the description of that stop() call from the docs:
To effect a shutdown, the application will then need to call the
io_service object's stop() member function. This will cause the
io_service run() call to return as soon as possible, abandoning
unfinished operations and without permitting ready handlers to be
dispatched.
That immediate shutdown and abandonment mention seems suspiciously familiar to your current situation.
Again, I don't know boost:asio from Adam, but were I on this I would check the startup configuration for the boost thread objects. they likely require configuration for how to start, how to wait, etc. There must be numerous samples of using boost:asio on the web concerning configuring the very thing you're describing here, namely a work crew paradigm. I see boost::asio a TON on SO, so there is likely many related or near-related questions as well.
Please feel free to downgrade this if it isn't anything useful, and I apologize if that is the case.
Class LocalT have member of other class that realized read-write-mutex. Mutex initialized at constructor and use pthread_rwlock_rdlock(&aMutex); for reading lock. So, seems, its all ok with mutex class. But program crashed when some LocalT object lock his mutex member for reading.
CSerialize.cpp:2054 line is MUTEX.lock_reading();
Thread 6 (Thread 0x80d4e00 (runnable)):
#0 0x4864f11d in pthread_mutex_lock () from /lib/libpthread.so.2
#1 0x4864b558 in pthread_rwlock_init () from /lib/libpthread.so.2
#2 0x4864b659 in pthread_rwlock_rdlock () from /lib/libpthread.so.2
#3 0x0807ae14 in LocalT::serialize (this=0x80d4e00, outbin=#0x7574736b)
at CSerialize.cpp:2054
Other two running threads:
1) at socket accept();
2) next runnable thread at popen() call, seems its execute or read from pipe. But does not know what is __error() ?????
Thread 1 (Thread 0x8614800 (LWP 100343)):
#0 0x4865b8f9 in __error () from /lib/libpthread.so.2
#1 0x4865a15a in pthread_testcancel () from /lib/libpthread.so.2
#2 0x486425bf in read () from /lib/libpthread.so.2
#3 0x08056340 in UT::execute_popen (command=#0x4865e6bc,
ptr_output=0xbf2f7d30) at Utils.cpp:75
3) all other thread sleeping.
I have no ideas why its crashed? Maybe somebody can assume something or suggest?
==EDIT==
and here is one system(?) thread ( i does not create it, but program always have +1 thread). It always:
Thread 8 (Thread 0x80d4a00 (LWP 100051)):
#0 0x4865a79b in pthread_testcancel () from /lib/libpthread.so.2
#1 0x48652412 in pthread_mutexattr_init () from /lib/libpthread.so.2
#2 0x489fd450 in ?? ()
==EDIT2 - bt as requested==
(gdb) bt
#0 0x4865a7db in pthread_testcancel () from /lib/libpthread.so.2
#1 0x48652412 in pthread_mutexattr_init () from /lib/libpthread.so.2
#2 0x489fd450 in ?? ()
strangely... why ?? () ?
==EDIT3 - when loading core==
Program terminated with signal 11, Segmentation fault.
[skiped]
#0 0x4865a7db in pthread_testcancel () from /lib/libpthread.so.2
[New Thread 0x8614800 (LWP 100343)]
[New Thread 0x8614600 (sleeping)]
[New Thread 0x8614400 (sleeping)]
[New Thread 0x8614200 (sleeping)]
[New Thread 0x8614000 (sleeping)]
[New Thread 0x80d4e00 (runnable)]
[New Thread 0x80d4c00 (sleeping)]
[New Thread 0x80d4a00 (LWP 100051)]
[New Thread 0x80d4000 (runnable)]
[New LWP 100802]
(gdb) info thread
* 10 LWP 100802 0x4865a7db in pthread_testcancel () from /lib/libpthread.so.2
9 Thread 0x80d4000 (runnable) 0x486d7bd3 in accept () from /lib/libc.so.6 -- MAIN() THREAD
8 Thread 0x80d4a00 (LWP 100051) 0x4865a79b in pthread_testcancel ()
from /lib/libpthread.so.2 ( UNIDENTIFIED THREAD system()? )
7 Thread 0x80d4c00 (sleeping) 0x48651cb6 in pthread_mutexattr_init ()
from /lib/libpthread.so.2 (SIGNAL PROCESSOR THREAD)
6 Thread 0x80d4e00 (runnable) 0x4864f11d in pthread_mutex_lock ()
from /lib/libpthread.so.2 (MAINTENANCE THREAD)
5 Thread 0x8614000 (sleeping) 0x48651cb6 in pthread_mutexattr_init ()
from /lib/libpthread.so.2 (other mutex cond_wait - worker 1)
4 Thread 0x8614200 (sleeping) 0x48651cb6 in pthread_mutexattr_init ()
from /lib/libpthread.so.2 (other mutex cond_wait - worker 2 )
3 Thread 0x8614400 (sleeping) 0x48651cb6 in pthread_mutexattr_init ()
from /lib/libpthread.so.2 (other mutex cond_wait - worker 3 )
2 Thread 0x8614600 (sleeping) 0x48651cb6 in pthread_mutexattr_init ()
from /lib/libpthread.so.2 (other mutex cond_wait - worker 4)
1 Thread 0x8614800 (LWP 100343) 0x4865b8f9 in __error ()
from /lib/libpthread.so.2 ( popen() thread see below)
I created: 1 maintenance thread (serializing), 1 popen() thread, 4 workers, 1 main, 1 signal thread = 8 threads....
the thread that you are referring to as system thread is actually your program's main thread.
Secondly with information shared by you so far, it looks like you are acquiring the mutex but never releasing it. that leads to an unstable state (some parameters having wrong values) which leads to a crash. I am sure you will also be observing an intermittent hang.
could you share the backtrace when it crashes ?