pthread_cond_wait sometimes will not receive the signal - c++

Question

I have a weird problem with pthread_cond_wait and pthread_cond_signal. I have arranged a series of threads. They are all in sleep state when started. A wake up function will signal these threads, do some work, and wait for the results.
In the setup below, td is thread data, containing the mutex and conditions, and th is an array containing the pointer to the threads:
for (size_t i = 0; i < NUM_THREADS; i++) {
pthread_cond_init(&td[i].cond, NULL);
pthread_mutex_init(&td[i].cond_mutex, NULL);
pthread_mutex_init(&td[i].work_mutex, NULL);
pthread_mutex_lock(&td[i].cond_mutex);
pthread_mutex_lock(&td[i].work_mutex);
pthread_create(&th[i], NULL, thread_worker, (void *)&td[i]);
}
Thread worker is like this:
void*
thread_worker(void* data)
{
THREAD_DATA *td = (THREAD_DATA *)data;
while (1) {
pthread_cond_wait(&td->cond, &td->cond_mutex); // marker
// do work ...
pthread_mutex_unlock(&td->work_mutex);
}
pthread_exit(NULL);
}
This job function is supposed to wake up all the threads, do the job, and wait for them to finish:
void
job()
{
for (size_t i = 0; i < NUM_THREADS; i++) {
pthread_cond_signal(&td[i].cond);
}
for (size_t i = 0; i < NUM_THREADS; i++) {
pthread_mutex_lock(&td[i].work_mutex); // block until the work is done
}
}
In some rare situations (1 out of 1000 runs maybe), the above setup will encounter a freeze. When that happens, the 'marker' line in thread_worker will not be signaled by pthread_cond_signal, it just kept on waiting. It's very rare but it happens from time to time. I've produced numerous log messages, and I verified that pthread_cond_wait is always called before pthread_cond_signal. What am I doing wrong here?

Answer 1

There is nothing there that forces the pthread_cond_wait() to be called before the pthread_cond_signal(). Despite what you say about logging, it's entirely possible for the logged lines to be out-of-sequence with what really happened.
You aren't using mutexes and condition variables correctly: mutexes should only be unlocked by the same thread that locked them, and condition variables should be paired with a test over some shared state (called a predicate). The shared state is supposed to be protected by the mutex that is passed to pthread_cond_wait().
For example, your example can be reworked to correctly use mutexes and condition variables. First, add an int work_status to the THREAD_DATA structure, where 0 indicates that the thread is waiting for work, 1 indicates that work is available and 2 indicates that the work is complete.
You don't appear to need two mutexes in each THREAD_DATA, and you don't want to lock the mutex in the main thread when you're setting it up:
for (size_t i = 0; i < NUM_THREADS; i++) {
pthread_cond_init(&td[i].cond, NULL);
pthread_mutex_init(&td[i].cond_mutex, NULL);
td[i].work_status = 0;
pthread_create(&th[i], NULL, thread_worker, (void *)&td[i]);
}
Have the threads wait on work_status using the condition variable:
void*
thread_worker(void* data)
{
THREAD_DATA *td = (THREAD_DATA *)data;
while (1) {
/* Wait for work to be available */
pthread_mutex_lock(&td->cond_mutex);
while (td->work_status != 1)
pthread_cond_wait(&td->cond, &td->cond_mutex);
pthread_mutex_unlock(&td->cond_mutex);
// do work ...
/* Tell main thread that the work has finished */
pthread_mutex_lock(&td->cond_mutex);
td->work_status = 2;
pthread_cond_signal(&td->cond);
pthread_mutex_unlock(&td->cond_mutex);
}
pthread_exit(NULL);
}
...and set and wait on work_status as appropriate in job():
void
job()
{
/* Tell threads that work is available */
for (size_t i = 0; i < NUM_THREADS; i++) {
pthread_mutex_lock(&td[i].cond_mutex);
td[i].work_status = 1;
pthread_cond_signal(&td[i].cond);
pthread_mutex_unlock(&td[i].cond_mutex);
}
/* Wait for threads to signal work complete */
for (size_t i = 0; i < NUM_THREADS; i++) {
pthread_mutex_lock(&td[i].cond_mutex);
while (td[i].work_status != 2)
pthread_cond_wait(&td[i].cond, &td[i].cond_mutex);
pthread_mutex_unlock(&td[i].cond_mutex);
}
}

Answer 2

Some check lists:
1) Do you lock the mutex td->cond_mutex before waiting on the cond variable? Otherwise, it's undefined.
2) Do you check predicate after pthread_cond_wait() returns? Typical usage is
while(!flag) pthread_cond_wait(&cv, &mutex); //waits on flag
which is not what you have. This is to protect against spurious wake-ups and also ensure the predicate hasn't changed in the meantime.
3) pthread_cond_signal() is guaranteed to wake up at least one thread. You may want to use pthread_cond_broadcast() if there are multiple threads waiting on the same condition variable.
4) If no thread is waiting on a conditional variable then pthread_cond_signal() or pthread_cond_broadcast() has no effect.