This question is regarding the pthread tutorial in llnl.
Say there are three threads.
Thread 1:
pthread_mutex_lock(&mutex)
do_something...
if condition
pthread_cond_signal(&con)
pthread_mutex_unlock(&mutex)
repeat
Thread 2:
pthread_mutex_lock(&mutex)
do_something...
if condition
pthread_cond_signal(&con)
pthread_mutex_unlock(&mutex)
repeat
Thread 3:
pthread_mutex_lock(&mutex)
while(condition not holds)
pthread_cond_wait(&con)
do_something...
pthread_mutex_unlock(&mutex)
Assume that Thread1 checks that condition is satisfied and then sends signal to wake up Thread3. Finally it unlocks the mutex. But at the same time, Thread 2 is trying to lock the mutex.
My question is: is there any guarantee that Thread3 will always win in the competition?
If not then after Thread2 do_something..., the condition variable may be changed, then when Thread3 gets to lock the mutex, the condition variable is different from what it expects.
My question is: is there any guarantee that Tread3 will always win in the competition?
No such guarantees. From POSIX:
The pthread_cond_signal() function shall unblock at least one of the
threads that are blocked on the specified condition variable cond (if
any threads are blocked on cond).
If more than one thread is blocked on a condition variable, the
scheduling policy shall determine the order in which threads are
unblocked. When each thread unblocked as a result of a
pthread_cond_broadcast() or pthread_cond_signal() returns from its
call to pthread_cond_wait() or pthread_cond_timedwait(), the thread
shall own the mutex with which it called pthread_cond_wait() or
pthread_cond_timedwait(). The thread(s) that are unblocked shall
contend for the mutex according to the scheduling policy (if
applicable), and as if each had called pthread_mutex_lock().
(Emphasis mine).
If not then after Tread2 do_something..., the condition variable may be changed, then when Tread3 gets to lock the mutex, the condition variable is different from what it expects.
If the order of execution of threads matter then you probably need to rewrite your code.
There is no guarantee. Returning from pthread_cond_wait() should be treated as implying that the condition might have changed, not that it definitely has - so you need to re-check the condition, and may need to wait again. This is exactly why pthread_cond_wait() should be called within a loop that checks the condition.
Related
When a thread waits on a condition variable, the associated mutex is (atomically) released (unlocked). When that condition variable is signaled (by a different thread), one (for signal) or all (for broadcast) waiting thread(s) is/are awakened, automatically re-acquiring (locking) the mutex.
What will happen if one or more other threads are waiting to acquire (lock) that same mutex, but not waiting on the same condition? Are the thread(s) waiting on the condition variable guaranteed to be awakened (and thus acquire the mutex) before the mutex can be acquired (locked) by the other threads, or could the other thread(s) acquire (lock) the mutex before the thread(s) waiting on the condition variable?
[Note: the example below is simplified for clarity. Thread_B does not really start Thread_C, but Thread_C is guaranteed to not be running until after Thread_B has acquired the mutex - it does not compete with Thread_B for the mutex after Thread_A waits on the condition variable]
Thread_A:
pthread_mutex_lock(&myMutex);
while (!someState) {
pthread_cond_wait(&myCondVar,&myMutex);
}
// do something
pthread_mutex_unlock(&myMutex);
Thread_B:
pthread_mutex_lock(&myMutex);
// do other things
someState = true;
// start Thread_C here
pthread_cond_signal(&myCondVar);
pthread_mutex_unlock(&myMutex);
Thread_C:
pthread_mutex_lock(&myMutex);
// can I reach this point after Thread_B releases the mutex,
// but before Thread_A re-acquires it after being signaled?
// do things that may interfere with Thread_A...
pthread_mutex_unlock(&myMutex);
Edit: The accepted answer below was chosen because it makes clear that whether or not a reader agrees with the interpretation given, there is enough ambiguity that the only safe assumption to make is that of the respondent. Note that others well-versed in C++-standard-speak may find the text totally unambiguous... I am not in that group.
There's nothing special about acquiring a mutex when awakened from pthread_cond_[timed]wait() compared to any other thread already blocked in pthread_mutex_lock() trying to acquire the same mutex.
Per the POSIX 7 pthread_cond_signal() documentation (bolding mine):
If more than one thread is blocked on a condition variable, the scheduling policy shall determine the order in which threads are unblocked. When each thread unblocked as a result of a pthread_cond_broadcast() or pthread_cond_signal() returns from its call to pthread_cond_wait() or pthread_cond_timedwait(), the thread shall own the mutex with which it called pthread_cond_wait() or pthread_cond_timedwait(). The thread(s) that are unblocked shall contend for the mutex according to the scheduling policy (if applicable), and as if each had called pthread_mutex_lock().
Acquiring the mutex after waking up from pthread_cond_[timed]wait() is required to be exactly as if the thread had called pthread_mutex_lock().
In short, any of the threads can acquire the mutex.
Looking at several videos and the documentation example, we unlock the mutex before calling the notify_all(). Will it be better to instead call it after?
The common way:
Inside the Notifier thread:
//prepare data for several worker-threads;
//and now, awaken the threads:
std::unique_lock<std::mutex> lock2(sharedMutex);
_threadsCanAwaken = true;
lock2.unlock();
_conditionVar.notify_all(); //awaken all the worker threads;
//wait until all threads completed;
//cleanup:
_threadsCanAwaken = false;
//prepare new batches once again, etc, etc
Inside one of the worker threads:
while(true){
// wait for the next batch:
std::unique_lock<std::mutex> lock1(sharedMutex);
_conditionVar.wait(lock1, [](){return _threadsCanAwaken});
lock1.unlock(); //let sibling worker-threads work on their part as well
//perform the final task
//signal the notifier that one more thread has completed;
//loop back and wait until the next task
}
Notice how the lock2 is unlocked before we notify the condition variable - should we instead unlock it after the notify_all() ?
Edit
From my comment below: My concern is that, what if the worker spuriously awakes, sees that the mutex is unlocked, super-quickly completes the task and loops back to the start of while. Now the slow-poke Notifier finally calls notify_all(), causing the worker to loop an additional time (excessive and undesired).
There are no advantages to unlocking the mutex before signaling the condition variable unless your implementation is unusual. There are two disadvantages to unlocking before signaling:
If you unlock before you signal, the signal may wake a thread that choose to block on the condition variable after you unlocked. This can lead to a deadlock if you use the same condition variable to signal more than one logical condition. This kind of bug is hard to create, hard to diagnose, and hard to understand. It is trivially avoided by always signaling before unlocking. This ensures that the change of shared state and the signal are an atomic operation and that race conditions and deadlocks are impossible.
There is a performance penalty for unlocking before signaling that is avoided by unlocking after signaling. If you signal before you unlock, a good implementation will know that your signal cannot possibly render any thread ready-to-run because the mutex is held by the calling thread and any thread affects by the condition variable necessarily cannot make forward progress without the mutex. This permits a significant optimization (often called "wait morphing") that is not possible if you unlock first.
So signal while holding the lock unless you have some unusual reason to do otherwise.
should we instead unlock it after the notify_all() ?
It is correct to do it either way but you may have different behavior in different situations. It is quite difficult to predict how it will affect performance of your program - I've seen both positive and negative effects for different applications. So it is better you profile your program and make decision on your particular situation based on profiling.
As mentioned here : cppreference.com
The notifying thread does not need to hold the lock on the same mutex
as the one held by the waiting thread(s); in fact doing so is a
pessimization, since the notified thread would immediately block
again, waiting for the notifying thread to release the lock.
That said, documentation for wait
At the moment of blocking the thread, the function automatically calls
lck.unlock(), allowing other locked threads to continue.
Once notified (explicitly, by some other thread), the function
unblocks and calls lck.lock(), leaving lck in the same state as when
the function was called. Then the function returns (notice that this
last mutex locking may block again the thread before returning).
so when notified wait will re-attempt to gain the lock and in that process it will get blocked again till original notifying thread releases the lock.
So I'll suggest that release the lock before calling notify. As done in example on cppreference.com and most importantly
Don't be Pessimistic.
David's answer seems to me wrong.
First, assuming the simple case of two threads, one waiting for the other on a condition variable, unlocking first by the notifier will not waken the other waiting thread, as the signal has not arrived. Then the notify call will immediately waken the waiting thread. You do not need any special optimizations.
On the other hand, signalling first has the potential of waking up a thread and making it sleep immediately again, as it cannot hold the lock—unless wait morphing is implemented.
Wait morphing does not exist in Linux at least, according to the answer under this StackOverflow question: Which OS / platforms implement wait morphing optimization?
The cppreference example also unlocks first before signalling: https://en.cppreference.com/w/cpp/thread/condition_variable/notify_all
It explicit says:
The notifying thread does not need to hold the lock on the same mutex as the one held by the waiting thread(s). Doing so may be a pessimization, since the notified thread would immediately block again, waiting for the notifying thread to release the lock, though some implementations recognize the pattern and do not attempt to wake up the thread that is notified under lock.
should we instead unlock it after the notify_all() ?
After reading several related posts, I've formed the opinion that it's purely a performance issue. If OS supports "wait morphing", unlock after; otherwise, unlock before.
I'm adding an answer here to augment that of #DavidSchwartz 's. Particularly, I'd like to clarify his point 1.
If you unlock before you signal, the signal may wake a thread that choose to block on the condition variable after you unlocked. This can lead to a deadlock if you use the same condition variable to signal more than one logical condition. This kind of bug is hard to create, hard to diagnose, and hard to understand. It is trivially avoided by always signaling before unlocking. This ensures that the change of shared state and the signal are an atomic operation and that race conditions and deadlocks are impossible.
The 1st thing I said is that, because it's a CV and not a Mutex, a better term for the so-called "deadlock" might be "sleep paralysis" - a mistake some programs make is that
a thread that's supposed to wake
went to sleep due to not rechecking the condition it's been waiting for before wait'ng again.
The 2nd thing is that, when waking some other thread(s),
the default choice should be broadcast/notify_all (broadcast is the POSIX term, which is equivalent to its C++ counterpart).
signal/notify is an optimized special case used for when there's only 1 other thread is waiting.
Finally 3rd, David is adamant that
it's better to unlock after notify,
because it can avoid the "deadlock" which I've been referring to as "sleep paralysis".
If it's unlock then notify, then there's a window where another thread (let's call this the "wrong" thread) may i.) acquire the mutex, ii.)going into wait, and iii.) wake up. The steps i. ii. and iii. happens too quickly, consumed the signal, leaving the intended (let's call it "correct") thread in sleep.
I discussed this extensively with David, he clarified that only when all 3 points are violated ( 1. condvar associated with several separate conditions and/or didn't check it before waiting again; 2. signal/notify only 1 thread when there're more than 1 other threads using the condvar; 3. unlock before notify creating a window for race condition ), the "sleep paralysis" would occur.
Finally, my recommendation is that, point 1 and 2 are essential for correctness of the program, and fixing issues associated with 1 and 2 should be prioritized over 3, which should only be a augmentative "last resort".
For the purpose of providing reference, manpage for signal/broadcast and wait contains some info from version 3 of Single Unix Specification that gave some explanations on point 1 and 2, and partly 3. Although specified for POSIX/Unix/Linux in C, it's concepts are applicable to C++.
As of this writing (2023-01-31), the 2018 edition of version 4 of Single Unix Specification is released, and the drafting of version 5 is underway.
Answer to this question is wrong as it has chance to deadlock. Condition Variable - Wait/Notify Race Condition
I found no solution to solving race condition or dead lock issue.
Imagine we have two threads. now goal is as follows.
first condition:
Thread 1 Waits
Thread 2 Notifies
second condition:
Thread 2 Notifies
Thread 1 Should not wait and continue normal execution.
How can this be implemented correctly without having a queue for notifies? because I want this part of code to run as fast as possible and use a Boolean value instead of adding item into queue. Also there are only 2 threads, so use of queue seems overkill for me.
pseudo code when there is race condition:
Thread 1:
lock(x);
if(!signaled)
{
unlock(x); // ********
// still small gap, how to avoid?
cv.wait(); // forget spurious wakeup for sake of simplicity
signaled = false;
}
else // ********
unlock(x);
Thread 2:
lock(x);
signaled = true;
cv.notify();
unlock(x);
now if you remove two lines commented with ******** race condition will be solved and chance of deadlock will be introduced where Thread1 waits while owning the lock and Thread2 is stuck at locking x.
The source of your confusion is your misunderstanding of conditional variables. The inherent feature of them is that lock release and entering into wait mode is done atomically, i.e. nothing happens in between.
So no modification to the variable can be happening before variables enters into wait mode, because lock will not be released. This is basic guarantee of any conforming conditional_variable implementation, for example, here is the extract from the reference page for std::conditional_variable:
http://en.cppreference.com/w/cpp/thread/condition_variable/wait
Atomically releases lock, blocks the current executing thread, and
adds it to the list of threads waiting on *this. The thread will be
unblocked when notify_all() or notify_one() is executed. It may also
be unblocked spuriously. When unblocked, regardless of the reason,
lock is reacquired and wait exits. If this function exits via
exception, lock is also reacquired. (until C++14)
I would like to ask whether it is OK to use 1 condition variable associated with 2 mutex, for 2 kind of data update.
Basically, I have thread1 and thread2. Thread1 could wait for 2 kinds of data update, so it got 2 mutex, one for each. (I know I could use one mutex for all these data, but this is not the point of this question, right?) And I certainly does not want it be waiting on data1 while data2 is already available, so it only got 1 condition variable. And thread2 would provide both data1 and data2. The problem is that in thread2, I don't know whether thread1 is now waiting for data1 or data2 or not waiting at all.
Pseudo code would look like below:
global data:
cond_var
mutex1
data1
mutex2
data2
thread1:
while true
lock1.lock(mutex1)
if data1 is not available
cond_var.wait(lock1)
if data1 is available
process data1
lock1.unlock()
lock2.lock(mutex2)
if data2 is not available
cond_var.wait(lock2)
if data2 is available
process data2
lock2.unlock()
thread2:
func1:
lock1.lock(mutex1)
update data1
cond_var.notify_all()
lock1.unlock()
func2:
lock2.lock(mutex2)
update data2
cond_var.notify_all()
lock2.unlock()
Outside world will call func1 or func2 to update the data.
When func1 or func2 is called, it will signal cond_var, whether it is in lock1 or lock2. The wait of cond_var is not surrounded by while, so if cond_var is waked on lock1 but data2 is available, thread1 will go on to process data2.
The actual implementation is via boost::thread, and since the platform for my test is Linux, boost::thread should be implemented via pthread.
In almost all tutorial and documents I read about condition variable, it is associated with only 1 mutex. So I am curious to know whether the above program is OK to use, or is fundamentally flawed.
This is OK.
Boost.Thread implements the C++11 thread library (it had some differences until v1.50 but is now very close) and that thread library is conceptually close to the Pthread model, but using a different API, so we can look at those specs for the answer.
The rule in the C++11 standard is:
Requires: lock.owns_lock() is true and lock.mutex() is locked by the calling thread, and either
— no other thread is waiting on this condition_variable object or
— lock.mutex() returns the same value for each of the lock arguments supplied by all concurrently waiting (via wait, wait_for, or wait_until) threads.
In your case the mutex is correctly locked and there is only ever one thread waiting on the condvar, so the condition is met.
The rule in POSIX is equivalent, but worded differently:
The effect of using more than one mutex for concurrent pthread_cond_timedwait() or pthread_cond_wait() operations on the same condition variable is undefined; that is, a condition variable becomes bound to a unique mutex when a thread waits on the condition variable, and this (dynamic) binding shall end when the wait returns.
Again, you have no concurrent wait operations.
In general it's OK to use a condvar with different mutexes as long as you never wait using different mutexes at the same time. When waiting on a condvar you associate it with a mutex and a predicate (the "condition") so as POSIX describes, the condvar and mutex are "bound" together and any a condvar must not be bound to more than one mutex at a time. The reason is that when a condvar wakes up from a wait it must reacquire the mutex it is associated with, if different threads have used it with different mutexes it might re-lock the wrong mutex in the wrong thread, causing chaos. A thread would wake up thinking it has re-locked the mutex it waited with, but it's actually got a lock on a different mutex, maybe one it doesn't have any reference to and so can never unlock. Deadlock and/or undefined behaviour and lions and tigers and bears, oh my.
In your case there are no concurrent waits, so no problem. If you had more than one thread waiting you would need to ensure that the same mutex is used by both threads for any given wait ... which would be difficult as you'd need some additional synchronisation, so it would be simpler to just use two condvars.
I read somewhere that we should lock the mutex before calling pthread_cond_signal and unlock the mutex after calling it:
The pthread_cond_signal() routine is
used to signal (or wake up) another
thread which is waiting on the
condition variable. It should be
called after mutex is locked, and must
unlock mutex in order for
pthread_cond_wait() routine to
complete.
My question is: isn't it OK to call pthread_cond_signal or pthread_cond_broadcast methods without locking the mutex?
If you do not lock the mutex in the codepath that changes the condition and signals, you can lose wakeups. Consider this pair of processes:
Process A:
pthread_mutex_lock(&mutex);
while (condition == FALSE)
pthread_cond_wait(&cond, &mutex);
pthread_mutex_unlock(&mutex);
Process B (incorrect):
condition = TRUE;
pthread_cond_signal(&cond);
Then consider this possible interleaving of instructions, where condition starts out as FALSE:
Process A Process B
pthread_mutex_lock(&mutex);
while (condition == FALSE)
condition = TRUE;
pthread_cond_signal(&cond);
pthread_cond_wait(&cond, &mutex);
The condition is now TRUE, but Process A is stuck waiting on the condition variable - it missed the wakeup signal. If we alter Process B to lock the mutex:
Process B (correct):
pthread_mutex_lock(&mutex);
condition = TRUE;
pthread_cond_signal(&cond);
pthread_mutex_unlock(&mutex);
...then the above cannot occur; the wakeup will never be missed.
(Note that you can actually move the pthread_cond_signal() itself after the pthread_mutex_unlock(), but this can result in less optimal scheduling of threads, and you've necessarily locked the mutex already in this code path due to changing the condition itself).
According to this manual :
The pthread_cond_broadcast() or
pthread_cond_signal() functions
may be called by a thread whether or not it currently owns the mutex that
threads calling pthread_cond_wait()
or pthread_cond_timedwait() have
associated with the condition variable
during their waits; however, if
predictable scheduling behavior is
required, then that mutex shall be
locked by the thread calling
pthread_cond_broadcast() or
pthread_cond_signal().
The meaning of the predictable scheduling behavior statement was explained by Dave Butenhof (author of Programming with POSIX Threads) on comp.programming.threads and is available here.
caf, in your sample code, Process B modifies condition without locking the mutex first. If Process B simply locked the mutex during that modification, and then still unlocked the mutex before calling pthread_cond_signal, there would be no problem --- am I right about that?
I believe intuitively that caf's position is correct: calling pthread_cond_signal without owning the mutex lock is a Bad Idea. But caf's example is not actually evidence in support of this position; it's simply evidence in support of the much weaker (practically self-evident) position that it is a Bad Idea to modify shared state protected by a mutex unless you have locked that mutex first.
Can anyone provide some sample code in which calling pthread_cond_signal followed by pthread_mutex_unlock yields correct behavior, but calling pthread_mutex_unlock followed by pthread_cond_signal yields incorrect behavior?