How to end a thread properly? - c++

My main program creates a thread. This thread initializes some data then enters a 'while' loop and runs until the main program sets the control variable to 'false'. Then it calls join() witch blocks the whole code endlessly.
bool m_ThreadMayRun;
void main(){
thread mythread = thread(&ThreadFunction);
//do stuff
m_ThreadMayRun = false;
mythread.join(); // this blocks endlessly even when I ask 'joinable' before
}
void ThreadFunction{
initdata();
m_ThreadMayRun=true;
while(m_ThreadMayRun){
//do stuff that can be / has to be done for ever
}
deinitdata();
}
Am I missing something here?
What would be a proper solution to make the loop leave from the main thread?
Is it at all necessary to call join?
Thanks for help

You have a race condition for two threads writing to m_ThreadMayRun. Consider what happens if first the main thread executes m_ThreadMayRun = false; and then the thread you spwaned executes m_ThreadMayRun = true;, then you have an infinite loop. However, strictly speaking that line of reasoning is irrelevant, because when you have a race condition your code has undefined behavior.
Am I missing something here?
You need to synchronize access to m_ThreadMayRun by making it either an std::atomic<bool> or using a std::mutex and make sure that m_ThreadMayRun = false is executed after m_ThreadMayRun = true;.
PS For this situation it is better to use a std::condition_variable.

The issue is that access to bool m_ThreadMayRun; is not synchronized, and according to C++ rules, each thread may assume it does not change between threads. So you end up with a race (a form of undefined behavior).
To make the intention clear, make it atomic.
std::atomic<bool> m_ThreadMayRun;
With this every load/store of m_ThreadMayRun becomes a memory fence, which not only synchronizes its own value, but also makes other work done by the thread visible, due to the acquire/release semantics of an atomic load/store.
Though there is still a small race possible between m_ThreadMayRun = true in the thread and setting m_ThreadMayRun = false. Either one can execute first, sometimes leading to undesired results. To avoid this, initialize it to true before starting the thread.
std::atomic<bool> m_ThreadMayRun;
void main(){
m_ThreadMayRun = true;
thread mythread(&ThreadFunction);
//do stuff
m_ThreadMayRun = false;
mythread.join(); // this blocks endlessly even when I ask 'joinable' before
}
void ThreadFunction{
initdata();
while(m_ThreadMayRun){
//do stuff that can be / has to be done for ever
}
deinitdata();
}
For more details about memory fences and acquire/release semantics, refer to the following excellent resources: the book "C++ Concurrency in Action" and Herb Sutter's atomic<> weapons talk.

Related

multi-thread program initialization using call_once vs atomic_flag

In book C++ Concurrency in Action 2nd, 3.3.1, the author introduced a way using call_once function to avoid double-checked locking pattern when doing initialization in multi-thread program,
std::shared_ptr<some_resource> resource_ptr;
std::once_flag resource_flag;
void init_resource()
{
resource_ptr.reset(new some_resource);
}
void foo()
{
std::call_once(resource_flag,init_resource); #1
resource_ptr->do_something();
}
the reason is explained in this [answer][1]. I used to use atomic_flag to do initialization in multi-thread program, something like this:
td::atomic_flag init = ATOMIC_FLAG_INIT;
std::atomic<bool> initialized = false;
void Init()
{
if (init.test_and_set()) return;
DoInit();
initialized = true;
}
void Foo(){
if(!initialized) return;
DoSomething(); // use some variable intialized in DoInit()
}
every threads will call Init() before call Foo().
After read the book, I wonder will the above pattern cause race condition, therefore not safe to use? Is it possible that the compiler reorder the instructions and initialized become true before DoInit() finish?
[1]: Explain race condition in double checked locking
The race condition in your code happens when thread 1 enters DoInit and thread 2 skips it and proceeds to Foo.
You handle it with if(!initialized) return in Foo but this is not always possible: you should always expect a method to accidently do nothing and you can forget to add such checks to other methods.
With std::call_once with concurrent invocation the execution will only continue after the action is executed.
Regarding reordering, atomics operations use memory_order_seq_cst by default that does not allow any reordering.

Pattern for thread to check if it should stop?

Is there a canonical pattern for a thread to check if it should stop working?
The scenario is that a thread is spinning a tight working loop but it should stop if another thread tells it to. I was thinking of checking an atomic bool in the loop condition but I'm not sure if that is an unnecessary performance hit or not. E.g.
std::atomic<bool> stop{false};
while(!stop){
//...
}
You have to make it atomic (or volatile in old C/C++) to ensure that the compiler doesn't optimize it away and only test stop once.
If you call a function in the loop that cannot be inlined (like reading sockets) you might be safe with a non-atomic bool, but why risk it - especially as the atomic read is unlikely to be a performance issue in that case?
To have the least effect you could do something like:
std::atomic<bool> stop;
void rx_thread() {
// ...
while(!stop.load(std::memory_order_relaxed)){
..
}
}
I don't see any reason why the bool should be atomic. There isn't really a potential for a race condition. When you want to stop the thread, you first set the variable to true and then issue a call that wakes up the blocking function inside the loop (however you do that, depends on the blocking call).
bool stop = false;
void rx_thread() {
// ...
while(!stop){
// ...
blocking_call();
// ...
}
}
void stop_thread() {
stop = true;
wakeup_rx_thread();
}
If the blocking call happens to wake up between setting stop to true and calling wakeup_rx_thread(), then the loop will finish anyway. The call to wakeup_rx_thread() will be needless, but that doesn't matter.

std::atomic_bool for cancellation flag: is std::memory_order_relaxed the correct memory order?

I have a thread that reads from a socket and generates data. After every operation, the thread checks a std::atomic_bool flag to see if it must exit early.
In order to cancel the operation, I set the cancellation flag to true, then call join() on the worker thread object.
The code of the thread and the cancellation function looks something like this:
std::thread work_thread;
std::atomic_bool cancel_requested{false};
void thread_func()
{
while(! cancel_requested.load(std::memory_order_relaxed))
process_next_element();
}
void cancel()
{
cancel_requested.store(true, std::memory_order_relaxed);
work_thread.join();
}
Is std::memory_order_relaxed the correct memory order for this use of an atomic variable?
As long as there is no dependency between cancel_requested flag and anything else, you should be safe.
The code as shown looks OK, assuming you use cancel_requested only to expedite the shutdown, but also have a provision for an orderly shutdown, such as a sentinel entry in the queue (and of course that the queue itself is synchronized).
Which means your code actually looks like this:
std::thread work_thread;
std::atomic_bool cancel_requested{false};
std::mutex work_queue_mutex;
std::condition_variable work_queue_filled_cond;
std::queue work_queue;
void thread_func()
{
while(! cancel_requested.load(std::memory_order_relaxed))
{
std::unique_lock<std::mutex> lock(work_queue_mutex);
work_queue_filled_cond.wait(lock, []{ return !work_queue.empty(); });
auto element = work_queue.front();
work_queue.pop();
lock.unlock();
if (element == exit_sentinel)
break;
process_next_element(element);
}
}
void cancel()
{
std::unique_lock<std::mutex> lock(work_queue_mutex);
work_queue.push_back(exit_sentinel);
work_queue_filled_cond.notify_one();
lock.unlock();
cancel_requested.store(true, std::memory_order_relaxed);
work_thread.join();
}
And if we're that far, then cancel_requested may just as well become a regular variable, the code even becomes simpler.
std::thread work_thread;
bool cancel_requested = false;
std::mutex work_queue_mutex;
std::condition_variable work_queue_filled_cond;
std::queue work_queue;
void thread_func()
{
while(true)
{
std::unique_lock<std::mutex> lock(work_queue_mutex);
work_queue_filled_cond.wait(lock, []{ return cancel_requested || !work_queue.empty(); });
if (cancel_requested)
break;
auto element = work_queue.front();
work_queue.pop();
lock.unlock();
process_next_element(element);
}
}
void cancel()
{
std::unique_lock<std::mutex> lock(work_queue_mutex);
cancel_requested = true;
work_queue_filled_cond.notify_one();
lock.unlock();
work_thread.join();
}
memory_order_relaxed is generally hard to reason about, because it blurs the general notion of sequentially executing code. So the usefulness of it is very, very limited as Herb explains in his atomic weapons talk.
Note std::thread::join() by itself acts as a memory barrier between the two threads.
Whether this code is correct depends on a lot of things. Most of all it depends on what exactly you mean by "correct". As far as I can tell, the bits of code that you show don't invoke undefined behavior (assuming your work_thread and cancel_requested are not actually initialized in the order your snippet above suggests as you would then have the thread potentially reading the uninitialized value of the atomic). If all you need to do is change the value of that flag and have the thread eventually see the new value at some point independent of whatever else may be going on, then std::memory_order_relaxed is sufficient.
However, I see that your worker thread calls a process_next_element() function. That suggests that there is some mechanism through which the worker thread receives elements to process. I don't see any way for the thread to exit when all elements have been processed. What does process_next_element() do when there's no next element available right away? Does it just return immediately? In that case you've got yourself a busy wait for more input or cancellation, which will work but is probably not ideal. Or does process_next_element() internally call some function that blocks until an element becomes available!? If that is the case, then cancelling the thread would have to involve first setting the cancellation flag and then doing whatever is needed to make sure the next element call your thread is potentially blocking on returns. In this case, it's potentially essential that the thread can never see the cancellation flag after the blocking call returns. Otherwise, you could potentially have the call return, go back into the loop, still read the old cancellation flag and then go call process_next_element() again. If process_next_element() is guaranteed to just return again, then you're fine. If that is not the case, you have a deadlock. So I believe it technically depends on what exactly process_next_element() does. One could imagine an implementation of process_next_element() where you would potentially need more than relaxed memory order. However, if you already have a mechanism for fetching new elements to process, why even use a separate cancellation flag? You could simply handle cancellation through that same mechanism, e.g., by having it return a next element with a special value or return no element at all to signal cancellation of processing and cause the thread to return instead of relying on a separate flag…

Using std::condition_variable with atomic<bool>

There are several questions on SO dealing with atomic, and other that deal with std::condition_variable. But my question if my use below is correct?
Three threads, one ctrl thread that does preparation work before unpausing the two other threads. The ctrl thread also is able to pause the worker threads (sender/receiver) while they are in their tight send/receive loops.
The idea with using the atomic is to make the tight loops faster in case the boolean for pausing is not set.
class SomeClass
{
public:
//...
// Disregard that data is public...
std::condition_variable cv; // UDP threads will wait on this cv until allowed
// to run by ctrl thread.
std::mutex cv_m;
std::atomic<bool> pause_test_threads;
};
void do_pause_test_threads(SomeClass *someclass)
{
if (!someclass->pause_test_threads)
{
// Even though we use an atomic, mutex must be held during
// modification. See documentation of condition variable
// notify_all/wait. Mutex does not need to be held for the actual
// notify call.
std::lock_guard<std::mutex> lk(someclass->cv_m);
someclass->pause_test_threads = true;
}
}
void unpause_test_threads(SomeClass *someclass)
{
if (someclass->pause_test_threads)
{
{
// Even though we use an atomic, mutex must be held during
// modification. See documentation of condition variable
// notify_all/wait. Mutex does not need to be held for the actual
// notify call.
std::lock_guard<std::mutex> lk(someclass->cv_m);
someclass->pause_test_threads = false;
}
someclass->cv.notify_all(); // Allow send/receive threads to run.
}
}
void wait_to_start(SomeClass *someclass)
{
std::unique_lock<std::mutex> lk(someclass->cv_m); // RAII, no need for unlock.
auto not_paused = [someclass](){return someclass->pause_test_threads == false;};
someclass->cv.wait(lk, not_paused);
}
void ctrl_thread(SomeClass *someclass)
{
// Do startup work
// ...
unpause_test_threads(someclass);
for (;;)
{
// ... check for end-program etc, if so, break;
if (lost ctrl connection to other endpoint)
{
pause_test_threads();
}
else
{
unpause_test_threads();
}
sleep(SLEEP_INTERVAL);
}
unpause_test_threads(someclass);
}
void sender_thread(SomeClass *someclass)
{
wait_to_start(someclass);
...
for (;;)
{
// ... check for end-program etc, if so, break;
if (someclass->pause_test_threads) wait_to_start(someclass);
...
}
}
void receiver_thread(SomeClass *someclass)
{
wait_to_start(someclass);
...
for (;;)
{
// ... check for end-program etc, if so, break;
if (someclass->pause_test_threads) wait_to_start(someclass);
...
}
I looked through your code manipulating conditional variable and atomic, and it seems that it is correct and won't cause problems.
Why you should protect writes to shared variable even if it is atomic:
There could be problems if write to shared variable happens between checking it in predicate and waiting on condition. Consider following:
Waiting thread wakes spuriously, aquires mutex, checks predicate and evaluates it to false, so it must wait on cv again.
Controlling thread sets shared variable to true.
Controlling thread sends notification, which is not received by anybody, because there is no threads waiting on conditional variable.
Waiting thread waits on conditional variable. Since notification was already sent, it would wait until next spurious wakeup, or next time when controlling thread sends notification. Potentially waiting indefinetly.
Reads from shared atomic variables without locking is generally safe, unless it introduces TOCTOU problems.
In your case you are reading shared variable to avoid unnecessary locking and then checking it again after lock (in conditional wait call). It is a valid optimisation, called double-checked locking and I do not see any potential problems here.
You might want to check if atomic<bool> is lock-free. Otherwise you will have even more locks you would have without it.
In general, you want to treat the fact that variable is atomic independently of how it works with a condition variable.
If all code that interacts with the condition variable follows the usual pattern of locking the mutex before query/modification, and the code interacting with the condition variable does not rely on code that does not interact with the condition variable, it will continue to be correct even if it wraps an atomic mutex.
From a quick read of your pseudo-code, this appears to be correct. However, pseudo-code is often a poor substitute for real code for multi-threaded code.
The "optimization" of only waiting on the condition variable (and locking the mutex) when an atomic read says you might want to may or may not be an optimization. You need to profile throughput.
atomic data doesn't need another synchronization, it's basis of lock-free algorithms and data structures.
void do_pause_test_threads(SomeClass *someclass)
{
if (!someclass->pause_test_threads)
{
/// your pause_test_threads might be changed here by other thread
/// so you have to acquire mutex before checking and changing
/// or use atomic methods - compare_exchange_weak/strong,
/// but not all together
std::lock_guard<std::mutex> lk(someclass->cv_m);
someclass->pause_test_threads = true;
}
}

Do I have to acquire lock before calling condition_variable.notify_one()?

I am a bit confused about the use of std::condition_variable. I understand I have to create a unique_lock on a mutex before calling condition_variable.wait(). What I cannot find is whether I should also acquire a unique lock before calling notify_one() or notify_all().
Examples on cppreference.com are conflicting. For example, the notify_one page gives this example:
#include <iostream>
#include <condition_variable>
#include <thread>
#include <chrono>
std::condition_variable cv;
std::mutex cv_m;
int i = 0;
bool done = false;
void waits()
{
std::unique_lock<std::mutex> lk(cv_m);
std::cout << "Waiting... \n";
cv.wait(lk, []{return i == 1;});
std::cout << "...finished waiting. i == 1\n";
done = true;
}
void signals()
{
std::this_thread::sleep_for(std::chrono::seconds(1));
std::cout << "Notifying...\n";
cv.notify_one();
std::unique_lock<std::mutex> lk(cv_m);
i = 1;
while (!done) {
lk.unlock();
std::this_thread::sleep_for(std::chrono::seconds(1));
lk.lock();
std::cerr << "Notifying again...\n";
cv.notify_one();
}
}
int main()
{
std::thread t1(waits), t2(signals);
t1.join(); t2.join();
}
Here the lock is not acquired for the first notify_one(), but is acquired for the second notify_one(). Looking though other pages with examples I see different things, mostly not acquiring the lock.
Can I choose myself to lock the mutex before calling notify_one(), and why would I choose to lock it?
In the example given, why is there no lock for the first notify_one(), but there is for subsequent calls. Is this example wrong or is there some rationale?
You do not need to be holding a lock when calling condition_variable::notify_one(), but it's not wrong in the sense that it's still well defined behavior and not an error.
However, it might be a "pessimization" since whatever waiting thread is made runnable (if any) will immediately try to acquire the lock that the notifying thread holds. I think it's a good rule of thumb to avoid holding the lock associated with a condition variable while calling notify_one() or notify_all(). See Pthread Mutex: pthread_mutex_unlock() consumes lots of time for an example where releasing a lock before calling the pthread equivalent of notify_one() improved performance measurably.
Keep in mind that the lock() call in the while loop is necessary at some point, because the lock needs to be held during the while (!done) loop condition check. But it doesn't need to be held for the call to notify_one().
2016-02-27: Large update to address some questions in the comments about whether there's a race condition if the lock isn't held for the notify_one() call. I know this update is late because the question was asked almost two years ago, but I'd like to address #Cookie's question about a possible race condition if the producer (signals() in this example) calls notify_one() just before the consumer (waits() in this example) is able to call wait().
The key is what happens to i - that's the object that actually indicates whether or not the consumer has "work" to do. The condition_variable is just a mechanism to let the consumer efficiently wait for a change to i.
The producer needs to hold the lock when updating i, and the consumer must hold the lock while checking i and calling condition_variable::wait() (if it needs to wait at all). In this case, the key is that it must be the same instance of holding the lock (often called a critical section) when the consumer does this check-and-wait. Since the critical section is held when the producer updates i and when the consumer checks-and-waits on i, there is no opportunity for i to change between when the consumer checks i and when it calls condition_variable::wait(). This is the crux for a proper use of condition variables.
The C++ standard says that condition_variable::wait() behaves like the following when called with a predicate (as in this case):
while (!pred())
wait(lock);
There are two situations that can occur when the consumer checks i:
if i is 0 then the consumer calls cv.wait(), then i will still be 0 when the wait(lock) part of the implementation is called - the proper use of the locks ensures that. In this case the producer has no opportunity to call the condition_variable::notify_one() in its while loop until after the consumer has called cv.wait(lk, []{return i == 1;}) (and the wait() call has done everything it needs to do to properly 'catch' a notify - wait() won't release the lock until it has done that). So in this case, the consumer cannot miss the notification.
if i is already 1 when the consumer calls cv.wait(), the wait(lock) part of the implementation will never be called because the while (!pred()) test will cause the internal loop to terminate. In this situation it doesn't matter when the call to notify_one() occurs - the consumer will not block.
The example here does have the additional complexity of using the done variable to signal back to the producer thread that the consumer has recognized that i == 1, but I don't think this changes the analysis at all because all of the access to done (for both reading and modifying) are done while in the same critical sections that involve i and the condition_variable.
If you look at the question that #eh9 pointed to, Sync is unreliable using std::atomic and std::condition_variable, you will see a race condition. However, the code posted in that question violates one of the fundamental rules of using a condition variable: It does not hold a single critical section when performing a check-and-wait.
In that example, the code looks like:
if (--f->counter == 0) // (1)
// we have zeroed this fence's counter, wake up everyone that waits
f->resume.notify_all(); // (2)
else
{
unique_lock<mutex> lock(f->resume_mutex);
f->resume.wait(lock); // (3)
}
You will notice that the wait() at #3 is performed while holding f->resume_mutex. But the check for whether or not the wait() is necessary at step #1 is not done while holding that lock at all (much less continuously for the check-and-wait), which is a requirement for proper use of condition variables). I believe that the person who has the problem with that code snippet thought that since f->counter was a std::atomic type this would fulfill the requirement. However, the atomicity provided by std::atomic doesn't extend to the subsequent call to f->resume.wait(lock). In this example, there is a race between when f->counter is checked (step #1) and when the wait() is called (step #3).
That race does not exist in this question's example.
As others have pointed out, you do not need to be holding the lock when calling notify_one(), in terms of race conditions and threading-related issues. However, in some cases, holding the lock may be required to prevent the condition_variable from getting destroyed before notify_one() is called. Consider the following example:
thread t;
void foo() {
std::mutex m;
std::condition_variable cv;
bool done = false;
t = std::thread([&]() {
{
std::lock_guard<std::mutex> l(m); // (1)
done = true; // (2)
} // (3)
cv.notify_one(); // (4)
}); // (5)
std::unique_lock<std::mutex> lock(m); // (6)
cv.wait(lock, [&done]() { return done; }); // (7)
}
void main() {
foo(); // (8)
t.join(); // (9)
}
Assume there is a context switch to the newly created thread t after we created it but before we start waiting on the condition variable (somewhere between (5) and (6)). The thread t acquires the lock (1), sets the predicate variable (2) and then releases the lock (3). Assume there is another context switch right at this point before notify_one() (4) is executed. The main thread acquires the lock (6) and executes line (7), at which point the predicate returns true and there is no reason to wait, so it releases the lock and continues. foo returns (8) and the variables in its scope (including cv) are destroyed. Before thread t could join the main thread (9), it has to finish its execution, so it continues from where it left off to execute cv.notify_one() (4), at which point cv is already destroyed!
The possible fix in this case is to keep holding the lock when calling notify_one (i.e. remove the scope ending in line (3)). By doing so, we ensure that thread t calls notify_one before cv.wait can check the newly set predicate variable and continue, since it would need to acquire the lock, which t is currently holding, to do the check. So, we ensure that cv is not accessed by thread t after foo returns.
To summarize, the problem in this specific case is not really about threading, but about the lifetimes of the variables captured by reference. cv is captured by reference via thread t, hence you have to make sure cv stays alive for the duration of the thread's execution. The other examples presented here do not suffer from this issue, because condition_variable and mutex objects are defined in the global scope, hence they are guaranteed to be kept alive until the program exits.
Situation
Using vc10 and Boost 1.56 I implemented a concurrent queue pretty much like this blog post suggests. The author unlocks the mutex to minimize contention, i.e., notify_one() is called with the mutex unlocked:
void push(const T& item)
{
std::unique_lock<std::mutex> mlock(mutex_);
queue_.push(item);
mlock.unlock(); // unlock before notificiation to minimize mutex contention
cond_.notify_one(); // notify one waiting thread
}
Unlocking the mutex is backed by an example in the Boost documentation:
void prepare_data_for_processing()
{
retrieve_data();
prepare_data();
{
boost::lock_guard<boost::mutex> lock(mut);
data_ready=true;
}
cond.notify_one();
}
Problem
Still this led to the following erratic behaviour:
while notify_one() has not been called yet cond_.wait() can still be interrupted via boost::thread::interrupt()
once notify_one() was called for the first time cond_.wait() deadlocks; the wait cannot be ended by boost::thread::interrupt() or boost::condition_variable::notify_*() anymore.
Solution
Removing the line mlock.unlock() made the code work as expected (notifications and interrupts end the wait). Note that notify_one() is called with the mutex still locked, it is unlocked right afterwards when leaving the scope:
void push(const T& item)
{
std::lock_guard<std::mutex> mlock(mutex_);
queue_.push(item);
cond_.notify_one(); // notify one waiting thread
}
That means that at least with my particular thread implementation the mutex must not be unlocked before calling boost::condition_variable::notify_one(), although both ways seem correct.
#Michael Burr is correct. condition_variable::notify_one does not require a lock on the variable. Nothing prevents you to use a lock in that situation though, as the example illustrates it.
In the given example, the lock is motivated by the concurrent use of the variable i. Because the signals thread modifies the variable, it needs to ensure that no other thread is access it during that time.
Locks are used for any situation requiring synchronization, I don't think we can state it in a more general way.
In some case, when the cv may be occupied(locked) by other threads. You needs to get lock and release it before notify_*().
If not, the notify_*() maybe not executed at all.
Just adding this answer because I think the accepted answer might be misleading. In all cases you will need to lock the mutex, prior to calling notify_one() somewhere for your code to be thread-safe, although you might unlock it again before actually calling notify_*().
To clarify, you MUST take the lock before entering wait(lk) because wait() unlocks lk and it would be Undefined Behavior if the lock wasn't locked. This is not the case with notify_one(), but you need to make sure you won't call notify_*() before entering wait() and having that call unlock the mutex; which obviously only can be done by locking that same mutex before you call notify_*().
For example, consider the following case:
std::atomic_int count;
std::mutex cancel_mutex;
std::condition_variable cancel_cv;
void stop()
{
if (count.fetch_sub(1) == -999) // Reached -1000 ?
cv.notify_one();
}
bool start()
{
if (count.fetch_add(1) >= 0)
return true;
// Failure.
stop();
return false;
}
void cancel()
{
if (count.fetch_sub(1000) == 0) // Reached -1000?
return;
// Wait till count reached -1000.
std::unique_lock<std::mutex> lk(cancel_mutex);
cancel_cv.wait(lk);
}
Warning: this code contains a bug.
The idea is the following: threads call start() and stop() in pairs, but only as long as start() returned true. For example:
if (start())
{
// Do stuff
stop();
}
One (other) thread at some point will call cancel() and after returning from cancel() will destroy objects that are needed at 'Do stuff'. However, cancel() is supposed not to return while there are threads between start() and stop(), and once cancel() executed its first line, start() will always return false, so no new threads will enter the 'Do stuff' area.
Works right?
The reasoning is as follows:
1) If any thread successfully executes the first line of start() (and therefore will return true) then no thread did execute the first line of cancel() yet (we assume that the total number of threads is much smaller than 1000 by the way).
2) Also, while a thread successfully executed the first line of start(), but not yet the first line of stop() then it is impossible that any thread will successfully execute the first line of cancel() (note that only one thread ever calls cancel()): the value returned by fetch_sub(1000) will be larger than 0.
3) Once a thread executed the first line of cancel(), the first line of start() will always return false and a thread calling start() will not enter the 'Do stuff' area anymore.
4) The number of calls to start() and stop() are always balanced, so after the first line of cancel() is unsuccessfully executed, there will always be a moment where a (the last) call to stop() causes count to reach -1000 and therefore notify_one() to be called. Note that can only ever happen when the first line of cancel resulted in that thread to fall through.
Apart from a starvation problem where so many threads are calling start()/stop() that count never reaches -1000 and cancel() never returns, which one might accept as "unlikely and never lasting long", there is another bug:
It is possible that there is one thread inside the 'Do stuff' area, lets say it is just calling stop(); at that moment a thread executes the first line of cancel() reading the value 1 with the fetch_sub(1000) and falling through. But before it takes the mutex and/or does the call to wait(lk), the first thread executes the first line of stop(), reads -999 and calls cv.notify_one()!
Then this call to notify_one() is done BEFORE we are wait()-ing on the condition variable! And the program would indefinitely dead-lock.
For this reason we should not be able to call notify_one() until we called wait(). Note that the power of a condition variable lies there in that it is able to atomically unlock the mutex, check if a call to notify_one() happened and go to sleep or not. You can't fool it, but you do need to keep the mutex locked whenever you make changes to variables that might change the condition from false to true and keep it locked while calling notify_one() because of race conditions like described here.
In this example there is no condition however. Why didn't I use as condition 'count == -1000'? Because that isn't interesting at all here: as soon as -1000 is reached at all, we are sure that no new thread will enter the 'Do stuff' area. Moreover, threads can still call start() and will increment count (to -999 and -998 etc) but we don't care about that. The only thing that matters is that -1000 was reached - so that we know for sure that there are no threads anymore in the 'Do stuff' area. We are sure that this is the case when notify_one() is being called, but how to make sure we don't call notify_one() before cancel() locked its mutex? Just locking cancel_mutex shortly prior to notify_one() isn't going to help of course.
The problem is that, despite that we're not waiting for a condition, there still is a condition, and we need to lock the mutex
1) before that condition is reached
2) before we call notify_one.
The correct code therefore becomes:
void stop()
{
if (count.fetch_sub(1) == -999) // Reached -1000 ?
{
cancel_mutex.lock();
cancel_mutex.unlock();
cv.notify_one();
}
}
[...same start()...]
void cancel()
{
std::unique_lock<std::mutex> lk(cancel_mutex);
if (count.fetch_sub(1000) == 0)
return;
cancel_cv.wait(lk);
}
Of course this is just one example but other cases are very much alike; in almost all cases where you use a conditional variable you will need to have that mutex locked (shortly) before calling notify_one(), or else it is possible that you call it before calling wait().
Note that I unlocked the mutex prior to calling notify_one() in this case, because otherwise there is the (small) chance that the call to notify_one() wakes up the thread waiting for the condition variable which then will try to take the mutex and block, before we release the mutex again. That's just slightly slower than needed.
This example was kinda special in that the line that changes the condition is executed by the same thread that calls wait().
More usual is the case where one thread simply wait's for a condition to become true and another thread takes the lock before changing the variables involved in that condition (causing it to possibly become true). In that case the mutex is locked immediately before (and after) the condition became true - so it is totally ok to just unlock the mutex before calling notify_*() in that case.
As I understand notify_one calls pthread_cond_signal.
If so, then what do think about this?
For predictable scheduling behavior and to prevent lost wake-ups, the mutex should be held when signaling a condition variable.
https://www.unix.com/man-page/hpux/3T/pthread_cond_signal/
All of the threads waiting on the condition variable are suspended until another thread uses the signal function:
pthread_cond_signal(&myConVar);
In this case the mutex has to be locked before calling the function and unlocked after it.
https://www.i-programmer.info/programming/cc/12288-fundamental-c-condition-variables.html
I personally had cases when notifications were missed because notify_one was called without locking the mutex.