Use of C++ condition variables with atomic predicate but no mutex - c++

I have two threads. One thread acts as a timer thread which at regular intervals of time needs to send a notification to another thread. I intend to use C++ condition variables. (There is a good article on how to use C++ condition variables along with its traps and pitfalls in the following link)
I have the following constraints/conditions :-
The notifying thread need not lock on to a mutex
The notified (or the receiver) thread does some useful section but there is no critical section
The receiver thread is allowed to miss a notification if and only if it is doing useful work
There should be no spurious wakeups.
Using the above link as a guideline I put together the following piece of code
// conditionVariableAtomic.cpp
#include <atomic>
#include <condition_variable>
#include <iostream>
#include <thread>
#include <iostream> // std::cout, std::endl
#include <thread> // std::this_thread::sleep_for
#include <chrono> // std::chrono::seconds
std::mutex mutex_;
std::condition_variable condVar;
std::atomic<bool> dataReady{false};
void waitingForWork(){
int i = 0;
while (i++ < 10)
{
std::cout << "Waiting " << std::endl;
{
std::unique_lock<std::mutex> lck(mutex_);
condVar.wait(lck, []{ return dataReady.load(); }); // (1)
dataReady = false;
}
std::cout << "Running " << std::endl;
// Do useful work but no critical section.
}
}
void setDataReady(){
int i = 0;
while (i++ < 10)
{
std::this_thread::sleep_for (std::chrono::seconds(1));
dataReady = true;
std::cout << "Data prepared" << std::endl;
condVar.notify_one();
}
}
int main(){
std::cout << std::endl;
std::thread t1(waitingForWork);
std::thread t2(setDataReady);
t1.join();
t2.join();
std::cout << std::endl;
}
I use an atomic predicate to avoid spurious wakeups, but don't use a lock_guard in the notifying thread.
My question is:
does the above piece of code satisfy the constraints/conditions listed above?
I understand that the receiver thread cannot avoid a mutex, hence the need to use std::unique_lock<std::mutex> lck(mutex_); in the receiver. I have however limited the scope of std::unique_lock<std::mutex> lck(mutex_); i.e. put the following section of code
std::unique_lock<std::mutex> lck(mutex_);
condVar.wait(lck, []{ return dataReady.load(); }); // (1)
dataReady = false;
inside a scope block aka { .... } so that the mutex is unlocked as soon as the wait condition is over (the receiver then does some useful work but since there is no critical section, it does not need to hold on to the mutex for the entire while loop). Could there still be consequences/side effects of this limited scoping in this context ? Or does the unique_lock<std::mutex> need to be locked for the entire while loop?

Your code has a race condition. Between checking the value of dataReady in your wait predicate and actually starting the wait, the other thread can set dataReady and call notify_one. In your example this isn't critical as you'll just miss one notify and wake up a second later on the next one.
Another race condition is that you can set dataReady to true in one thread, set dataReady back to false in the other thread and then call notify_one in the first thread, again this will cause the wait to block for longer than you intended.
You should hold the mutex in both threads when setting dataReady and using the condition variable to avoid these races.
You could avoid the second race condition by using an atomic counter instead of a boolean, incrementing it on one thread then decrementing on the other and in the predicate checking if it is non-zero.

Related

Minimal mutexes for std::queue producer/consumer

I have two threads that work the producer and consumer sides of a std::queue. The queue isn't often full, so I'd like to avoid the consumer grabbing the mutex that is guarding mutating the queue.
Is it okay to call empty() outside the mutex then only grab the mutex if there is something in the queue?
For example:
struct MyData{
int a;
int b;
};
class SpeedyAccess{
public:
void AddDataFromThread1(MyData data){
const std::lock_guard<std::mutex> queueMutexLock(queueAccess);
workQueue.push(data);
}
void CheckFromThread2(){
if(!workQueue.empty()) // Un-protected access...is this dangerous?
{
queueAccess.lock();
MyData data = workQueue.front();
workQueue.pop();
queueAccess.unlock();
ExpensiveComputation(data);
}
}
private:
void ExpensiveComputation(MyData& data);
std::queue<MyData> workQueue;
std::mutex queueAccess;
}
Thread 2 does the check and isn't particularly time-critical, but will get called a lot (500/sec?). Thread 1 is very time critical, a lot of stuff needs to run there, but isn't called as frequently (max 20/sec).
If I add a mutex guard around empty(), if the queue is empty when thread 2 comes, it won't hold the mutex for long, so might not be a big hit. However, since it gets called so frequently, it might occasionally happen at the same time something is trying to get put on the back....will this cause a substantial amount of waiting in thread 1?
As written in the comments above, you should call empty() only under a lock.
But I believe there is a better way to do it.
You can use a std::condition_variable together with a std::mutex, to achieve synchronization of access to the queue, without locking the mutex more than you must.
However - when using std::condition_variable, you must be aware that it suffers from spurious wakeups. You can read about it here: Spurious wakeup - Wikipedia.
You can see some code examples here:
Condition variable examples.
The correct way to use a std::condition_variable is demonstrated below (with some comments).
This is just a minimal example to show the principle.
#include <thread>
#include <mutex>
#include <condition_variable>
#include <queue>
#include <iostream>
using MyData = int;
std::mutex mtx;
std::condition_variable cond_var;
std::queue<MyData> q;
void producer()
{
MyData produced_val = 0;
while (true)
{
std::this_thread::sleep_for(std::chrono::milliseconds(1000)); // simulate some pause between productions
++produced_val;
std::cout << "produced: " << produced_val << std::endl;
{
// Access the Q under the lock:
std::unique_lock<std::mutex> lck(mtx);
q.push(produced_val);
cond_var.notify_all(); // It's not a must to nofity under the lock but it might be more efficient (see #DavidSchwartz's comment below).
}
}
}
void consumer()
{
while (true)
{
MyData consumed_val;
{
// Access the Q under the lock:
std::unique_lock<std::mutex> lck(mtx);
// NOTE: The following call will lock the mutex only when the the condition_varible will cause wakeup
// (due to `notify` or spurious wakeup).
// Then it will check if the Q is empty.
// If empty it will release the lock and continue to wait.
// If not empty, the lock will be kept until out of scope.
// See the documentation for std::condition_variable.
cond_var.wait(lck, []() { return !q.empty(); }); // will loop internally to handle spurious wakeups
consumed_val = q.front();
q.pop();
}
std::cout << "consumed: " << consumed_val << std::endl;
std::this_thread::sleep_for(std::chrono::milliseconds(200)); // simulate some calculation
}
}
int main()
{
std::thread p(producer);
std::thread c(consumer);
while(true) {}
p.join(); c.join(); // will never happen in our case but to remind us what is needed.
return 0;
}
Some notes:
In your real code, none of the threads should run forever. You should have some mechanism to notify them to gracefully exit.
The global variables (mtx,q etc.) are better to be members of some context class, or passed to the producer() and consumer() as parameters.
This example assumes for simplicity that the producer's production rate is always low relatively to the consumer's rate. In your real code you can make it more general, by making the consumer extract all elements in the Q each time the condition_variable is signaled.
You can "play" with the sleep_for times for the producer and consumer to test varios timing cases.

Can we use two different mutex when waiting on same conditional variable?

Consider below scenario:
Thread 1
mutexLk1_
gcondVar_.wait(mutexLk1);
Thread 2
mutexLk2_
gcondVar_.wait(mutexLk2);
Thread 3
condVar_
gcondVar_.notify_all();
What I observe is that notify_all() does not wake up both the threads but just one on the two. If i were to replace mutexLk2 with mutexLk1. I get a functional code.
To reproduce the issue consider below modified example from cppref
#include <iostream>
#include <condition_variable>
#include <thread>
#include <chrono>
std::condition_variable cv;
std::mutex cv_m1;
std::mutex cv_m; // This mutex is used for three purposes:
// 1) to synchronize accesses to i
// 2) to synchronize accesses to std::cerr
// 3) for the condition variable cv
int i = 0;
void waits1()
{
std::unique_lock<std::mutex> lk(cv_m);
std::cerr << "Waiting... \n";
cv.wait(lk, []{return i == 1;});
std::cerr << "...finished waiting. waits1 i == 1\n";
}
void waits()
{
std::unique_lock<std::mutex> lk(cv_m1);
std::cerr << "Waiting... \n";
cv.wait(lk, []{return i == 1;});
std::cerr << "...finished waiting. i == 1\n";
}
void signals()
{
std::this_thread::sleep_for(std::chrono::seconds(1));
{
std::lock_guard<std::mutex> lk(cv_m);
std::cerr << "Notifying...\n";
}
cv.notify_all();
std::this_thread::sleep_for(std::chrono::seconds(1));
{
std::lock_guard<std::mutex> lk(cv_m);
i = 1;
std::cerr << "Notifying again...\n";
}
cv.notify_all();
}
int main()
{
std::thread t1(waits), t2(waits1), t3(waits), t4(signals);
t1.join();
t2.join();
t3.join();
t4.join();
}
Compilation command
g++ --std=c++17 t.cpp -lpthread
Interesting thing here is that the above code gets stuck on either of the waits (sometimes waits1 runs sometime waits) on a centos 7.9 System with g++ 9.3 version (same behavior with g++ 10 on this system) but with a ubuntu 18.04 system (with g++ 9.4) this works without any issues
Any idea what is the expected behaviour or ideal practice to follow? Because in my used case I need two different mutex to protect different data structures but the trigger is from same conditional variable.
Thanks
It seems you violate standad:
33.5.3 Class condition_variable [thread.condition.condvar]
void wait(unique_lock& lock);
Requires: lock.owns_lock() is true and lock.mutex() is locked by the
calling thread, and either
(9.1) — no other thread is waiting on this condition_variable object
or
(9.2) — lock.mutex() returns the same value for each of the lock
arguments supplied by all concurrently waiting (via wait, wait_for, or
wait_until) threads.
Cleary two threads are waiting and lock.mutex() does not return same.
To elaborate on my comment, there are three things that require synchronization in the question.
There are two independent hunks of data, which the question says are each properly synchronized with two independent mutexes. That's fine, but it's irrelevant.
The third synchronization issue is ensuring that some other specified condition is satisfied in order for some threads to run. That's implemented with a condition variable; so far, so good. But when more than one thread waits for a condition variable, all the waiting threads must use the same mutex. So the code needs a mutex for the threads to use when they wait. While it might be possible to use one or another of the existing mutexes for that synchronization, it seems far more likely that the code needs another mutex, specifically for use when waiting on that condition variable.
When you're dealing with synchronization, resist the urge to share things that manage synchronization; that can get you into trouble.

C++ scoped lock in loop blocks another thread

Simple example:
#include <iostream>
#include <thread>
#include <mutex>
std::mutex lock_m;
void childTh() {
while(true) {
//std::this_thread::yield();
//std::this_thread::sleep_for(std::chrono::milliseconds(1));
std::unique_lock<std::mutex> lockChild(lock_m);
std::cout << "childTh CPN1" << std::endl;
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
}
int main(int, char**) {
std::thread thr(childTh);
std::this_thread::sleep_for(std::chrono::milliseconds(200));
std::unique_lock<std::mutex> lockMain(lock_m);
std::cout << "MainTh CPN1" << std::endl;
std::this_thread::sleep_for(std::chrono::seconds(10));
return 0;
}
Main thread blocks on lockMain and never reach "MainTh CPN1". I expect that main thread should acquire lock_m when childTh reach end of iteration because lockChild is destroyed and lock_m is released. But this never happens.
Can you please describe in details why main thread don't have time to acquire the lock before childTh lock it again ?
With sleep_for main can reach "MainTh CPN1", but with yield not.
I know that condition_variable can be used to notify and unblock another thread, but is it possible to use just scoped lock ? So it looks that it is risky to use scoped lock in different threads, even if it the same lock.
In childTh, lockChild doesn't release the mutex until the iteration ends. Right after that iteration ends, it starts the next one. This means you only have the time between the destruction of lockChild and then the initialization of lockChild in the next iteration. Since that happens as basically the next instruction, there basically isn't any time for lockMain to acquire a lock on the mutex. To save CPU cycles a typical lock acquire is going to yield for a short duration, which is not as short as single instruction, so there is basically no chance of lockMain being able to lock the mutex as it would have to be timed perfectly. If you change childTh to
void childTh() {
while(true) {
//std::this_thread::yield();
//std::this_thread::sleep_for(std::chrono::milliseconds(1));
{
std::unique_lock<std::mutex> lockChild(lock_m);
std::cout << "childTh CPN1" << std::endl;
}
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
}
now you have a 1 second delay between when the mutex is release by lockChild and when it reacquired in the next iteration, which then allows lockMain to acquire the mutex.
Also note that you are not calling join on thr at the end of main. Not doing so causes thr's destructor to throw an exception which will cause your program to terminate imporperly.

Conditional variable is stuck on waiting

Why the condition variable is stuck on waiting if it was notified in worker_thread? What am I missing here?
#include <thread>
#include <mutex>
#include <condition_variable>
#include <iostream>
std::mutex m;
std::condition_variable cv;
void worker_thread()
{
cv.notify_one();
}
int main()
{
std::thread worker(worker_thread);
std::cout << "Start waiting..." << std::endl;
std::unique_lock<std::mutex> lk(m);
cv.wait(lk);
std::cout << "Finished waiting..." << std::endl;
worker.join();
getchar();
}
Your problem is that cv.notify_one() only wakes threads that are currently waiting. cv doesn't remember you notified it, and someone later comes along and waits.
Your worker thread is outpacing your main thread. So the notify happens before the main thread.
This is just a symptom of your real problem; you are using a condition variable wrong. Barring extremely advanced use, all use of condition variable should be in a triple.
A std::condition_variable.
A std::mutex.
A payload.
Your code is missing the payload.
To signal, you:
std::unique_lock<std::mutex> l(m);
payload = /* move it to a 'set' or 'non-empty' state */;
cv.notify_one(); // or all
to listen you:
std::unique_lock<std::mutex> l(m);
cv.wait(l, [&]{ return /* payload is in a set or non-empty state */; });
// while locked, consume one "unit" of payload from the payload.
with minor variations for wait_for and the like.
Following this cargo-cult pattern is important, as it avoids a number of pitfalls. It deals with both spurious wakeups with the wait happening after the notification.
Your code is missing a payload. So your code is vulnerable to both the waiting thread outrunning the signaling thread, and spurious wakeups.
Note that getting "clever" here is highly discouraged. For example, deciding that "I'll use an atomic variable to avoid using a mutex when signaling" actually doesn't work. Either follow the above recipe dogmatically, or go and spend a few months learning the threading and memory model of C++ well enough to improvise.
notify_one will unblock a waiting thread if there is one. If there are no waiting threads, nothing happens. A condition_variable does not have a state to remember how many threads should be notified when it is waited on.

Destroying condition_variable while waiting

First my code to make my explanation more clear:
struct Foo {
std::condition_variable cv;
};
static Foo* foo; // dynamically created object
// Thread1
foo->cv.wait(...);
// Thread2
foo->cv.notify();
delete foo; // thread1 might have not left the wait function yet
I am trying to delete a std::condition_variable while it is in wait. So from my understanding, I have to notify it first to make it leave it's waiting for routine, and then I can delete it. But after calling notify* I can't delete it right away, because it might still be in wait because it needs a few cycles. What is a common way to achieve this?
You can delete it right away.
Quote from C++ standard:
~ condition_variable();
Requires: There shall be no thread blocked on *this. [Note: That is,
all threads shall have been notified; they may subsequently block on
the lock specified in the wait. This relaxes the usual rules, which
would have required all wait calls to happen before destruction. Only
the notification to unblock the wait must happen before destruction.
Basically wait functions are required to perform locking and waiting atomically:
The execution of notify_one and notify_all shall be atomic. The execution
of wait, wait_for, and wait_until shall be performed in three atomic parts:
the release of the mutex and entry into the waiting state;
the unblocking of the wait; and
the reacquisition of the lock.
Once notify wakes a thread, it should be considered "unblocked" and should contest the mutex.
There are similar guarantees about std::mutex: threads are not required to leave unlock before mutex is destroyed.
Quote from C++ standard:
The implementation shall provide lock and unlock operations, as
described below. For purposes of determining the existence of a data
race, these behave as atomic operations. The lock and unlock
operations on a single mutex shall appear to occur in a single total
order.
Later:
Note: After a thread A has called unlock(), releasing a mutex, it is
possible for another thread B to lock the same mutex, observe that it
is no longer in use, unlock it, and destroy it, before thread A
appears to have returned from its unlock call.
Such guarantees are required to avoid issues like this, when mutex inside an object is used to protect object reference counter.
Note that this does not guarantee that your implementation has no bugs in this regard. In the past glibc had multiple bugs related to the destruction of synchronization objects, in particular pthread_mutex_unlock was accessing mutex before returning.
One easy fix: move delete foo into thread 1 after foo->cv.wait(...);.
A better fix would be to change the design to work with std::shared_ptr, no manual delete invocations.
Here is my solution for C++17 (C++11 needs adaptation): notify everybody that we are deleting current instance and make the destructor to wait
Two things to do:
Give to the wait predicate a possibility to exit before object is really deleted
Make the destructor to resync with the lock to make it wait everything is finished (you must be sure all waiting methods are checking the deleting at beginning)
Note: if you have a thread inside this class waiting on the condition_variable, it might be better to join() the thread after notification instead of using the resynchronization lock (see comment in the code)
#include <iostream>
#include <memory>
#include <string>
#include <thread>
#include <condition_variable>
#include <mutex>
#include <shared_mutex>
using namespace std;
chrono::system_clock::time_point startTime;
struct Foo
{
condition_variable cond;
mutex mutex1;
shared_mutex mutexSafe;
bool deleting = false;
~Foo()
{
deleting = true;
cond.notify_all();
// Will make the destructor to wait all the opened shared_lock are released
unique_lock l(mutexSafe);
}
void waitOnThing(const string& name)
{
// Shared lock to make possible several threads are using the method at the same time
shared_lock lSafe(mutexSafe);
cout << chrono::duration_cast<chrono::milliseconds>(chrono::system_clock::now() - startTime).count()
<< " Thread " << name << " -> waitOnThing()" << endl;
unique_lock l(mutex1);
cond.wait(l, [&]()
{
if (deleting)
{
this_thread::sleep_for(chrono::milliseconds(1000)); // Slow down exit process to show destructor is waiting
return true;
}
return false;
});
cout << chrono::duration_cast<chrono::milliseconds>(chrono::system_clock::now() - startTime).count()
<< " Thread " << name << " unlocked" << endl;
}
};
int main()
{
startTime = chrono::system_clock::now();
Foo* foo = new Foo();
cout << chrono::duration_cast<chrono::milliseconds>(chrono::system_clock::now() - startTime).count()
<< " Starting" << endl;
thread t1([&]() { foo->waitOnThing("t1"); });
thread t2([&]() { foo->waitOnThing("t2"); });
thread t3([&]() { foo->waitOnThing("t3"); });
// Wait a bit to be sure thread started and is waiting
this_thread::sleep_for(chrono::milliseconds(100));
cout << chrono::duration_cast<chrono::milliseconds>(chrono::system_clock::now() - startTime).count()
<< " Deleting foo..." << endl;
delete foo;
cout << chrono::duration_cast<chrono::milliseconds>(chrono::system_clock::now() - startTime).count()
<< " Foo deleted" << endl;
// Avoid demo to crash
t1.join();
t2.join();
t3.join();
}
Result:
0 Starting
4 Thread t2 -> waitOnThing()
4 Thread t1 -> waitOnThing()
4 Thread t3 -> waitOnThing()
100 Deleting foo...
1100 Thread t1 unlocked
2100 Thread t2 unlocked
3100 Thread t3 unlocked
3100 Foo deleted
C++11 :
Class shared_mutex is available only since C++17, if you are using C++11 you can do the same by using a regular unique_lock
and making a vector of mutex (one instance per call to the waiting method) and try to lock all of them in the destructor.
Example (not tested):
vector<shared_ptr<mutex>> mutexesSafe;
~Foo()
{
// ...
for(const auto& m : mutexesSafe)
{
unique_lock l(m);
}
}
void waitOnThing(const string& name)
{
auto m = make_shared<mutex>();
mutexesSafe.push_back(m);
unique_lock lSafe(*m);
//...
}