Poring through legacy code of old and large project, I had found that there was used some odd method of creating thread-safe queue, something like this:
template < typename _Msg>
class WaitQue: public QWaitCondition
{
public:
typedef _Msg DataType;
void wakeOne(const DataType& msg)
{
QMutexLocker lock_(&mx);
que.push(msg);
QWaitCondition::wakeOne();
}
void wait(DataType& msg)
{
/// wait if empty.
{
QMutex wx; // WHAT?
QMutexLocker cvlock_(&wx);
if (que.empty())
QWaitCondition::wait(&wx);
}
{
QMutexLocker _wlock(&mx);
msg = que.front();
que.pop();
}
}
unsigned long size() {
QMutexLocker lock_(&mx);
return que.size();
}
private:
std::queue<DataType> que;
QMutex mx;
};
wakeOne is used from threads as kind of "posting" function" and wait is called from other threads and waits indefinitely until a message appears in queue. In some cases roles between threads reverse at different stages and using separate queues.
Is this even legal way to use a QMutex by creating local one? I kind of understand why someone could do that to dodge deadlock while reading size of que but how it even works? Is there a simpler and more idiomatic way to achieve this behavior?
Its legal to have a local condition variable. But it normally makes no sense.
As you've worked out in this case is wrong. You should be using the member:
void wait(DataType& msg)
{
QMutexLocker cvlock_(&mx);
while (que.empty())
QWaitCondition::wait(&mx);
msg = que.front();
que.pop();
}
Notice also that you must have while instead of if around the call to QWaitCondition::wait. This is for complex reasons about (possible) spurious wake up - the Qt docs aren't clear here. But more importantly the fact that the wake and the subsequent reacquire of the mutex is not an atomic operation means you must recheck the variable queue for emptiness. It could be this last case where you previously were getting deadlocks/UB.
Consider the scenario of an empty queue and a caller (thread 1) to wait into QWaitCondition::wait. This thread blocks. Then thread 2 comes along and adds an item to the queue and calls wakeOne. Thread 1 gets woken up and tries to reacquire the mutex. However, thread 3 comes along in your implementation of wait, takes the mutex before thread 1, sees the queue isn't empty, processes the single item and moves on, releasing the mutex. Then thread 1 which has been woken up finally acquires the mutex, returns from QWaitCondition::wait and tries to process... an empty queue. Yikes.
Related
I have a WorkDispatcher class which holds Worker class objects as properties and launches their function in new threads.
Here is a example:
WorkDispatcher.h:
class WorkDispatcher
{
private:
std::thread m_reconstructionThread;
std::shared_ptr <Reconstruction::RGBDImageModel> m_currentRGBD;
public:
WorkDispatcher();
std::mutex m_rgbdMutex, m_commandMutex;
std::deque<Network::NetworkCommandModel> m_CommandQueue;
std::condition_variable m_RgbConditional, m_CommandConditional;
Reconstruction::SceneReconstructor m_Reconstructor;
void Initialize();
void Work();
void Stop();
};
WorkDispatcher.cpp:
void WorkDispatcher::Work()
{
m_reconstructionThread = std::thread(
&Reconstruction::SceneReconstructor::Reconstruct,
std::ref(m_Reconstructor),
std::ref(m_currentRGBD),
std::ref(m_CommandQueue),
std::ref(m_rgbdMutex),
std::ref(m_RgbConditional),
std::ref(m_commandMutex),
std::ref(m_CommandConditional)
);
}
These functions hold infinite loops and I use the condition variables to wait until work is avaible. For example my Reconstruct function:
void SceneReconstructor::Reconstruct(
std::shared_ptr<RGBDImageModel>& currentImage,
std::deque<Network::NetworkCommandModel> commandQueue,
std::mutex& rgbdMutex,
std::condition_variable& rgbdCond,
std::mutex& commandMutex,
std::condition_variable& commandConditional)
{
while (true)
{
std::unique_lock<std::mutex> rgbdLocker(rgbdMutex);
rgbdCond.wait(rgbdLocker, [this] {return m_QuitReconstructionFlag; });
// Quit flag to break out of loop
if (m_QuitReconstructionFlag)
break;
// do stuff here
}
}
So far so good, however if I want to quit the application I need to quit all of my worker threads. As seen above, for this these classes have a flag to quit, which I uses as follows:
void WorkDispatcher::Stop()
{
// for all worker threads do this
m_Reconstructor.m_QuitReconstructionFlag = true;
if (m_reconstructionThread.joinable())
m_reconstructionThread.join();
}
In theory this should stop the wait() function within a worker threads loop and then break out of the loop with the m_QuitReconstructionFlag, however this doesn't work.
What does work is the following:
remove the lambda from the wait functions
call notify_all() on the condition variables after settings the
quit-flags to true;
This works fine for me, however the question is, why doesn't the lambda work?
why doesn't the lambda work?
It works just fine, by itself.
However, C++ requires complex rules to be followed to properly synchronize multiple execution threads. Just because one execution thread sets a particular variable to a value does not guarantee you, in any way, that other execution threads will see the variable's new value. The synchronization rules govern that behavior.
So, this lambda works just fine. In its own execution thread. But if you want this lambda to observe changes to the value, made by other execution threads, this must be correctly synchronized.
Additionally, if you review your documentation of wait(), you should find a detailed explanation that says that if the condition function evaluates to false, it will not be called again until the condition variable is notified.
What does work is ... call notify_all()
Well, of course. Since wait() requires the condition variable to be notified, before it checks the waited-on condition again, then that's what you must do!
Finally, notifying the condition variable will work correctly in most cases, but, as I mentioned, synchronization rules (of which mutexes and condition variables play an important part of) have some edge cases where this, by itself, will not work. You must follow the following sequence of events strictly in order to have proper synchronization in all edge cases:
Lock the same mutex that another execution thread has locked before waiting on its condition variable.
Notify the condition variable.
Unlock the mutex
You must protect m_QuitReconstructionFlag with the same mutex used by the condition variable wait.
Or it won't work.
When using a condition variable if you do not want to learn about the C++ memory model in extreme detail, you should follow "best practices" that defend you against problems.
The best practices for a condition variable is to bundle up 3 things together.
The condition variable.
A mutex (often mutable).
A state.
Then bundle all 3 of them up behind a single abstraction of some kind.
To change the state:
Lock the mutex
Change the state
Notify the condition variable appropriately
Unlock the mutex
Do not think that the state being atomic means you don't have to lock the mutex.
When you want to wait on the condition variable:
Lock the mutex
Wait, passing a lambda that checks the state.
When exiting wait, you are free to update the state.
Unlock the mutex
In general, use a unique_lock to lock the mutex in all of the above cases, and rely on RAII to unlock it.
What, exactly, the state is, and when you notify, is up to you.
Do not interact with that mutex directly outside of this bundle and api, don't interact with the state directly outside of this bundle and api, and don't interact with the condition variable outside of this bundle and api.
Copy or move data out of the api if needed, don't hold pointers or references or iterators into it.
Your state can have more than just one variable in it. Like, you can have a queue and a bool for shutdown.
For example, suppose you have a queue.
template<class T>
struct cv_queue {
std::optional<T> pop() {
auto l = lock();
cv.wait( l, [&]{ return aborted || !queue.empty(); } );
if (aborted) return {};
auto retval = std::move(queue.front());
queue.pop_front();
return retval;
}
void push( T in ) {
auto l = lock();
queue.push_back( std::move(in) );
cv.notify_one();
}
void abort_everything() {
auto l = lock();
abort = true;
cv.notify_all();
}
bool empty() const {
auto l = lock();
return queue.empty();
}
private:
std::condition_variable cv;
mutable std::mutex m;
std::deque<T> queue;
bool aborted=false;
auto lock() const { return std::unique_lock( m ); }
};
adding pop_wait_for or try_pop isn't hard.
A simple 3-part wrapper around data or whatever isn't hard to write. Making it more generic, in my experience, doesn't add much to it being understandable.
Here the lambda returning true is not the condition to stop waiting, rather the lambda is to account for spurious wake ups. The notify or notify_all function of the conditional_variable is what is used to make the wait quit.
Rather than removing the lambda, you must simply change the stop function to
void WorkDispatcher::Stop()
{
// for all worker threads do this
m_Reconstructor.m_QuitReconstructionFlag = true;
m_RgbConditional.notify_all()
if (m_reconstructionThread.joinable())
m_reconstructionThread.join();
}
from here you can see that the wait with predicate passed to it (wait(predicate)) is equivalent to
if(!predicate())
wait()
Hence when you call Stop(), It sets the predicate to true, so when the thread is woken up wait() returns, and the predicate is checked, if it is true, the wait(predicate) returns.
In the earlier case, the predicate was set to true but the function was not woken up
Since I have recently started coding multi threaded programs this might be a stupid question. I found out about the awesome mutex and condition variable usage. From as far as I can understand there use is:
Protect sections of code/shared resources from getting corrupted by multiple threads access. Hence lock that portion thus one can control which thread will be accessing.
If a thread is waiting for a resource/condition from another thread one can use cond.wait() instead of polling every msec
Now Consider the following class example:
class Queue {
private:
std::queue<std::string> m_queue;
boost::mutex m_mutex;
boost::condition_variable m_cond;
bool m_exit;
public:
Queue()
: m_queue()
, m_mutex()
, m_cond()
, m_exit(false)
{}
void Enqueue(const std::string& Req)
{
boost::mutex::scoped_lock lock(m_mutex);
m_queue.push(Req);
m_cond.notify_all();
}
std::string Dequeue()
{
boost::mutex::scoped_lock lock(m_mutex);
while(m_queue.empty() && !m_exit)
{
m_cond.wait(lock);
}
if (m_queue.empty() && m_exit) return "";
std::string val = m_queue.front();
m_queue.pop();
return val;
}
void Exit()
{
boost::mutex::scoped_lock lock(m_mutex);
m_exit = true;
m_cond.notify_all();
}
}
In the above example, Exit() can be called and it will notify the threads waiting on Dequeue that it's time to exit without waiting for more data in the queue.
My question is since Dequeue has acquired the lock(m_mutex), how can Exit acquire the same lock(m_mutex)? Isn't unless the Dequeue releases the lock then only Exit can acquire it?
I have seen this pattern in Destructor implementation too, using same class member mutex, Destructor notifies all the threads(class methods) thats it time to terminate their respective loops/functions etc.
As Jarod mentions in the comments, the call
m_cond.wait(lock)
is guaranteed to atomically unlock the mutex, releasing it for the thread, and starts listening to notifications of the condition variable (see e.g. here).
This atomicity also ensures any code in the thread is executed after the listening is set up (so no notify calls will be missed). This assumes of course that the thread first locks the mutex, otherwise all bets are off.
Another important bit to understand is that condition variables may suffer from "spurious wakeups", so it is important to have a second boolean condition (e.g. here, you could check the emptiness of your queue) so that you don't end up awoken with an empty queue. Something like this:
m_cond.wait(lock, [this]() { return !m_queue.empty() || m_exit; });
I would like to write a class that wraps around std::thread and behaves like a std::thread but without actually allocating a thread every time I need to process something async. The reason is that I need to use multi threading in a context where I'm not allow to dynamically allocate and I also don't want to have the overhead of creating a std::thread.
Instead, I want a thread to run in a loop and wait until it can start processing. The client calls invoke which wakes up the thread. The Thread locks a mutex, does it's processing and falls asleep again. A function join behaves like std::thread::join by locking until the thread frees the lock (i.e. falls asleep again).
I think I got the class to run but because of a general lack of experience in multi threading, I would like to ask if anybody can spot race conditions or if the approach I used is considered "good style". For example, I'm not sure if temporary locking the mutex is a decent way to "join" the thread.
EDIT
I found another race condition: when calling join directly after invoke, there is no reason the thread already locked the mutex and thus locks the caller of join until the thread goes to sleep. To prevent this, I had to add a check for the invoke counter.
Header
#pragma once
#include <thread>
#include <atomic>
#include <mutex>
class PersistentThread
{
public:
PersistentThread();
~PersistentThread();
// set function to invoke
// locks if thread is currently processing _func
void set(const std::function<void()> &f);
// wakes the thread up to process _func and fall asleep again
// locks if thread is currently processing _func
void invoke();
// mimics std::thread::join
// locks until the thread is finished with it's loop
void join();
private:
// intern thread loop
void loop(bool *initialized);
private:
bool _shutdownRequested{ false };
std::mutex _mutex;
std::unique_ptr<std::thread> _thread;
std::condition_variable _cond;
std::function<void()> _func{ nullptr };
};
Source File
#include "PersistentThread.h"
PersistentThread::PersistentThread()
{
auto lock = std::unique_lock<std::mutex>(_mutex);
bool initialized = false;
_thread = std::make_unique<std::thread>(&PersistentThread::loop, this, &initialized);
// wait until _thread notifies, check bool initialized to prevent spurious wakeups
_cond.wait(lock, [&] {return initialized; });
}
PersistentThread::~PersistentThread()
{
{
std::lock_guard<std::mutex> lock(_mutex);
_func = nullptr;
_shutdownRequested = true;
// wake up and let join
_cond.notify_one();
}
// join thread,
if (_thread->joinable())
{
_thread->join();
}
}
void PersistentThread::set(const std::function<void()>& f)
{
std::lock_guard<std::mutex> lock(_mutex);
this->_func = f;
}
void PersistentThread::invoke()
{
std::lock_guard<std::mutex> lock(_mutex);
_cond.notify_one();
}
void PersistentThread::join()
{
bool joined = false;
while (!joined)
{
std::lock_guard<std::mutex> lock(_mutex);
joined = (_invokeCounter == 0);
}
}
void PersistentThread::loop(bool *initialized)
{
std::unique_lock<std::mutex> lock(_mutex);
*initialized = true;
_cond.notify_one();
while (true)
{
// wait until we get the mutex again
_cond.wait(lock, [this] {return _shutdownRequested || (this->_invokeCounter > 0); });
// shut down if requested
if (_shutdownRequested) return;
// process
if (_func) _func();
_invokeCounter--;
}
}
You are asking about potential race conditions, and I see at least one race condition in the shown code.
After constructing a PersistentThread, there is no guarantee that the new thread will acquire its initial lock in its loop() before the main execution thread returns from the constructor and enters invoke(). It is possible that the main execution thread enters invoke() immediately after the constructor is complete, ends up notifying nobody, since the internal execution thread hasn't locked the mutex yet. As such, this invoke() will not result in any processing taking place.
You need to synchronize the completion of the constructor with the execution thread's initial lock acquisition.
EDIT: your revision looks right; but I also spotted another race condition.
As documented in the description of wait(), wait() may wake up "spuriously". Just because wait() returned, doesn't mean that some other thread has entered invoke().
You need a counter, in addition to everything else, with invoke() incrementing the counter, and the execution thread executing its assigned duties only when the counter is greater than zero, decrementing it. This will guard against spurious wake-ups.
I would also have the execution thread check the counter before entering wait(), and enter wait() only if it is 0. Otherwise, it decrements the counter, executes its function, and loops back.
This should plug up all the potential race conditions in this area.
P.S. The spurious wake-up also applies to the initial notification, in your correction, that the execution thread has entered the loop. You'll need to do something similar for that situation, too.
I don't understand what you're trying to ask exactly. It's a nice style you used.
It would be much safer using bools and check the single routines because void returns nothing so you could be maybe stuck caused by bugs. Check everything you can since the thread runs under the hood. Make sure the calls are running correctly, if the process had really success. Also you could read some stuff about "Thread Pooling".
I am a bit stuck with the problem, so it is my cry for help.
I have a manager that pushes some events to a queue, which is proceeded in another thread.
I don't want this thread to be 'busy waiting' for events in the queue, because it may be empty all the time (as well as it may always be full).
Also I need m_bShutdownFlag to stop the thread when needed.
So I wanted to try a condition_variable for this case: if something was pushed to a queue, then the thread starts its work.
Simplified code:
class SomeManager {
public:
SomeManager::SomeManager()
: m_bShutdownFlag(false) {}
void SomeManager::Initialize() {
boost::recursive_mutex::scoped_lock lock(m_mtxThread);
boost::thread thread(&SomeManager::ThreadProc, this);
m_thread.swap(thread);
}
void SomeManager::Shutdown() {
boost::recursive_mutex::scoped_lock lock(m_mtxThread);
if (m_thread.get_id() != boost::thread::id()) {
boost::lock_guard<boost::mutex> lockEvents(m_mtxEvents);
m_bShutdownFlag = true;
m_condEvents.notify_one();
m_queue.clear();
}
}
void SomeManager::QueueEvent(const SomeEvent& event) {
boost::lock_guard<boost::mutex> lockEvents(m_mtxEvents);
m_queue.push_back(event);
m_condEvents.notify_one();
}
private:
void SomeManager::ThreadProc(SomeManager* pMgr) {
while (true) {
boost::unique_lock<boost::mutex> lockEvents(pMgr->m_mtxEvents);
while (!(pMgr->m_bShutdownFlag || pMgr->m_queue.empty()))
pMgr->m_condEvents.wait(lockEvents);
if (pMgr->m_bShutdownFlag)
break;
else
/* Thread-safe processing of all the events in m_queue */
}
}
boost::thread m_thread;
boost::recursive_mutex m_mtxThread;
bool m_bShutdownFlag;
boost::mutex m_mtxEvents;
boost::condition_variable m_condEvents;
SomeThreadSafeQueue m_queue;
}
But when I test it with two (or more) almost simultaneous calls to QueueEvent, it gets locked at the line boost::lock_guard<boost::mutex> lockEvents(m_mtxEvents); forever.
Seems like the first call doesn't ever release lockEvents, so all the rest just keep waiting for its freeing.
Please, help me to find out what am I doing wrong and how to fix this.
There's a few things to point out on your code:
You may wish to join your thread after calling shutdown, to ensure that your main thread doesn't finish before your other thread.
m_queue.clear(); on shutdown is done outside of your m_mtxEvents mutex lock, meaning it's not as thread safe as you think it is.
your 'thread safe processing' of the queue should be just taking an item off and then releasing the lock while you go off to process the event. You've not shown that explicitly, but failure to do so will result in the lock preventing items from being added.
The good news about a thread blocking like this, is that you can trivially break and inspect what the other threads are doing, and locate the one that is holding the lock. It might be that as per my comment #3 you're just taking a long time to process an event. On the other hand it may be that you've got a dead lock. In any case, what you need is to use your debugger to establish exactly what you've done wrong, since your sample doesn't have enough in it to demonstrate your problem.
inside ThreadProc, while(ture) loop, the lockEvents is not unlocked in any case. try put lock and wait inside a scope.
Conditional variables use a mutex and the .wait() function unlocks the
mutex so another thread can access the shared data. When the condition
variable is notified it tries to lock the mutex again to use the shared
data.
This pattern is used in the following concurrent_queue example from Anthony Williams:
template<typename Data>
class concurrent_queue
{
private:
boost::condition_variable the_condition_variable;
public:
void wait_for_data()
{
boost::mutex::scoped_lock lock(the_mutex);
while(the_queue.empty())
{
the_condition_variable.wait(lock);
}
}
void push(Data const& data)
{
boost::mutex::scoped_lock lock(the_mutex);
bool const was_empty=the_queue.empty();
the_queue.push(data);
if(was_empty)
{
the_condition_variable.notify_one();
}
}
};
Since the code uses std::queue it's clear that the mutex has to be
locked when accessing the queue.
But let's say instead of std::queue one uses Microsofts
Concurrency::concurrent_queue from PPL. Member functions like empty,
push and try_pop are thread safe. Do I still need to lock a mutex in
this case or can the condition variable be used like this, without
creating any possible race conditions.
My code (that seems to work, but what does that mean in multithreading?) looks like this. I have one producer that pushes items into Microsofts concurrent_queue and one background thread that waits for new items in this queue.
The consumer/background thread:
while(runFlag) //atomic
{
while(the_queue.empty() && runFlag) //wait only when thread should still run
{
boost::mutex mtx; //local mutex thats locked afterwards. awkward.
boost::mutex::scoped_lock lock(mtx);
condition.wait(lock);
}
Data d;
while(!the_queue.empty() && the_queue.try_pop(d))
{
//process data
}
}
The producer/main thread:
const bool was_empty = the_queue.empty();
Data d;
the_queue.push(d);
if(was_empty) cond_var.notify_one();
The shutdown procedure:
bool expected_run_state = true;
if(runLoop.compare_exchange_strong(expected_run_state, false))
{
//atomically set our loop flag to false and
//notify all clients of the queue to wake up and exit
cond_var.notify_all();
}
As said above this code seems to work but that doesn't necessarily mean it's correct. Especially the local mutex that is only used because the condition variable interface forces me to use a mutex, seems like a very bad idea. I wanted to use condition variables since the time between data items added to the queue hard to predict and I would have to create to sleep and wake up periodically like this:
if(the_queue.empty()) Sleep(short_amount_of_time);
Are there any other, maybe OS (in my case: Windows) specific tools, that make a background thread sleep until some condition is met without regularly waking up and checking the condition?
The code is not correct in different scenarios, for example. If the queue has a single element when const bool was_empty = the_queue.empty(); is evaluated, but a thread consumes the element and a different thread tries to consume and waits on the condition, the writer will not notify that thread after inserting the element in the queue.
The key issue is that the fact that all of the operations in an interface are thread safe does not necessarily mean that your use of the interface is safe. If you depend on multiple operations being performed atomically, you need to provide synchronization mechanisms externally.
Are there any other, maybe OS (in my case: Windows) specific tools,
that make a background thread sleep until some condition is met
without regularly waking up and checking the condition?
This is exactly what Events are for
But if you are targeting only Windows platform (Vista+) you should check out
Slim Reader/Writer (SRW) Locks