I am a newcomer to the Boost library, and am trying to implement a simple producer and consumer threads that operate on a shared queue. My example implementation looks like this:
#include <iostream>
#include <deque>
#include <boost/thread.hpp>
boost::mutex mutex;
std::deque<std::string> queue;
void producer()
{
while (true) {
boost::lock_guard<boost::mutex> lock(mutex);
std::cout << "producer() pushing string onto queue" << std::endl;
queue.push_back(std::string("test"));
}
}
void consumer()
{
while (true) {
boost::lock_guard<boost::mutex> lock(mutex);
if (!queue.empty()) {
std::cout << "consumer() popped string " << queue.front() << " from queue" << std::endl;
queue.pop_front();
}
}
}
int main()
{
boost::thread producer_thread(producer);
boost::thread consumer_thread(consumer);
sleep(5);
producer_thread.detach();
consumer_thread.detach();
return 0;
}
This code runs as I expect, but when main exits, I get
/usr/include/boost/thread/pthread/mutex.hpp:45:
boost::mutex::~mutex(): Assertion `!pthread_mutex_destroy(&m)' failed.
consumer() popped string test from queue
Aborted
(I'm not sure if the output from consumer is relevant in that position, but I've left it in.)
Am I doing something wrong in my usage of Boost?
A bit off-topic but relevant imo (...waits for flames in comments).
The consumer model here is very greedy, looping and checking for data on the queue continually. It will be more efficient (waste less CPU cycles) if you have your consumer threads awakened determistically when data is available, using inter-thread signalling rather than this lock-and-peek loop. Think about it this way: while the queue is empty, this is essentially a tight loop only broken by the need to acquire the lock. Not ideal?
void consumer()
{
while (true) {
boost::lock_guard<boost::mutex> lock(mutex);
if (!queue.empty()) {
std::cout << "consumer() popped string " << queue.front() << " from queue" << std::endl;
queue.pop_front();
}
}
}
I understand that you are learning but I would not advise use of this in 'real' code. For learning the library though, it's fine. To your credit, this is a more complex example than necessary to understand how to use the lock_guard, so you are aiming high!
Eventually you will most likely build (or better if available, reuse) code for a queue that signals workers when they are required to do work, and you will then use the lock_guard inside your worker threads to mediate accesses to shared data.
You give your threads (producer & consumer) the mutex object and then detach them. They are supposed to run forever. Then you exit from your program and the mutex object is no longer valid. Nevertheless your threads still try to use it, they don't know that it is no longer valid. If you had used the NDEBUG define you would have got a coredump.
Are you trying to write a daemon application and this is the reason for detaching threads?
When main exits, all the global objects are destroyed. Your threads, however, do continue to run. You therefore end up with problems because the threads are accessing a deleted object.
Bottom line is that you must terminate the threads before exiting. The only what to do this though is to get the main program to wait (by using a boost::thread::join) until the threads have finished running. You may want to provide some way of signaling the threads to finish running to save from waiting too long.
The other issue is that your consumer thread continues to run even when there is not data. You might want to wait on a boost::condition_variable until signaled that there is new data.
Related
std::queue<double> some_q;
std::mutex mu_q;
/* an update function may be an event observer */
void UpdateFunc()
{
/* some other processing */
std::lock_guard lock{ mu_q };
while (!some_q.empty())
{
const auto& val = some_q.front();
/* update different states according to val */
some_q.pop();
}
/* some other processing */
}
/* some other thread might add some values after processing some other inputs */
void AddVal(...)
{
std::lock_guard lock{ mu_q };
some_q.push(...);
}
For this case is it okay to handle the queue this way?
Or would it be better if I try to use a lock-free queue like the boost one?
How bad it is to lock a mutex in an infinite loop or an update function
It's pretty bad. Infinite loops actually make your program have undefined behavior unless it does one of the following:
terminate
make a call to a library I/O function
perform an access through a volatile glvalue
perform a synchronization operation or an atomic operation
Acquiring the mutex lock before entering the loop and just holding it does not count as performing a synchronization operation (in the loop). Also, when holding the mutex, noone can add information to the queue, so while processing the information you extract, all threads wanting to add to the queue will have to wait - and no other worker threads wanting to share the load can extract from the queue either. It's usually better to extract one task from the queue, release the lock and then work with what you got.
The common way is to use a condition_variable that lets other threads acquire the lock and then notify other threads waiting with the same condition_variable. The CPU will be pretty close to idle while waiting and wake up to do the work when needed.
Using your program as a base, it could look like this:
#include <chrono>
#include <condition_variable>
#include <iostream>
#include <mutex>
#include <queue>
#include <thread>
std::queue<double> some_q;
std::mutex mu_q;
std::condition_variable cv_q; // the condition variable
bool stop_q = false; // something to signal the worker thread to quit
/* an update function may be an event observer */
void UpdateFunc() {
while(true) {
double val;
{
std::unique_lock lock{mu_q};
// cv_q.wait lets others acquire the lock to work with the queue
// while it waits to be notified.
while (not stop_q && some_q.empty()) cv_q.wait(lock);
if(stop_q) break; // time to quit
val = std::move(some_q.front());
some_q.pop();
} // lock released so others can use the queue
// do time consuming work with "val" here
std::cout << "got " << val << '\n';
}
}
/* some other thread might add some values after processing some other inputs */
void AddVal(double val) {
std::lock_guard lock{mu_q};
some_q.push(val);
cv_q.notify_one(); // notify someone that there's a new value to work with
}
void StopQ() { // a function to set the queue in shutdown mode
std::lock_guard lock{mu_q};
stop_q = true;
cv_q.notify_all(); // notify all that it's time to stop
}
int main() {
auto th = std::thread(UpdateFunc);
// simulate some events coming with some time apart
std::this_thread::sleep_for(std::chrono::seconds(1));
AddVal(1.2);
std::this_thread::sleep_for(std::chrono::seconds(1));
AddVal(3.4);
std::this_thread::sleep_for(std::chrono::seconds(1));
AddVal(5.6);
std::this_thread::sleep_for(std::chrono::seconds(1));
StopQ();
th.join();
}
If you really want to process everything that is currently in the queue, then extract everything first and then release the lock, then work with what you extracted. Extracting everything from the queue is done quickly by just swapping in another std::queue. Example:
#include <atomic>
std::atomic<bool> stop_q{}; // needs to be atomic in this version
void UpdateFunc() {
while(not stop_q) {
std::queue<double> work; // this will be used to swap with some_q
{
std::unique_lock lock{mu_q};
// cv_q.wait lets others acquire the lock to work with the queue
// while it waits to be notified.
while (not stop_q && some_q.empty()) cv_q.wait(lock);
std::swap(work, some_q); // extract everything from the queue at once
} // lock released so others can use the queue
// do time consuming work here
while(not stop_q && not work.empty()) {
auto val = std::move(work.front());
work.pop();
std::cout << "got " << val << '\n';
}
}
}
You can use it like you currently are assuming proper use of the lock across all threads. However, you may run into some frustrations about how you want to call updateFunc().
Are you going to be using a callback?
Are you going to be using an ISR?
Are you going to be polling?
If you use a 3rd party lib it often trivializes thread synchronization and queues
For example, if you are using a CMSIS RTOS(v2). It is a fairly straight forward process to get multiple threads to pass information between each other. You could have multiple producers, and a single consumer.
The single consumer can wait in a forever loop where it waits to receive a message before performing its work
when timeout is set to osWaitForever the function will wait for an
infinite time until the message is retrieved (i.e. wait semantics).
// Two producers
osMessageQueuePut(X,Y,Z,timeout=0)
osMessageQueuePut(X,Y,Z,timeout=0)
// One consumer which will run only once something enters the queue
osMessageQueueGet(X,Y,Z,osWaitForever)
tldr; You are safe to proceed, but using a library will likely make your synchronization problems easier.
I am a beginner in C++11 multithreading. I am working with small codes and came into this problem. Here is the code:
#include <iostream>
#include <thread>
#include <vector>
#include <mutex>
std::mutex print_mutex;
void function1()
{
std::cout << "Thread1 started" << std::endl;
while (true)
{
std::unique_lock<std::mutex> lock(print_mutex);
for (size_t i = 0; i<= 1000000000; i++)
continue;
std::cout << "This is function1" << std::endl;
lock.unlock();
}
}
void function2()
{
std::cout << "Thread2 started" << std::endl;
while (true)
{
std::unique_lock<std::mutex> lock(print_mutex);
for (size_t i = 0; i <= 1000000000; i++)
continue;
std::cout << "This is function2" << std::endl;
lock.unlock();
}
}
int main()
{
std::thread t1(function1);
std::thread t2(function2);
t1.join();
t2.join();
return 0;
}
I have written code with the intuition of expecting the following output:
Thread1 started Thread2 started This is
function1 This is function2 This is function1 . . . .
But the output shown is as follows:
Thread1 started Thread2 started
This is function1 This is function1 This is
function1 . . .
Where am I going wrong?
Unlocking a mutex does not guarantee that another thread that's waiting to lock the same mutex will immediately acquire a lock.
It only guarantees that the other thread will TRY to acquire the lock.
In this case, after you unlock the mutex in one thread, the same thread will immediately try to lock it again. Even though another thread was waiting patiently, for the mutex, it's not a guarantee that the other thread will win this time. The same thread that just locked it can succeed in immediately locking it again.
Today, you're seeing that the same thread always wins the locking race. Tomorrow, you may find that it's always the other thread that does. You have no guarantees, whatsoever, which thread will acquire the mutex when there's more than one thread going after the same mutex, at the same time. The winner depends on your CPU and other hardware architecture, how busy the system is loaded, at the time, and many other factors.
Both of your thread is doing following steps:
Lock
Long empty loop
Print
Unlock
Lock
Long empty loop
(and so on)
Practically, you haven't left any time for context switching, there is a lock just right after the unlock. Solution: Swap the "lock" and the "long empty loop" steps, so only the "print" step will be locked, the scheduler can switch to the other thread during "long empty loop".
Welcome to threads!
Edit: Pro Tipp: Debugging multithreading programs is hard. But sometimes it's worth to insert a plain printf() to indicate locks and unlocks (the right order: lock, then printf and printf then unlock), even when the program seems correct. In this case you could see the zero gap between unlock-lock.
This is a valid result, your code does not try to control the execution order in any way so as long as all threads execute at some point and there's is no problem and it's a legitimate result.
This could happen even if you switched the order of the loop and the lock(see here), because again you haven't written anything that attempts to control it using e.g conditional variables or just some silly atomic_bool(it is a silly solution just to demonstrate how can you actually make it alternating and be sure it will) boolean to alternate the runs.
I need to parallelize some tasks in a C++ program and am completely new to parallel programming. I've made some progress through internet searches so far, but am a bit stuck now. I'd like to reuse some threads in a loop, but clearly don't know how to do what I'm trying for.
I am acquiring data from two ADC cards on the computer (acquired in parallel), then I need to perform some operations on the collected data (processed in parallel) while collecting the next batch of data. Here is some pseudocode to illustrate
//Acquire some data, wait for all the data to be acquired before proceeding
std::thread acq1(AcquireData, boardHandle1, memoryAddress1a);
std::thread acq2(AcquireData, boardHandle2, memoryAddress2a);
acq1.join();
acq2.join();
while(user doesn't interrupt)
{
//Process first batch of data while acquiring new data
std::thread proc1(ProcessData,memoryAddress1a);
std::thread proc2(ProcessData,memoryAddress2a);
acq1(AcquireData, boardHandle1, memoryAddress1b);
acq2(AcquireData, boardHandle2, memoryAddress2b);
acq1.join();
acq2.join();
proc1.join();
proc2.join();
/*Proceed in this manner, alternating which memory address
is written to and being processed until the user interrupts the program.*/
}
That's the main gist of it. The next run of the loop would write to the "a" memory addresses while processing the "b" data and continue to alternate (I can get the code to do that, just took it out to prevent cluttering up the problem).
Anyway, the problem (as I'm sure some people can already tell) is that the second time I try to use acq1 and acq2, the compiler (VS2012) says "IntelliSense: call of an object of a class type without appropriate operator() or conversion functions to pointer-to-function type". Likewise, if I put std::thread in front of acq1 and acq2 again, it says " error C2374: 'acq1' : redefinition; multiple initialization".
So the question is, can I reassign threads to a new task when they have completed their previous task? I always wait for the previous use of the thread to end before calling it again, but I don't know how to reassign the thread, and since it's in a loop, I can't make a new thread each time (or if I could, that seems wasteful and unnecessary, but I could be mistaken).
Thanks in advance
The easiest way is to use a waitable queue of std::function objects. Like this:
#include <iostream>
#include <thread>
#include <mutex>
#include <condition_variable>
#include <queue>
#include <functional>
#include <chrono>
class ThreadPool
{
public:
ThreadPool (int threads) : shutdown_ (false)
{
// Create the specified number of threads
threads_.reserve (threads);
for (int i = 0; i < threads; ++i)
threads_.emplace_back (std::bind (&ThreadPool::threadEntry, this, i));
}
~ThreadPool ()
{
{
// Unblock any threads and tell them to stop
std::unique_lock <std::mutex> l (lock_);
shutdown_ = true;
condVar_.notify_all();
}
// Wait for all threads to stop
std::cerr << "Joining threads" << std::endl;
for (auto& thread : threads_)
thread.join();
}
void doJob (std::function <void (void)> func)
{
// Place a job on the queu and unblock a thread
std::unique_lock <std::mutex> l (lock_);
jobs_.emplace (std::move (func));
condVar_.notify_one();
}
protected:
void threadEntry (int i)
{
std::function <void (void)> job;
while (1)
{
{
std::unique_lock <std::mutex> l (lock_);
while (! shutdown_ && jobs_.empty())
condVar_.wait (l);
if (jobs_.empty ())
{
// No jobs to do and we are shutting down
std::cerr << "Thread " << i << " terminates" << std::endl;
return;
}
std::cerr << "Thread " << i << " does a job" << std::endl;
job = std::move (jobs_.front ());
jobs_.pop();
}
// Do the job without holding any locks
job ();
}
}
std::mutex lock_;
std::condition_variable condVar_;
bool shutdown_;
std::queue <std::function <void (void)>> jobs_;
std::vector <std::thread> threads_;
};
void silly (int n)
{
// A silly job for demonstration purposes
std::cerr << "Sleeping for " << n << " seconds" << std::endl;
std::this_thread::sleep_for (std::chrono::seconds (n));
}
int main()
{
// Create two threads
ThreadPool p (2);
// Assign them 4 jobs
p.doJob (std::bind (silly, 1));
p.doJob (std::bind (silly, 2));
p.doJob (std::bind (silly, 3));
p.doJob (std::bind (silly, 4));
}
The std::thread class is designed to execute exactly one task (the one you give it in the constructor) and then end. If you want to do more work, you'll need a new thread. As of C++11, that's all we have. Thread pools didn't make it into the standard. (I'm uncertain what C++14 has to say about them.)
Fortunately, you can easily implement the required logic yourself. Here is the large-scale picture:
Start n worker threads that all do the following:
Repeat while there is more work to do:
Grab the next task t (possibly waiting until one becomes ready).
Process t.
Keep inserting new tasks in the processing queue.
Tell the worker threads that there is nothing more to do.
Wait for the worker threads to finish.
The most difficult part here (which is still fairly easy) is properly designing the work queue. Usually, a synchronized linked list (from the STL) will do for this. Synchronized means that any thread that wishes to manipulate the queue must only do so after it has acquired a std::mutex so to avoid race conditions. If a worker thread finds the list empty, it has to wait until there is some work again. You can use a std::condition_variable for this. Each time a new task is inserted into the queue, the inserting thread notifies a thread that waits on the condition variable and will therefore stop blocking and eventually start processing the new task.
The second not-so-trivial part is how to signal to the worker threads that there is no more work to do. Clearly, you can set some global flag but if a worker is blocked waiting at the queue, it won't realize any time soon. One solution could be to notify_all() threads and have them check the flag each time they are notified. Another option is to insert some distinct “toxic” item into the queue. If a worker encounters this item, it quits itself.
Representing a queue of tasks is straight-forward using your self-defined task objects or simply lambdas.
All of the above are C++11 features. If you are stuck with an earlier version, you'll need to resort to third-party libraries that provide multi-threading for your particular platform.
While none of this is rocket science, it is still easy to get wrong the first time. And unfortunately, concurrency-related bugs are among the most difficult to debug. Starting by spending a few hours reading through the relevant sections of a good book or working through a tutorial can quickly pay off.
This
std::thread acq1(...)
is the call of an constructor. constructing a new object called acq1
This
acq1(...)
is the application of the () operator on the existing object aqc1. If there isn't such a operator defined for std::thread the compiler complains.
As far as I know you may not reused std::threads. You construct and start them. Join with them and throw them away,
Well, it depends if you consider moving a reassigning or not. You can move a thread but not make a copy of it.
Below code will create new pair of threads each iteration and move them in place of old threads. I imagine this should work, because new thread objects will be temporaries.
while(user doesn't interrupt)
{
//Process first batch of data while acquiring new data
std::thread proc1(ProcessData,memoryAddress1a);
std::thread proc2(ProcessData,memoryAddress2a);
acq1 = std::thread(AcquireData, boardHandle1, memoryAddress1b);
acq2 = std::thread(AcquireData, boardHandle2, memoryAddress2b);
acq1.join();
acq2.join();
proc1.join();
proc2.join();
/*Proceed in this manner, alternating which memory address
is written to and being processed until the user interrupts the program.*/
}
What's going on is, the object actually does not end it's lifetime at the end of the iteration, because it is declared in the outer scope in regard to the loop. But a new object gets created each time and move takes place. I don't see what can be spared (I might be stupid), so I imagine this it's exactly the same as declaring acqs inside the loop and simply reusing the symbol. All in all ... yea, it's about how you classify a create temporary and move.
Also, this clearly starts a new thread each loop (of course ending the previously assigned thread), it doesn't make a thread wait for new data and magically feed it to the processing pipe. You would need to implement it a differently like. E.g: Worker threads pool and communication over queues.
References: operator=, (ctor).
I think the errors you get are self-explanatory, so I'll skip explaining them.
I think you need a much more simpler answer for running a set of threads more than once, this is the best solution:
do{
std::vector<std::thread> thread_vector;
for (int i=0;i<nworkers;i++)
{
thread_vector.push_back(std::thread(yourFunction,Parameter1,Parameter2, ...));
}
for(std::thread& it: thread_vector)
{
it.join();
}
q++;
} while(q<NTIMES);
You also could make your own Thread class and call its run method like:
class MyThread
{
public:
void run(std::function<void()> func) {
thread_ = std::thread(func);
}
void join() {
if(thread_.joinable())
thread_.join();
}
private:
std::thread thread_;
};
// Application code...
MyThread myThread;
myThread.run(AcquireData);
I was checking out the boost library(version 1.45) for a reader/writer lock. When I ran my tests on it, it seemed like the shared_ptr was preferring my reader threads, i.e. when my writer tried to take the lock for its operation, it didn't stop any subsequent reads from occurring.
Is it possible in boost to change this behavior?
using namespace std;
using namespace boost;
mutex outLock;
shared_mutex workerAccess;
bool shouldIWork = true;
class WorkerKiller
{
public:
void operator()()
{
upgrade_lock<shared_mutex> lock(workerAccess);
upgrade_to_unique_lock<shared_mutex> uniqueLock(lock);
cout << "Grabbed exclusive lock, killing system" << endl;
sleep(2);
shouldIWork = false;
cout << "KILLING ALL WORK" << endl;
}
private:
};
class Worker
{
public:
Worker()
{
}
void operator()()
{
shared_lock<shared_mutex> lock(workerAccess);
if (!shouldIWork) {
outLock.lock();
cout << "Workers are on strike. This worker refuses to work" << endl;
outLock.unlock();
} else {
sleep(1);
outLock.lock();
cout << "Worked finished her work" << endl;
outLock.unlock();
}
}
};
int main(int argc, char* argv[])
{
Worker w1;
Worker w2;
Worker w3;
Worker w4;
WorkerKiller wk;
boost::thread workerThread1(w1);
boost::thread workerThread2(w2);
boost::thread workerKillerThread(wk);
boost::thread workerThread3(w3);
boost::thread workerThread4(w4);
workerThread1.join();
workerThread2.join();
workerKillerThread.join();
workerThread3.join();
return 0;
}
And here is the output every time:
Worked finished her work
Worked finished her work
Worked finished her work
Worked finished her work
Grabbed exclusive lock, killing system
KILLING ALL WORK
My Requirement
If the writer tried to grab an exclusive lock, I'd like for all previous read operations to finish. And then all subsequent read operations to block.
I'm a little late to this question, but I believe I have some pertinent information.
The proposals of shared_mutex to the C++ committee, which the boost libs are based on, purposefully did not specify an API to give readers nor writers priority. This is because Alexander Terekhov proposed an algorithm about a decade ago that is completely fair. It lets the operating system decide whether the next thread to get the mutex is a reader or writer, and the operating system is completely ignorant as to whether the next thread is a reader or writer.
Because of this algorithm, the need for specifying whether a reader or writer is preferred disappears. To the best of my knowledge, the boost libs are now (boost 1.52) implemented with this fair algorithm.
The Terekhov algorithm consists of having the read/write mutex consist of two gates: gate1 and gate2. Only one thread at a time can pass through each gate. The gates can be implemented with a mutex and two condition variables.
Both readers and writers attempt to pass through gate1. In order to pass through gate1 it must be true that a writer thread is not currently inside of gate1. If there is, the thread attempting to pass through gate1 blocks.
Once a reader thread passes through gate1 it has read ownership of the mutex.
When a writer thread passes through gate1 it must also pass through gate2 before obtaining write ownership of the mutex. It can not pass through gate2 until the number of readers inside of gate1 drops to zero.
This is a fair algorithm because when there are only 0 or more readers inside of gate1, it is up to the OS as to whether the next thread to get inside of gate1 is a reader or writer. A writer becomes "prioritized" only after it has passed through gate1, and is thus next in line to obtain ownership of the mutex.
I used your example compiled against an example implementation of what eventually became shared_timed_mutex in C++14 (with minor modifications to your example). The code below calls it shared_mutex which is the name it had when it was proposed.
I got the following outputs (all with the same executable):
Sometimes:
Worked finished her work
Worked finished her work
Grabbed exclusive lock, killing system
KILLING ALL WORK
Workers are on strike. This worker refuses to work
Workers are on strike. This worker refuses to work
And sometimes:
Worked finished her work
Grabbed exclusive lock, killing system
KILLING ALL WORK
Workers are on strike. This worker refuses to work
Workers are on strike. This worker refuses to work
Workers are on strike. This worker refuses to work
And sometimes:
Worked finished her work
Worked finished her work
Worked finished her work
Worked finished her work
Grabbed exclusive lock, killing system
KILLING ALL WORK
I believe it should be theoretically possible to also obtain other outputs, though I did not confirm that experimentally.
In the interest of full disclosure, here is the exact code I executed:
#include "../mutexes/shared_mutex"
#include <thread>
#include <chrono>
#include <iostream>
using namespace std;
using namespace ting;
mutex outLock;
shared_mutex workerAccess;
bool shouldIWork = true;
class WorkerKiller
{
public:
void operator()()
{
unique_lock<shared_mutex> lock(workerAccess);
cout << "Grabbed exclusive lock, killing system" << endl;
this_thread::sleep_for(chrono::seconds(2));
shouldIWork = false;
cout << "KILLING ALL WORK" << endl;
}
private:
};
class Worker
{
public:
Worker()
{
}
void operator()()
{
shared_lock<shared_mutex> lock(workerAccess);
if (!shouldIWork) {
lock_guard<mutex> _(outLock);
cout << "Workers are on strike. This worker refuses to work" << endl;
} else {
this_thread::sleep_for(chrono::seconds(1));
lock_guard<mutex> _(outLock);
cout << "Worked finished her work" << endl;
}
}
};
int main()
{
Worker w1;
Worker w2;
Worker w3;
Worker w4;
WorkerKiller wk;
thread workerThread1(w1);
thread workerThread2(w2);
thread workerKillerThread(wk);
thread workerThread3(w3);
thread workerThread4(w4);
workerThread1.join();
workerThread2.join();
workerKillerThread.join();
workerThread3.join();
workerThread4.join();
return 0;
}
A google search of "boost shared lock starvation" turned up this link:
Example for boost shared_mutex (multiple reads/one write)?
It looks like "upgrade" might be the key. See also:
Example of how to use boost upgradeable mutexes
http://HowardHinnant.github.io/shared_mutex
I'm using a common base class has_threads to manage any type that should be allowed to instantiate a boost::thread.
Instances of has_threads each own a set of threads (to support waitAll and interruptAll functions, which I do not include below), and should automatically invoke removeThread when a thread terminates to maintain this set's integrity.
In my program, I have just one of these. Threads are created on an interval every 10s, and each performs a database lookup. When the lookup is complete, the thread runs to completion and removeThread should be invoked; with a mutex set, the thread object is removed from internal tracking. I can see this working properly with the output ABC.
Once in a while, though, the mechanisms collide. removeThread is executed perhaps twice concurrently. What I can't figure out is why this results in a deadlock. All thread invocations from this point never output anything other than A. [It's worth noting that I'm using thread-safe stdlib, and that the issue remains when IOStreams are not used.] Stack traces indicate that the mutex is locking these threads, but why would the lock not be eventually released by the first thread for the second, then the second for the third, and so on?
Am I missing something fundamental about how scoped_lock works? Is there anything obvious here that I've missed that could lead to a deadlock, despite (or even due to?) the use of a mutex lock?
Sorry for the poor question, but as I'm sure you're aware it's nigh-on impossible to present real testcases for bugs like this.
class has_threads {
protected:
template <typename Callable>
void createThread(Callable f, bool allowSignals)
{
boost::mutex::scoped_lock l(threads_lock);
// Create and run thread
boost::shared_ptr<boost::thread> t(new boost::thread());
// Track thread
threads.insert(t);
// Run thread (do this after inserting the thread for tracking so that we're ready for the on-exit handler)
*t = boost::thread(&has_threads::runThread<Callable>, this, f, allowSignals);
}
private:
/**
* Entrypoint function for a thread.
* Sets up the on-end handler then invokes the user-provided worker function.
*/
template <typename Callable>
void runThread(Callable f, bool allowSignals)
{
boost::this_thread::at_thread_exit(
boost::bind(
&has_threads::releaseThread,
this,
boost::this_thread::get_id()
)
);
if (!allowSignals)
blockSignalsInThisThread();
try {
f();
}
catch (boost::thread_interrupted& e) {
// Yes, we should catch this exception!
// Letting it bubble over is _potentially_ dangerous:
// http://stackoverflow.com/questions/6375121
std::cout << "Thread " << boost::this_thread::get_id() << " interrupted (and ended)." << std::endl;
}
catch (std::exception& e) {
std::cout << "Exception caught from thread " << boost::this_thread::get_id() << ": " << e.what() << std::endl;
}
catch (...) {
std::cout << "Unknown exception caught from thread " << boost::this_thread::get_id() << std::endl;
}
}
void has_threads::releaseThread(boost::thread::id thread_id)
{
std::cout << "A";
boost::mutex::scoped_lock l(threads_lock);
std::cout << "B";
for (threads_t::iterator it = threads.begin(), end = threads.end(); it != end; ++it) {
if ((*it)->get_id() != thread_id)
continue;
threads.erase(it);
break;
}
std::cout << "C";
}
void blockSignalsInThisThread()
{
sigset_t signal_set;
sigemptyset(&signal_set);
sigaddset(&signal_set, SIGINT);
sigaddset(&signal_set, SIGTERM);
sigaddset(&signal_set, SIGHUP);
sigaddset(&signal_set, SIGPIPE); // http://www.unixguide.net/network/socketfaq/2.19.shtml
pthread_sigmask(SIG_BLOCK, &signal_set, NULL);
}
typedef std::set<boost::shared_ptr<boost::thread> > threads_t;
threads_t threads;
boost::mutex threads_lock;
};
struct some_component : has_threads {
some_component() {
// set a scheduler to invoke createThread(bind(&some_work, this)) every 10s
}
void some_work() {
// usually pretty quick, but I guess sometimes it could take >= 10s
}
};
Well, a deadlock might occurs if the same thread lock a mutex it has already locked (unless you use a recursive mutex).
If the release part is called a second time by the same thread as it seems to happen with your code, you have a deadlock.
I have not studied your code in details, but you probably have to re-design your code (simplify ?) to be sure that a lock can not be acquired twice by the same thread. You can probably use a safeguard checking for the ownership of the lock ...
EDIT:
As said in my comment and in IronMensan answer, one possible case is that the thread stop during creation, the at_exit being called before the release of the mutex locked in the creation part of your code.
EDIT2:
Well, with mutex and scoped lock, I can only imagine a recursive lock, or a lock that is not released. It can happen if a loop goes to infinite due to a memory corruption for instance.
I suggest to add more logs with a thread id to check if there is a recursive lock or something strange. Then I will check that my loop is correct. I will also check that the at_exit is only called once per thread ...
One more thing, check the effect of erasing (thus calling the destructor) of a thread while being in the at_exit function...
my 2 cents
You may need to do something like this:
void createThread(Callable f, bool allowSignals)
{
// Create and run thread
boost::shared_ptr<boost::thread> t(new boost::thread());
{
boost::mutex::scoped_lock l(threads_lock);
// Track thread
threads.insert(t);
}
//Do not hold threads_lock while starting the new thread in case
//it completes immediately
// Run thread (do this after inserting the thread for tracking so that we're ready for the on-exit handler)
*t = boost::thread(&has_threads::runThread<Callable>, this, f, allowSignals);
}
In other words, use thread_lock exclusively to protect threads.
Update:
To expand on something in the comments with speculation about how boost::thread works, the lock patterns could look something like this:
createThread:
(createThread) obtain threads_lock
(boost::thread::opeator =) obtain a boost::thread internal lock
(boost::thread::opeator =) release a boost::thread internal lock
(createThread) release threads_lock
thread end handler:
(at_thread_exit) obtain a boost::thread internal lock
(releaseThread) obtain threads_lock
(releaseThread) release threads_lock
(at_thread_exit) release a boost:thread internal lock
If those two boost::thread locks are the same lock, the potential for deadlock is clear. But this is speculation because much of the boost code scares me and I try not to look at it.
createThread could/should be reworked to move step 4 up between steps one and two and eliminate the potential deadlock.
It is possible that the created thread is finishing before or during the assignment operator in createThread is complete. Using an event queue or some other structure that is might be necessary. Though a simpler, though hack-ish, solution might work as well. Don't change createThread since you have to use threads_lock to protect threads itself and the thread objects it points to. Instead change runThread to this:
template <typename Callable>
void runThread(Callable f, bool allowSignals)
{
//SNIP setup
try {
f();
}
//SNIP catch blocks
//ensure that createThread is complete before this thread terminates
boost::mutex::scoped_lock l(threads_lock);
}