Waking up multiple threads to work once per condition - c++

Question

I have a situation where one thread needs to occasionally wake up a number of worker threads and each worker thread needs to do it's work (only) once and then go back to sleep to wait for the next notification. I'm using a condition_variable to wake everything up, but the problem I'm having is the "only once" part. Assume that each thread is heavy to create, so I don't want to just be creating and joining them each time.
// g++ -Wall -o threadtest -pthread threadtest.cpp
#include <iostream>
#include <condition_variable>
#include <mutex>
#include <thread>
#include <chrono>
std::mutex condMutex;
std::condition_variable condVar;
bool dataReady = false;
void state_change_worker(int id)
{
while (1)
{
{
std::unique_lock<std::mutex> lck(condMutex);
condVar.wait(lck, [] { return dataReady; });
// Do work only once.
std::cout << "thread " << id << " working\n";
}
}
}
int main()
{
// Create some worker threads.
std::thread threads[5];
for (int i = 0; i < 5; ++i)
threads[i] = std::thread(state_change_worker, i);
while (1)
{
// Signal to the worker threads to work.
{
std::cout << "Notifying threads.\n";
std::unique_lock<std::mutex> lck(condMutex);
dataReady = true;
condVar.notify_all();
}
// It would be really great if I could wait() on all of the
// worker threads being done with their work here, but it's
// not strictly necessary.
std::cout << "Sleep for a bit.\n";
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
}
Update: Here is a version implementing an almost-but-not-quite working version of a squad lock. The problem is that I can't guarantee that each thread will have a chance to wake up and derement count in waitForLeader() before one runs through again.
// g++ -Wall -o threadtest -pthread threadtest.cpp
#include <iostream>
#include <condition_variable>
#include <mutex>
#include <thread>
#include <chrono>
class SquadLock
{
public:
void waitForLeader()
{
{
// Increment count to show that we are waiting in queue.
// Also, if we are the thread that reached the target, signal
// to the leader that everything is ready.
std::unique_lock<std::mutex> count_lock(count_mutex_);
std::unique_lock<std::mutex> target_lock(target_mutex_);
if (++count_ >= target_)
count_cond_.notify_one();
}
// Wait for leader to signal done.
std::unique_lock<std::mutex> lck(done_mutex_);
done_cond_.wait(lck, [&] { return done_; });
{
// Decrement count to show that we are no longer waiting.
// If we are the last thread set done to false.
std::unique_lock<std::mutex> lck(count_mutex_);
if (--count_ == 0)
{
done_ = false;
}
}
}
void waitForHerd()
{
std::unique_lock<std::mutex> lck(count_mutex_);
count_cond_.wait(lck, [&] { return count_ >= target_; });
}
void leaderDone()
{
std::unique_lock<std::mutex> lck(done_mutex_);
done_ = true;
done_cond_.notify_all();
}
void incrementTarget()
{
std::unique_lock<std::mutex> lck(target_mutex_);
++target_;
}
void decrementTarget()
{
std::unique_lock<std::mutex> lck(target_mutex_);
--target_;
}
void setTarget(int target)
{
std::unique_lock<std::mutex> lck(target_mutex_);
target_ = target;
}
private:
// Condition variable to indicate that the leader is done.
std::mutex done_mutex_;
std::condition_variable done_cond_;
bool done_ = false;
// Count of currently waiting tasks.
std::mutex count_mutex_;
std::condition_variable count_cond_;
int count_ = 0;
// Target number of tasks ready for the leader.
std::mutex target_mutex_;
int target_ = 0;
};
SquadLock squad_lock;
std::mutex print_mutex;
void state_change_worker(int id)
{
while (1)
{
// Wait for the leader to signal that we are ready to work.
squad_lock.waitForLeader();
{
// Adding just a bit of sleep here makes it so that every thread wakes up, but that isn't the right way.
// std::this_thread::sleep_for(std::chrono::milliseconds(100));
std::unique_lock<std::mutex> lck(print_mutex);
std::cout << "thread " << id << " working\n";
}
}
}
int main()
{
// Create some worker threads and increment target for each one
// since we want to wait until all threads are finished.
std::thread threads[5];
for (int i = 0; i < 5; ++i)
{
squad_lock.incrementTarget();
threads[i] = std::thread(state_change_worker, i);
}
while (1)
{
// Signal to the worker threads to work.
std::cout << "Starting threads.\n";
squad_lock.leaderDone();
// Wait for the worked threads to be done.
squad_lock.waitForHerd();
// Wait until next time, processing results.
std::cout << "Tasks done, waiting for next time.\n";
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
}

Answer 1

Following is an excerpt from a blog I created concerning concurrent design patterns. The patterns are expressed using the Ada language, but the concepts are translatable to C++.
Summary
Many applications are constructed of groups of cooperating threads of execution. Historically this has frequently been accomplished by creating a group of cooperating processes. Those processes would cooperate by sharing data. At first, only files were used to share data. File sharing presents some interesting problems. If one process is writing to the file while another process reads from the file you will frequently encounter data corruption because the reading process may attempt to read data before the writing process has completely written the information. The solution used for this was to create file locks, so that only one process at a time could open the file. Unix introduced the concept of a Pipe, which is effectively a queue of data. One process can write to a pipe while another reads from the pipe. The operating system treats data in a pipe as a series of bytes. It does not let the reading process access a particular byte of data until the writing process has completed its operation on the data.
Various operating systems also introduced other mechanisms allowing processes to share data. Examples include message queues, sockets, and shared memory. There were also special features to help programmers control access to data, such as semaphores. When operating systems introduced the ability for a single process to operate multiple threads of execution, also known as lightweight threads, or just threads, they also had to provide corresponding locking mechanisms for shared data.
Experience shows that, while the variety of possible designs for shared data is quite large, there are a few very common design patterns that frequently emerge. Specifically, there are a few variations on a lock or semaphore, as well as a few variations on data buffering. This paper explores the locking and buffering design patterns for threads in the context of a monitor. Although monitors can be implemented in many languages, all examples in this paper are presented using Ada protected types. Ada protected types are a very thorough implementation of a monitor.
Monitors
There are several theoretical approaches to creating and controlling shared memory. One of the most flexible and robust is the monitor as first described by C.A.R. Hoare. A monitor is a data object with three different kinds of operations.
Procedures are used to change the state or values contained by the monitor. When a thread calls a monitor procedure that thread must have exclusive access to the monitor to prevent other threads from encountering corrupted or partially written data.
Entries, like procedures, are used to change the state or values contained by the monitor, but an entry also specifies a boundary condition. The entry may only be executed when the boundary condition is true. Threads that call an entry when the boundary condition is false are placed in a queue until the boundary condition becomes true. Entries are used, for example, to allow a thread to read from a shared buffer. The reading thread is not allowed to read the data until the buffer actually contains some data. The boundary condition would be that the buffer must not be empty. Entries, like procedures, must have exclusive access to the monitor's data.
Functions are used to report the state of a monitor. Since functions only report state, and do not change state, they do not need exclusive access to the monitor's data. Many threads may simultaneously access the same monitor through functions without danger of data corruption.
The concept of a monitor is extremely powerful. It can also be extremely efficient. Monitors provide all the capabilities needed to design efficient and robust shared data structures for threaded systems.
Although monitors are powerful, they do have some limitations. The operations performed on a monitor should be very fast, with no chance of making a thread block. If those operations should block, the monitor will become a road block instead of a communication tool. All the threads awaiting access to the monitor will be blocked as long as the monitor operation is blocked. For this reason, some people choose not to use monitors. There are design patterns for monitors that can actually be used to work around these problems. Those design patterns are grouped together as locking patterns.
Squad Locks
A squad lock allows a special task (the squad leader) to monitor the progress of a herd or group of worker tasks. When all (or a sufficient number) of the worker tasks are done with some aspect of their work, and the leader is ready to proceed, the entire set of tasks is allowed to pass a barrier and continue with the next sequence of their activities. The purpose is to allow tasks to execute asynchronously, yet coordinate their progress through a complex set of activities.
package Barriers is
protected type Barrier(Trigger : Positive) is
entry Wait_For_Leader;
entry Wait_For_Herd;
procedure Leader_Done;
private
Done : Boolean := False;
end Barrier;
protected type Autobarrier(Trigger : Positive) is
entry Wait_For_Leader;
entry Wait_For_Herd;
private
Done : Boolean := False;
end Autobarrier;
end Barriers;
This package shows two kinds of squad lock. The Barrier protected type demonstrates a basic squad lock. The herd calls Wait_For_Leader and the leader calls Wait_For_Herd and then Leader_Done. The Autobarrier demonstrates a simpler interface. The herd calls Wait_For_Leader and the leader calls Wait_For_Herd. The Trigger parameter is used when creating an instance of either type of barrier. It sets the minimum number of herd tasks the leader must wait for before it can proceed.
package body Barriers is
protected body Barrier is
entry Wait_For_Herd when Wait_For_Leader'Count >= Trigger is
begin
null;
end Wait_For_Herd;
entry Wait_For_Leader when Done is
begin
if Wait_For_Leader'Count = 0 then
Done := False;
end if;
end Wait_For_Leader;
procedure Leader_Done is
begin
Done := True;
end Leader_Done;
end Barrier;
protected body Autobarrier is
entry Wait_For_Herd when Wait_For_Leader'Count >= Trigger is
begin
Done := True;
end Wait_For_Herd;
entry Wait_For_Leader when Done is
begin
if Wait_For_Leader'Count = 0 then
Done := False;
end if;
end Wait_For_Leader;
end Autobarrier;
end Barriers;