C11 sharing a thread safe Q across multiple classes and threads - c++

I am fairly new to C++ and very new to using mutex. I am trying to implement a thread safe queue by #ChewOnThis_Trident from this answer.
Essentially I have different threads adding messages to a queue and I need to preserve the order they are being added. However the messages require some conditional modifications before being added. In the real code listeners on separate threads call unique "handleMessage" functions that modify the message before adding to them to the queue. A separate thread checks to see if messages are in the queue and handles them in order. In the full code, I know the listeners are receiving the messages in the correct order, but they are failing to add them to the queue in the correct order.
I think the problem is there is some time elapsing between when a message is received and if it is being modified, causing messages to fall out of order.
For practical reasons in the real code, I can't do these modifications inside of "Safequeue::enqueue".
In my example two threads can add to the queue. One thread reads from it. The "message" in this case is a random int. "UsesQ" handles adding to the queue, and message modification (Ex. makes all ints odd).
I think another mutex is needed when "UsesQ::addQ" is called, but it would need to be shared across all the threads and I'm not sure if I am not sure how to implement it.
In the example I am struggling of thinking of a way to test if the order is correct.
Here is the example:
#include <queue>
#include <mutex>
#include <condition_variable>
#include <stdio.h>
#include <stdlib.h>
#include <iostream>
#include <assert.h>
#include <pthread.h>
#include <unistd.h>
class SafeQueue
{// A threadsafe-queue.
public:
SafeQueue(void)
: q()
, m()
, cv()
{}
~SafeQueue(void)
{}
// Add an element to the queue.
void enqueue(int i)
{
std::lock_guard<std::mutex> lock(m);
q.push(i);
cv.notify_one();
}
// Get the "front"-element.
// If the queue is empty, wait till a element is avaiable.
int dequeue(void)
{
std::unique_lock<std::mutex> lock(m);
while(q.empty())
{
// release lock as long as the wait and reaquire it afterwards.
cv.wait(lock);
}
int val = q.front();
q.pop();
return val;
}
private:
std::queue<int> q;
mutable std::mutex m;
std::condition_variable cv;
};
class UsesQ
{
private:
int readVal;
int lastReadVal = 1;
public:
SafeQueue & Q;
UsesQ(SafeQueue & Q): Q(Q){};
~UsesQ(){};
void addQ(int i)
{
if(i% 2 == 0)
{
i++;//some conditional modification to the initial "message"
}
Q.enqueue(i);
}
void removeQ()
{
readVal = Q.dequeue();
}
};
void* run_add(void* Ptr)
{
UsesQ * UsesQPtr = (UsesQ *)Ptr;
for(;;)
{
int i = rand();//simulate an incoming "message"
UsesQPtr->addQ(i);
}
pthread_exit (NULL);
return NULL;
}
void* run_remove(void* Ptr)
{
UsesQ * UsesQPtr = (UsesQ *)Ptr;
for(;;)
{
UsesQPtr->removeQ();
}
pthread_exit (NULL);
return NULL;
}
int main()
{
SafeQueue Q;
UsesQ * UsesQPtr = new UsesQ(std::ref(Q));
pthread_t thread1;
pthread_create(&thread1, NULL, run_add, UsesQPtr);
pthread_t thread2;
pthread_create(&thread2, NULL, run_add, UsesQPtr);
pthread_t thread3;
pthread_create(&thread3, NULL, run_remove, UsesQPtr);
while(1)
{
usleep(1);
printf(".\n");
}
};
Complied with the pthread tag
g++ main.cpp -pthread
Thank you for any help.

Related

Multi-consumer condition variable wait in the same instance's member function

I'm having trouble thinking of a way to properly implement a signalling mechanism for multiple listeners waiting in the same function for a producer to signal some new data continuously, without getting "signalled" for the same previous data-
I want all listeners to always see the latest available data (not caring about missed signals if they are busy), without repeats.
My attempt so far:
#include <functional>
#include <shared_mutex>
#include <condition_variable>
#include <thread>
class Signaller {
public:
// Used by producer, will hold on to the mutex uniquely as it modifies data
void Signal(const std::function<void()>& fnIn) {
std::unique_lock lock(m_mtx);
fnIn();
m_newData = true;
m_cv.notify_all();
}
// Used by consumers, will only hold shared mutex to read data
void Wait(const std::function<void()>& fnIn) {
{
std::shared_lock lock(m_mtx);
m_cv.wait(lock, [this](){ return m_newData; });
fnIn();
}
// Need some way to flip m_newData to false when all threads are "done"
// (or some other method of preventing spurious wakeups)
// I don't think this is particularly ideal
{
std::unique_lock lock(m_mtx);
m_newData = false;
}
}
private:
std::condition_variable_any m_cv;
std::shared_mutex m_mtx;
bool m_newData{false}; // To prevent spurious wakeups
};
class Example {
public:
// Multiple threads will call this function in the same instance of Example
void ConsumerLoop()
{
int latestData{0};
while (true){
m_signaller.Wait([this, &latestData](){ latestData = m_latestData; });
// process latestData...
// I want to make sure latestData here is always the latest
// (It's OK to miss a few signals in between if its off processing this latest data)
}
}
// One thread will be using this to signal new data
void ProducerLoop(){
while(true){
int newData = rand();
m_signaller.Signal([this, newData](){ m_latestData = newData; });
std::this_thread::sleep_for(std::chrono::milliseconds(1));
}
}
private:
Signaller m_signaller;
int m_latestData;
};
My main issue (I think) is how to prevent spurious wakeups, while preventing repeated data from waking up the same thread. I've thought about using some sort of counter within each thread to keep track of whether it's receiving the same data, but couldn't get anywhere with that idea (unless I perhaps make some sort of map using std::this_thread::get_id?). Is there a better way to do this?
EDIT:
Expanding on my map of thread ID's idea, I think I've found a solution:
#include <functional>
#include <shared_mutex>
#include <condition_variable>
#include <unordered_map>
#include <thread>
class Signaller {
public:
// Used by producer, will hold on to the mutex uniquely as it modifies data
void Signal(const std::function<void()>& fnIn) {
std::unique_lock lock(m_mtx);
fnIn();
m_ctr++;
m_cv.notify_all();
}
void RegisterWaiter(){
std::unique_lock lock(m_mtx);
auto [itr, emplaced] = m_threadCtrMap.try_emplace(std::this_thread::get_id(), m_ctr);
if (!emplaced) {
itr->second = m_ctr;
}
}
// Used by consumers, will only hold shared mutex to read data
void Wait(const std::function<void()>& fnIn) {
std::shared_lock lock(m_mtx);
m_cv.wait(lock, [this](){ return m_threadCtrMap[std::this_thread::get_id()] != m_ctr; });
fnIn();
m_threadCtrMap[std::this_thread::get_id()] = m_ctr;
}
private:
std::condition_variable_any m_cv;
std::shared_mutex m_mtx;
std::uint32_t m_ctr{0};
std::unordered_map<std::thread::id, std::uint32_t> m_threadCtrMap; // Stores the last signalled ctr for that thread
};
class Example {
public:
// Multiple threads will call this function in the same instance of Example
void ConsumerLoop()
{
int latestData{0};
m_signaller.RegisterWaiter();
while (true){
m_signaller.Wait([this, &latestData](){ latestData = m_latestData; });
}
}
// One thread will be using this to signal new data
void ProducerLoop(){
while(true){
int newData = rand();
m_signaller.Signal([this, newData](){ m_latestData = newData; });
std::this_thread::sleep_for(std::chrono::milliseconds(1));
}
}
private:
Signaller m_signaller;
int m_latestData;
};
Here's my implementation:
#include <unordered_map>
#include <condition_variable>
#include <shared_mutex>
#include <thread>
/*
Example usage:
struct MyClass {
MultiCVSignaller m_signaller;
int m_latestData;
std::atomic<bool> m_stop{false};
~MyClass(){
m_stop = true;
m_signaller.Shutdown();
}
void FuncToWaitOnData() { // e.g. Multiple threads call this fn to "subscribe" to the signal
auto& signalCtr = m_signaller.RegisterListener();
while(!m_stop.load(std::memory_order_relaxed)) {
int latestDataInLocalThread;
// WaitForSignal() calls the provided function while holding on to the shared mutex
m_signaller.WaitForSignal(signalCtr, [this, &latestDataInLocalThread](){
latestDataInLocalThread = m_latestData;
});
// Make use of latest data...
}
}
void ProducerLoop() {
while(!m_stop.load(std::memory_order_relaxed)) {
// Signal() holds on to the mutex uniquely while calling the provided function.
m_signaller.Signal([&latestData](){
m_latestData = rand();
});
}
}
};
*/
class MultiCVSignaller
{
public:
using SignalCtr = std::uint32_t;
public:
MultiCVSignaller() = default;
~MultiCVSignaller() { Shutdown(); }
/*
Call to set and signal shutdown state, cancelling waits (and skipping the functions provided if any)
This should be added in the class' destructor before threads are joined.
*/
void Shutdown() {
std::unique_lock lock(m_mtx);
m_shutdown = true;
m_cv.notify_all();
}
// Calls the function if specified while holding on to the mutex with a UNIQUE lock
template<class Func = void(*)()>
void Signal(Func fnIn = +[]{})
{
std::unique_lock lock(m_mtx);
fnIn();
m_ctr++;
m_cv.notify_all();
}
MultiCVSignaller::SignalCtr& RegisterListener(){
std::unique_lock lock(m_mtx);
auto [itr, emplaced] = m_threadCtrMap.try_emplace(std::this_thread::get_id(), m_ctr);
if (!emplaced) {
itr->second = m_ctr;
}
return itr->second;
}
/*
Calls the optional function while holding on to the SHARED lock when signalled. The signalCtr argument should be provided by the return of RegisterListener() (see example)
*/
template<class Func = void(*)()>
void WaitForSignal(MultiCVSignaller::SignalCtr& signalCtr, Func fnIn = +[]{})
{
std::shared_lock lock(m_mtx);
m_cv.wait(lock, [this, &signalCtr](){ return ( m_shutdown || signalCtr != m_ctr); });
if (!m_shutdown)
{
fnIn();
signalCtr = m_ctr;
}
}
private:
std::condition_variable_any m_cv;
std::shared_mutex m_mtx;
bool m_shutdown{false};
SignalCtr m_ctr{0}; // Latest ctr from Signal()
// This map stores the signal count received for registered listeners.
// We use an unordered_map as references are never invalidated (unless erased),
// which is not the case for a vector
std::unordered_map<std::thread::id, SignalCtr> m_threadCtrMap;
};

Condition variable should be used or not to reduce missed wakeups

I have two threads, one is the producer and other is consumer. My consumer is always late (due to some costly function call, simulated in below code using sleeps) so I have used ring buffer as I can afford to loose some events.
Questions:
I am wondering if it would be better to use condition variable instead of what I currently have : continuous monitoring of the ring buffer size to see if the events got generated. I know that the current while loop of checking the ring buffer size is expensive, so I can probably add some yield calls to reduce the tight loop. I want to reduce the chances of dropped events.
Can I get rid of pointers? In my current code I am passing pointers to my ring buffer from main function to the threads. Wondering if there is any fancy or better way to do the same?
#include <iostream>
#include <thread>
#include <chrono>
#include <vector>
#include <atomic>
#include <boost/circular_buffer.hpp>
#include <condition_variable>
#include <functional>
std::atomic<bool> mRunning;
std::mutex m_mutex;
std::condition_variable m_condVar;
long int data = 0;
class Detacher {
public:
template<typename Function, typename ... Args>
void createTask(Function &&func, Args&& ... args) {
m_threads.emplace_back(std::forward<Function>(func), std::forward<Args>(args)...);
}
Detacher() = default;
Detacher(const Detacher&) = delete;
Detacher & operator=(const Detacher&) = delete;
Detacher(Detacher&&) = default;
Detacher& operator=(Detacher&&) = default;
~Detacher() {
for (auto& thread : m_threads) {
thread.join();
}
}
private:
std::vector<std::thread> m_threads;
};
void foo_1(boost::circular_buffer<int> *cb)
{
while (mRunning) {
std::unique_lock<std::mutex> mlock(m_mutex);
if (!cb->size())
continue;
int data = cb[0][0];
cb->pop_front();
mlock.unlock();
if (!mRunning) {
break;
}
//simulate time consuming function call
std::this_thread::sleep_for(std::chrono::milliseconds(16));
}
}
void foo_2(boost::circular_buffer<int> *cb)
{
while (mRunning) {
std::unique_lock<std::mutex> mlock(m_mutex);
cb->push_back(data);
data++;
mlock.unlock();
//simulate time consuming function call
std::this_thread::sleep_for(std::chrono::milliseconds(1));
}
}
int main()
{
mRunning = true;
boost::circular_buffer<int> cb(100);
Detacher thread_1;
thread_1.createTask(foo_1, &cb);
Detacher thread_2;
thread_2.createTask(foo_2, &cb);
std::this_thread::sleep_for(std::chrono::milliseconds(20000));
mRunning = false;
}
The producer is faster (16x) than the consumer, so ~93% of all events will always be discarded.

Threading inside class with atomics and mutex c++

I wrote this sample program to mimic what I'm trying to do in a larger program.
I have some data that will come from the user and be passed into a thread for some processing. I am using mutexes around the data the flags to signal when there is data.
Using the lambda expression, is a pointer to *this send to the thread? I seem to be getting the behavior I expect in the cout statement.
Are the mutexes used properly around the data?
Is putting the atomics and mutexes as a private member of the class a good move?
foo.h
#pragma once
#include <atomic>
#include <thread>
#include <vector>
#include <mutex>
class Foo
{
public:
Foo();
~Foo();
void StartThread();
void StopThread();
void SendData();
private:
std::atomic<bool> dataFlag;
std::atomic<bool> runBar;
void bar();
std::thread t1;
std::vector<int> data;
std::mutex mx;
};
foo.c
#include "FooClass.h"
#include <thread>
#include <string>
#include <iostream>
Foo::Foo()
{
dataFlag = false;
}
Foo::~Foo()
{
StopThread();
}
void Foo::StartThread()
{
runBar = true;
t1 = std::thread([=] {bar(); });
return;
}
void Foo::StopThread()
{
runBar = false;
if(t1.joinable())
t1.join();
return;
}
void Foo::SendData()
{
mx.lock();
for (int i = 0; i < 5; ++i) {
data.push_back(i);
}
mx.unlock();
dataFlag = true;
}
void Foo::bar()
{
while (runBar)
{
if(dataFlag)
{
mx.lock();
for(auto it = data.begin(); it < data.end(); ++it)
{
std::cout << *it << '\n';
}
mx.unlock();
dataFlag = false;
}
}
}
main.cpp
#include "FooClass.h"
#include <iostream>
#include <string>
int main()
{
Foo foo1;
std::cout << "Type anything to end thread" << std::endl;
foo1.StartThread();
foo1.SendData();
// type something to end threads
char a;
std::cin >> a;
foo1.StopThread();
return 0;
}
You ensure that the thread is joined using RAII techniques? Check.
All data access/modification is either protected through atomics or mutexs? Check.
Mutex locking uses std::lock_guard? Nope. Using std::lock_guard wraps your lock() and unlock() calls with RAII. This ensures that even if an exception occurs while within the lock, that the lock is released.
Is putting the atomics and mutexes as a private member of the class a good move?
Its neither good nor bad, but in this scenario, where Foo is a wrapper for a std::thread that does work and controls the synchronization, it makes sense.
Using the lambda expression, is a pointer to *this send to the thread?
Yes, you can also do t1 = std::thread([this]{bar();}); to make it more explicit.
As it stands, with your dataFlag assignments after the locks, you may encounter problems. If you call SendData twice such that bar processes the first one but is halted before setting dataFlag = false so that the second call adds the data, sets the flag to true only to have bar set it back to false. Then, you'll have data that has been "sent" but bar doesn't think there's anything to process.
There may be other tricky situations, but this was just one example; moving it into the lock clears up that problem.
for example, your SendData should look like:
void Foo::SendData()
{
std::lock_guard<std::mutex> guard(mx);
for (int i = 0; i < 5; ++i) {
data.push_back(i);
}
dataFlag = true;
}

C++ Concurrency segfault on mutex

Hello,
i am quite new to C++ but I have 6 years Java experience, 2 years C experience and some knowledge of concurrency basics. I am trying to create a threadpool to handle tasks. it is below with the associated test main.
it seems like the error is generated from
void ThreadPool::ThreadHandler::enqueueTask(void (*task)(void)) {
std::lock_guard<std::mutex> lock(queueMutex);
as said by my debugger, but doing traditional cout debug, i found out that sometimes it works without segfaulting and removing
threads.emplace(handler->getSize(), handler);
from ThreadPool::enqueueTask() improves stability greatly.
Overall i think it is related too my bad use of condition_variable (called idler).
compiler: minGW-w64 in CLion
.cpp
#include <iostream>
#include "ThreadPool.h"
ThreadPool::ThreadHandler::ThreadHandler(ThreadPool *parent) : parent(parent) {
thread = std::thread([&]{
while (this->parent->alive){
if (getSize()){
std::lock_guard<std::mutex> lock(queueMutex);
(*(queue.front()))();
queue.pop_front();
} else {
std::unique_lock<std::mutex> lock(idlerMutex);
idler.wait(lock);
}
}
});
}
void ThreadPool::ThreadHandler::enqueueTask(void (*task)(void)) {
std::lock_guard<std::mutex> lock(queueMutex);
queue.push_back(task);
idler.notify_all();
}
size_t ThreadPool::ThreadHandler::getSize() {
std::lock_guard<std::mutex> lock(queueMutex);
return queue.size();
}
void ThreadPool::enqueueTask(void (*task)(void)) {
std::lock_guard<std::mutex> lock(threadsMutex);
std::map<int, ThreadHandler*>::iterator iter = threads.begin();
threads.erase(iter);
ThreadHandler *handler = iter->second;
handler->enqueueTask(task);
threads.emplace(handler->getSize(), handler);
}
ThreadPool::ThreadPool(size_t size) {
for (size_t i = 0; i < size; ++i) {
std::lock_guard<std::mutex> lock(threadsMutex);
ThreadHandler *handler = new ThreadHandler(this);
threads.emplace(handler->getSize(), handler);
}
}
ThreadPool::~ThreadPool() {
std::lock_guard<std::mutex> lock(threadsMutex);
auto it = threads.begin(), end = threads.end();
for (; it != end; ++it) {
delete it->second;
}
}
.h
#ifndef WLIB_THREADPOOL_H
#define WLIB_THREADPOOL_H
#include <mutex>
#include <thread>
#include <list>
#include <map>
#include <condition_variable>
class ThreadPool {
private:
class ThreadHandler {
std::condition_variable idler;
std::mutex idlerMutex;
std::mutex queueMutex;
std::thread thread;
std::list<void (*)(void)> queue;
ThreadPool *parent;
public:
ThreadHandler(ThreadPool *parent);
void enqueueTask(void (*task)(void));
size_t getSize();
};
std::multimap<int, ThreadHandler*> threads;
std::mutex threadsMutex;
public:
bool alive = true;
ThreadPool(size_t size);
~ThreadPool();
virtual void enqueueTask(void (*task)(void));
};
#endif //WLIB_THREADPOOL_H
main:
#include <iostream>
#include <ThreadPool.h>
ThreadPool pool(3);
void fn() {
std::cout << std::this_thread::get_id() << '\n';
pool.enqueueTask(fn);
};
int main() {
std::cout << "Hello, World!" << std::endl;
pool.enqueueTask(fn);
return 0;
}
Your main() function invokes enqueueTask().
Immediately afterwards, your main() returns.
This gets the gears in motion for winding down your process. This involves invoking the destructors of all global objects.
ThreadPool's destructor then proceeds to delete all dynamically-scoped threads.
While the threads are still running. Hilarity ensues.
You need to implement the process for an orderly shutdown of all threads.
This means setting active to false, kicking all of the threads in the shins, and then joining all threads, before letting nature take its course, and finally destroy everything.
P.S. -- you need to fix how alive is being checked. You also need to make access to alive thread-safe, protected by a mutex. The problem is that the thread could be holding a lock on one of two differented mutexes. This makes this process somewhat complicated. Some redesign is in order, here.

Callback for arriving task in jobqueue?

I was wondering if anyone has any good design suggestions for a jobqueue that notifies a processJob() function when tasks > 0. I'm using Boost and c++ and just trying to get a general idea of such a design. Thanks.
I would run processJob() in a separate thread, which uses a "condition variable" to gate whether it's running; and when adding something to the queue, notifying that c.v.
The loop logic is something like:
boost::unique_lock<boost::mutex> lock(mymutex);
while (!terminate)
{
lock.lock();
while (!Q.empty())
jobCV.wait(lock);
pItem = Q.pop();
lock.unlock();
pItem->process();
}
Remember that adding items to the queue also needs to lock on the same mutex. Also, you'll need a test before that wait() for the signal that will set terminate; and the setting of that signal also needs to call notify() on the c.v.
If you already use boost library, it is convenient for you to just use boost::asio. an io_service object can be used to manage queue of jobs, since it guarantees the callbacks will be called in the order they are posted. and no hassle of locks if you only run io_service in one thread. Some sample code:
#include <boost/asio.hpp>
#include <boost/bind.hpp>
#include <boost/thread.hpp>
#include <boost/shared_ptr.hpp>
#include <iostream>
#include <cstdlib>
int job_number = 0;
struct Job
{
virtual void run() { std::cout << "job " << ++job_number << " done" << '\n'; }
virtual ~Job() {}
};
class Processor
{
private:
boost::asio::io_service ioserv_;
boost::asio::io_service::work work_;
boost::thread thread_;
public:
Processor() : ioserv_(), work_(ioserv_) {
}
void run() {
ioserv_.reset();
thread_ = boost::thread (boost::bind(&boost::asio::io_service::run, &ioserv_));
}
void stop() {
ioserv_.stop();
}
~Processor() {
stop();
if (thread_.get_id() != boost::thread::id())
thread_.join();
}
void processJob(boost::shared_ptr<Job> j)
{
j->run();
}
void addJob(boost::shared_ptr<Job> j)
{
ioserv_.post(boost::bind(&Processor::processJob, this, j));
}
};
int main()
{
Processor psr;
psr.run();
for (int i=0; i<10; ++i)
psr.addJob(boost::shared_ptr<Job>(new Job));
sleep(1);
return 0;
}