I have a ะก++ class that has a databases_list_ vector defined.
This vector contains database objects.
I need to make two functions that will run on separate threads. And each function will use the databases_list_
vector.
That is, this vector is a shared resource for my functions.
In this case, I don't know the correct approach to secure the share so that every thread can use it.
// This is class Worker
class Worker {
private:
std::thread check_oo_thread_{}; // thread #1
std::thread check_db_thread_{}; // thread #2
void check_oo(); // func #1
void check_db(); // func #2
std::vector<std::unique_ptr<database>> databases_list_; // This is a shared resource
bool is_work_{ false };
}
// Here I am running two threads
void Worker::start() {
is_work_ = true;
check_oo_thread_ = std::thread(&Worker::check_oo, this);
check_db_thread_ = std::thread(&Worker::check_db, this);
}
And here are two functions that use the databases_list_ vector in different threads. The difference is that these functions get different data from the database - it depends on the command_type parameter
// In this function, each database object is called in turn
void Worker::check_oo() {
while (is_work_) {
for (auto& db : databases_list_) {
auto db_cfg = db->get_cfg();
data_handler_.set_database(db_cfg.db_server);
auto db_respond = db->send_command(receive_cmd, command_type::get_data);
}
std::this_thread::sleep_for(std::chrono::seconds(processing_period_1));
}
}
// In this function, each database object is called in turn
void Worker::check_db() {
while (is_work_) {
for (auto& db : databases_list_) {
auto db_cfg = db->get_cfg();
data_handler_.set_database(db_cfg.db_server);
auto db_respond = db->send_command(receive_cmd, command_type::get_stat);
}
std::this_thread::sleep_for(std::chrono::seconds(processing_period_2));
}
}
This is a tricky case for me and I can't figure out how to get two threads to work on the shared resource databases_list_.
You can lock a std::mutex every time you want to access the shared resource. You also access is_work_ from multiple threads and I therefore suggest making that std::atomic<bool> is_work_ instead.
#include <atomic>
#include <mutex>
class Worker {
private:
std::thread check_oo_thread_{}; // thread #1
std::thread check_db_thread_{}; // thread #2
mutable std::mutex m_mtx; // use this to sync actions on your resource
std::atomic<bool> is_work_{}; // now atomic
void check_oo() {
while(is_work_) {
{
std::lock_guard<std::mutex> lock{m_mtx}; // locks the mutex
// use databases_list_ here
} // mutex is unlocked here
std::this_thread::sleep_for(...);
}
}
void check_db() {
while(is_work_) {
{
std::lock_guard<std::mutex> lock{m_mtx};
// use databases_list_ here
}
std::this_thread::sleep_for(...);
}
}
std::vector<std::unique_ptr<database>> databases_list_;
};
Related
I have a struct instance that gets used by multiple threads. Each thread contains an unknown amount of function calls that alter the struct member variable.
I have a dedicated function that tries to "reserve" the struct instance for the current thread and I would like to ensure no other thread can reserve the instance till the original thread allows it.
Mutexes come to mind as those can be used to guard resources, but I only know of std::lock_guard that are in the scope of a single function, but do not add protection for all function calls in between lock and unlock.
Is it possible to protect a resource like that, when I know it will always call reserve and release in that order?
Snippet that explains it better:
#include <iostream> // std::cout
#include <thread> // std::thread
#include <mutex> // std::mutex
struct information_t {
std::mutex mtx;
int importantValue = 0;
// These should only be callable from the thread that currently holds the mutex
void incrementIt() { importantValue++; }
void decrementIt() { importantValue--; }
void reset() { importantValue = 0; }
} protectedResource; // We only have one instance of this that we need to work with
// Free the resource so other threads can reserve and use it
void release()
{
std::cout << "Result: " << protectedResource.importantValue << '\n';
protectedResource.reset();
protectedResource.mtx.unlock(); // Will this work? Can I guarantee the mtx is locked?
}
// Supposed to make sure no other thread can reserve or use it now anymore!
void reserve()
{
protectedResource.mtx.lock();
}
int main()
{
std::thread threads[3];
threads[0] = std::thread([]
{
reserve();
protectedResource.incrementIt();
protectedResource.incrementIt();
release();
});
threads[1] = std::thread([]
{
reserve();
// do nothing
release();
});
threads[2] = std::thread([]
{
reserve();
protectedResource.decrementIt();
release();
});
for (auto& th : threads) th.join();
return 0;
}
My suggestion per comment:
A better idiom might be a monitor which keeps the lock of your resource and provides access to the owner. To obtain a resource, the reserve() could return such monitor object (something like a proxy to access the contents of the resource). Any competing access to reserve() would block now (as the mutex is locked). When the resource owning thread is done, it just destroys the monitor object which in turn unlocks the resource. (This allows to apply RAII to all this which makes your code safe and maintainable.)
I modified OPs code to sketch how this could look like:
#include <iostream> // std::cout
#include <thread> // std::thread
#include <mutex> // std::mutex
class information_t {
private:
std::mutex mtx;
int importantValue = 0;
public:
class Monitor {
private:
information_t& resource;
std::lock_guard<std::mutex> lock;
friend class information_t; // to allow access to constructor.
private:
Monitor(information_t& resource):
resource(resource), lock(resource.mtx)
{ }
public:
~Monitor()
{
std::cout << "Result: " << resource.importantValue << '\n';
resource.reset();
}
Monitor(const Monitor&) = delete; // copying prohibited
Monitor& operator=(const Monitor&) = delete; // copy assign prohibited
public:
// exposed resource API for monitor owner:
void incrementIt() { resource.incrementIt(); }
void decrementIt() { resource.decrementIt(); }
void reset() { resource.reset(); }
};
friend class Monitor; // to allow access to private members
public:
Monitor aquire() { return Monitor(*this); }
private:
// These should only be callable from the thread that currently holds the mutex
// Hence, they are private and accessible through a monitor instance only
void incrementIt() { importantValue++; }
void decrementIt() { importantValue--; }
void reset() { importantValue = 0; }
} protectedResource; // We only have one instance of this that we need to work with
#if 0 // OBSOLETE
// Free the resource so other threads can reserve and use it
void release()
{
protectedResource.reset();
protectedResource.mtx.unlock(); // Will this work? Can I guarantee the mtx is locked?
}
#endif // 0
// Supposed to make sure no other thread can reserve or use it now anymore!
information_t::Monitor reserve()
{
return protectedResource.aquire();
}
using MyResource = information_t::Monitor;
int main()
{
std::thread threads[3];
threads[0]
= std::thread([]
{
MyResource protectedResource = reserve();
protectedResource.incrementIt();
protectedResource.incrementIt();
// scope end releases protectedResource
});
threads[1]
= std::thread([]
{
try {
MyResource protectedResource = reserve();
throw "Haha!";
protectedResource.incrementIt();
// scope end releases protectedResource
} catch(...) { }
});
threads[2]
= std::thread([]
{
MyResource protectedResource = reserve();
protectedResource.decrementIt();
// scope end releases protectedResource
});
for (auto& th : threads) th.join();
return 0;
}
Output:
Result: 2
Result: -1
Result: 0
Live Demo on coliru
Is it possible to protect a resource like that, when I know it will always call reserve and release in that order?
It's not anymore necessary to be concerned about this. The correct usage is burnt in:
To get access to the resource, you need a monitor.
If you get it you are the exclusive owner of the resource.
If you exit the scope (where you stored the monitor as local variable) the monitor is destroyed and thus the locked resource auto-released.
The latter will happen even for unexpected bail-outs (in the MCVE the throw "Haha!";).
Furthermore, I made the following functions private:
information_t::increment()
information_t::decrement()
information_t::reset()
So, no unauthorized access is possible. To use them properly, an information_t::Monitor instance must be acquired. It provides public wrappers to those functions which can be used in the scope where the monitor resides i.e. by the owner thread only.
I want my class to spawn two threads, that will run in a loop and interact with the owner class. This was easy, when I had my class with single thread at first: one atomic flag, one mutex etc. But as the requirement for second thread rose, I am thinking of some more elegant solution, that will encapsulate the thread and its utilities. Such as: a ThreadWrapper class, that could be instantiated in the owner class.
So far, I came up with the following, however I am not certain if this is the right way to go:
class ThreadWrapper
{
public:
ThreadWrapper()
: m_threadShouldRun(false)
, m_threadRunning(false)
{
}
ThreadWrapper(std::function<void()> fn)
: m_threadShouldRun(true)
, m_threadRunning(false)
, m_threadFunction(fn)
{
m_threadPointer = std::make_shared<std::thread>(std::thread([&] { this->threadLoop(); }));
}
virtual ~ThreadWrapper()
{
if (m_threadRunning)
m_threadShouldRun = false;
m_threadPointer->join();
m_threadPointer = nullptr;
}
private:
void threadLoop()
{
m_threadRunning = true;
while(m_threadShouldRun)
{
std::this_thread::sleep_for(std::chrono::milliseconds(100));
m_threadFunction();
}
m_threadRunning = false;
}
std::function<void()> m_threadFunction = []{};
std::shared_ptr<std::thread> m_threadPointer;
std::atomic<bool> m_threadShouldRun;
std::atomic<bool> m_threadRunning;
};
Usage (fixed):
class Foo
{
Foo()
: t1(std::function<void()>(std::bind(&Foo::internalLoop1, this)))
, t2(std::function<void()>(std::bind(&Foo::internalLoop2, this)))
{
}
ThreadWrapper t1;
ThreadWrapper t2;
std::mutex mtx;
void internalLoop1()
{
// looped task
{
std::scoped_lock lock(mtx);
// write'n'read data to tx/rx queues
}
}
void internalLoop2()
{
// looped task
{
std::scoped_lock lock(mtx);
// process the rx queue
}
}
};
This is somewhat similar to a Wrapper that I found here.
The shortcoming is that the sleep period needs to be passed to the ThreadWrapper object.
Is this a correct approach? Or maybe it should be done with a basis class and just inherited virtual method instead? As shown on the pic below.
Source: https://www.codeproject.com/Articles/18383/A-thread-wrapper-class
Ideally I would like to use a template here, however the only "different" part between both ThreadWrapper is a single method, which rather suggests me inheritance/function pointers. Or am I wrong here?
EDIT: Motivation. In C# one could do the following: var t1 = new Thread(() => internalLoop1()); In the end I am looking after such one-liner in my main class. Where the other ThreadWrapper class would take care of the thread safety management. Hence, being easily replicable if multiple threads were to be spawned.
I'm struggling to implement a thread-safe reference-counted queue. The idea is that I have a number of tasks that each maintain a shared_ptr to a task manager that owns the queue. Here is a minimal implementation that should encounter that same issue:
#include <condition_variable>
#include <deque>
#include <functional>
#include <iostream>
#include <memory>
#include <mutex>
#include <thread>
namespace {
class TaskManager;
struct Task {
std::function<void()> f;
std::shared_ptr<TaskManager> manager;
};
class Queue {
public:
Queue()
: _queue()
, _mutex()
, _cv()
, _running(true)
, _thread([this]() { sweepQueue(); })
{
}
~Queue() { close(); }
void close() noexcept
{
try {
{
std::lock_guard<std::mutex> lock(_mutex);
if (!_running) {
return;
}
_running = false;
}
_cv.notify_one();
_thread.join();
} catch (...) {
std::cerr << "An error occurred while closing the queue\n";
}
}
void push(Task&& task)
{
std::unique_lock<std::mutex> lock(_mutex);
_queue.emplace_back(std::move(task));
lock.unlock();
_cv.notify_one();
}
private:
void sweepQueue() noexcept
{
while (true) {
try {
std::unique_lock<std::mutex> lock(_mutex);
_cv.wait(lock, [this] { return !_running || !_queue.empty(); });
if (!_running && _queue.empty()) {
return;
}
if (!_queue.empty()) {
const auto task = _queue.front();
_queue.pop_front();
task.f();
}
} catch (...) {
std::cerr << "An error occurred while sweeping the queue\n";
}
}
}
std::deque<Task> _queue;
std::mutex _mutex;
std::condition_variable _cv;
bool _running;
std::thread _thread;
};
class TaskManager : public std::enable_shared_from_this<TaskManager> {
public:
void addTask(std::function<void()> f)
{
_queue.push({ f, shared_from_this() });
}
private:
Queue _queue;
};
} // anonymous namespace
int main(void)
{
const auto manager = std::make_shared<TaskManager>();
manager->addTask([]() { std::cout << "Hello world\n"; });
}
The problem I find is that on rare occasions, the queue will try to invoke its own destructor within the sweepQueue method. Upon further inspection, it seems that the reference count on the TaskManager hits zero once the last task is dequeued. How can I safely maintain the reference count without invoking the destructor?
Update: The example does not clarify the need for the std::shared_ptr<TaskManager> within Task. Here is an example use case that should illustrate the need for this seemingly unnecessary ownership cycle.
std::unique_ptr<Task> task;
{
const auto manager = std::make_shared<TaskManager>();
task = std::make_unique<Task>(someFunc, manager);
}
// Guarantees manager is not destroyed while task is still in scope.
The ownership hierarchy here is TaskManager owns Queue and Queue owns Tasks. Tasks maintaining a shared pointer to TaskManager create an ownership cycle which does not seem to serve a useful purpose here.
This is the ownership what is root of the problem here. A Queue is owned by TaskManager, so that Queue can have a plain pointer to TaskManager and pass that pointer to Task in sweepQueue. You do not need std::shared_pointer<TaskManager> in Task at all here.
I'd refactor the queue from the thread first.
But to fix your problem:
struct am_I_alive {
explicit operator bool() const { return m_ptr.lock(); }
private:
std::weak_ptr<void> m_ptr;
};
struct lifetime_tracker {
am_I_alive track_lifetime() {
if (!m_ptr) m_ptr = std::make_shared<bool>(true);
return {m_ptr};
}
lifetime_tracker() = default;
lifetime_tracker(lifetime_tracker const&) {} // do nothing, don't copy
lifetime_tracker& operator=(lifetime_tracker const&){ return *this; }
private:
std::shared_ptr<void> m_ptr;
};
this is a little utility to detect if we have been deleted. It is useful in any code that calls an arbitrary callback whose side effect could include delete(this).
Privately inherit your Queue from it.
Then split popping the task from running it.
std::optional<Task> get_task() {
std::unique_lock<std::mutex> lock(_mutex);
_cv.wait(lock, [this] { return !_running || !_queue.empty(); });
if (!_running && _queue.empty()) {
return {}; // end
}
auto task = _queue.front();
_queue.pop_front();
return task;
}
void sweepQueue() noexcept
{
while (true) {
try {
auto task = get_task();
if (!task) return;
// we are alive here
auto alive = track_lifetime();
try {
(*task).f();
} catch(...) {
std::cerr << "An error occurred while running a task\n";
}
task={};
// we could be deleted here
if (!alive)
return; // this was deleted, get out of here
}
} catch (...) {
std::cerr << "An error occurred while sweeping the queue\n";
}
}
}
and now you are safe.
After that you need to deal with the thread problem.
The thread problem is that you need your code to destroy the thread from within the thread it is running. At the same time, you also need to guarantee that the thread has terminated before main ends.
These are not compatible.
To fix that, you need to create a thread owning pool that doesn't have your "keep alive" semantics, and get your thread from there.
These threads don't delete themselves; instead, they return themselves to that pool for reuse by another client.
At shutdown, those threads are blocked on to ensure you don't have code running elsewhere that hasn't halted before the end of main.
To write such a pool without your inverted dependency mess, split the queue part of your code off. This queue owns no thread.
template<class T>
struct threadsafe_queue {
void push(T);
std::optional<T> pop(); // returns empty if thread is aborted
void abort();
~threadsafe_queue();
private:
std::mutex m;
std::condition_variable v;
std::deque<T> data;
bool aborted = false;
};
then a simple thread pool:
struct thread_pool {
template<class F>
std::future<std::result_of_t<F&()>> enqueue( F&& f );
template<class F>
std::future<std::result_of_t<F&()>> thread_off_now( F&& f ); // starts a thread if there aren't any free
void abort();
void start_thread( std::size_t n = 1 );
std::size_t count_threads() const;
~thread_pool();
private:
threadsafe_queue< std::function<void()> > tasks;
std::vector< std::thread > threads;
static void thread_loop( thread_pool* pool );
};
make a thread pool singleton. Get your threads for your queue from thread_off_now method, guaranteeing you a thread that (when you are done with it) can be recycled, and whose lifetime is handled by someone else.
But really, you should instead be thinking with ownership in mind. The idea that tasks and task queues mutually own each other is a mess.
If someone disposes of a task queue, it is probably a good idea to abandon the tasks instead of persisting it magically and silently.
Which is what my simple thread pool does.
I have two functions foo and bar that should be mutually exclusive since they operate on the same data. However foo duplicates a lot of code from bar, so I would like to refactor foo to make a call to bar.
This is a problem because then I can't use a single mutex for both functions, because then foo would deadlock when it calls bar. So rather than "mutually exclusive" I only want "mutually exclusive from different threads".
Is there a pattern for implementing this? I'm using C++ and I'm okay with C++14/boost if I need something like shared_mutex.
Define a private "unlocked" function and use that from both foo and bar:
void bar_unlocked()
{
// assert that mx_ is locked
// real work
}
void bar()
{
std::lock_guard<std::mutex> lock(mx_);
bar_unlocked();
}
void foo()
{
std::lock_guard<std::mutex> lock(mx_);
// stuff
bar_unlocked();
// more stuff
}
another way - this has the advantage that you can prove that the lock has been taken:
void bar_impl(std::unique_lock<std::mutex> lock)
{
assert(lock.owns_lock());
// real work
}
void bar()
{
bar_impl(std::unique_lock<std::mutex>(mx_));
}
void foo()
{
// stuff
bar_impl(std::unique_lock<std::mutex>(mx_));
// more stuff
}
Rationale:
std::mutex is not (mandated by the standard to be) moveable, but a std::unique_lock<std::mutex> is. For this reason, we can move a lock into a callee and return it back to a caller (if necessary).
This allows us to prove ownership of the lock at every stage of a call chain.
In addition, once the optimiser gets involved, it's likely that all the lock-moving will be optimised away. This gives us the best of both worlds - provable ownership and maximal performance.
A more complete example:
#include <mutex>
#include <cassert>
#include <functional>
struct actor
{
//
// public interface
//
// perform a simple synchronous action
void simple_action()
{
impl_simple_action(take_lock());
}
/// perform an action either now or asynchronously in the future
/// hander() is called when the action is complete
/// handler is a latch - i.e. it will be called exactly once
/// #pre an existing handler must not be pending
void complex_action(std::function<void()> handler)
{
impl_complex_action(take_lock(), std::move(handler));
}
private:
//
// private external interface (for callbacks)
//
void my_callback()
{
auto lock = take_lock();
assert(!_condition_met);
_condition_met = true;
impl_condition_met(std::move(lock));
}
// private interface
using mutex_type = std::mutex;
using lock_type = std::unique_lock<mutex_type>;
void impl_simple_action(const lock_type& lock)
{
// assert preconditions
assert(lock.owns_lock());
// actions here
}
void impl_complex_action(lock_type my_lock, std::function<void()> handler)
{
_handler = std::move(handler);
if (_condition_met)
{
return impl_condition_met(std::move(my_lock));
}
else {
// initiate some action that will result in my_callback() being called
// some time later
}
}
void impl_condition_met(lock_type lock)
{
assert(lock.owns_lock());
assert(_condition_met);
if(_handler)
{
_condition_met = false;
auto copy = std::move(_handler);
// unlock here because the callback may call back into our public interface
lock.unlock();
copy();
}
}
auto take_lock() const -> lock_type
{
return lock_type(_mutex);
}
mutable mutex_type _mutex;
std::function<void()> _handler = {};
bool _condition_met = false;
};
void act(actor& a)
{
a.complex_action([&a]{
// other stuff...
// note: calling another public interface function of a
// during a handler initiated by a
// the unlock() in impl_condition_met() makes this safe.
a.simple_action();
});
}
For example I've an EventGenerator class that call IEventHandler::onEvent for all registered event handlers:
class IEventHandler {
public: virtual void onEvent(...) = 0;
};
class EventGenerator {
private:
std::vector<IEventHandler*> _handlers;
std::mutex _mutex; // [1]
public:
void AddHandler(IEventHandler* handler) {
std::lock_guard<std::mutex> lck(_mutex); // [2]
_handlers.push_back(handler);
}
void RemoveHanler(IEventHandler* handler) {
std::lock_guard<std::mutex> lck(_mutex); // [3]
// remove from "_handlers"
}
private:
void threadMainTask() {
while(true) {
// Do some work ...
// Post event to all registered handlers
{
std::lock_guard<std::mutex> lck(_mutex); // [4]
for(auto& h : _handlers) { h->onEvent(...); )
}
// Do some work ...
}
}
The code should be thread safe in the following manner:
one thread is executing the EventGenerator::threadMainTask
many threads might access EventGenerator::AddHandler and EventGenerator::RemoveHandler APIs.
To support this, I have the following synchonization (see comment in the code):
[1] is the mutex that protects the vector _handlers from multiple thread access.
[2] and [3] are protect adding or removing handlers simultaneously.
[4] is preventing from changing the vector while the main thread is posting events.
This code works until... If for some reason, during the execution of IEventHandler::onEvent(...) the code is trying to call EventManager::RemoveHandler or EventManager::AddHandler. The result is runtime exception.
What is the best approach to handle registration of the event handlers and executing the event handler callback in the thread safe manner?
>> UPDATE <<
So based on the inputs, I've updated to the following design:
class IEventHandler {
public: virtual void onEvent(...) = 0;
};
class EventDelegate {
private:
IEventHandler* _handler;
std::atomic<bool> _cancelled;
public:
EventDelegate(IEventHandler* h) : _handler(h), _cancelled(false) {};
void Cancel() { _cancelled = true; }
void Invoke(...) { if (!_cancelled) _handler->onEvent(...); }
}
class EventGenerator {
private:
std::vector<std::shared_ptr<EventDelegate>> _handlers;
std::mutex _mutex;
public:
void AddHandler(std::shared_ptr<EventDelegate> handler) {
std::lock_guard<std::mutex> lck(_mutex);
_handlers.push_back(handler);
}
void RemoveHanler(std::shared_ptr<EventDelegate> handler) {
std::lock_guard<std::mutex> lck(_mutex);
// remove from "_handlers"
}
private:
void threadMainTask() {
while(true) {
// Do some work ...
std::vector<std::shared_ptr<EventDelegate>> handlers_copy;
{
std::lock_guard<std::mutex> lck(_mutex);
handlers_copy = _handlers;
}
for(auto& h : handlers_copy) { h->Invoke(...); )
// Do some work ...
}
}
As you can see, there is additional class EventDelegate that have two purposes:
hold the event callback
enable to cancel the callback
In the threadMainTask, I'm using a local copy of the std::vector<std::shared_ptr<EventDelegate>> and I'm releasing the lock before invoking the callbacks. This approach solves an issue when during the IEventHandler::onEvent(...) the EventGenerator::{AddHandler,RemoveHanler} is called.
Any thoughts about the new design?
Copy-on-Write vector implemented on atomic swap of shared_ptr's (in assumptions callback registration is occurring far less frequently than events the callbacks are notified about):
using callback_t = std::shared_ptr<std::function<void(event_t const&)> >;
using callbacks_t = std::shared_ptr<std::vector<callback_t> >;
callbacks_t callbacks_;
mutex_t mutex_; // a mutex of your choice
void register(callback_t cb)
{
// the mutex is to serialize concurrent callbacks registrations
// this is not always necessary, as depending on the application
// architecture, single writer may be enforced by design
scoped_lock lock(mutex_);
auto callbacks = atomic_load(&callbacks_);
auto new_callbacks = std::make_shared< std::vector<callback_t> >();
new_callbacks->reserve(callbacks->size() + 1);
*new_callbacks = callbacks;
new_callbacks->push_back(std::move(cb));
atomic_store(&callbacks_, new_callbacks);
}
void invoke(event_t const& evt)
{
auto callbacks = atomic_load(&callbacks_);
// many people wrap each callback invocation into a try-catch
// and de-register on exception
for(auto& cb: *callbacks) (*cb)(evt);
}
Specifically on the subject of asynchronous behavior when callback is executed while being de-registered, well here the best approach to take is remember of the Separation of Concerns principle.
The callback should not be able to die until it has been executed. This is achieved via another classic trick called "extra level of indirection". Namely, instead of registering user provided callback one would wrap it to something like the below and callback de-registration apart from updating the vector will call the below defined discharge() method on the callback wrapper and will even notify the caller of de-registration method of whether the callback execution finished successfully.
template <class CB> struct cb_wrapper
{
mutable std::atomic<bool> done_;
CB cb_;
cb_wrapper(CB&& cb): cb(std::move(cb_)) {}
bool discharge()
{
bool not_done = false;
return done_.compare_exchange_strong(not_done, true);
}
void operator()(event_t const&)
{
if (discharge())
{
cb();
}
}
};
I can't see a right thing here. From your update I can see a problem: you are not synchronizing the invoke method with callback removal. There's an atomic but it's not enough. Example: just after this line of code:
if (!_cancelled)
Another thread calls the remove method. What can happen is that the onEvent() is called anyway, even if the removed method has removed the callback from the list and returned the result, there's nothing to keep synchronized this execution flow. Same problem for the answer of #bobah.