Product/Consumer - what is the optimal signalling pattern - c++

Question

I am building a high performance app that needs two function to synchronise threads
void wake_thread(thread)
void sleep_thread(thread)
The app has a single thread (lets call it C) that may fall asleep with a call to sleep_thread. There are multiple threads that will call wake_thread. When wake_thread returns it MUST guarantee that C is either running or will be woken. wake_thread must NEVER block.
The easy way is of course to do use a synchronisation event like this:
hEvent = CreateEvent(NULL, FALSE, TRUE, NULL);
void wake_thread(thread) {
SetEvent(hEvent);
}
And:
void sleep_thread(thread)
{
WaitForSingleObject(hEvent);
}
This provides the desired semantics and is free of race conditions for the scenario (There is only one thread waiting, but multiple that can signal). I included it here to show what I am trying to tune.
HOWEVER, I am wondering there is a faster way under Windows for this very specific scenario. wake_thread may be called a lot, even when C is not sleeping. This causes a lot of calls to SetEvent that do nothing. Would there be a faster way to use manual reset event and reference counters to make sure SetEvent is only called when there is actually something to set.
Every CPU cycle counts in this scenario.

Answer 1

I haven't tested this (apart from making sure it compiles) but I think this should do the trick. It was, admittedly, a bit trickier than I at first thought. Note that there are some obvious optimizations you could make; I've left it in unoptimized form for clarity and to aid any debugging that may be necessary. I've also omitted error checking.
#include <intrin.h>
HANDLE hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
__declspec(align(4)) volatile LONG thread_state = 2;
// 0 (00): sleeping
// 1 (01): sleeping, wake request pending
// 2 (10): awake, no additional wake request received
// 3 (11): awake, at least one additional wake request
void wake_thread(void)
{
LONG old_state;
old_state = _InterlockedOr(&thread_state, 1);
if (old_state == 0)
{
// This is the first wake request since the consumer thread
// went to sleep. Set the event.
SetEvent(hEvent);
return;
}
if (old_state == 1)
{
// The consumer thread is already in the process of being woken up.
// Any items added to the queue by this thread will be processed,
// so we don't need to do anything.
return;
}
if (old_state == 2)
{
// This is an additional wake request when the consumer thread
// is already awake. We've already changed the state accordingly,
// so we don't need to do anything else.
return;
}
if (old_state == 3)
{
// The consumer thread is already awake, and already has an
// additional wake request registered, so we don't need to do
// anything.
return;
}
BigTrouble();
}
void sleep_thread(void)
{
LONG old_state;
// Debugging only, remove this test in production code.
// The event should never be signaled at this point.
if (WaitForSingleObject(hEvent, 0) != WAIT_TIMEOUT)
{
BigTrouble();
}
old_state = _InterlockedAnd(&thread_state, 1);
if (old_state == 2)
{
// We've changed the state from "awake" to "asleep".
// Go to sleep.
WaitForSingleObject(hEvent, INFINITE);
// We've been buzzed; change the state to "awake"
// and then reset the event.
if (_InterlockedExchange(&thread_state, 2) != 1)
{
BigTrouble();
}
ResetEvent(hEvent);
return;
}
if (old_state == 3)
{
// We've changed the state from "awake with additional
// wake request" to "waking". Change it to "awake"
// and then carry on.
if (_InterlockedExchange(&thread_state, 2) != 1)
{
BigTrouble();
}
return;
}
BigTrouble();
}
Basically this uses a manual-reset event and a two-bit flag to reproduce the behaviour of an automatic-reset event. It may be clearer if you draw a state diagram. The thread safety depends on the rules about which of the functions is allowed to make which transitions, and also on when the event object is allowed to be signaled.
As an editorial: I think it is separating the synchronization code into the wake_thread and sleep_thread functions that makes things a bit awkward. It would probably be more natural, slightly more efficient, and almost certainly clearer if the synchronization code were moved into the queue implementation.

Answer 2

SetEvent() will introduce some latency as it does have to make a system call (sysenter triggers the switch from user to kernel mode) for the object manager to check the state of the event and dispatch it (via a call to KeSetEvent()). I think that the time of the system call might be considered to be acceptable even in your circumstances, but that is speculation. Where most of the latency is likely going to be introduced is on the receiving side of the event. In other words, it takes time to wake a thread from a WaitFor*Object() than it does to signal the event. The Windows scheduler tries to help getting to the thread sooner by giving a priority "boost" to a thread that is having a wait return, but that boost only does so much.
In order to get around this, you should be sure that you are only waiting when it is necessary to do so. The typical method to do this is, in your consumer, when you are signaled to go, consume every work item that you can without waiting on the event again, then when done make your call to sleep_thread()
I should point out that SetEvent()/WaitFor*Object() is almost surely faster than everything short of eating 100% CPU and even then it may be quicker as a result of the contention on whatever locking object needs to protect your shared data.
Normally, I would recommend the use of a ConditionVariable but I have not tested its performance compared to your technique. I have a suspicion that it may be slower since it also has the overhead of entering CRITICAL_SECTION object. You may have to measure the performance different -- when in doubt, measure, measure, measure.
The only other thing that I can think to say is that MS does acknowledge that dispatching and waiting on events can be slow, especially when it is performed repeatedly. In order to get around this, they changed the CRITICAL_SECTION object to try for a number of times in user mode to acquire the lock before actually waiting on the event. They call this the spin count. While I wouldn't recommend it, you may be able to do something similar.

Answer 3

Something like:
void consumer_thread(void)
{
while(1)
{
WaitForSingleObject(...);
// Consume all items from queue in a thread safe manner (e.g. critical section)
}
}
void produce()
{
bool queue_was_empty = ...; // in a thread safe manner determine if queue is empty
// thread safe insertion into queue ...
// These two steps should be done in a way that prevents the consumer
// from emptying the queue in between, e.g. a spin lock.
// This guarantees you will never miss the "edge"
if( queue_was_empty )
{
SetEvent(...);
}
}
The general idea is to only SetEvent on the transition from empty to full. If the threads have the same priority Windows should let the producer(s) keep running and therefore you can minimize your number of SetEvent calls per queue insertions. I've found this arrangement (between threads of equal priority) to give the best performance (at least under Windows XP and Win7, YMMV).