I have an application wherein multiple threads wait on the same event object to signal. The problem I am seeing appears to be a type of race condition in that sometimes some threads' wait states (WaitForMultipleObjects) return as a result of the event signal and other threads' wait states apparently don't see the event signal because they don't return. These events were created using CreateEvent as manual-reset event objects.
My application handles these events such that when an event object is signaled, its "owner" thread is responsible for resetting the event object's signal state, as shown in the following code snippet. Other threads waiting on the same event do not attempt to reset its signal state.
switch ( dwObjectWaitState = ::WaitForMultipleObjects( i, pHandles, FALSE, INFINITE ) )
{
case WAIT_OBJECT_0 + BAS_MESSAGE_READY_EVT_ID:
::ResetEvent( pHandles[BAS_MESSAGE_READY_EVT_ID] );
/* handles the event */
break;
}
To put it another way, the problem I am seeing appears to be to what is described in the Remarks section for PulseEvent on the MSDN website:
If the call to PulseEvent occurs
during the time when the thread has
been removed from the wait state, the
thread will not be released because
PulseEvent releases only those threads
that are waiting at the moment it is
called. Therefore, PulseEvent is
unreliable and should not be used by
new applications. Instead, use
condition variables.
If this is what is happening, the only solution I can see is for each thread to register its usage of a given event object with that object's owner thread, so that the owner thread can determine when it is safe to reset the event object's signal state.
Is there a better way to do this? Thanks.
Yes there is a better way:
[...] Instead, use condition variables.
http://msdn.microsoft.com/en-us/library/ms682052(v=vs.85).aspx
Look for WakeAllConditionVariable specificly
Why PulseEvent() is Unreliable and What to Do Without It
The auto-reset event is king!
PulseEvent did only appear in Windows NT 4.0. It did not exist in the original Windows NT 3.1. To the contrary, the reliable functions like CreateEvent, SetEvent and WaitForMultipleObjects did exist from start of the Windows NT, so consider using them.
The CreateEvent function has the bManualReset argument. If this parameter is TRUE, the function creates a manual-reset event object, which requires the use of the ResetEvent function to set the event state to non-signaled. This is not what you need. If this parameter is FALSE, the function creates an auto-reset event object, and system automatically resets the event state to non-signaled after a single waiting thread has been released.
These auto-reset events are very reliable and easy to use.
If you wait for an auto-reset event object with WaitForMultipleObjects or WaitForSingleObject, it reliably resets the event upon exit from these wait functions.
So create events the following way:
EventHandle := CreateEvent(nil, FALSE, FALSE, nil);
Wait for the event from one thread and do SetEvent from another thread. This is very simple and very reliable.
Don’t' ever call ResetEvent (since it automatically reset) or PulseEvent (since it is not reliable and deprecated). Even Microsoft has admitted that PulseEvent should not be used. See https://msdn.microsoft.com/en-us/library/windows/desktop/ms684914(v=vs.85).aspx
This function is unreliable and should not be used, because only those threads will be notified that are in the "wait" state at the moment PulseEvent is called. If they are in any other state, they will not be notified, and you may never know for sure what the thread state is. A thread waiting on a synchronization object can be momentarily removed from the wait state by a kernel-mode Asynchronous Procedure Call, and then returned to the wait state after the APC is complete. If the call to PulseEvent occurs during the time when the thread has been removed from the wait state, the thread will not be released because PulseEvent releases only those threads that are waiting at the moment it is called.
You can find out more about the kernel-mode Asynchronous Procedure Calls at the following links:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms681951(v=vs.85).aspx
http://www.drdobbs.com/inside-nts-asynchronous-procedure-call/184416590
http://www.osronline.com/article.cfm?id=75
We have never used PulseEvent in our applications. As about auto-reset events, we are using them since Windows NT 3.51 and they work very well.
What to Do when Multiple Threads Waiting for a Single Object
Unfortunately, your case is a little bit more complicated. You have multiple threads waiting for an event, and you have to make sure that all the threads did in fact receive the notification. There is no other reliable way other than to create own event for each thread.
You wrote theat "the only solution I can see is for each thread to register its usage of a given event object with that object's owner thread". This is correct.
You also wrote that "the owner thread can determine when it is safe to reset the event object's signal state" - this is impractical and unsafe. The best way is to use the auto-reset events, so they will reset themselves automatically.
So, you will need to have as many events as are the threads. Besides that, you will need to keep a list of registered threads. So, to notify all the threads, you will have to do SetEvent in a loop for all the event handles. This is a very fast, reliable and cheap way. Events are much cheaper than threads. So, the number of threads is an issue, not the number of events. There is virtually no limit on the kernel objects - the per-process limit on kernel handles is 2^24.
Use conditional variable as in PulseEvent description. The only problem is that native conditional variable on windows was implemented starting from Vista so older system like XP doesn't have it. But you can emulate conditional variable using some other synchronization objects (http://www1.cse.wustl.edu/~schmidt/win32-cv-1.html) but I think the easiest way is to use conditional variable from boost library and its notify_all method to wake up all threads (http://www.boost.org/doc/libs/1_41_0/doc/html/thread/synchronization.html#thread.synchronization.condvar_ref)
Another possibility (but not very beautiful) is to create one event for each thread and when right now you have PulseEvent you can call SetEvent for all of them. For this solution probably auto-reset events would work better.
Related
Why is HANDLE event object(synchronization object which is created by CreateEvent function) in winapi assumed to be valid in thread function?
From multithreading example to microsoft docs code examples, this event object is passed to WaitForSingleObject function without any protection.
I've been doing the same. And today, I just reached to the thought that how can I deal with this "branch" safe, in such a sense like branch coverage in code perspective.
In the strict sense, this event object is shared along multiple threads, at least in the thread which calls SetEvent and in the thread which Calls WaitForSingleObject.
Therefore, it has to be classified as a type of shared resource. Then, all shared resources must be protected by "lock", such as mutex or critical section.
Also, it is possible to deliberately call CloseHandle after SetEvent while thread is alive, which will lead to passing closed event handle to WaitForSingleObject in thread function. (maybe the event object won't be deleted due to deferred deletion)
Acquiring lock and calling WaitForSingleObject in thread function, and trying to acquire lock in other thread in order to call SetEvent would definitely lead to deadlock.
[EDIT]
Maybe I misled my point by mentioning "assumed" and particular code example. I wonder how to do thread safe validity check for HANDLE event object, treating HANDLE as variable.
According to Synchronizing Execution of Multiple Threads, There are a number of objects whose handles can be used to synchronize multiple threads. These objects include:
Console input buffers
Events
Mutexes
Processes
Semaphores
Threads
Timers
The state of each of these objects is either signaled or not signaled.(atomic)
For handle concerned, WaitForSingleObject function say If this handle is closed while the wait is still pending, the function's behavior is undefined.
For an invalid handle, It's programmer's responsibility to troubleshoot where the handle becomes invalid(BUG).
I have a C++ program that uses overlapped IO for network communication. The main thread has a loop that calls SleepEx(5, true);. There are also two TCP sockets. I assume that the completion callbacks are called during the alertable wait. Assume also that by the time SleepEx gets called both of my TCP connections have received some data. Now the question is what happens if the first completion callback takes longer than 5ms? Does the SleepEx return after calling the first callback or does it also call the second callback? In other words does the SleepEx guarantee to call ALL of the scheduled completion callbacks? This is not clear because the documentation says it will return when at least one of the events meet...
Your code must not assume that both APCs will be called before SleepEx() returns. Conversely, it must not assume that a pending APC will not be called simply because the specified wait period has expired.
The only behaviour that you can rely upon is that if one or more APCs are pending, at least one will be executed.
Generally speaking, best practice is to wait for APCs in a loop that does nothing else, using an infinite timeout in the wait. If you need to do something periodically, you can use a waitable timer to generate an APC periodically.
Alternatively, you can use WaitForSingleObjectEx() or WaitForMultipleObjectsEx() to detect when a waitable timer or other synchronization object is triggered, while still handling APCs.
However, if you must perform some periodic action that cannot be handled in an APC or be triggered by a synchronization object, you can use nested loops: the inner loop does nothing but call the wait repeatedly (with a timeout period reduced by however long the loop has already been running) and the outer loop performs the periodic action.
If you must perform some periodic action that cannot be delayed by pending APCs, you will need to do it in a separate thread. Note that because Windows is not a real-time OS, you will still not be able to guarantee that any given action will take place within any particular timeframe, although you can reduce the risk by increasing the thread priority.
I want to implement a message queue for 2 threads. Thread #1 will pop the messages in queue and process it. Thread #2 will push the messages into queue.
Here is my code:
Thread #1 //Pop message and process
{
while(true)
{
Lock(mutex);
message = messageQueue.Pop();
Unlock(mutex);
if (message == NULL) //the queue is empty
{
//assume that the interruption occurs here (*)
WaitForSingleObject(hWakeUpEvent, INFINITE);
continue;
}
else
{
//process message
}
}
}
Thread #2 //push new message in queue and wake up thread #1
{
Lock(mutex);
messageQueue.Push(newMessage)
Unlock(mutex);
SetEvent(hWakeUpEvent);
}
The problem is there are some cases SetEvent(hWakeUpEvent) will be called before WaitForSingleObject() ( note (*) ), it will be dangerous.
Your code is fine!
There's no actual problem with timing between SetEvent and WaitForSingleObject: the key issue is that WaitForSingleObject on an event will check the state of the event, and wait until it is triggered. If the event is already triggered, it will return immediately. (In technical terms, it's level-triggered, not edge-triggered.) This means that it's fine if SetEvent is called either before or during the call to WaitForSingleObject; WaitForSingleObject will return in either case; either immediately or when SetEvent is called later on.
(BTW, I'm assuming using an Automatic Reset event here. I can't think of a good reason for using a Manual Reset event; you'd just end up having to call ResetEvent immediately after WaitForSingleObject returns; and there's a danger that if your forget this, you could end up Waiting for an event you've already waited for but forgotten to clear. Additionally,it's important to Reset before checking the underlying data state, otherwise if SetEvent is called between when the data is processed and Reset() is called, you lose that information. Stick with Automatic Reset, and you avoid all this.)
--
[Edit: I misread the OP's code as doing a single 'pop' on each wake, rather than only waiting on empty, so the comments below refer to code that scenario. The OP's code is actually equivalent to the second suggested fix below. So the text below is really describing a somewhat common coding error where events are used as through they were semaphores, rather than the OP's actual code.]
But there is a different problem here [or, there would be if there was only one pop per wait...], and that's that Win32 Events objects have only two states: unsignaled and signaled, so you can use them only to track binary state, but not to count. If you SetEvent and event that's already signaled, it remains Signaled, and the information of that extra SetEvent call is lost.
In that case, what could happen is:
Item is added, SetEvent called, event is now signaled.
Another item is added, SetEvent is called again, event stays signaled.
Worker thread calls WaitForSingleObject, which returns, clearing the event,
only one item is processed,
worker thread calls WaitForsingleObject, which blocks because the event is unsignaled, even though there's still an item in the queue.
There's two ways around this: the classic Comp.Sci way is to use a semaphore instead of an event - semaphores are essentially events that count up all the 'Set' calls; you could conversely think of an event as a semaphore with a max count of 1 which ignores any other signals beyond that one.
An alternative way is to continue using events, but when the worker thread wakes up, it can only assume that there may be some items in the queue, and it should attempt to process them all before it returns to waiting - typically by putting the code that pops the item in a loop that pops items and processes them until its empty. The event is now used not to count, but rather to signal "the queue is no longer empty". (Note that when you do this, you can also get cases where, while processing the queue, you also process an item that was just added and for which SetEvent was called, so that when the worker thread reaches WaitForSingleObject, the thread wakes up but finds the queue is empty as the item has already been processed; this can seem a bit surprising at first, but is actually fine.)
I view these two as mostly equivalent; there's minor pros and cons to both, but they're both correct. (Personally I prefer the events approach, since it decouples the concept of "something needing to be done" or "more data is available" from the quantity of that work or data.)
The 'classic' way, (ie. will surely work correctly), is to use a semaphore, (see CreateSemaphore, ReleaseSemaphore API). Create the semaphore empty. In the producer thread, lock the mutex, push the message, unlock the mutex, release a unit to the semaphore. In the consumer thread, wait on the semaphore handle with WFSO, (like you wait on the event above), then lock the mutex, pop a message, unlock the mutex.
Why is this better than events?
1) No need to check the queue count - the semaphore counts the messages.
2) A signal to the semaphore is not 'lost' just because no thread is waiting on it.
3) Not checking the queue count means that result from, and code path taken as a result of, such checking cannot be incorrect because of preemption.
4) It will work for multiple producers and multiple consumers without change.
5) It is more cross-platform friendly - all preemptive OS have mutexes/semaphores.
It would be dangerous if there were several threads consuming data at the same time, or if you used PulseEvent instead of SetEvent.
But with only one consumer, and since the event will be kept signaled until you wait into it (if auto rest) or forever (if manual reset), it should just work.
I have a pthread that I created and now I want that in a specific time interval the thread execute some code. But the user should also be able to cancel the thread. How can I cancel a thread and ensure that the thread is not cancelled when it execute the code?
In Java you handle this with
while(!isInterrupted)
Is there any similar solution with pthreads.
In the Question's example code you are checking some variable. This is not the normal pattern for interrupting threads in Java.
In Java, you interrupt a thread by calling the interrupt() method.
The thread then checks if it is interrupted inside IO and system calls (which can throw InterruptedException when this happens; this means a thread that is sleeping or waiting on IO can be awoken when interrupted) or by sampling the isInterrupted() flag (typically used in a condition in a loop, as in Question).
The distinction is important; checking some flag variable you've declared is only possible in loops and your own code; the Java interrupting system works for all threads and all non-CPU-blocking code without special effort on the part of the programmer.
Pthreads has the pthread_cancel() pattern which works like the Java interrupting pattern.
pthread_cancel is available for sending cancel requests:
A thread's cancellation type, determined by pthread_setcanceltype(3), may be
either asynchronous or deferred (the default for new threads). Asynchronous
cancelability means that the thread can be canceled at any time (usually
immediately, but the system does not guarantee this). Deferred cancelability
means that cancellation will be delayed until the thread next calls a function
that is a cancellation point. A list of functions that are or may be
cancellation points is provided in pthreads(7).
A thread's cancelability state, determined by pthread_setcancelstate(3), can
be enabled (the default for new threads) or disabled. If a thread has
disabled cancellation, then a cancellation request remains queued until the
thread enables cancellation. If a thread has enabled cancellation, then its
cancelability type determines when cancellation occurs.
So there are several options:
1: while value checking (works very well, but you don't have much control).
2: check the pthread_cancel manpage, it works to but with strict rules.
3: using pthread_signal, first you need to block, than signal for resume. It has the same issues as the second option.
Using pthreads cancel and signal will only work from within the thread that must be locked. So setting a variable to initiate the signal block. Unlocking can be done by any other thread.
The same can be done using mutex or semaphores (pthread_mutex, pthread_semaphore).
A site I recommend: http://www.yolinux.com/TUTORIALS/LinuxTutorialPosixThreads.html
There's no specific function to cancel a threadYou can use pthread_cancel to cancel the thread, as mentioned (but I would advise against it, unless you know what you're doing), and you have to set up your own timers. But the while(!isInterrupted) is pretty acceptable way of doing it.
It should basically be like this:
while(!isInterrupted)
{
// whatever you want to do
sleep(howLongYouWantToWait);
}
// clean up and exit the thread function here
and in the main thread have a global (or other, see below)
volatile bool isInterrupted = false;
and set it to true when you're done, and pthread_join if you want to wait for the thread to finish.
Instead of global, you can use a class variable, or a flag pointer passed to the thread function, or any other way, global is the simplest and the least preferable.
Of course, if you want to cancel the thread while it waits, and not to have it canceled only after it finishes the whole loop, then you need to deal with signals, and other stuff, but I think you're not looking for that.
I have a main program which creates a collection of N child threads to perform some calculations. Each child is going to be fully occupied on their tasks from the moment their threads are created till the moment they have finished. The main program will also create a special (N+1)th thread which has some intermittent tasks to perform. When certain conditions are met (like a global variable takes on a certain value) the special thread will perform a calculation and then go back to waiting for those conditions to be met again. It is vital that when the N+1th thread has nothing to do, it should not slow down the other processors.
Can someone suggest how to achieve this.
EDIT:
The obvious but clumsy way would be like this:
// inside one of the standard worker child threads...
if (time_for_one_of_those_intermittent_calculations_to_be_done())
{
global_flag_set = TRUE;
}
and
// inside the special (N+1)th thread
for(;;)
{
if (global_flag_set == TRUE)
{
perform_big_calculation();
global_flag_set = FALSE;
}
// sleep for a while?
}
You should check out the WaitForSingleObject and WaitForMultipleObjects functions in the Windows API.
WaitForMultipleObjects
A ready-to-use condition class for WIN32 ;)
class Condition {
private:
HANDLE m_condition;
Condition( const Condition& ) {} // non-copyable
public:
Condition() {
m_condition = CreateEvent( NULL, TRUE, FALSE, NULL );
}
~Condition() {
CloseHandle( m_condition );
}
void Wait() {
WaitForSingleObject( m_condition, INFINITE );
ResetEvent( m_condition );
}
bool Wait( uint32 ms ) {
DWORD result = WaitForSingleObject( m_condition, (DWORD)ms );
ResetEvent( m_condition );
return result == WAIT_OBJECT_0;
}
void Signal() {
SetEvent( m_condition );
}
};
Usage:
// inside one of the standard worker child threads...
if( time_for_one_of_those_intermittent_calculations_to_be_done() ) {
global_flag_set = TRUE;
condition.Signal();
}
// inside the special (N+1)th thread
for(;;) {
if( global_flag_set==FALSE ) {
condition.Wait(); // sends thread to sleep, until signalled
}
if (global_flag_set == TRUE) {
perform_big_calculation();
global_flag_set = FALSE;
}
}
NOTE: you have to add a lock (e.g. a critical section) around global_flag_set. And also in most cases the flag should be replaced with a queue or at least a counter (a thread could signal multiple times while 'special' thread is performing its calculations).
Yes. Use condition variables. If you sleep on a condition variable, the thread will be removed from the runqueue until the condition variable has been signaled.
You should use Windows synchronization events for this, so your thread is doing nothing while waiting. See MSDN for more info; I'd start with CreateEvent(), and then go to the rest of the Event-related functions here for OpenEvent(), PulseEvent(), SetEvent() and ResetEvent().
And, of course, WaitForSingleObject() or WaitForMultipleObjects(), as pointed out by mrduclaw in the comment below.
Lacking the more preferred options already given, I generally just yield the CPU in a loop until the desired condition is met.
Basically, you have two possibilities for your N+1th thread.
If its work is rare, the best thing to do is simply to ask it to sleep, and wake it up on demand. Rare context switches are insignificants.
If it has to work often, then you may need to spinlock it, that is, a busy waiting state that prevent it from being rescheduled, or switched.
Each global variable should have an accompanying event for your N+1 thread. Whenever you change the status of the global variable, set the event to the signaled state. It is better to hide these variables inside a singleton-class private properties and expose functions to get and set the values. The function that sets the value will do the comparison and will set the events if needed. So, your N+1 thread will just to the loop of WaitForMultipleObjects with infinite timeout. Another global variable should be used to signal that the application as a whole exits, so the threads will be able to exit. You may only exit your application after your last thread has finished. So, if you need to prematurely exit, you have to notify all your threads that they have to exit. Those threads that are permanently running, can be notified by just reading a variable periodically. Those that are waiting, like the N+1 thread, should be notified by an event.
People have suggested to use CreateEvent (to create auto-reset events), SetEvent and WaitForMultipleObjects. I agree with them.
Other people have suggested, in addition to the above functions, to use ResetEvent and PulseEvent. I do not agree with them. You don’t need ResetEvent with auto-reset events. This is the function supposed to be used with manual-reset events, but the application of the manual-reset events is very limited, you will see below.
To create an auto-reset event, call the CreateEvent Win32 API function with the bManualReset parameter set to FALSE (if it is TRUE, the function creates a manual-reset event object, which requires the use of the ResetEvent function to set the event state to non-signaled – this is not what you need). If this parameter is FALSE, the function creates an auto-reset event object, and system automatically resets the event state to non-signaled after a single waiting thread has been released, i.e. has exited from a function like WaitForMultipleObjects or WaitForSigleObject – but, as I wrote before, only one thread will be notified, not all, so you need one event for each of the threads that are waiting. Since you are going to have just one thread that will be waiting, you will need just one event.
As about the PulseEvent – it is unreliable and should never be used -- see https://msdn.microsoft.com/en-us/library/windows/desktop/ms684914(v=vs.85).aspx
Only those threads are notified by PulseEvent that are in the "wait" state at the moment PulseEvent is called. If they are in any other state, they will not be notified, and you may never know for sure what the thread state is. A thread waiting on a synchronization object can be momentarily removed from the wait state by a kernel-mode Asynchronous Procedure Call, and then returned to the wait state after the APC is complete. If the call to PulseEvent occurs during the time when the thread has been removed from the wait state, the thread will not be released because PulseEvent releases only those threads that are waiting at the moment it is called. You can find out more about the kernel-mode Asynchronous Procedure Calls (APC) at the following links:
- https://msdn.microsoft.com/en-us/library/windows/desktop/ms681951(v=vs.85).aspx
- http://www.drdobbs.com/inside-nts-asynchronous-procedure-call/184416590
- http://www.osronline.com/article.cfm?id=75
You can get more ideas about auto-reset events and manual reset events from the following article:
- https://www.codeproject.com/Articles/39040/Auto-and-Manual-Reset-Events-Revisited
As about the the Manual-Reset events, they too can be used under certain conditions and in certain cases. You can reliably use them when you need to notify multiple instances of a global state change that occurs only once, for example application exit.
You just have one waiting thread, but maybe in future you will have more waiting threads, so this information will be useful.
Auto-reset events can only be used to notify one thread (if more threads are waiting simultaneously for an auto-reset event and you set the event, just one thread will exit and resets it, and the behavior of other threads will be undefined). From the Microsoft documentation, we may assume that only one thread will exit while others would not, this is not very clear. However, we must take the following quote into consideration: “Do not assume a first-in, first-out (FIFO) order. External events such as kernel-mode APCs can change the wait order” Source - https://msdn.microsoft.com/en-us/library/windows/desktop/ms682655(v=vs.85).aspx
So, when you need to very quickly notify all the threads – just set the manual-reset event to the signaled state (by calling the SetEvent), rather than signaling each auto-reset event for each thread. Once you have signaled the manual-reset event, do not call ResetEvent since then. The drawback of this solution is that the threads need to have an additional event handle passed in the array of their WaitForMultipleObjects. The array size is limited, although to MAXIMUM_WAIT_OBJECTS which is 64, and in practice we did never reach close to this limit.
At the first glance, Microsoft documentation may seem to be full of jargon, but over time you will find it very easy and friendly. Anyway, correct multi-threaded work is not an easy topic, so you have to tolerate a certain amount of jargon 😉