I'm programming an interprocess communication module (Process A and Process B).
Is there any way the communication thread in B can run (be unlock) as soon as process A finishes a specific action, I mean without B doing any polling nor B waiting too much after A finishes its action?
Which are the concepts/models/design patterns governing these issues? (Something more precise than interprocess synchronization). Which libraries/methods do you recommend?
Thanks.
Edit: I'm looking for methods suitable for each of the three main OSes: Ms Windows, Apple Mac OS X, GNU/Linux.
This is quite hard job:
For Unix OSes you can use:
pthread condition and mutex with setpshared argument.
Note: it is supported well under Linux 2.6, Solaris, but it does not supported FreeBSD and Cygwin (don't know about Mac OS X)
For Unixes you may also use named semaphores, but I don't know the support level of them
For Windows there are some events...
This is hard job, especially for IPC...
So if you want something portable, I'd suggest to take a look on Boost.Interprocess that has conditions and mutexes...
But make sure that all feature supported on all OSes you want to support.
Things you should note about Boost.Interprocess
Check carefully level of support for each Unix OS you need to work with, because Boost.Interprosess uses pthread_* functions that are not always supported... and then fails back to emulation -- check the quality of such emulation
Also, check how this stuff works on Windows -- as far as I know that there is no "in-shared-memory" mutexes in Win32 API, generally named objects should be used, so check what is supported and how.
EDIT: I mistakenly thought you needed inter thread synchronizing, Revised for IPC
I think you need something like waitable events.
In Windows you can use CreateEvent(), to create (or get an existing) named, auto-reset event.
When process A completes processing, it should call SetEvent(), while process B should call WaitForSingleObject() to sleep until completion (or timeout).
Alternately, you can use semaphores created by CreateSemaphore(), initialized to 0.
Process A signals completion by calling ReleaseSemaphore(), while process B again uses WaitForSingleObject() to wait for completion.
Under Linux and OS X you can use semaphores to a similar effect.
use sem_open() to create a named semaphore, with 0 as its initial value.
When process A completes, it should call sem_post() to increment the semaphore, while process B should call sem_wait() to sleep until completion.
NOTE: the semaphore method may allow multiple completions to be signaled, you should handle this by setting a maximum count under Windows, or checking the current sem value for sanity with sem_getvalue()
I think condition variables fit what you're trying to do, here's a sample that would work on Linux and OSX
#include <pthread.h>
/* no error checking, quick and dirty sample */
pthread_mutex_t g_mutex;
pthread_cond_t g_cond;
int a_done = 0;
void init(void)
{
pthread_mutex_init(&g_mutex, NULL);
pthread_cond_init(&g_cond, NULL);
}
void thread_a(void *arg)
{
/* do something here... */
pthread_mutex_lock(&g_mutex);
a_done = 1;
pthread_cond_signal(&g_cond);
pthread_mutex_unlock(&g_mutex);
}
void thread_b(void *arg)
{
/* wait for a to complete */
pthread_mutex_lock(&g_mutex);
while (!a_done)
pthread_cond_wait(&g_cond, &g_mutex);
a_done = 0;
pthread_mutex_unlock(&g_mutex);
}
Under Windows, you can use pthreads-win32, or native condition variables under Vista, see the MSDN Condition Variables page for more information.
References:
pthread_cond_wait
pthread_cond_signal
If your OS supports signals you could unlock a mutex from a signal handler and send the signal from process A as soon as you finish the task.
Process B would be waiting on a mutex or other synchronization tool and A would be working on whatever, then when finishes sends signal USR1 for example, and USR1 handler in process B unlocks the corresponding synchronization tool.
The most common is to use select()/poll(). Both can check several file descriptors if there's input available. Both receive a timeout parameter - this will prevent busy wait, which may consume 100% CPU. This is very suitable solution for small/medium applications.
Another approach is to make polling in separate thread.
If you're going to develop a big application it's worth to look towards ACE framework or
boost. These frameworks are cross platform solutions, well designed and well tested.
Well, in my opinion and experience, the best way to do that in a portable and simple way is to use sockets. Plus you get the possibility to have the two processes on different machine (if needed). Plus you can extand the communication to handle more than synchro.
If you don't want to poll, use a thread that wait for a synchro message on a socket. You read the socket in a blocking way. When you receive the message you use a standard multithread synchronisation to handle your synchronisation. In your case as B should wait till A ends, you just have to read in a blocking way in your process.
To be portable use a portable socket library like boost or ptypes or whatever.
Related
Linux is a new platform to me. I've coded on Windows in c++ for a number of years and have become comfortable with multithreading on that platform.
Along comes C++11 at a time when I need to learn c++ on the linux platform.
Linux appears to use pthreads for the most part - okay there's also boost::threads and QT have their own threads too. But with C++11 comes std::thread, a whole new (cross platform and C++ standard) way to do threads.
So I guess I'll have to learn pthreads and std::threads. Ultimately, std::thread seems more important, but there's a lot of legacy code out there, so I'll have to know both.
For thread synchronization on windows, I would use WaitForMultipleObjects to wait for a number of tasks to complete before continuing with further work.
Does a similar synchronization mechanism exist for pthreads? std::threads?
I've had a look at pthread_join, and it seems to have the facility to only wait on one thread at a time. Am I missing another pthread call maybe?
std::thread is boost::thread accepted into C++11 with some extras. My understanding is that if boost::thread gets replaced in code with std::thread it should still compile and work.
boost::thread is based on pthreads design, providing thin C++ wrappers over thread, mutex and condition variables. Thread cancellation though was left outside the scope of C++11, since there was no agreement how it should work in C++.
So, by learning pthreads you also learn std::thread concepts. std::thread adds mostly syntax sugar and convenience functions on top of pthreads C API.
With regards to WaitForMultipleObjects(), neither pthreads nor std::thread provide anything similar to its bWaitAll=FALSE mode, however, it's routinely simulated using pipes and select() on UNIX, or more modern eventfd() and epoll() on Linux. bWaitAll=TRUE mode can be simulated by waiting on all tasks in turn, since it doesn't proceed until all objects are ready anyway.
No, neither pthreads nor C++11 has direct equivalent of WaitForMultipleObjects (i.e. wait for any waitable "handle" type.) pthread_join can only be used to join threads, and only a single, specific thread.
The closest equivalent on posix platforms is to wait for multiple file descriptors using system calls such as select(), poll() or the linux-specific epoll(), but they require you to have a file descriptor to wait on, which is fine for I/O events but requires extra work from you to use them wait for mutexes, condition variables or other synchronisation objects. There are more general event libraries built on top of those system calls, e.g. libevent and libev and Boost ASIO, which support waiting for timers as well as I/O, but still not thread completion, mutex locks etc. with a single function like WaitForMultipleObjects
The alternatives you do have for pthreads and C++11 threads are to wait on different synchronisation types separately. You can wait for timers, wait for threads to complete, wait for mutexes, wait on condition variables, wait for asynchronous results to be ready (std::async in C++11, no direct equivalent in pthreads) ... but there's no call that will allow you to wait a heterogeneous set of those types all at once.
I could give you a really fancy answer but alas, this is where I learned them and it is a good introduction:
http://www.yolinux.com/TUTORIALS/LinuxTutorialPosixThreads.html
You use pthread_mutex_t for syncronization and pthread_join probably handles the wait for multiple tasks problem. It works exactly as you would expect.
Based on this, you must call pthread_join for each single thread you have created. Or to use mutexes, if there is a need to synchronize your threads.
Regarding WaitForMultipleObjects, this is generally called a Barrier Sync. Boost has an implementation called barrier. It uses conditional variables to implement it, in posix its a pthread_cond_t
Here is an answer I left recently explaining barrier sync.
I'm looking for an API on windows that enables to to create and kill threads at will. Also having ability to bind threads to cores. I was introduced to Win32 Threading API here.
However when I checked MSDN I see _beginthreadex(), and _endthreadex(). So I'm guessing there should be a call to _endthreadex everytime I create a thread?
To get answers to such questions I'm looking for a tutorial on Windows Threading. Can anyone help with this?
P.S. This may be off topic, but does Boost support thread affinity too? If so, can someone point me to a tutorial/documentation related to thread affinity?
Having thread created (such as with _beginthreadex) you need to let the thread exit gracefully as you never know if it is in the middle of something just now (having a lock on a certain resource - for instance). Still you have an option to blow it away with TerminateThread API any time.
SetThreadAffinityMask and friends let you locate your threads at the CPU battlefield. You might end up leaving OS scheduler to choose cores to run your threads on though, as chances are high that it is going to be more efficient.
Update on reusing threads: Creating a thread you are passing your thread proc to start, and as soon as you return from it, the thread is about to be terminated. That is, starting another worker thread activity is possible in two ways: either create a new thread from the start, or do not exit from thread proc and synchronize to catch up a new worker activity request. The latter might be implemented using IPC objects, e.g. events:
int ThreadProc()
{
while(true)
{
wait for new event;
if(termination requested) break;
otherwise, on worker activity request, do next requested task;
}
}
Refer to Thread Synchronization for Beginners for sample code and description.
If you are using MFC, you can better use CWinThread. You can send messages to the thread very easily and can control the thread's behaviour from outside. Using the thread's handle, you can provide an affinity mask for a thread using SetThreadAffinityMask, which will schedule a thread on desired processor(s).
1) Do not mix up _beginthread/_beginthreadex and the Win32 API Function CreateThread. These are two different APIs. See Other SO Post for details.
2) If you use _beginthread/_beginthreadex, _endthread/_endthreadex should be used for termination
3) TerminateThread (and also _endthread) should not be used under normal conditions. See MSDN Post.
4) Functions such as SetThreadAffinityMask, or SetThreadIdealProcessor can be used to set the core a thread should use.
5) The boost threading API is much more robust and simple. Actually its the base of the C++11 threads.
I'm working on a project, where a primary server thread needs to dispatch events to a series of worker threads. The work that goes on in the worker threads relies on polling (ie. epoll or kqueue depending on the UNIX system in question) with timeouts on these operations needing to be handles. This means, that a normal conditional variable or semaphore structure is not viable for this dispatch, as it would make one or the other block resulting in an unwanted latency between either handling the events coming from polling or the events originating from the server thread.
So, I'm wondering what the most optimal construct for dispatching such events between threads in a pollable fashion is? Essentially, all that needs to be delivered is a pollable "signal" that tells the worker thread, that it has more events to fetch. I've looked at using UNIX pipes (unnamed ones, as it's internal to the process) which seems like a decent solution given that a single byte can be written to the pipe and read back out when the queue is cleared -- but, I'm wondering if this is the best approach available? Or the fastest?
Alternatively, there is the possibility to use signalfd(2) on Linux, but as this is not available on BSD systems, I'd rather like to avoid this construct. I'm also wondering how great the overhead in using system signals actually is?
Jan Hudec's answer is correct, although I wouldn't recommend using signals for a few reasons:
Older versions of glibc emulated pselect and ppoll in a non-atomic fashion, making them basically worthless. Even when you used the mask correctly, signals could get "lost" between the pthread_sigprocmask and select calls, meaning they don't cause EINTR.
I'm not sure signalfd is any more efficient than the pipe. (Haven't tested it, but I don't have any particular reason to believe it is.)
signals are generally a pain to get right. I've spent a lot of effort on them (see my sigsafe library) and I'd recommend avoiding them if you can.
Since you're trying to have asynchronous handling portable to several systems, I'd recommend looking at libevent. It will abstract epoll or kqueue for you, and it will even wake up workers on your behalf when you add a new event. See event.c
2058 static inline int
2059 event_add_internal(struct event *ev, const struct timeval *tv,
2060 int tv_is_absolute)
2061 {
...
2189 /* if we are not in the right thread, we need to wake up the loop */
2190 if (res != -1 && notify && EVBASE_NEED_NOTIFY(base))
2191 evthread_notify_base(base);
...
2196 }
Also,
The worker thread deals with both socket I/O and asynchronous disk I/O, which means that it is optimally always waiting for the event queuing mechanism (epoll/kqueue).
You're likely to be disappointed here. These event queueing mechanisms don't really support asynchronous disk I/O. See this recent thread for more details.
As far as performance goes, the cost of system call is comparably huge to other operations, so it's the number of system calls that matters. There are two options:
Use the pipes as you wrote. If you have any useful payload for the message, you get one system call to send, one system call to wait and one system call to receive. Try to pass any relevant data down the pipe instead of reading them from a shared structure to avoid additional overhead from locking.
The select and poll have variants, that also waits for signals (pselect, ppoll). Linux epoll can do the same using signalfd, so it remains a question whether kqueue can wait for signals, which I don't know. If it can, than you could use them (you are using different mechanism on Linux and *BSD anyway). It would save you the syscall for reading if you don't have good use for the passed data.
I would expect passing the data over socket to be more efficient if it allows you do do away with any other locking.
Hello I have some code that is cross-platform by unsing #ifdef OS,
I have a Queue protected by a CriticalSection on Windows, and by a pthread_mutex_t on Linux.
I would like to implement a Wait(timeout) call that would block a thread until something has been enqueued. I though about using WaitForSingleObject on windows but it don't seem to support CriticalSection. Which Win32 and which Linux functions should I use to Wait and Signal for a condition to happen.
Thank
I think that boost's conditions might be what you need. It is crossplatform so you won't have to bother with different implementations depending on OS.
Another alternative is to use Windows Events with WaitForSingleObject() and the quite new linux eventfd() with select() or poll().
Using Boost will allow you to do threading and synchronization for both platforms without a bunch of ifdefs.
Seems like conditions variable is what I was looking for.
On windows They work with critical Section and SleepConditionVariableCS
On linux pthread_cond_timedwait work with pthread.
Thanks all.
With pthread, a condition variable.
On Windows, it looks like you want a Semaphore. Win32 Semaphores have a counter that starts at zero - at which point the handle is not signaled. As you add items to the queue, you would increase the semaphore counter with ReleaseSemaphore - Each count added to the semaphore will satisfy one call to a WaitforXXXObject function, so, if you added 3 items to a queue, you would ReleaseSemaphore with a count of 3. WaitFor... would then return 3 times before the handle became non signalled again.
You can (sort of) simulate a try with a timeout using TryEnterCriticalSection, but for the most part, if you want a timeout you might be better off using a mutex instead (when you get down to it, a critical section is mostly a wrapper around a mutex).
Another possibility would be to use the Win32 pthreads library, which will probably let your Linux code compile under Win32 unchanged (and you'd simply eliminate your own Win32 code).
This is pretty similar to the threading support that's been added to the C++ 0x library, though it doesn't (even try to) follow the new standard precisely. If you want to follow the standard, you could use Anthony Williams' Just Thread library (warning: fairly reasonably priced, but not free in either sense).
Edit (in response to Billy O'neal's questions): Thinking about it a bit more, there actually is source code easily available that shows most of what's going on. The CRITICAL_SECTION data structure is defined in winbase.h as a typedef of an RTL_CRITICAL_SECTION. That, in turn, is defined in WinNT.h as:
typedef struct _RTL_CRITICAL_SECTION {
PRTL_CRITICAL_SECTION_DEBUG DebugInfo;
//
// The following three fields control entering and exiting the critical
// section for the resource
//
LONG LockCount;
LONG RecursionCount;
HANDLE OwningThread; // from the thread's ClientId->UniqueThread
HANDLE LockSemaphore;
ULONG_PTR SpinCount; // force size on 64-bit systems when packed
} RTL_CRITICAL_SECTION, *PRTL_CRITICAL_SECTION;
If memory serves, the basic idea of how this is used runs something like:
If this thread already owns the critical section, increment RecursionCount and return
Otherwise, do SpinCount attempts to enter CS via fast path using atomic ops on LockCount
Otherwise, wait on LockSemaphore
I am writing an application which blocks on input from two istreams.
Reading from either istream is a synchronous (blocking) call, so, I decided to create two Boost::threads to do the reading.
Either one of these threads can get to the "end" (based on some input received), and once the "end" is reached, both input streams stop receiving. Unfortunately, I cannot know which will do so.
Thus, I cannot join() on both threads, because only one thread (cannot be predetermined which one) will actually return (unblock).
I must somehow force the other to exit, but it is blocked waiting for input, so it cannot itself decide it is time to return (condition variables or what not).
Is their a way to either:
Send a signal a boost::thread, or
Force an istream to "fail", or
Kill a Boost::thread?
Note:
One of the istreams is cin
I am trying to restart the process, so I cannot close the input streams in a way that prohibits reseting them.
Edit:
I do know when the "end" is reached, and I do know which thread has successfully finished, and which needs to be killed. Its the killing I need to figure out (or a different strategy for reading from an istream).
I need both threads to exit and cleanup properly :(
Thanks!
I don't think there is a way to do it cross platform, but pthread_cancel should be what you are looking for. With a boost thread you can get the native_handle from a thread, and call pthread_cancel on it.
In addition a better way might be to use the boost asio equivalent of a select call on multiple files. That way one thread will be blocked waiting for the input, but it could come from either input stream. I don't know how easy it is to do something like this with iostreams though.
Yes there is!
boost::thread::terminate() will do the job to your specifications.
It will cause the targeted thread to throw an exception. Assuming it's uncaught, the stack will unwind properly destroying all resources and terminating thread execution.
The termination isn't instant. (The wrong thread is running at that moment, anyway.)
It happens under predefined conditions - the most convenient for you would probably be when calling boost::this_thread::sleep();, which you could have that thread do periodically.
If a boost thread is blocking on an i/o operation (e.g. cin>>whatever), boost::thread::terminate() will not kill the thread. cin i/o is not a valid termination point. Catch 22.
Well on linux, I use pthread_signal(SIGUSR1), as it interrupts blocking IO. There no such call on windows as I discovered when porting my code. Only a deprecated one in socket reading call. In windows you have to explicitly define an event that will interrupt your blocking call. So there no such thing (AFAIK) as a generic way to interrupt blocking IO.
The boost.thread design handle this by managing well identified interrupt points. I don't know boost.asio well and it seems that you don't want to rely on it anyway. If you don't want to refactor to use non-blocking paradigm, What you can do is using something between non-blocking (polling) and blocking IO. That is do something like (pseudo code ?) :
while(!stopped && !interrupted)
{
io.blockingCall(timeout);
if(!stopped && !interrupted)
{
doSomething();
}
}
Then you interrupt your two threads and join them ...
Perhaps it is simpler in your case ? If you have a master thread that knows one thread is ended you just have to close the IO of the other thread ?
Edit:
By the way I'm interested in the final solution you have ...
I had a similar issue myself and have reached this solution, which some other readers of this question might find useful:
Assuming that you are using a condition variable with a wait() command, it is important for you to know that in Boost, the wait() statement is a natural interrupt point. So just put a try/catch block around the code with the wait statement and allow the function to terminate normally in your catch block.
Now, assuming you have a container with your thread pointers, iterate over your thread pointers and call interrupt() on each thread, followed by join().
Now all of your threads will terminate gracefully and any Boost-related memory cleanup should work cleanly.
Rather than trying to kill your thread, you can always tryjoin the thread instead, and if it fails, you join the other one instead. (Assuming you will always be able to join at least one of your two threads).
In boost:thread you're looking for the timed_join function.
If you want to look at the correct answer, however, that would be to use non-blocking io with timed waits. Allowing you to get the flow structure of synchronous io, with the non-blocking of asynchronous io.
You talk about reading form an istream, but an istream is only an interface. for stdin, you can just fclose the stdin file descriptor to interrupt the read. As for the other, it depends an where you're reading from...
It seems that threads are not helping you do what you want in a simple way. If Boost.Asio is not to your liking, consider using select().
The idea is to get two file descriptors and use select() to tell you which of them has input available. The file descriptor for cin is typically STDIN_FILENO; how to get the other one depends on your specifics (if it's a file, just open() it instead of using ifstream).
Call select() in a loop to find out which input to read, and when you want to stop, just break out of the loop.
Under Windows, use QueueUserAPC to queue a proc which throws an exception. That approach works fine for me.
HOWEVER: I've just found that boost mutexes etc are not "alertable" on win32, so QueueUserAPC cannot interrupt them.
Very late, but in Windows (and it's precursors like VMS or RSX for those that rember such things) I'd use something like ReadFileEx with a completion routine that signals when finished, and CancelIO if the read needs to be cancelled early.
Linux/BSD has an entirely different underlying API which isn't as flexible. Using pthread_kill to send a signal works for me, that will stop the read/open operation.
It's worth implementing different code in this area for each platform, IMHO.