In a multithreaded C++ program where the main thread is executing a libuv event loop, is it guaranteed that this event loop thread is executing signal handlers registered using uv_signal_start?
Background information:
From http://docs.libuv.org/en/v1.x/design.html
The I/O (or event) loop is [...] meant to be tied to a single thread.
But as we are in a multithreaded program, signal handlers can be executed by other threads
According to POSIX.1, a process-directed signal (sent using kill(2), for example) should be handled by a single, arbitrarily selected thread within the process.
So my question is basically whether libuv signal handling works as advertised
Signal handles implement Unix style signal handling on a per-event loop bases.
even in multithreaded programs.
TLDR: Yes, should work as advertised.
From my understanding of libuv's source code unix/signal.c there is a generic signal handler
static void uv__signal_handler(int signum) {
uv__signal_msg_t msg;
uv_signal_t* handle;
int saved_errno;
saved_errno = errno;
memset(&msg, 0, sizeof msg);
if (uv__signal_lock()) {
errno = saved_errno;
return;
}
for (handle = uv__signal_first_handle(signum);
handle != NULL && handle->signum == signum;
handle = RB_NEXT(uv__signal_tree_s, &uv__signal_tree, handle)) {
int r;
msg.signum = signum;
msg.handle = handle;
/* write() should be atomic for small data chunks, so the entire message
* should be written at once. In theory the pipe could become full, in
* which case the user is out of luck.
*/
do {
r = write(handle->loop->signal_pipefd[1], &msg, sizeof msg);
} while (r == -1 && errno == EINTR);
assert(r == sizeof msg ||
(r == -1 && (errno == EAGAIN || errno == EWOULDBLOCK)));
if (r != -1)
handle->caught_signals++;
}
uv__signal_unlock();
errno = saved_errno;
}
in which a pipe handle->loop->signal_pipefd[1] is used to tell the handle's associated loop abount the incoming signal. Indeed, this generic signal handler can be called from any thread, however the libuv thread will then call the user's specific signal handler registered with uv_signal_start in the event loop thread (main thread in my setting) when it reads the signal_pipefd[1] in the next loop iteration.
This was for the unix source code and the windows win/signal.c source has a similar mechanism.
So the answer should be yes, it should also work as advertised in a multithreaded setting, i.e. the registered handler will be executed by the loop thread.
Related
Threads are resource-heavy to create and use, so often a pool of threads will be reused for asynchronous tasks. A task is packaged up, and then "posted" to a broker that will enqueue the task on the next available thread.
This is the idea behind dispatch queues (i.e. Apple's Grand Central Dispatch), and thread handlers (Android's Looper mechanism).
Right now, I'm trying to roll my own. In fact, I'm plugging a gap in Android whereby there is an API for posting tasks in Java, but not in the native NDK. However, I'm keeping this question platform independent where I can.
Pipes are the ideal choice for my scenario. I can easily poll the file descriptor of the read-end of a pipe(2) on my worker thread, and enqueue tasks from any other thread by writing to the write-end. Here's what that looks like:
int taskRead, taskWrite;
void setup() {
// Create the pipe
int taskPipe[2];
::pipe(taskPipe);
taskRead = taskPipe[0];
taskWrite = taskPipe[1];
// Set up a routine that is called when task_r reports new data
function_that_polls_file_descriptor(taskRead, []() {
// Read the callback data
std::function<void(void)>* taskPtr;
::read(taskRead, &taskPtr, sizeof(taskPtr));
// Run the task - this is unsafe! See below.
(*taskPtr)();
// Clean up
delete taskPtr;
});
}
void post(const std::function<void(void)>& task) {
// Copy the function onto the heap
auto* taskPtr = new std::function<void(void)>(task);
// Write the pointer to the pipe - this may block if the FIFO is full!
::write(taskWrite, &taskPtr, sizeof(taskPtr));
}
This code puts a std::function on the heap, and passes the pointer to the pipe. The function_that_polls_file_descriptor then calls the provided expression to read the pipe and execute the function. Note that there are no safety checks in this example.
This works great 99% of the time, but there is one major drawback. Pipes have a limited size, and if the pipe is filled, then calls to post() will hang. This in itself is not unsafe, until a call to post() is made within a task.
auto evil = []() {
// Post a new task back onto the queue
post({});
// Not enough new tasks, let's make more!
for (int i = 0; i < 3; i++) {
post({});
}
// Now for each time this task is posted, 4 more tasks will be added to the queue.
});
post(evil);
post(evil);
...
If this happens, then the worker thread will be blocked, waiting to write to the pipe. But the pipe's FIFO is full, and the worker thread is not reading anything from it, so the entire system is in deadlock.
What can be done to ensure that calls to post() eminating from the worker thread always succeed, allowing the worker to continue processing the queue in the event it is full?
Thanks to all the comments and other answers in this post, I now have a working solution to this problem.
The trick I've employed is to prioritise worker threads by checking which thread is calling post(). Here is the rough algorithm:
pipe ← NON-BLOCKING-PIPE()
overflow ← Ø
POST(task)
success ← WRITE(task, pipe)
IF NOT success THEN
IF THREAD-IS-WORKER() THEN
overflow ← overflow ∪ {task}
ELSE
WAIT(pipe)
POST(task)
Then on the worker thread:
LOOP FOREVER
task ← READ(pipe)
RUN(task)
FOR EACH overtask ∈ overflow
RUN(overtask)
overflow ← Ø
The wait is performed with pselect(2), adapted from the answer by #Sigismondo.
Here's the algorithm implemented in my original code example that will work for a single worker thread (although I haven't tested it after copy-paste). It can be extended to work for a thread pool by having a separate overflow queue for each thread.
int taskRead, taskWrite;
// These variables are only allowed to be modified by the worker thread
std::__thread_id workerId;
std::queue<std::function<void(void)>> overflow;
bool overflowInUse;
void setup() {
int taskPipe[2];
::pipe(taskPipe);
taskRead = taskPipe[0];
taskWrite = taskPipe[1];
// Make the pipe non-blocking to check pipe overflows manually
::fcntl(taskWrite, F_SETFL, ::fcntl(taskWrite, F_GETFL, 0) | O_NONBLOCK);
// Save the ID of this worker thread to compare later
workerId = std::this_thread::get_id();
overflowInUse = false;
function_that_polls_file_descriptor(taskRead, []() {
// Read the callback data
std::function<void(void)>* taskPtr;
::read(taskRead, &taskPtr, sizeof(taskPtr));
// Run the task
(*taskPtr)();
delete taskPtr;
// Run any tasks that were posted to the overflow
while (!overflow.empty()) {
taskPtr = overflow.front();
overflow.pop();
(*taskPtr)();
delete taskPtr;
}
// Release the overflow mechanism if applicable
overflowInUse = false;
});
}
bool write(std::function<void(void)>* taskPtr, bool blocking = true) {
ssize_t rc = ::write(taskWrite, &taskPtr, sizeof(taskPtr));
// Failure handling
if (rc < 0) {
// If blocking is allowed, wait for pipe to become available
int err = errno;
if ((errno == EAGAIN || errno == EWOULDBLOCK) && blocking) {
fd_set fds;
FD_ZERO(&fds);
FD_SET(taskWrite, &fds);
::pselect(1, nullptr, &fds, nullptr, nullptr, nullptr);
// Try again
return write(tdata);
}
// Otherwise return false
return false;
}
return true;
}
void post(const std::function<void(void)>& task) {
auto* taskPtr = new std::function<void(void)>(task);
if (std::this_thread::get_id() == workerId) {
// The worker thread gets 1st-class treatment.
// It won't be blocked if the pipe is full, instead
// using an overflow queue until the overflow has been cleared.
if (!overflowInUse) {
bool success = write(taskPtr, false);
if (!success) {
overflow.push(taskPtr);
overflowInUse = true;
}
} else {
overflow.push(taskPtr);
}
} else {
write(taskPtr);
}
}
Make the pipe write file descriptor non-blocking, so that write fails with EAGAIN when the pipe is full.
One improvement is to increase the pipe buffer size.
Another is to use a UNIX socket/socketpair and increase the socket buffer size.
Yet another solution is to use a UNIX datagram socket which many worker threads can read from, but only one gets the next datagram. In other words, you can use a datagram socket as a thread dispatcher.
You can use the old good select to determine whether the file descriptors are ready to be used for writing:
The file descriptors in writefds will be watched to see if
space is available for write (though a large write may still block).
Since you are writing a pointer, your write() cannot be classified as large at all.
Clearly you must be ready to handle the fact that a post may fail, and then be ready to retry it later... otherwise you will be facing indefinitely growing pipes, until you system will break again.
More or less (not tested):
bool post(const std::function<void(void)>& task) {
bool post_res = false;
// Copy the function onto the heap
auto* taskPtr = new std::function<void(void)>(task);
fd_set wfds;
struct timeval tv;
int retval;
FD_ZERO(&wfds);
FD_SET(taskWrite, &wfds);
// Don't wait at all
tv.tv_sec = 0;
tv.tv_usec = 0;
retval = select(1, NULL, &wfds, NULL, &tv);
// select() returns 0 when no FD's are ready
if (retval == -1) {
// handle error condition
} else if (retval > 0) {
// Write the pointer to the pipe. This write will succeed
::write(taskWrite, &taskPtr, sizeof(taskPtr));
post_res = true;
}
return post_res;
}
If you only look at Android/Linux using a pipe is not start of the art but using a event file descriptor together with epoll is the way to go.
Background
My objective is to handle certain signals on a dedicated thread rather than to have them handled on any of the threads that happen to be running in my process when the signal is raised.
I am doing this as follows (in this example, for signal 16 only):
On the main thread, before any other threads are started (error handling ommited)
sigset_t sigset;
sigaddset(&sigset, 16);
sigprocmask(SIG_BLOCK, &sigset, nullptr);
Then I create a thread that waits for those signals (only 16 in this example):
std::thread _thread = std::thread([&]()
{
int ret = sigwaitinfo(&sigset, nullptr);
if (ret == 16)
{
// handle signal 16
}
});
This works well.
Problem
However, I would like to be able to cancel the call to sigwaitinfo when needed.
Two Inadequate Solutions
I have tried two solutions, but neither are adequate:
1. Polling
One option (that works) is not to use sigwaitinfo but rather to use sigtimedwait which accepts a timeout argument.
This allows me to use polling and to cancel when the call next returns and some cancel flag is set.
The code in the thread then looks like this:
std::atomic<bool> _cancel (false);
std::thread _thread = std::thread([&]()
{
timespec _timespec {0, 1}; // 1 second
int ret = sigtimedwait(&sigset, nullptr, _timespec);
if (_cancel)
{
return;
}
if (ret == 16)
{
// handle signal 16
}
});
In order to cancel, I only need to set the _cancel flag in the main thread.
The problem with this solution, is that polling incurs the typical trade-off between responsiveness (of the cancellation) and the amount of busy work done checking the cancellation flag.
2. raise()/sigqueue()/kill()
In this solution I add to the signal mask a dedicated signal, for instance SIGUSR1 with the following call:
sigset_t sigset;
sigaddset(&sigset, 16);
sigaddset(&sigset, SIGUSR1); // <-- added call here
sigprocmask(SIG_BLOCK, &sigset, nullptr);
Then when I need to cancel the call to sigwaitinfo I set a cancel flag and call raise(SIGUSR1)
The code in the thread then looks like this:
std::atomic<bool> _cancel (false);
std::thread _thread = std::thread([&]()
{
int ret = sigwaitinfo(&sigset, nullptr);
if (_cancel) // <-- now check _cancel flag before handling signal
{
return;
}
if (ret == 16)
{
// handle signal 16
}
});
The cancellation is now done as follows:
_cancel = true; // <-- set the flag before raising the signal
raise(SIGUSR1);
The problem with this solution is that it doesn't work, because the call to raise() does not cause sigwaitinfo to return in the dedicated thread. I believe that according to the documentation it will only raise the signal in the executing thread itself.
sigqueue() and kill() also do not work.
Summary
Is there a way to cause sigwaitinfo to return prematurely, without requiring a loop in which calls to sigtimedwait are called with a timeout?
Use pthread_kill to send a signal to a specific thread.
E.g., instead of raise(SIGUSR1); do:
if(int rc = ::pthread_kill(_thread.native_handle(), SIGUSR1))
// Handle pthread_kill error.
This is the solution I found.
Instead of waiting with sigtimedwait, use signalfd to get a file descriptor that represents the signals to be handled. (sigprocmask or similar need to be called first as with the solution presented in the question).
Call eventfd to return an "event" file descriptor.
Call poll wait on both file descriptors. This blocks. Do so in a loop.
Signal cancellation by writing to the event file descriptor on a different thread.
When poll returns check which file descriptor was signaled by checking the revents fields.
If the event file descriptor was signaled break from the loop.
Else (the signalfd descriptor was signaled) read the signal description and handle the signal by calling the handler. Then loop around calling poll again.
I have verified that this solution is reliable.
More detailed information can be found in the documentation for:
signalfd,eventfd and poll
I have a server application, which waits on a queue, fetches incoming messages, and spawns a thread to process the received message and send a reply.
The pthread portion/options I am using are as follows:
pthread_attr_t child_attr;
pthread_attr_setdetachstate(&child_attr, PTHREAD_CREATE);
// other code here
while (true)
{
// code here to wait on valid message (msg)
if (valid_message(msg))
{
pthread_t child_thread;
MessageProcessor * processor = new MessageProcessor();
if (0 == pthread_create(&child_thread, &child_attr, processor->process, (void *) msg))
{
printf("Thread dispatch successful\n");
}
else
{
printf("Error %d: could not create thread\n", errno);
}
}
}
// other code here
pthread_attr_destroy(&child_attr);
Every time I run this, the error code displayed is 11, which apparently would indicate that my process has crossed the max threads threshold, based on stuff I've read on the Internet.
However,
This is happening right from the beginning, not after I have run the application for a while.
The threads are created detached, so I shouldn't have to use pthread_join().
I used top as well as ps -p <PID> -lfT to check how many threads were in use by the application, and only 3 were (one for main server, one for the message receiver, and one for a message sender for the queue system)
PS:
The "process" prototype is as follows:
class MessageProcessor
{
MessageProcessor();
static void * MessageProcessor::process(void * arg);
}
void * MessageProcessor::process(void * arg)
{
// do something here with arg
}
Like all pthreads functions, pthread_create does not set errno to report errors, it returns an error number instead. To see why it failed you need to print the return value, not errno.
const int err = pthread_create(&child_thread, &child_attr, processor->process, (void *) msg);
if (err == 0)
printf("Thread dispatch successful\n");
else
printf("Error %d: could not create thread\n", err);
POSIX specifies errno like so:
The value of errno shall be defined only after a call to a function for which it is explicitly stated to be set [...] The value of errno should only be examined when it is indicated to be valid by a function's return value.
Since pthread_create is not documented to set errno it means the value is not defined after a call to pthread_create and should not be examined.
Your code is using an uninitialized child_attr, you'll have to do:
pthread_attr_init(&child_attr);
pthread_attr_setdetachstate(&child_attr, PTHREAD_CREATE_DETACHED);
errno 11 is usually EAGAIN, in this case it means no more processes (linux treats threads as light weight processes - see the clone manual page)available.
The while(true) loop will run forever making processes.
Note if you have a special version of Linux like ARM the error number 11 is NOT required to be EAGAIN. So take this answer with a grain of salt.
I keep running into this problem of trying to run a thread with the following properties:
runs in an infinite loop, checking some external resource, e.g. data from the network or a device,
gets updates from its resource promptly,
exits promptly when asked to,
uses the CPU efficiently.
First approach
One solution I have seen for this is something like the following:
void class::run()
{
while(!exit_flag)
{
if (resource_ready)
use_resource();
}
}
This satisfies points 1, 2 and 3, but being a busy waiting loop, uses 100% CPU.
Second approach
A potential fix for this is to put a sleep statement in:
void class::run()
{
while(!exit_flag)
{
if (resource_ready)
use_resource();
else
sleep(a_short_while);
}
}
We now don't hammer the CPU, so we address 1 and 4, but we could wait up to a_short_while unnecessarily when the resource is ready or we are asked to quit.
Third approach
A third option is to do a blocking read on the resource:
void class::run()
{
while(!exit_flag)
{
obtain_resource();
use_resource();
}
}
This will satisfy 1, 2, and 4 elegantly, but now we can't ask the thread to quit if the resource does not become available.
Question
The best approach seems to be the second one, with a short sleep, so long as the tradeoff between CPU usage and responsiveness can be achieved.
However, this still seems suboptimal, and inelegant to me. This seems like it would be a common problem to solve. Is there a more elegant way to solve it? Is there an approach which can address all four of those requirements?
This depends on the specifics of the resources the thread is accessing, but basically to do it efficiently with minimal latency, the resources need to provide an API for either doing an interruptible blocking wait.
On POSIX systems, you can use the select(2) or poll(2) system calls to do that, if the resources you're using are files or file descriptors (including sockets). To allow the wait to be preempted, you also create a dummy pipe which you can write to.
For example, here's how you might wait for a file descriptor or socket to become ready or for the code to be interrupted:
// Dummy pipe used for sending interrupt message
int interrupt_pipe[2];
int should_exit = 0;
void class::run()
{
// Set up the interrupt pipe
if (pipe(interrupt_pipe) != 0)
; // Handle error
int fd = ...; // File descriptor or socket etc.
while (!should_exit)
{
// Set up a file descriptor set with fd and the read end of the dummy
// pipe in it
fd_set fds;
FD_CLR(&fds);
FD_SET(fd, &fds);
FD_SET(interrupt_pipe[1], &fds);
int maxfd = max(fd, interrupt_pipe[1]);
// Wait until one of the file descriptors is ready to be read
int num_ready = select(maxfd + 1, &fds, NULL, NULL, NULL);
if (num_ready == -1)
; // Handle error
if (FD_ISSET(fd, &fds))
{
// fd can now be read/recv'ed from without blocking
read(fd, ...);
}
}
}
void class::interrupt()
{
should_exit = 1;
// Send a dummy message to the pipe to wake up the select() call
char msg = 0;
write(interrupt_pipe[0], &msg, 1);
}
class::~class()
{
// Clean up pipe etc.
close(interrupt_pipe[0]);
close(interrupt_pipe[1]);
}
If you're on Windows, the select() function still works for sockets, but only for sockets, so you should install use WaitForMultipleObjects to wait on a resource handle and an event handle. For example:
// Event used for sending interrupt message
HANDLE interrupt_event;
int should_exit = 0;
void class::run()
{
// Set up the interrupt event as an auto-reset event
interrupt_event = CreateEvent(NULL, FALSE, FALSE, NULL);
if (interrupt_event == NULL)
; // Handle error
HANDLE resource = ...; // File or resource handle etc.
while (!should_exit)
{
// Wait until one of the handles becomes signaled
HANDLE handles[2] = {resource, interrupt_event};
int which_ready = WaitForMultipleObjects(2, handles, FALSE, INFINITE);
if (which_ready == WAIT_FAILED)
; // Handle error
else if (which_ready == WAIT_OBJECT_0))
{
// resource can now be read from without blocking
ReadFile(resource, ...);
}
}
}
void class::interrupt()
{
// Signal the event to wake up the waiting thread
should_exit = 1;
SetEvent(interrupt_event);
}
class::~class()
{
// Clean up event etc.
CloseHandle(interrupt_event);
}
You get a efficient solution if your obtain_ressource() function supports a timeout value:
while(!exit_flag)
{
obtain_resource_with_timeout(a_short_while);
if (resource_ready)
use_resource();
}
This effectively combines the sleep() with the obtain_ressurce() call.
Check out the manpage for nanosleep:
If the nanosleep() function returns because it has been interrupted by a signal, the function returns a value of -1 and sets errno to indicate the interruption.
In other words, you can interrupt sleeping threads by sending a signal (the sleep manpage says something similar). This means you can use your 2nd approach, and use an interrupt to immediately wake the thread if it's sleeping.
Use the Gang of Four Observer Pattern:
http://home.comcast.net/~codewrangler/tech_info/patterns_code.html#Observer
Callback, don't block.
Self-Pipe trick can be used here.
http://cr.yp.to/docs/selfpipe.html
Assuming that you are reading the data from file descriptor.
Create a pipe and select() for readability on the pipe input as well as on the resource you are interested.
Then when data comes on resource, the thread wakes up and does the processing. Else it sleeps.
To terminate the thread send it a signal and in signal handler, write something on the pipe (I would say something which will never come from the resource you are interested in, something like NULL for illustrating the point). The select call returns and thread on reading the input knows that it got the poison pill and it is time to exit and calls pthread_exit().
EDIT: Better way will be just to see that the data came on the pipe and hence just exit rather than checking the value which came on that pipe.
The Win32 API uses more or less this approach:
someThreadLoop( ... )
{
MSG msg;
int retVal;
while( (retVal = ::GetMessage( &msg, TaskContext::winHandle_, 0, 0 )) > 0 )
{
::TranslateMessage( &msg );
::DispatchMessage( &msg );
}
}
GetMessage itself blocks until any type of message is received therefore not using any processing (refer). If a WM_QUIT is received, it returns false, exiting the thread function gracefully. This is a variant of the producer/consumer mentioned elsewhere.
You can use any variant of a producer/consumer, and the pattern is often similar. One could argue that one would want to split the responsibility concerning quitting and obtaining of a resource, but OTOH quitting could depend on obtaining a resource too (or could be regarded as one of the resources - but a special one). I would at least abstract the producer consumer pattern and have various implementations thereof.
Therefore:
AbstractConsumer:
void AbstractConsumer::threadHandler()
{
do
{
try
{
process( dequeNextCommand() );
}
catch( const base_except& ex )
{
log( ex );
if( ex.isCritical() ){ throw; }
//else we don't want loop to exit...
}
catch( const std::exception& ex )
{
log( ex );
throw;
}
}
while( !terminated() );
}
virtual void /*AbstractConsumer::*/process( std::unique_ptr<Command>&& command ) = 0;
//Note:
// Either may or may not block until resource arrives, but typically blocks on
// a queue that is signalled as soon as a resource is available.
virtual std::unique_ptr<Command> /*AbstractConsumer::*/dequeNextCommand() = 0;
virtual bool /*AbstractConsumer::*/terminated() const = 0;
I usually encapsulate command to execute a function in the context of the consumer, but the pattern in the consumer is always the same.
Any (welln at least, most) approaches mentioned above will do the following: thread is created, then it's blocked wwiting for resource, then it's deleted.
If you're worried about efficiency, this is not a best approach when waiting for IO. On Windows at least, you'll allocate around 1mb of memory in user mode, some in kernel for just one additional thread. What if you have many such resources? Having many waiting threads will also increase context switches and slow down your program. What if resource takes longer to be available and many requests are made? You may end up with tons of waiting threads.
Now, the solution to it (again, on Windows, but I'm sure there should be something similar on other OSes) is using threadpool (the one provided by Windows). On Windows this will not only create limited amount of threads, it'll be able to detect when thread is waiting for IO and will stwal thread from there and reuse it for other operations while waitting.
See http://msdn.microsoft.com/en-us/library/windows/desktop/ms686766(v=vs.85).aspx
Also, for more fine-grained control bit still having ability give up thread when waiting for IO, see IO completion ports (I think they'll anyway use threadpool inside): http://msdn.microsoft.com/en-us/library/windows/desktop/aa365198(v=vs.85).aspx
Im trying to implement out of process rendering for my application (like what chrome does). I have the ipc (interprocess communication) all set up and working however it just deadlocks when trying to init a new form on the other process.
I have started the process with inherit handles as true is there any thing else i need to do?
I happy to provide sample code if needed.
Edit: it deadlocks in window api calls. Runs fine when in the same process
It is very easy to couple two threads if they own windows with any kind of relationship.
The effective result of this is, your IPC calls cannot block when waiting for a reply - your IPC reads always need to use MsgWaitForMultipleObjects so that you can process window messages from the other process/thread while waiting for the IPC message indicating completion.
What you do is replace your current call to WaitForMultipleObjects with MSGWaitForMultipleObjects. When it returns, you check the return value. If nCount is the number of IPC handles you are waiting to be signalled:
// Pump messages while waiting on 0 or more handles.
for(;;)
{
while(PeekMessage(&msg,0,0,0,PM_REMOVE))
{
TranslateMessage(&msg);
DispatchMessage(&msg);
}
DWORD ret = MsgWaitForMultipleObjects(nCount,pHandles,FALSE,dwTimeout,QS_ALLEVENTS);
if(ret >= WAIT_OBJECT_0 && ret < (WAIT_OBJECT_0 + nCount))
{
// one of the handles was signalled.
return ret;
}
else if(ret == WAIT_OBJECT_0 + nCount)
{
// The wait was aborted because there is at least one message,
// go back to pumping messages
continue;
}
else
{
// test for WAIT_OBJECT_ABANDONED_0, WAIT_TIMEOUT etc. as appropriate
}
}