Proper way to handle SIGTERM with multiple threads - c++

Question

I have a multi threaded program on Raspberry in which I want to handle SIGTERM and shut everything down gracefully. The issue is that I have a background thread that has called recvfrom() on a blocking socket. As per my understanding from the man pages, if I exit my handler all the system calls should be woken up and return with -1 and errno set to EINTR. However in my case the recvfrom call keeps hanging.
1) In general am I understanding this right, that all threads that have blocking system calls that are able to be woken up by a signal should wake up in this scenario?
2) Could it be that the operating system is setting some special signal mask on my thead?
The interresting part is that I am using the VideoCore primitives, not pthread, maybe that could be the cause? Here is a small test example:
#include <iostream>
#include <cstdlib>
#include <cstring>
#include <errno.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <signal.h>
#include "interface/vcos/vcos.h"
void SignalHandler(int nSignalNumber)
{
std::cout << "received signal " << nSignalNumber << std::endl;
}
void* ThreadMain(void* pArgument)
{
int nSocket = socket(AF_INET, SOCK_DGRAM, 0);
if (nSocket >= 0)
{
sockaddr_in LocalAddress;
memset(&LocalAddress, 0, sizeof(LocalAddress));
LocalAddress.sin_family = AF_INET;
LocalAddress.sin_addr.s_addr = INADDR_ANY;
LocalAddress.sin_port = htons(1234);
if (bind(nSocket, reinterpret_cast<sockaddr *>(&LocalAddress), sizeof(LocalAddress)) == 0)
{
sockaddr_in SenderAddress;
socklen_t nSenderAddressSize = sizeof(SenderAddress);
unsigned char pBuffer[512];
std::cout << "calling recvfrom()" << std::endl;
int nBytesReceived = recvfrom(nSocket, pBuffer, sizeof(pBuffer), 0, reinterpret_cast<struct sockaddr *>(&SenderAddress), &nSenderAddressSize);
if (nBytesReceived == -1)
{
if (errno == EINTR)
{
std::cout << "recvfrom() was interrupred by a signal" << std::endl;
}
else
{
std::cout << "recvfrom() failed with " << errno << std::endl;
}
}
}
else
{
std::cout << "bind() failed with " << errno << std::endl;
}
close(nSocket);
}
else
{
std::cout << "socket() failed with " << errno << std::endl;
}
return NULL;
}
int main(int argc, char** argv)
{
struct sigaction SignalAction;
memset(&SignalAction, 0, sizeof(SignalAction));
SignalAction.sa_handler = SignalHandler;
sigaction(SIGTERM, &SignalAction, NULL);
VCOS_THREAD_T Thread;
VCOS_STATUS_T nVcosStatus = vcos_thread_create(&Thread, "", NULL, ThreadMain, NULL);
if (nVcosStatus == VCOS_SUCCESS)
{
void* pData = NULL;
vcos_thread_join(&Thread, &pData);
}
else
{
std::cout << "vcos_thread_create() failed with " << nVcosStatus << std::endl;
}
return EXIT_SUCCESS;
}
It can be compiled like this:
g++ test.cpp -I/opt/vc/include -L/opt/vc/lib -lvcos -o test
When I run it and then call kill on the running instance the output is:
calling recvfrom()
received signal 15
and the process hangs. I'll try if a pthread behaves differently.
UPDATE
Ok I updated the sample to spawn a pthread thread as well and that one is not quitting as well. So I assume the signals are not populated to all threads?
#include <iostream>
#include <cstdlib>
#include <cstring>
#include <errno.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <signal.h>
#include <pthread.h>
#include "interface/vcos/vcos.h"
void SignalHandler(int nSignalNumber)
{
std::cout << "received signal " << nSignalNumber << std::endl;
}
void* ThreadMain(void* pArgument)
{
const char* pThreadType = reinterpret_cast<const char*>(pArgument);
int nSocket = socket(AF_INET, SOCK_DGRAM, 0);
if (nSocket >= 0)
{
sockaddr_in LocalAddress;
memset(&LocalAddress, 0, sizeof(LocalAddress));
LocalAddress.sin_family = AF_INET;
LocalAddress.sin_addr.s_addr = INADDR_ANY;
LocalAddress.sin_port = htons(pThreadType[0] * 100);
if (bind(nSocket, reinterpret_cast<sockaddr *>(&LocalAddress), sizeof(LocalAddress)) == 0)
{
sockaddr_in SenderAddress;
socklen_t nSenderAddressSize = sizeof(SenderAddress);
unsigned char pBuffer[512];
std::cout << "calling recvfrom()" << std::endl;
int nBytesReceived = recvfrom(nSocket, pBuffer, sizeof(pBuffer), 0, reinterpret_cast<struct sockaddr *>(&SenderAddress), &nSenderAddressSize);
if (nBytesReceived == -1)
{
if (errno == EINTR)
{
std::cout << "recvfrom() was interrupred by a signal" << std::endl;
}
else
{
std::cout << "recvfrom() failed with " << errno << std::endl;
}
}
}
else
{
std::cout << "bind() failed with " << errno << std::endl;
}
close(nSocket);
}
else
{
std::cout << "socket() failed with " << errno << std::endl;
}
std::cout << pThreadType << " thread is exiting" << std::endl;
return NULL;
}
int main(int argc, char** argv)
{
struct sigaction SignalAction;
memset(&SignalAction, 0, sizeof(SignalAction));
SignalAction.sa_handler = SignalHandler;
sigaction(SIGTERM, &SignalAction, NULL);
VCOS_THREAD_T VcosThread;
VCOS_STATUS_T nVcosStatus = vcos_thread_create(&VcosThread, "", NULL, ThreadMain, const_cast<char*>("vcos"));
bool bJoinVcosThread = false;
if (nVcosStatus == VCOS_SUCCESS)
{
bJoinVcosThread = true;
}
else
{
std::cout << "vcos_thread_create() failed with " << nVcosStatus << std::endl;
}
pthread_t PthreadThread;
int nPthreadStatus = pthread_create(&PthreadThread, NULL, ThreadMain, const_cast<char*>("pthread"));
bool bJoinPthreadThread = false;
if (nPthreadStatus == 0)
{
bJoinPthreadThread = true;
}
else
{
std::cout << "pthread_create() failed with " << nPthreadStatus << std::endl;
}
if (bJoinVcosThread)
{
void* pData = NULL;
vcos_thread_join(&VcosThread, &pData);
}
if (bJoinPthreadThread)
{
void* pData = NULL;
pthread_join(PthreadThread, &pData);
}
return EXIT_SUCCESS;
}

Answer 1

A signal such as SIGTERM is submitted to one thread in the process only. The only precondition is that the chosen thread must either have not masked the signal, or must wait for it using sigwait. The other threads will not be directly notified that the signal has been delivered.
A common approach to combine signals with threads is to have a separate thread which handles signals only and notifies the other threads using thread synchronization mechanisms such as condition variables.
For interrupting file I/O, this may not be sufficient because there is a race condition between checking for a termination request and making the system call to perform the I/O operation. Some language run-time libraries use non-blocking I/O with poll or epoll with a special file descriptor which becomes ready on signal delivery (either using the previously-mentioned thread-based approach, or something Linux-specific like signalfd). Others try to avoid this overhead by using the read and write system calls directly with a complicated dance which uses dup2 to replace the file descriptor with one that always causes I/O to fail, thereby avoiding the race condition (but the bookkeeping needed for that is fairly complicated).

Answer 2

The manpage for signal reads:
If a signal handler is invoked while a system call or library function call is blocked, then either:
the call is automatically restarted after the signal handler returns; or
the call fails with the error EINTR.
Which of these two behaviors occurs depends on the interface and whether or not the signal handler was established using the SA_RESTART flag (see sigaction(2)). The details vary across UNIX systems<...>
A few lines below, recvfrom is listed among the functions that use SA_RESTART behavior by default. (Note: this behavior is disabled if there's a timeout on the socket, though.)
Thus, you should fill the sa_flags field of the sigaction structure to carefully avoid setting the SA_RESTART flag.

Answer 3

A good way to deal with blocking sockets -see socket(7)- (and even non blocking ones) is to use a multiplexing syscall like poll(2) (or the obsolete select(2)....)
Regarding signals, be sure to read signal(7) and signal-safety(7).
A common way to handle signals with some event loop (using poll(2)) is to have a signal handler which simply write(2)-s a byte on a pipe(7) to self (you'll setup the pipe at initialization, and you'll poll it in your event loop). The Qt documentation explains how and why. You might also use the Linux specific signalfd(2).