unblocking WSAccept for blocking TCP server sockets - c++

Question

I'm writing a TCP server (blocking socket model).
I'm having trouble implementing a valid normal program exit when the server is waiting (blocking) for new connection attempts on Accept (I use WSAccept).
The code for the server's listening socket is something like this (I omitted error handling and other irrelevant code):
int ErrCode = WSAStartup(MAKEWORD(2,2), &m_wsaData) ;
// Create a new socket to listen and accept new connection attempts
struct addrinfo hints, *res = NULL, *ptr = NULL ;
int rc, count = 0 ;
memset(&hints, 0, sizeof(hints)) ;
hints.ai_family = AF_UNSPEC ;
hints.ai_socktype = SOCK_STREAM ;
hints.ai_protocol = IPPROTO_TCP ;
hints.ai_flags = AI_PASSIVE ;
CString strPort ;
strPort.Format("%d", Port) ;
getaddrinfo(pLocalIp, strPort.GetBuffer(), &hints, &res) ;
strPort.ReleaseBuffer() ;
ptr = res ;
if ((m_Socket = WSASocket(res->ai_family, res->ai_socktype, res->ai_protocol, NULL, 0, 0)) == INVALID_SOCKET)
{
// some error
}
if(bind(m_Socket, (SOCKADDR *)res->ai_addr, res->ai_addrlen) == SOCKET_ERROR)
{
// some error
}
if (listen(m_Socket, SOMAXCONN) == SOCKET_ERROR)
{
// some error
}
So far so good... Then I implemented the WSAccept call inside a thread like this:
SOCKADDR_IN ClientAddr ;
int ClientAddrLen = sizeof(ClientAddr) ;
SOCKET TempS = WSAAccept(m_Socket, (SOCKADDR*) &ClientAddr, &ClientAddrLen, NULL, NULL);
Of course the WSAccept blocks until a new connection attempt is made but if I wish to exit
the program then i need some way to cause WSAccept to exit. I have tried several different approaches:
Attempt to call shutdown and/or closesocket with m_Socket from within another thread failed (program just hangs).
using WSAEventSelect indeed solves this issue but then WSAccept delivers only non-blocking sockets - which is not my intention. (Is there a way to make the sockets blocking?)
I Read about APC and tried to use something like QueueUserAPC(MyAPCProc, m_hThread, 1)) but it didn't work either.
What am I doing wrong ?
Is there a better way to cause this blocking WSAccept to exit ?

Answer 1

Use select() with a timeout to detect when a client connection is actually pending before then calling WSAAccept() to accept it. It works with blocking sockets without putting them into non-blocking mode. That will give your code more opportunities to check if the app is shutting down.

Answer 2

Go with non-blocking accepting socket (WSAEventSelect as you mentioned) and use non-blocking WSAccept. You can make a non-blocking socket that WSAccept returns into blocking socket with ioctlsocket (see msdn).

Answer 3

Do all the other stuff you absoultely have to on shutdown, (maybe you have DB connections to close, or files to flush?), and then call ExitProcess(0). That will stop your listening thread, no problem.

Answer 4

See log4cplus source for my take on this issue. I basically wait on two event objects, one is signaled when connection is being accepted (using WSAEventSelect()) and another is there to interrupt the waiting. The most relevant parts of the source is below. See ServerSocket::accept().
namespace {
static
bool
setSocketBlocking (SOCKET_TYPE s)
{
u_long val = 0;
int ret = ioctlsocket (to_os_socket (s), FIONBIO, &val);
if (ret == SOCKET_ERROR)
{
set_last_socket_error (WSAGetLastError ());
return false;
}
else
return true;
}
static
bool
removeSocketEvents (SOCKET_TYPE s, HANDLE ev)
{
// Clean up socket events handling.
int ret = WSAEventSelect (to_os_socket (s), ev, 0);
if (ret == SOCKET_ERROR)
{
set_last_socket_error (WSAGetLastError ());
return false;
}
else
return true;
}
static
bool
socketEventHandlingCleanup (SOCKET_TYPE s, HANDLE ev)
{
bool ret = removeSocketEvents (s, ev);
ret = setSocketBlocking (s) && ret;
ret = WSACloseEvent (ev) && ret;
return ret;
}
} // namespace
ServerSocket::ServerSocket(unsigned short port)
{
sock = openSocket (port, state);
if (sock == INVALID_SOCKET_VALUE)
{
err = get_last_socket_error ();
return;
}
HANDLE ev = WSACreateEvent ();
if (ev == WSA_INVALID_EVENT)
{
err = WSAGetLastError ();
closeSocket (sock);
sock = INVALID_SOCKET_VALUE;
}
else
{
assert (sizeof (std::ptrdiff_t) >= sizeof (HANDLE));
interruptHandles[0] = reinterpret_cast<std::ptrdiff_t>(ev);
}
}
Socket
ServerSocket::accept ()
{
int const N_EVENTS = 2;
HANDLE events[N_EVENTS] = {
reinterpret_cast<HANDLE>(interruptHandles[0]) };
HANDLE & accept_ev = events[1];
int ret;
// Create event and prime socket to set the event on FD_ACCEPT.
accept_ev = WSACreateEvent ();
if (accept_ev == WSA_INVALID_EVENT)
{
set_last_socket_error (WSAGetLastError ());
goto error;
}
ret = WSAEventSelect (to_os_socket (sock), accept_ev, FD_ACCEPT);
if (ret == SOCKET_ERROR)
{
set_last_socket_error (WSAGetLastError ());
goto error;
}
do
{
// Wait either for interrupt event or actual connection coming in.
DWORD wsawfme = WSAWaitForMultipleEvents (N_EVENTS, events, FALSE,
WSA_INFINITE, TRUE);
switch (wsawfme)
{
case WSA_WAIT_TIMEOUT:
case WSA_WAIT_IO_COMPLETION:
// Retry after timeout or APC.
continue;
// This is interrupt signal/event.
case WSA_WAIT_EVENT_0:
{
// Reset the interrupt event back to non-signalled state.
ret = WSAResetEvent (reinterpret_cast<HANDLE>(interruptHandles[0]));
// Clean up socket events handling.
ret = socketEventHandlingCleanup (sock, accept_ev);
// Return Socket with state set to accept_interrupted.
return Socket (INVALID_SOCKET_VALUE, accept_interrupted, 0);
}
// This is accept_ev.
case WSA_WAIT_EVENT_0 + 1:
{
// Clean up socket events handling.
ret = socketEventHandlingCleanup (sock, accept_ev);
// Finally, call accept().
SocketState st = not_opened;
SOCKET_TYPE clientSock = acceptSocket (sock, st);
int eno = 0;
if (clientSock == INVALID_SOCKET_VALUE)
eno = get_last_socket_error ();
return Socket (clientSock, st, eno);
}
case WSA_WAIT_FAILED:
default:
set_last_socket_error (WSAGetLastError ());
goto error;
}
}
while (true);
error:;
DWORD eno = get_last_socket_error ();
// Clean up socket events handling.
if (sock != INVALID_SOCKET_VALUE)
{
(void) removeSocketEvents (sock, accept_ev);
(void) setSocketBlocking (sock);
}
if (accept_ev != WSA_INVALID_EVENT)
WSACloseEvent (accept_ev);
set_last_socket_error (eno);
return Socket (INVALID_SOCKET_VALUE, not_opened, eno);
}
void
ServerSocket::interruptAccept ()
{
(void) WSASetEvent (reinterpret_cast<HANDLE>(interruptHandles[0]));
}

Answer 5

A not so neat way of solving this problem is by issuing a dummy WSAConnect request from the thread that needs to do the shutdown. If the dummy connect fails, you might resort to ExitProcess as suggested by Martin.
void Drain()
{
if (InterlockedIncrement(&drain) == 1)
{
// Make a dummy connection to unblock wsaaccept
SOCKET ConnectSocket = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_TCP, NULL, 0, 0);
if (ConnectSocket != INVALID_SOCKET) {
int iResult = WSAConnect(ConnectSocket, result->ai_addr, result->ai_addrlen, 0, 0, 0, 0);
if (iResult != 0) {
printf("Unable to connect to server! %d\n", WSAGetLastError());
}
else
{
closesocket(ConnectSocket);
}
}
}
}