Unable to accept a new connection properly using IOCP - Invalid socket handle - c++

Question

I am learning about IOCP and have decided to write my own wrapper class based on the following article:
http://www.codeproject.com/Articles/13382/A-simple-application-using-I-O-Completion-Ports-an
My project is a C++ TCP server using IOCP. The client uses send() and recv() to send and receive data which I cannot change (from what I've been told this shouldn't cause any problem, but I am mentioning it just in case). It also creates a socket using socket() (and not WSASocket()).
Everything seems to be working fine (no error with CreateIoCompletionPort, I can add a socket descriptor to the existing completion port without any error. I've checked everything by adding a call to WSAGetLastError() after each of these functions).
(Before anything, please don't mind the inconsistent coding style. I like to make stuff work first and then clean it all up.)
socket_ = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_IP, NULL, 0, WSA_FLAG_OVERLAPPED);
setsockopt(socket_, IPPROTO_IP, SO_DEBUG | TCP_NODELAY, sockopt, 4);
ioctlsocket(socket_, FIONBIO, &ulSockMode_);
sin_.sin_family = AF_INET;
sin_.sin_port = htons((uint16_t)uiPort_));
hAccept_[0] = WSACreateEvent(); //only 1 event, I'm using an array for convenience
if (hAccept_ == WSA_INVALID_EVENT)
{
//this is never executed
}
WSAEventSelect(socket_, hAccept_[0], FD_ACCEPT);
After an incoming connection is detected (I use WSAWaitForMultipleEevents and WSAEnumNetworkEvents which work don't trigger any error), I use the following code to accept the client (and this is where the problems start):
SOCKET sock_client{ INVALID_SOCKET };
int32_t len_si{ sizeof(SOCKADDR_IN) };
//sock_client = accept(socket_, reinterpret_cast<SOCKADDR*>(pSockAddr), &len_si); // this works fine
//sock_client = sock_client = WSAAccept(socket_, reinterpret_cast<SOCKADDR*>(pSockAddr), &len_si, NULL, 0);//works fine too
char buf[2 * (sizeof(SOCKADDR_IN) + 16)];
WSAOVERLAPPED wsaovl;
uint32_t bytes{ 0 };
BOOL b = AcceptEx(socket_, sock_client, (PVOID)buf, 0, sizeof(SOCKADDR_IN) + 16, sizeof(SOCKADDR_IN) + 16, reinterpret_cast<LPDWORD>(&bytes), &wsaovl); //this fails, returns 0
int32_t test = WSAGetLastError(); // this returns 6 (WSA_INVALID_HANDLE)
I have no idea why it works with accept() and WSAAccept(), however it doesn't with AcceptEx().
If I use accept() though, after accepting the client I need to call WSARecv() immediately. I'm not sending anything back to the client just yet but I read that it needs to be called before GetQueuedCompletionStatus() in the worker thread:
WSABUF* buf = new WSABUF;
OVERLAPPED* ovl = new OVERLAPPED;
int32_t flags{ 0 };
int32_t bytes{ 0 };
int32_t bytes_recv = WSARecv(client_socket, buf, 1, &flags, &bytes, ovl, NULL); // this returns -1
int32_t err = WSAGetLastError(); // this returns 6 (WSA_INVALID_HANDLE)
And since this doesn't work, the GetQueuedCompletionStatus() routine in my worker thread keeps on hanging (or at least, I assume this is the reason)
Is there anything wrong with what I'm doing? I've been trying to search around and fix it since yesterday night, I know it's not a lot of time but I really don't see what I'm not doing correctly.
UPDATE:
I have changed the way I initialize my socket for AcceptEx().
SOCKET sock_client = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_IP, NULL, 0, WSA_FLAG_OVERLAPPED);
and
WSAOVERLAPPED wsaovl = {};
AcceptEx() still returns false, however the error returned by WSAGetLastError() is now 997 (WSA_IO_PENDING). I'm not sure what I/O operation is pending exactly and how I would go about fixing it.

Answer 1

I had stumbled upon similar hurdle when I was learning I/O Completion Ports (IOCP)...
I think the problem is that, in the scheme of IOCP socket model, the most complicated part is the beginning phase of 'socket acceptance'. That is why most tutorial skip over it and begin the discussion on how to handle send/recv, instead.
If you want to develop sufficient understanding of IOCP so that you could implement a production software then my advice to you is to study it until you completely grasp it (this answer below is not enough). One document that I would recommend is chapter 5 of 'Network Programming for Microsoft Windows - 2nd edition'. The book may be old but valid for IOCP. Also, the article 'Windows via C/C++: Synchronous and Asynchronous Device I/O' touches some aspects of IOCP, though not enough information to do production software.
I will try to explain as best as I can, however, I must warn you that this may not be enough. Here it goes...
So, the part you are missing is "How to do 'socket acceptance' in an IOCP socket model".
First off, lets examine the typical Winsock (non-IOCP) sequence of calls on server;
// (1) Create listen socket on server.
WSASocket()
// (2) Bind an address to your listen socket.
bind()
// (3) Associate the listen socket with an event object on FD_ACCEPT event.
WSAEventSelect(,, FD_ACCEPT )
// (4) Put socket in listen state - now, Windows listening for new
// connection requests. If new requests comes, the associated
// event object will be set.
listen()
// (5) Wait on the event object associated on listen socket. This
// will get signaled when a new connection request comes.
WaitForSingleObject() {
// (6) A network activity has occurred. Verify that FD_ACCEPT has
// raised the event object. This also resets the event object
// so WaitForSingleObject() does not loop non-stop.
WSAEnumNetworkEvents()
// (7) Understanding this part is important. The WSAAccept() doesn't
// just accept connection, it first creates a new socket and
// then associates it with the newly accepted connection.
WSAAccept()
}
The step (7) is ok for non-IOCP based models. However, when looking it from the performance point of view - socket creation is expensive. And it slows down the connection acceptance process.
In IOCP model, sockets are created in advance for new incoming connection requests. Not only sockets are created in advance they are associated with the listen socket even before the connection request comes. To achieve this Microsoft has provided extension functions. Two such functions that are required for IOCP model are AcceptEx() & GetAcceptExSockaddrs().
Note: When using these extension functions they need to be loaded at runtime in order to avoid performance penalty. This can be achieved using WSAIoctl(). For further read refer to the MSDN documentation on AcceptEx().
Caveat: AcceptEx() can be used to set the new socket to receive some data as part of connection-acceptance process. This feature needs to be disabled as it makes application susceptible to DoS attack i.e., a connection request is issued but no data is sent. The receiving application will wait on that socket indefinitely. To avoid that just pass 0 value for its 'dwReceiveDataLength' parameter.
How to setup connection-acceptance code for IOCP model?
One way to do this is;
// (1) Create IO completion port
CreateIoCompletionPort()
// (2) Have a method that creates worker threads say 'CreateWorkerThreads()'.
// This assign same method (say WorkerThread_Func()) to all worker threads.
// In the WorkerThread_Func() threads are blocked on call to
// GetQueuedCompletionStatus().
CreateWorkerThreads()
// (3) Create listen socket.
WSASocket()
// (4) Associate listen socket to IO Completion Port created earlier.
CreateIoCompletionPort()
// (5) Bind an address to your listen socket.
bind()
// (6) Put socket in listen state - now, Windows listening for new
// connection requests. If a new request comes, GetQueuedCompletionStatus()
// will release a thread.
listen()
// (7) Create sockets in advance and call AcceptEx on each of
// these sockets. If a new connection requests comes
// Windows will pick one of these sockets and associate the
// connection with it.
//
// As an example, below loop will create 1000 sockets.
GUID GuidAcceptEx = WSAID_ACCEPTEX;
DWORD dwBytes;
LPFN_ACCEPTEX lpfnAcceptEx;
// First, load extension method.
int retCode = WSAIoctl(listenSocket,
SIO_GET_EXTENSION_FUNCTION_POINTER,
&GuidAcceptEx,
sizeof(GuidAcceptEx),
&lpfnAcceptEx,
sizeof(lpfnAcceptEx),
&dwBytes,
NULL,
NULL
);
for( /* loop for 1000 times */ ) {
SOCKET preemptiveSocket = WSASocket(, , , , , WSA_FLAG_OVERLAPPED);
lpfnAcceptEx(listenSocket, preemptiveSocket,,,,,,);
}
This essentially prepare your application to accept sockets in IOCP way. When a new connection requests comes one of the worker threads, that are waiting on GetQueuedCompletionStatus(), will be released and handed over the pointer to the data structure. This will have the socket that was created by lpfnAcceptEx().
Is the process complete? Not yet. The socket accepted through AcceptEx() call does not inherit properties of listenSocket. To do that you need to call;
setsockopt( acceptSocket, SOL_SOCKET, SO_UPDATE_ACCEPT_CONTEXT,
(char*)&listenSocket, sizeof(listenSocket) );
Now, acceptSocket is good to use for WSASend / WSARecv!
Something is missing! I skipped the part about how does a worker thread gets the acceptedSocket from GetQueuedCompletionStatus() ?
The answer is, by passing your specially crafted structure to lpfnAcceptEx(). When the GetQueuedCompletionStatus() returns it will have this data structure containing the socket that you will have passed.
How to make such a structure? By creating a structure having 'WSAOVERLAPPED' as its first member. You can have any members of your own after the first member. For example, my structure looked like;
typedef struct _WSAOVERLAPPEDPLUS
{
WSAOVERLAPPED ProviderOverlapped; // 'WSAOVERLAPPED' has to be the first member.
SOCKET client; // Use this to pass preemptive socket.
SOCKET listenSocket; // Use this to pass the listenSocket.
DWORD dwBytes;
SOCKET_OPERATION operation; // Enum to assist in knowing what socket operation ...
} WSAOVERLAPPEDPLUS, *LPWSAOVERLAPPEDPLUS;
...
typedef enum SOCKET_OPERATION {
UNINITIALIZED_ENUM, // To protect against memory leaks and uninitialized buffers.
OP_ACCEPTEX,
OP_RECEIVE,
OP_SEND
};
...
//
// So the previously mentioned for() loop will become;
//
for( /* loop for 1000 times */ ) {
SOCKET preemptiveSocket = WSASocket(, , , , , WSA_FLAG_OVERLAPPED);
LPWSAOVERLAPPEDPLUS pOl = new WSAOVERLAPPEDPLUS();
// Initialize our "extended" overlapped structure
memset(pOl, 0, sizeof(WSAOVERLAPPEDPLUS));
pOl->operation = OP_ACCEPTEX;
pOl->client = preemptiveSocket;
pOl->listenSocket = listenSocket;
int buflen = (sizeof(SOCKADDR_IN) + 16) * 2;
char* pBuf = new char[buflen];
memset(pBuf, 0, buflen);
m_lpfnAcceptEx(listenSocket,
preemptiveSocket,
pBuf,
0, // Passed 0 to avoid reading data on accept which in turn
// avoids DDoS attack i.e., connection attempt without data will
// cause AcceptEx to wait indefinitely.
sizeof(SOCKADDR_IN) + 16,
sizeof(SOCKADDR_IN) + 16,
&pOl->dwBytes,
&pOl->ProviderOverlapped
);
}
... and in the worker thread when GetQueuedCompletionStatus() returns;
while (TRUE)
{
bOk = ::GetQueuedCompletionStatus(hCompPort, &bytes_transferred, &completion_key, &pOverlapped, INFINITE);
if (bOk) {
// Process a successfully completed I/O request
if (completion_key == 0) {
// Safe way to extract the customized structure from pointer
// is to use 'CONTAINING_RECORD'. Read more on 'CONTAINING_RECORD'.
WSAOVERLAPPEDPLUS *pOl = CONTAINING_RECORD(pOverlapped, WSAOVERLAPPEDPLUS, ProviderOverlapped);
if (pOl->operation == OP_ACCEPTEX) {
// Before doing any WSASend/WSARecv, inherit the
// listen socket properties by calling 'setsockopt()'
// as explained earlier.
// The listenSocket and the preemptive socket are available
// in the 'pOl->listenSocket' & 'pOl->client', respectively.
}
delete pOl;
}
}
else {
// Handle error ...
}
I hope this gave you idea on how AcceptEx() is utilized with IOCP.