I'm trying to write a server which can support many clients connections simultaneously so I'm trying to do it with IOCP. So let me brief about my code flow and then I can explain my problem. First of all, server is opening a port for listening and waiting on an "accept" call for new incoming connections. For reference I have used same code as mentioned here So it accepts every new incoming connection and returns a new socket descriptor (sd), and then it marks as nonblocking with:
arg = 1;
ioctlsocket(sd, FIONBIO, &arg);
and then enable TCP_NODELAY:
level = IPPROTO_TCP;
optName = TCP_NODELAY;
value = 1;
setsockopt(sd, level, optName, (const char*)&value, sizeof(value));
thereafter associating with an IOCP port as:
CreateIoCompletionPort((HANDLE)sd, iocp_port, (DWORD)completion_key, 4);
completion_key is a class object which is nothing but a container, it contains data buffer, overlapped-buffer, query-type recv/send etc.
and in last issuing a read call:
WSARecv(sd, wsabuf, 1, &bytes, &flags, overlapped, NULL);
wsabuf and overlapped are part of completion_key object.
In 90% cases it works fine i.e. when there is some incoming data available on this socket "GetQueuedCompletionStatus" gets unblocked and it has valid data. But sometimes WSARecv call returns with an error and GetLastError() returns 6 which is "invalid handle" error. I'm bit bewildered why it's happening so.
The way I'm creating an iocp port:
iocp_port = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0);
and there are threads which are waiting on "GetQueuedCompletionStatus".
I monitored all system calls which were happening in background. WSARecv internally calls NtDeviceIoControlFile and there is an argument "Event" which is same as what is passed in lpOverlapped structure of WSARecv as hEvent. I wasn't setting hEvent to NULL, so it was taking some garbage value, when it was NULL then NtDeviceIoControlFile returned successfully and for other cases it returned "INVALID_HANDLE" error. Unfortunately, it was NULL most of the time.
Related
I am reading about IOCP through this article
http://www.winsocketdotnetworkprogramming.com/winsock2programming/winsock2advancediomethod5i.html
This first WSARecv function he used with 0 lpNumberOfBytesRecvd to initiate the completion ports
if (WSARecv(Accept, &(PerIoData->DataBuf), 1, &RecvBytes, &Flags, &(PerIoData->Overlapped), NULL) == SOCKET_ERROR)
{
if (WSAGetLastError() != ERROR_IO_PENDING)
{
printf("WSARecv() failed with error %d\n", WSAGetLastError());
return 1;
}
}
else
printf("WSARecv() is OK!\n");
}
Now on the worker threads function where the application receives the IO post
it checks parameter lpNumberOfBytes of GetQueuedCompletionStatus for 0 also and then it closes the connection if it is
// First check to see if an error has occurred on the socket and if so
// then close the socket and cleanup the SOCKET_INFORMATION structure
// associated with the socket
if (BytesTransferred == 0)
When i tried it the first IOCP message posted the function GetQueuedCompletionStatus it always returning 0 in lpNumberOfBytes parameter because first WSARecv call was called with lpNumberOfBytesRecvd set to zero
I have tried to implement this code and i am having troubles to get the code working unless i choose to not use NULL in this parameter
in the first call to WSARecv so that the GetQueuedCompletionStatus function returning value 0 in lpNumberOfBytes can be used as in case of all data recieved.
MSDN says to use NULL
lpNumberOfBytesRecvd [out]
A pointer to the number, in bytes, of data received by this call if
the receive operation completes immediately.
Use NULL for this parameter if the lpOverlapped parameter is not NULL
to avoid potentially erroneous results. This parameter can be NULL
only if the lpOverlapped parameter is not NULL.
The code is not working well if i call WSARecv again (inside worker threads function) and all data were recieved this causes the GetQueuedCompletionStatus function to return immediatly without waiting even if INFINITE used and the worker threads enter in endless loop. No errors ever returned by GetQueuedCompletionStatus function it just acts as if there is more data but there is none.
WSARecv needs to be initiated with what value at buf size?
also should i use non blocking socket? or just normal blocking sockets since i use separate thread to accept connections so i think its fine because it won't freeze my GUI.
I use a blocking FSocket in client-side that connected to tcp server, if there's no message from server, socket thread would block in function FScoket::Recv(), if TCP server shutdown, socket thread is still blocking in this function. but when use blocking socket of BSD Socket API, thread would pass from recv function and return errno when TCP server shutdown, so is it the defect of FSocket?
uint32 HRecvThread::Run()
{
uint8* recv_buf = new uint8[RECV_BUF_SIZE];
uint8* const recv_buf_head = recv_buf;
int readLenSeq = 0;
while (Started)
{
//if (TcpClient->Connected() && ClientSocket->GetConnectionState() != SCS_Connected)
//{
// // server disconnected
// TcpClient->SetConnected(false);
// break;
//}
int32 bytesRead = 0;
//because use blocking socket, so thread would block in Recv function if have no message
ClientSocket->Recv(recv_buf, readLenSeq, bytesRead);
.....
//some logic of resolution for tcp msg bytes
.....
}
delete[] recv_buf;
return 0
}
As I expected, you are ignoring the return code, which presumably indicates success or failure, so you are looping indefinitely (not blocking) on an error or end of stream condition.
NB You should allocate the recv_buf on the stack, not dynamically. Don't use the heap when you don't have to.
There is a similar question on the forums in the UE4 C++ Programming section. Here is the discussion:
https://forums.unrealengine.com/showthread.php?111552-Recv-function-would-keep-blocking-when-TCP-server-shutdown
Long story short, in the UE4 Source, they ignore EWOULDBLOCK as an error. The code comments state that they do not view it as an error.
Also, there are several helper functions you should be using when opening the port and when polling the port (I assume you are polling since you are using blocking calls)
FSocket::Connect returns a bool, so make sure to check that return
value.
FSocket::GetLastError returns the UE4 Translated error code if an
error occured with the socket.
FSocket::HasPendingData will return a value that informs you if it
is safe to read from the socket.
FSocket::HasPendingConnection can check to see your connection state.
FSocket::GetConnectionState will tell you your active connection state.
Using these helper functions for error checking before making a call to FSocket::Recv will help you make sure you are in a good state before trying to read data. Also, it was noted in the forum posts that using the non-blocking code worked as expected. So, if you do not have a specific reason to use blocking code, just use the non-blocking implementation.
Also, as a final hint, using FSocket::Wait will block until your socket is in a desirable state of your choosing with a timeout, i.e. is readable or has data.
I am currently trying some new libraries (IOCP) for socket programming. And I've stumbled upon the AcceptEx functionality to enable async connections.
As the documentation says:
The AcceptEx function uses overlapped I/O, unlike the accept function. If your application uses AcceptEx, it can service a large number of clients with a relatively small number of threads. As with all overlapped Windows functions, either Windows events or completion ports can be used as a completion notification mechanism.
But I am not receving any completion when a client connects. I do however get a completion when the client sends data..
This is my code:
DWORD dwBytes;
GUID GuidAcceptEx = WSAID_ACCEPTEX;
int iResult = WSAIoctl(m_hSocket, SIO_GET_EXTENSION_FUNCTION_POINTER,
&GuidAcceptEx, sizeof (GuidAcceptEx),
&m_lpfnAcceptEx, sizeof (m_lpfnAcceptEx),
&dwBytes, NULL, NULL);
if (iResult == SOCKET_ERROR)
{
CloseSocket();
}
And then:
WSAOVERLAPPED olOverlap;
memset(&olOverlap, 0, sizeof (olOverlap));
char lpOutputBuf[1024];
int outBufLen = 1024;
DWORD dwBytes;
BOOL bRet = m_lpfnAcceptEx( m_hSocket, hSocket, lpOutputBuf,
outBufLen - ((sizeof (sockaddr_in) + 16) * 2),
sizeof (sockaddr_in) + 16, sizeof (sockaddr_in) + 16,
&dwBytes, &olOverlap);
if ( bRet == FALSE )
{
DWORD dwRet = WSAGetLastError();
if( dwRet != WSA_IO_PENDING )
{
return dwRet;
}
}
Any suggestion of what to do to receive completions?
EDIT:
I bind the hSocket to the completionport after m_lpfnAcceptEx()
Firstly the WSAOVERLAPPED and data buffer that you're declaring on the stack above your call to AcceptEx() will not be in existence when a completion occurs (unless you are calling GetQueuedCompletionStatus() in the same function, which would be a trifle odd). You need to dynamically allocate them or pool them.
Secondly you state that you associate the socket to the completion port after you call AcceptEx(). That's wrong. You need to do these things before you call AcceptEx().
Create a socket with WSA_FLAG_OVERLAPPED set.
Bind it to the address you want to listen on.
Call listen on it with your desired backlog.
Load AcceptEx() dynamically using the listening socket and a call to WSAIoctl (not strictly necessary and the code you show should work but this way you can be sure you get your listening socket from the same underlying winsock provider and that it supports AcceptEx().
Load GetAcceptExSockaddrs() in the same way as you load AcceptEx() - you'll need it once the accept completes.
Associate the listening socket to your IOCP.
Now you can post a number of AcceptEx() calls using the listening socket and new 'accept' socket which you create like this:
Create a socket with WSA_FLAG_OVERLAPPED set.
Associate the socket to your IOCP.
As stated above you need to ensure that the buffer and the OVERLAPPED are unique per call and last until the completion occurs.
When the completion occurs you have to do the following....
Call setsockopt() with SO_UPDATE_ACCEPT_CONTEXT on the accepted socket using the listening socket as the data...
Deblock your addresses using GetAcceptExSockaddrs().
Process any data (if you allocated enough space in the buffer for data).
Note that by design AcceptEx() can be used to accept a new connection and return the initial data from that connection in one operation (this leads to slightly better performance in situations where you know you will always want some data before you can start doing things but is horribly complex to manage if you want to defend aginst the denial of service attack that can be launched simply by connecting and NOT sending data - I wrote about this here).
If you do not want AcceptEx() to wait for data to arrive then simply provide a data buffer that is ONLY big enough for the addresses to be returned and pass 0 as the 'buffer size'. This will cause the AcceptEx() to operate like an overlaped accept() and return as soon as the connection is established.
Note that Martin James' initial comment to your question is in fact the answer you're looking for. Don't pass outBufLen - ((sizeof (sockaddr_in) + 16) * 2), pass 0.
I'm trying to write an IOCP server. Basically, I have it accepting new connections. For the purpose of my testing, I'm running and connecting to 127.0.0.1.
I create the pseudo socket prior to calling AcceptEx(). Once a connection is accepted, the new pseudo socket is used for communication. This new socket is associated with an io completion port [CreateIoCompletionPort], I then assign it a few options, [SO_EXCLUSIVEADDRUSE] and [SO_CONDITIONAL_ACCEPT], and then I call WSARecv() to accept incoming data.
The problem is that once my remote connection connects to the server, it sends data, but that data is never received. I'm wondering if someone could offer some ideas as to why it's not receiving data? Perhaps my logic is flawed? I stepped through my code several times. no errors are recorded.
EDIT: Fixed the wording. I create the socket before AcceptEx() call.
Basic logic in my code:
// Create socket, associate with IOCP
WSASocket(af, type, proto, lpProtoInfo, g, dwFlags);
HANDLE hIOCP = GetPool()->GetQueueHandle();
CreateIoCompletionPort(hSource, hIOCP, 0, 0) != NULL;
// Server bind and listen
bind(m_shSocket, pAddr, nAddrLen);
listen(m_shSocket, nBacklog);
// Creation of the pseudo socket
SOCKET s = ::WSASocket(m_iSocketAf, m_iSocketType, m_iSocketProto, m_pWpi, m_SocketGroup, m_dwSocketFlags);
DWORD dwBytes;
BOOL bRet = m_fnAcceptEx(m_shSocket, s, chOutput, 0, sizeof(SOCKADDR_STORAGE) + 16, sizeof(SOCKADDR_STORAGE) + 16, &dwBytes, m_pcbAccept);
// ... New Connection comes in, it's accepted ...
// Associate new pseudo socket with IOCP
HANDLE hNewIOCP = GetPool()->GetQueueHandle();
CreateIoCompletionPort((HANDLE) s, hNewIOCP , 0, 0) != NULL;
// ... Remote socket sends ...
// ... Remote socket and Pseudo socket call WSARecv ...
// ... Pseudo socket does not receive ...
NOTE: I tried sending from the pseudo socket to the remote socket, same problem as sending data in the reverse way.
You need to post some code but your description doesn't make sense. That's NOT how AcceptEx() based servers operate.
With an AcceptEx() based server you create your accepted socket before you post the AcceptEx(). You then post the AcceptEx() with the listening socket and the new socket and a buffer which allows you to receive the remote address and, optionally, data.
So if you are describing your code in your original question then your code is wrong or you're not using AcceptEx(). I'm currently ignoring the 'few options' that you throw into the mix as they simply further confuse things at present without any code to analyse.
You might be interested in downloading my free IOCP based server framework, which includes working AcceptEx() and traditional Accept() based server code. You can get it from here: http://www.serverframework.com/products---the-free-framework.html
Are you calling GetQueuedCompletionStatus to get the data?
In case you are not doing this just to learn for yourself, I would also recommend that you use boost::asio - an excellent library that allows you to let someone else do the tedious code for handling the io completion ports.
I figured it out. I'm an idiot. I was sending zero bytes.
I want to write a server using a pool of worker threads and an IO completion port. The server should processes and forwards messages between multiple clients. The 'per client' data is in a class ClientContext. Data between instances of this class are exchanged using the worker threads. I think this is a typical scenario.
However, I have two problems with those IO completion ports.
(1) The first problem is that the server basically receives data from clients but I never know if a complete message was received. In fact WSAGetLastError() always returns that WSARecv() is still pending. I tried to wait for the event OVERLAPPED.hEvent with WaitForMultipleObjects(). However, it blocks forever, i.e WSARecv() never completes in my program.
My goal is to be absolutely sure that the whole message has been received before further processing starts. My message has a 'message length' field in its header, but I don't really see how to use it with the IOCP function parameters.
(2) If WSARecv() is commented out in the code snippet below, the program still receives data. What does that mean? Does it mean that I don't need to call WSARecv() at all? I am not able to get a deterministic behaviour with those IO completion ports.
Thanks for your help!
while(WaitForSingleObject(module_com->m_shutdown_event, 0)!= WAIT_OBJECT_0)
{
dequeue_result = GetQueuedCompletionStatus(module_com->m_h_io_completion_port,
&transfered_bytes,
(LPDWORD)&lp_completion_key,
&p_ol,
INFINITE);
if (lp_completion_key == NULL)
{
//Shutting down
break;
}
//Get client context
current_context = (ClientContext *)lp_completion_key;
//IOCP error
if(dequeue_result == FALSE)
{
//... do some error handling...
}
else
{
// 'per client' data
thread_state = current_context->GetState();
wsa_recv_buf = current_context->GetWSABUFPtr();
// 'per call' data
this_overlapped = current_context->GetOVERLAPPEDPtr();
}
while(thread_state != STATE_DONE)
{
switch(thread_state)
{
case STATE_INIT:
//Check if completion packet has been posted by internal function or by WSARecv(), WSASend()
if(transfered_bytes > 0)
{
dwFlags = 0;
transf_now = 0;
transf_result = WSARecv(current_context->GetSocket(),
wsa_recv_buf,
1,
&transf_now,
&dwFlags,
this_overlapped,
NULL);
if (SOCKET_ERROR == transf_result && WSAGetLastError() != WSA_IO_PENDING)
{
//...error handling...
break;
}
// put received message into a message queue
}
else // (transfered_bytes == 0)
{
// Another context passed data to this context
// and notified it via PostQueuedCompletionStatus().
}
break;
}
}
}
(1) The first problem is that the
server basically receives data from
clients but I never know if a complete
message was received.
Your recv calls can return anywhere from 1 byte to the whole 'message'. You need to include logic that works out when it has enough data to work out the length of the complete 'message' and then work out when you actually have a complete 'message'. Whilst you do NOT have enough data you can reissue a recv call using the same memory buffer but with an updated WSABUF structure that points to the end of the data that you have already recvd. In that way you can accumulate a full message in your buffer without needing to copy data after every recv call completes.
(2) If WSARecv() is commented out in
the code snippet below, the program
still receives data. What does that
mean? Does it mean that I don't need
to call WSARecv() at all?
I expect it just means you have a bug in your code...
Note that it's 'better' from a scalability point of view not to use the event in the overlapped structure and instead to associate the socket with the IOCP and allow the completions to be posted to a thread pool that deals with your completions.
I have a free IOCP client/server framework available from here which may give you some hints; and a series of articles on CodeProject (first one is here: http://www.codeproject.com/KB/IP/jbsocketserver1.aspx) where I deal with the whole 'reading complete messages' problem (see "Chunking the byte stream").