I am reading about IOCP through this article
http://www.winsocketdotnetworkprogramming.com/winsock2programming/winsock2advancediomethod5i.html
This first WSARecv function he used with 0 lpNumberOfBytesRecvd to initiate the completion ports
if (WSARecv(Accept, &(PerIoData->DataBuf), 1, &RecvBytes, &Flags, &(PerIoData->Overlapped), NULL) == SOCKET_ERROR)
{
if (WSAGetLastError() != ERROR_IO_PENDING)
{
printf("WSARecv() failed with error %d\n", WSAGetLastError());
return 1;
}
}
else
printf("WSARecv() is OK!\n");
}
Now on the worker threads function where the application receives the IO post
it checks parameter lpNumberOfBytes of GetQueuedCompletionStatus for 0 also and then it closes the connection if it is
// First check to see if an error has occurred on the socket and if so
// then close the socket and cleanup the SOCKET_INFORMATION structure
// associated with the socket
if (BytesTransferred == 0)
When i tried it the first IOCP message posted the function GetQueuedCompletionStatus it always returning 0 in lpNumberOfBytes parameter because first WSARecv call was called with lpNumberOfBytesRecvd set to zero
I have tried to implement this code and i am having troubles to get the code working unless i choose to not use NULL in this parameter
in the first call to WSARecv so that the GetQueuedCompletionStatus function returning value 0 in lpNumberOfBytes can be used as in case of all data recieved.
MSDN says to use NULL
lpNumberOfBytesRecvd [out]
A pointer to the number, in bytes, of data received by this call if
the receive operation completes immediately.
Use NULL for this parameter if the lpOverlapped parameter is not NULL
to avoid potentially erroneous results. This parameter can be NULL
only if the lpOverlapped parameter is not NULL.
The code is not working well if i call WSARecv again (inside worker threads function) and all data were recieved this causes the GetQueuedCompletionStatus function to return immediatly without waiting even if INFINITE used and the worker threads enter in endless loop. No errors ever returned by GetQueuedCompletionStatus function it just acts as if there is more data but there is none.
WSARecv needs to be initiated with what value at buf size?
also should i use non blocking socket? or just normal blocking sockets since i use separate thread to accept connections so i think its fine because it won't freeze my GUI.
Related
I use a blocking FSocket in client-side that connected to tcp server, if there's no message from server, socket thread would block in function FScoket::Recv(), if TCP server shutdown, socket thread is still blocking in this function. but when use blocking socket of BSD Socket API, thread would pass from recv function and return errno when TCP server shutdown, so is it the defect of FSocket?
uint32 HRecvThread::Run()
{
uint8* recv_buf = new uint8[RECV_BUF_SIZE];
uint8* const recv_buf_head = recv_buf;
int readLenSeq = 0;
while (Started)
{
//if (TcpClient->Connected() && ClientSocket->GetConnectionState() != SCS_Connected)
//{
// // server disconnected
// TcpClient->SetConnected(false);
// break;
//}
int32 bytesRead = 0;
//because use blocking socket, so thread would block in Recv function if have no message
ClientSocket->Recv(recv_buf, readLenSeq, bytesRead);
.....
//some logic of resolution for tcp msg bytes
.....
}
delete[] recv_buf;
return 0
}
As I expected, you are ignoring the return code, which presumably indicates success or failure, so you are looping indefinitely (not blocking) on an error or end of stream condition.
NB You should allocate the recv_buf on the stack, not dynamically. Don't use the heap when you don't have to.
There is a similar question on the forums in the UE4 C++ Programming section. Here is the discussion:
https://forums.unrealengine.com/showthread.php?111552-Recv-function-would-keep-blocking-when-TCP-server-shutdown
Long story short, in the UE4 Source, they ignore EWOULDBLOCK as an error. The code comments state that they do not view it as an error.
Also, there are several helper functions you should be using when opening the port and when polling the port (I assume you are polling since you are using blocking calls)
FSocket::Connect returns a bool, so make sure to check that return
value.
FSocket::GetLastError returns the UE4 Translated error code if an
error occured with the socket.
FSocket::HasPendingData will return a value that informs you if it
is safe to read from the socket.
FSocket::HasPendingConnection can check to see your connection state.
FSocket::GetConnectionState will tell you your active connection state.
Using these helper functions for error checking before making a call to FSocket::Recv will help you make sure you are in a good state before trying to read data. Also, it was noted in the forum posts that using the non-blocking code worked as expected. So, if you do not have a specific reason to use blocking code, just use the non-blocking implementation.
Also, as a final hint, using FSocket::Wait will block until your socket is in a desirable state of your choosing with a timeout, i.e. is readable or has data.
I'm currently using libpcap to sniff traffic in promiscuous mode
int main()
{
// some stuff
printf("Opening device: %s\n", devname.c_str());
handle = pcap_open_live(devname.c_str(), 65536 , 1 , 0 , errbuf);
if (handle == NULL)
{
fprintf(stderr, "Couldn't open device %s : %s..." , devname.c_str(), errbuf);
return 1;
}
printf(" Done\n");
pcap_loop(handle , -1 , process_packet , NULL);
// here run a thread to do some stuff. however, pcap_loop is blocking
return 0;
}
I'd like to add an external thread to do some other stuff. How do I change the code above to make it non-blocking?
When you use non-blocking mode on libpcap you have to use pcap_dispatch, but note, pcap_dispatch can work in blocking or in non-blocking mode, it depends how you set libpcap, to set libpcap to work in non-blocking you have use the function pcap_setnonblock:
int pcap_setnonblock(pcap_t *p, int nonblock, char *errbuf);
The difference between blocking and non-blocking is not a loop that runs forever, but in blocking the function pcap_dispatch waits for a packet and only returns when this packet is received, however, in the non-blocking mode the function returns immediately and the callback must process the packet.
In "non-blocking" mode, an attempt to read from the capture
descriptor with pcap_dispatch() will, if no packets are currently
available to be read, return 0 immediately rather than blocking
waiting for packets to arrive. pcap_loop() and pcap_next() will not
work in "non-blocking" mode.
http://www.tcpdump.org/manpages/pcap_setnonblock.3pcap.html
pcap_loop is meant to go on until all input ends. If you don't want that behavior, call pcap_dispatch in a loop instead. By definition pcap_loop will never return, its meant to always searching for more data.
I use pcap_next_ex It returns a result indicating if a packet was read. This way I manage the acquisition my own thread. See an example here. The read_timeout in pcap_open also affects this function.
WSASend() will return immediately whether the data will be sent or not. But how to make sure that data will be sent, for example I have a button in my UI that will send "Hello World!" when pressed. Now I want to make sure that when the user click on this button the "Hello World!" will be sent at some point, but WSASend() could return WSAEWOULDBLOCK indicating that data will not be sent, so should I enclose WSASend() in a loop that does not exit until WSASend() returns 0 (success).
Note: I am using IOCP.
should I enclose WSASend() in a loop that does not exit until
WSASend() returns 0 (success)
Err.. NO!
Have the UI issue an overlapped WSASend request, complete with buffer/s and OVERLAPPED/s. If, by some miracle, it does actually return success immedately, (and I've never seen it), you're good.
If, (when:), it returns WSA_IO_PENDING, you can do nothing in your UI button-handler because GUI event-handlers cannot wait. Graphical UI's are state-machines - you must exit the button-handler and return to the message input queue in prompt manner. You can do some GUI stuff, if you want. Maybe disable the 'Send' button, or add some 'Message sent' text to a memo component. That's about it - you must then exit.
Some time later, the successful completion notification, (or failure notification), will get posted to the IOCP completion queue and a handler thread will get hold of it. Use PostMessage, QueueUserAPC or similar inter-thread comms mechanism to signal 'something', (eg. the buffer object used in the original WSASend), back to the UI thread so that it can take action/s on the returned result, eg. re-enabling the 'Send' button.
Yes, it can be seen as messy, but it is the only way you can do it that will work well.
Other approaches - polling loops, Application.DoEvents, timers etc are all horrible bodges.
Overlapped Socket I/O
If an overlapped operation completes immediately, WSASend returns a value of zero and the lpNumberOfBytesSent parameter is updated with the number of bytes sent. If the overlapped operation is successfully initiated and will complete later, WSASend returns SOCKET_ERROR and indicates error code WSA_IO_PENDING.
...
The error code WSA_IO_PENDING indicates that the overlapped operation has been successfully initiated and that completion will be indicated at a later time. Any other error code indicates that the overlapped operation was not successfully initiated and no completion indication will occur.
...
So as demonstrated in docs, you don't need to enclose in a loop, just check for a SOCKET_ERROR and if the last error is not equal to WSA_IO_PENDING, everything is fine:
rc = WSASend(AcceptSocket, &DataBuf, 1,
&SendBytes, 0, &SendOverlapped, NULL);
if ((rc == SOCKET_ERROR) &&
(WSA_IO_PENDING != (err = WSAGetLastError()))) {
printf("WSASend failed with error: %d\n", err);
break;
}
I'm trying to write a server which can support many clients connections simultaneously so I'm trying to do it with IOCP. So let me brief about my code flow and then I can explain my problem. First of all, server is opening a port for listening and waiting on an "accept" call for new incoming connections. For reference I have used same code as mentioned here So it accepts every new incoming connection and returns a new socket descriptor (sd), and then it marks as nonblocking with:
arg = 1;
ioctlsocket(sd, FIONBIO, &arg);
and then enable TCP_NODELAY:
level = IPPROTO_TCP;
optName = TCP_NODELAY;
value = 1;
setsockopt(sd, level, optName, (const char*)&value, sizeof(value));
thereafter associating with an IOCP port as:
CreateIoCompletionPort((HANDLE)sd, iocp_port, (DWORD)completion_key, 4);
completion_key is a class object which is nothing but a container, it contains data buffer, overlapped-buffer, query-type recv/send etc.
and in last issuing a read call:
WSARecv(sd, wsabuf, 1, &bytes, &flags, overlapped, NULL);
wsabuf and overlapped are part of completion_key object.
In 90% cases it works fine i.e. when there is some incoming data available on this socket "GetQueuedCompletionStatus" gets unblocked and it has valid data. But sometimes WSARecv call returns with an error and GetLastError() returns 6 which is "invalid handle" error. I'm bit bewildered why it's happening so.
The way I'm creating an iocp port:
iocp_port = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0);
and there are threads which are waiting on "GetQueuedCompletionStatus".
I monitored all system calls which were happening in background. WSARecv internally calls NtDeviceIoControlFile and there is an argument "Event" which is same as what is passed in lpOverlapped structure of WSARecv as hEvent. I wasn't setting hEvent to NULL, so it was taking some garbage value, when it was NULL then NtDeviceIoControlFile returned successfully and for other cases it returned "INVALID_HANDLE" error. Unfortunately, it was NULL most of the time.
I want to write a server using a pool of worker threads and an IO completion port. The server should processes and forwards messages between multiple clients. The 'per client' data is in a class ClientContext. Data between instances of this class are exchanged using the worker threads. I think this is a typical scenario.
However, I have two problems with those IO completion ports.
(1) The first problem is that the server basically receives data from clients but I never know if a complete message was received. In fact WSAGetLastError() always returns that WSARecv() is still pending. I tried to wait for the event OVERLAPPED.hEvent with WaitForMultipleObjects(). However, it blocks forever, i.e WSARecv() never completes in my program.
My goal is to be absolutely sure that the whole message has been received before further processing starts. My message has a 'message length' field in its header, but I don't really see how to use it with the IOCP function parameters.
(2) If WSARecv() is commented out in the code snippet below, the program still receives data. What does that mean? Does it mean that I don't need to call WSARecv() at all? I am not able to get a deterministic behaviour with those IO completion ports.
Thanks for your help!
while(WaitForSingleObject(module_com->m_shutdown_event, 0)!= WAIT_OBJECT_0)
{
dequeue_result = GetQueuedCompletionStatus(module_com->m_h_io_completion_port,
&transfered_bytes,
(LPDWORD)&lp_completion_key,
&p_ol,
INFINITE);
if (lp_completion_key == NULL)
{
//Shutting down
break;
}
//Get client context
current_context = (ClientContext *)lp_completion_key;
//IOCP error
if(dequeue_result == FALSE)
{
//... do some error handling...
}
else
{
// 'per client' data
thread_state = current_context->GetState();
wsa_recv_buf = current_context->GetWSABUFPtr();
// 'per call' data
this_overlapped = current_context->GetOVERLAPPEDPtr();
}
while(thread_state != STATE_DONE)
{
switch(thread_state)
{
case STATE_INIT:
//Check if completion packet has been posted by internal function or by WSARecv(), WSASend()
if(transfered_bytes > 0)
{
dwFlags = 0;
transf_now = 0;
transf_result = WSARecv(current_context->GetSocket(),
wsa_recv_buf,
1,
&transf_now,
&dwFlags,
this_overlapped,
NULL);
if (SOCKET_ERROR == transf_result && WSAGetLastError() != WSA_IO_PENDING)
{
//...error handling...
break;
}
// put received message into a message queue
}
else // (transfered_bytes == 0)
{
// Another context passed data to this context
// and notified it via PostQueuedCompletionStatus().
}
break;
}
}
}
(1) The first problem is that the
server basically receives data from
clients but I never know if a complete
message was received.
Your recv calls can return anywhere from 1 byte to the whole 'message'. You need to include logic that works out when it has enough data to work out the length of the complete 'message' and then work out when you actually have a complete 'message'. Whilst you do NOT have enough data you can reissue a recv call using the same memory buffer but with an updated WSABUF structure that points to the end of the data that you have already recvd. In that way you can accumulate a full message in your buffer without needing to copy data after every recv call completes.
(2) If WSARecv() is commented out in
the code snippet below, the program
still receives data. What does that
mean? Does it mean that I don't need
to call WSARecv() at all?
I expect it just means you have a bug in your code...
Note that it's 'better' from a scalability point of view not to use the event in the overlapped structure and instead to associate the socket with the IOCP and allow the completions to be posted to a thread pool that deals with your completions.
I have a free IOCP client/server framework available from here which may give you some hints; and a series of articles on CodeProject (first one is here: http://www.codeproject.com/KB/IP/jbsocketserver1.aspx) where I deal with the whole 'reading complete messages' problem (see "Chunking the byte stream").