Why would an Overlapped call to recv return ERROR_NO_MORE_ITEMS(259)? - c++

I did a few tests with an I/O-Completion port and winsock sockets.
I encountered, that sometimes after I received data from a connection and then adjacently call WSARecv again on that socket it returns immediately with the error 259 (ERROR_NO_MORE_ITEMS).
I am wondering why the system flags the overlapped transaction with this error instead of keeping the recv call blocking/waiting for incoming data.
Do You know what´s the sense of this ?
I would be glad to hear about your thoughts.
Edit: Code
do
{
OVERLAPPED* pOverlapped = nullptr;
DWORD dwBytes = 0; ULONG_PTR ulKey = 0;
//Dequeue a completion packet
if(!m_pIOCP->GetCompletionStatus(&dwBytes, &ulKey, &pOverlapped, INFINITE))
DebugBreak();
//Evaluate
switch(((MYOVERLAPPED*)pOverlapped)->WorkType)
{
case ACCEPT_OVERLAPPED_TYPE:
{
//cast
ACCEPT_OVERLAPPED* pAccept = (ACCEPT_OVERLAPPED*)pOverlapped;
//Associate the newly accepted connection with the IOCP
if(!m_pIOCP->AssociateHandle((HANDLE)(pAccept->pSockClient)->operator SOCKET(), 1))
{
//Association failed: close the socket and and delte the overlapped strucuture
}
//Call recv
RECV_OVERLAPPED* pRecvAction = new RECV_OVERLAPPED;
pRecvAction->pSockClient = pAccept->pSockClient;
short s = (pRecvAction->pSockClient)->Recv(pRecvAction->strBuf, pRecvAction->pWSABuf, 10, pRecvAction);
if(s == Inc::REMOTECONNECTION_CLOSED)
{
//Error stuff
}
//Call accept again (create a new ACCEPT_OVERLAPPED to ensure overlapped being zeroed out)
ACCEPT_OVERLAPPED *pNewAccept = new ACCEPT_OVERLAPPED;
pNewAccept->pSockListen = pAccept->pSockListen;
pNewAccept->pSockClient = new Inc::CSocket((pNewAccept->pSockListen)->Accept(nullptr, nullptr, pNewAccept));
//delete the old overlapped struct
delete pAccept;
}
break;
case RECV_OVERLAPPED_TYPE:
{
RECV_OVERLAPPED* pOldRecvAction = (RECV_OVERLAPPED*)pOverlapped;
if(!pOldRecvAction->InternalHigh)
{
//Connection has been closed: delete the socket(implicitly closes the socket)
Inc::CSocket::freewsabuf(pOldRecvAction->pWSABuf); //free the wsabuf
delete pOldRecvAction->pSockClient;
}
else
{
//Call recv again (create a new RECV_OVERLAPPED)
RECV_OVERLAPPED* pNewRecvAction = new RECV_OVERLAPPED;
pNewRecvAction->pSockClient = pOldRecvAction->pSockClient;
short sRet2 = (pNewRecvAction->pSockClient)->Recv(pNewRecvAction->strBuf, pNewRecvAction->pWSABuf, 10, pNewRecvAction);
//Free the old wsabuf
Inc::CSocket::freewsabuf(pOldRecvAction->pWSABuf);
delete pOldRecvAction;
}
Cutted error checkings...
The Recv-member-function is a simple wrapper around the WSARecv-call which creates the WSABUF and the receiving buffer itself (which needs to be cleaned up by the user via freewsabuf - just to mention)...

It looks like I was sending less data than was requested by the receiving side.
But since it´s an overlapped operation receiving a small junk of the requested bunch via the TCP-connection would trigger the completion indication with the error ERROR_NO_MORE_ITEMS, meaning there was nothing more to recv than what it already had.

Related

Strange IOCP behaviour when communicating with browsers

I'm writing IOCP server for video streaming from desktop client to browser.
Both sides uses WebSocket protocol to unify server's achitecture (and because there is no other way for browsers to perform a full-duplex exchange).
The working thread starts like this:
unsigned int __stdcall WorkerThread(void * param){
int ThreadId = (int)param;
OVERLAPPED *overlapped = nullptr;
IO_Context *ctx = nullptr;
Client *client = nullptr;
DWORD transfered = 0;
BOOL QCS = 0;
while(WAIT_OBJECT_0 != WaitForSingleObject(EventShutdown, 0)){
QCS = GetQueuedCompletionStatus(hIOCP, &transfered, (PULONG_PTR)&client, &overlapped, INFINITE);
if(!client){
if( Debug ) printf("No client\n");
break;
}
ctx = (IO_Context *)overlapped;
if(!QCS || (QCS && !transfered)){
printf("Error %d\n", WSAGetLastError());
DeleteClient(client);
continue;
}
switch(auto opcode = client->ProcessCurrentEvent(ctx, transfered)){
// Client owed to receive some data
case OPCODE_RECV_DEBT:{
if((SOCKET_ERROR == client->Recv()) && (WSA_IO_PENDING != WSAGetLastError())) DeleteClient(client);
break;
}
// Client received all data or the beginning of new message
case OPCODE_RECV_DONE:{
std::string message;
client->GetInput(message);
// Analizing the first byte of WebSocket frame
switch( opcode = message[0] & 0xFF ){
// HTTP_HANDSHAKE is 'G' - from GET HTTP...
case HTTP_HANDSHAKE:{
message = websocket::handshake(message);
while(!client->SetSend(message)) Sleep(1); // Set outgoing data
if((SOCKET_ERROR == client->Send()) && (WSA_IO_PENDING != WSAGetLastError())) DeleteClient(client);
break;
}
// Browser sent a closing frame (0x88) - performing clean WebSocket closure
case FIN_CLOSE:{
websocket::frame frame;
frame.parse(message);
frame.masked = false;
if( frame.pl_len == 0 ){
unsigned short reason = 1000;
frame.payload.resize(sizeof(reason));
frame.payload[0] = (reason >> 8) & 0xFF;
frame.payload[1] = reason & 0xFF;
}
frame.pack(message);
while(!client->SetSend(message)) Sleep(1);
if((SOCKET_ERROR == client->Send()) && (WSA_IO_PENDING != WSAGetLastError())) DeleteClient(client);
shutdown(client->Socket(), SD_SEND);
break;
}
IO context struct:
struct IO_Context{
OVERLAPPED overlapped;
WSABUF data;
char buffer[IO_BUFFER_LENGTH];
unsigned char opcode;
unsigned long long debt;
std::string message;
IO_Context(){
debt = 0;
opcode = 0;
data.buf = buffer;
data.len = IO_BUFFER_LENGTH;
overlapped.Offset = overlapped.OffsetHigh = 0;
overlapped.Internal = overlapped.InternalHigh = 0;
overlapped.Pointer = nullptr;
overlapped.hEvent = nullptr;
}
~IO_Context(){ while(!HasOverlappedIoCompleted(&overlapped)) Sleep(1); }
};
Client Send function:
int Client::Send(){
int var_buf = O.message.size();
// "O" is IO_Context for Output
O.data.len = (var_buf>IO_BUFFER_LENGTH)?IO_BUFFER_LENGTH:var_buf;
var_buf = O.data.len;
while(var_buf > 0) O.data.buf[var_buf] = O.message[--var_buf];
O.message.erase(0, O.data.len);
return WSASend(connection, &O.data, 1, nullptr, 0, &O.overlapped, nullptr);
}
When the desktop client disconnects (it uses just closesocket() to do it, no shutdown()) the GetQueuedCompletionStatus returns TRUE and sets transfered to 0 - in this case WSAGetLastError() returns 64 (The specified network name is no longer available), and it has sense - client disconnected (line with if(!QCS || (QCS && !transfered))). But when the browser disconnects, the error codes confuse me... It can be 0, 997 (pending operation), 87 (invalid parameter)... and no codes related to end of connection.
Why do IOCP select this events? How can it select a pending operation? Why the error is 0 when 0 bytes transferred? Also it leads to endless trying to delete an object associated with the overlapped structure, because the destructor calls ~IO_Context(){ while(!HasOverlappedIoCompleted(&overlapped)) Sleep(1); } for secure deleting. In DeleteClient call the socket is closing with closesocket(), but, as you can see, I'm posting a shutdown(client->Socket(), SD_SEND); call before it (in FIN_CLOSE section).
I understand that there are two sides of a connection and closing it on a server side does not mean that an other side will close it too. But I need to create a stabile server, immune to bad and half opened connections. For example, the user of web application can rapidly press F5 to reload page few times (yeah, some dudes do so :) ) - the connection will reopen few times, and the server must not lag or crash due to this actions.
How to handle this "bad" events in IOCP?
you have many wrong code here.
while(WAIT_OBJECT_0 != WaitForSingleObject(EventShutdown, 0)){
QCS = GetQueuedCompletionStatus(hIOCP, &transfered, (PULONG_PTR)&client, &overlapped, INFINITE);
this is not efficient and wrong code for stop WorkerThread. at first you do excess call WaitForSingleObject, use excess EventShutdown and main this anyway fail todo shutdown. if your code wait for packet inside GetQueuedCompletionStatus that you say EventShutdown - not break GetQueuedCompletionStatus call - you continue infinite wait here. correct way for shutdown - PostQueuedCompletionStatus(hIOCP, 0, 0, 0) instead call SetEvent(EventShutdown) and if worked thread view client == 0 - he break loop. and usually you need have multiple WorkerThread (not single). and multiple calls PostQueuedCompletionStatus(hIOCP, 0, 0, 0) - exactly count of working threads. also you need synchronize this calls with io - do this only after all io already complete and no new io packets will be queued to iocp. so "null packets" must be the last queued to port
if(!QCS || (QCS && !transfered)){
printf("Error %d\n", WSAGetLastError());
DeleteClient(client);
continue;
}
if !QCS - the value in client not initialized, you simply can not use it and call DeleteClient(client); is wrong under this condition
when object (client) used from several thread - who must delete it ? what be if one thread delete object, when another still use it ? correct solution will be if you use reference counting on such object (client). and based on your code - you have single client per hIOCP ? because you retriever pointer for client as completion key for hIOCP which is single for all I/O operation on sockets bind to the hIOCP. all this is wrong design.
you need store pointer to client in IO_Context. and add reference to client in IO_Context and release client in IO_Context destructor.
class IO_Context : public OVERLAPPED {
Client *client;
ULONG opcode;
// ...
public:
IO_Context(Client *client, ULONG opcode) : client(client), opcode(opcode) {
client->AddRef();
}
~IO_Context() {
client->Release();
}
void OnIoComplete(ULONG transfered) {
OnIoComplete(RtlNtStatusToDosError(Internal), transfered);
}
void OnIoComplete(ULONG error, ULONG transfered) {
client->OnIoComplete(opcode, error, transfered);
delete this;
}
void CheckIoError(ULONG error) {
switch(error) {
case NOERROR:
case ERROR_IO_PENDING:
break;
default:
OnIoComplete(error, 0);
}
}
};
then are you have single IO_Context ? if yes, this is fatal error. the IO_Context must be unique for every I/O operation.
if (IO_Context* ctx = new IO_Context(client, op))
{
ctx->CheckIoError(WSAxxx(ctx) == 0 ? NOERROR : WSAGetLastError());
}
and from worked threads
ULONG WINAPI WorkerThread(void * param)
{
ULONG_PTR key;
OVERLAPPED *overlapped;
ULONG transfered;
while(GetQueuedCompletionStatus(hIOCP, &transfered, &key, &overlapped, INFINITE)) {
switch (key){
case '_io_':
static_cast<IO_Context*>(overlapped)->OnIoComplete(transfered);
continue;
case 'stop':
// ...
return 0;
default: __debugbreak();
}
}
__debugbreak();
return GetLastError();
}
the code like while(!HasOverlappedIoCompleted(&overlapped)) Sleep(1); is always wrong. absolute and always. never write such code.
ctx = (IO_Context *)overlapped; despite in your concrete case this give correct result, not nice and can be break if you change definition of IO_Context. you can use CONTAINING_RECORD(overlapped, IO_Context, overlapped) if you use struct IO_Context{
OVERLAPPED overlapped; } but better use class IO_Context : public OVERLAPPED and static_cast<IO_Context*>(overlapped)
now about Why do IOCP select this events? How to handle this "bad" events in IOCP?
the IOCP nothing select. he simply signaling when I/O complete. all. which specific wsa errors you got on different network operation absolute independent from use IOCP or any other completion mechanism.
on graceful disconnect is normal when error code is 0 and 0 bytes transferred in recv operation. you need permanent have recv request active after connection done, and if recv complete with 0 bytes transferred this mean that disconnect happens

Should i pass unique OVERLAPPED structure for each WSASend call , in this case?

I have a list of sockets.(Opened connections)
I have n worker threads.
Thread loop:
while (1)
{
_this.result = GetQueuedCompletionStatus(a_server.server_iocp, &_this.numberOfBytesTransfered,
&_this.completionKey, (OVERLAPPED**)&_this.iocp_task, INFINITE);
...
}
I have this simple struct:
struct iocp_send_global :public iocp_task<IOCP_SEND_GLOBAL> {
OVERLLAPED ov; //overlapped struct at top
std::atomic_uint32_t ref;
bool decr_ref(){ return ref.fetch_sub(1, std::memory_order_acq_rel) == 1;}
//packet data here
}
...
This is the 'Broadcast' function:
iocp_send_global * packet = new iocp_send_global;
[set packet data here]
for(int i=0;i<connectionsCount;++i){
WSASend(connections[i],...,&packet,...); //posting same packet to all connections
}
I want to do this in the worker loop after GetQueuedCompletionStatus call returns with the overlapped result;
if (_this.iocp_task->type == IOCP_SEND_GLOBAL) {
auto* task = (iocp_send_global*)_this.iocp_task;
if (!task->decr_ref()) {
_this.iocp_task = nullptr;
//dont delete the task yet,
//all send post must finish first
//[all posts share the same buffer]
}
else {
//delete the task containing the send data after all send posts finished
delete _this.iocp_task;
_this.iocp_task = nullptr;
}
}
From what i read on Microsoft WSASend documentation each WSASend overlapped call sould have its own OVERLAPPED structure passed, but is that valid when i WSASend the same buffer?
Thank you!
You must pass a different OVERLAPPED buffer for each call since you'll be making multiple pending calls. This is clearly spelled out in the documentation for the OVERLAPPED structure.

why shutdown on udp socket blocks?

I'm writing a UDP server application for windows desktop/server.
My code uses the WSA API suggested by windows the following way (This is my simplified receivePacket method):
struct Packet
{
unsigned int size;
char buffer[MAX_SIZE(1024)];
}
bool receivePacket(Packet packet)
{
WSABUFFER wsa_buffer[2];
wsa_buffer[0].buf = &packet.size;
wsa_buffer[0].len = sizeof(packet.size);
wsa_buffer[1].buf = packet.buffer;
wsa_buffer[1].len = MAX_SIZE;
bool retval = false;
int flags = 0;
int recv_bytes = 0;
inet_addr client_addr;
int client_addr_len = sizeof(client_addr);
if(WSARecvFrom(_socket, wsa_buffer, sizeof(wsa_buffer)/sizeof(wsa_buffer[0]), &bytes_recv, &flags, (sockaddr *)&client_addr, &client_addr_len, NULL, NULL) == 0)
{
//Packet received successfully
}
else
{
//Report
}
}
Now, when I'm trying to close my application gracefully, not network-wise, but rather application-wise (going through all the d'tors and stuff), i'm trying to unblock this call.
To do this, I call the shutdown(_socket, SD_BOTH) method. Unfortunately, the call to shutdown itself BLOCKS!
After reading every possible page in the MSDN, I didn't find any reference to why this happens, other ways of attacking the problem or any way out.
Another thing I checked was using the SO_RCVTIMEO. Surprisingly, this sockopt didn't work as expected as well.
Is there any problem with my code/approach?
Did you run shutdown on duplicated handle? Shutdown on the same handle will wait any active operation on this handle to complete.

IOCP and overwritten buffer

Well i make a IOCP for handling client connections with the following details:
- Threads = (CPU cores * 2)
- Assigning an completion port to each socket
- Accessing the socket context by Client Index or overlapped struct (either way is the same)
So i am trying to debug the incoming packets, its works like a charm, except for a little but nasty detail... I set a break point on WorkersThread function (where i recv the packet) i am watching the buffer with the packet i recv, when suddenly the buffer gets overwritten with a new packet that i got from client.
Why is that? according to what i read, IOCP should wait till i process the packet, send a response to client before recv any other packet. So i set a flag on my socket context called "Processing" and still got the overwritten buffer with an incoming packet. So it doesn't let me debug at all and its driving me crazy
Is ollydbg (debugger) fault that let the other threads running while i set a break point? Or is some error in my IOCP implementation?
Here is how my WorkerThread is coded:
DWORD WINAPI WorkerThread(void* argument)
{
int BytesTransfer;
int BytesRecv;
int ClientID;
int result;
OVERLAPPED* overlapped = 0;
ClientInfo* clientinfo = 0;
WSABUF wsabuf;
int flags;
//Exit only when shutdown signal is recv
while (WaitForSingleObject(IOCPBase::internaldata->sockcontext.ShutDownSignal, NULL) != WAIT_OBJECT_0)
{
flags = 0; BytesTransfer = 0; BytesRecv = 0; ClientID = 0;
//Get from queued list
if (GetQueuedCompletionStatus(IOCPBase::internaldata->sockcontext.CompletionPort, (LPDWORD)&BytesTransfer, (PULONG_PTR)&ClientID, &overlapped, INFINITE) == TRUE)
{
if (overlapped == 0)
{
//Fatal error
break;
}
clientinfo = (ClientInfo*)overlapped;
if (BytesTransfer != 0)
{
//Assign the buffer pointer and buffer len to WSABUF local
clientinfo->RecvContext.RecvBytes = BytesTransfer;
wsabuf.buf = (char*)clientinfo->RecvContext.Buffer;
wsabuf.len = clientinfo->RecvContext.Len;
//Switch for OperationCode
//switch (IOCPBase::internaldata->ClientContext[ClientID].OperationCode)
switch (clientinfo->OperationCode)
{
case FD_READ:
// Check if we have send all data to the client from a previous send
if (clientinfo->SendContext.SendBytes < clientinfo->SendContext.TotalBytes)
{
clientinfo->OperationCode = FD_READ; //We set FD_READ caused on the next send, there could still be bytes left to send
wsabuf.buf += clientinfo->SendContext.SendBytes; //The buffer position is + sended bytes
wsabuf.len = clientinfo->SendContext.TotalBytes - clientinfo->SendContext.SendBytes; //the buffer len is total - sended bytes
//Send the remain bytes
result = WSASend(clientinfo->sock, &wsabuf, 1, (LPDWORD)&BytesRecv, flags, &clientinfo->overlapped, NULL);
if (result == SOCKET_ERROR && (WSAGetLastError() != WSA_IO_PENDING))
{
CloseClient(ClientID);
}
clientinfo->SendContext.SendBytes += BytesRecv;
}
else
{
if (clientinfo->Processing == 0)
{
clientinfo->OperationCode = FD_WRITE; //If no more bytes left to send now we can set the operation code to write (in fact is read)
memset(clientinfo->RecvContext.Buffer, NULL, MAX_DATA_BUFFER_SIZE); //Clean the buffer for recv new data
//Recv data from our client
clientinfo->RecvContext.RecvBytes = WSARecv(clientinfo->sock, &wsabuf, 1, (LPDWORD)&BytesRecv, (LPDWORD)&flags, &clientinfo->overlapped, NULL);
if (clientinfo->RecvContext.RecvBytes == SOCKET_ERROR && WSAGetLastError() != WSA_IO_PENDING)
{
CloseClient(ClientID);
break;
}
}
}
break;
case FD_WRITE:
//Send data to the RecvProtocol
clientinfo->Processing = 1;
IOCPBase::internaldata->callback.RecvProtocol(clientinfo->RecvContext.Buffer, clientinfo->RecvContext.Len, ClientID);
clientinfo->Processing = 0;
default:
break;
}
}
}
}
return false;
}
The problem appears when looking at clientinfo->RecvContext.Buffer. I am watching the packet, past a few seconds and boom the buffer is overwritten with a new packet.
Thanks !
Never mind i fix the debug problem by copy the packet to the stack frame of the function i use to analyze the packet, this way i have no overwritten problem.

_beginthreadx and socket

i have a question about the _beginthreadx function In the third and fourth parameter:
if i have this line to create the thread
hThread=(HANDLE)_beginthreadex(0,0, &RunThread, &m_socket,CREATE_SUSPENDED,&threadID );
m_socket is the socket that i want inside the thread (fourth parameter)
and i have the RunThread function (third parameter) in this way
static unsigned __stdcall RunThread (void* ptr) {
return 0;
}
It is sufficient to create the thread independently if m_socket has something or not?
Thanks in advance
Thank you for the response Ciaran Keating helped me understand better the thread
I'll explain a little more the situation
I´m creating the tread in this function inside a class
public: void getClientsConnection()
{
numberOfClients = 1;
SOCKET temporalSocket = NULL;
firstClient = NULL;
secondClient = NULL;
while (numberOfClients < 2)
{
temporalSocket = SOCKET_ERROR;
while (temporalSocket == SOCKET_ERROR)
{
temporalSocket = accept(m_socket, NULL, NULL);
//-----------------------------------------------
HANDLE hThread;
unsigned threadID;
hThread=(HANDLE)_beginthreadex(0,0, &RunThread, &m_socket,CREATE_SUSPENDED,&threadID );
WaitForSingleObject( hThread, INFINITE );
if(!hThread)
printf("ERROR AL CREAR EL HILO: %ld\n", WSAGetLastError());
//-----------------------------------------------
}
if(firstClient == NULL)
{
firstClient = temporalSocket;
muebleC1 = temporalSocket;
actionC1 = temporalSocket;
++numberOfClients;
printf("CLIENTE 1 CONECTADO\n");
}
else
{
secondClient = temporalSocket;
muebleC2 = temporalSocket;
actionC2 = temporalSocket;
++numberOfClients;
printf("CLIENTE 2 CONECTADO\n");
}
}
}
What i'm trying to do is to have the socket inside the thread while wait for a client connection
Is this feasible as i have the code of the thread?
I can change the state of the thread that is not a problem
Thanks again
Yes, that will create the thread and pass it your socket handle. But by returning immediately from RunThread your new thread will terminate immediately after you resume it (you've created it suspended.) You'll need to put your socket handling code (read/write loop etc.) inside RunThread.
Some more tips:
You'll have to make sure that m_socket remains valid for the life of the thread, because you passed it by reference. You might prefer to pass it by value instead, and let ownership pass to the new thread, but of course in that case it probably wouldn't belong in your object instance (I infer from the m_ prefix.) Or you might prefer to leave the socket handle in the object instance, and pass a reference to the object to beginthread instead:
beginthread(...,&RunThread,this,...);
(With your new info, I can see that my other answer isn't what you need.)
If I understand you right, you just want to wait on the accept() call until a client connects. You don't need threads for that - there are native sockets ways to do it. One option is to make m_socket a blocking socket, so accept() doesn't return until a client connects. An easier way is to use the select() function to wait until the socket is ready to read, which in the case of a listening socket means that a client has connected.
fd_set fds;
FD_ZERO(&fds);
FD_SET(m_socket,&fds);
int ret = select(0,&fds,NULL,NULL,NULL); // will block
if(FD_ISSET(m_socket,&fds))
temporalSocket = accept(...);