GetQueuedCompletionStatus exiting prematurely (or so I thought) - c++

Question

Let me give the general overview first. I'm recieving data thru three ports. I have a socket, a completion port and a worker thread for each. I call WSARecv and the worker thread process calls GetQueuedCompletionStatus followed by my parsing routine ReadMsgs. It sometimes happens that the buffer is unchanged when ReadMsgs is called and the buffer is updated while ReadMsgs is processing the buffer. The number of bytes processed, as returned by GetQueuedCompletionStatus is correct for the update when it occurs.
Does anyone know why this might occur and what I am doing wrong. Let me show you the code that seems most relevant. If you need to see more code, please be specific. My base socket class looks like this (I have omitted details that seem to me to be irrelevant. I have also omitted all error checking.)
class Socket_Base : public OVERLAPPED
{
public:
Socket_Base()
{
// Initialize base OVERLAPPED object
Internal = 0;
InternalHigh = 0;
Offset = 0;
OffsetHigh = 0;
hEvent = WSACreateEvent();
// Initialize addr structure
ZeroMemory( &addr, sizeof(struct sockaddr_in));
// Create the completion port
hCP = CreateIoCompletionPort( INVALID_HANDLE_VALUE, NULL, 0, 1);
// Create the worker thread and bind it to the callback function and the completion port
hThread = (HANDLE)_beginthreadex( NULL, 0, Callback_Socket, hCP, 0, NULL);
// Create the socket
Sock = WSASocket( AF_INET, SOCK_STREAM, IPPROTO_TCP, NULL, 0, WSA_FLAG_OVERLAPPED);
// Bind the socket to the completion port
CreateIoCompletionPort( (HANDLE)Sock, hCP, 0, 0);
}
void Connect() { WSAConnect( Sock, (SOCKADDR*)(&addr), sizeof(addr), NULL, NULL, NULL, NULL);}
void StartRecv()
{
DWORD Flags = 0;
DWORD numBytes = 0;
if (WSARecv( Sock, &wsaBuf, 1, &numBytes, &Flags, (OVERLAPPED*)this, NULL) == 0) ReadMsgs( numBytes);
}
int ReadMsgs( int NumBytes);
protected:
virtual ~Socket_Base() {}
virtual void ProcessMsg() = 0;
struct sockaddr_in addr;
SOCKET Sock;
HANDLE hCP;
WSABUF wsaBuf;
HANDLE hThread;
char *readBuf;
int bufsize;
};
Each port has its own derived socket class distinguised by port number and the virtual ProcessMsg function (which is called by ReadMsgs as each message is parsed). Here is one such class:
class Socket_Admin : public Socket_Base
{
public:
static const int bufcap = 1024;
Socket_Admin::Socket_Admin() : Socket_Base()
{
// Buffer
readBuf = new char[ bufcap];
// The socket
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = inet_addr("127.0.0.1");
addr.sin_port = htons(9300);
wsaBuf.buf = readBuf;
wsaBuf.len = bufcap;
}
~Socket_Admin();
void ProcessMsg();
};
The worker thread process is
unsigned int Callback_Socket( void *lpParameter)
{
HANDLE hCP = (HANDLE)lpParameter;
DWORD NumBytes = 0;
ULONGLONG CompletionKey;
WSAOVERLAPPED *pOverlapped;
while (GetQueuedCompletionStatus( hCP, &NumBytes, &CompletionKey, &pOverlapped, INFINITE) && CompletionKey == 0)
{
Socket_Base *pTCP = (Socket_Base*)pOverlapped;
if (NumBytes > 0) pTCP->ReadMsgs( NumBytes);
NumBytes = 0;
}
return 0;
}
There is one other thing I should explain. ReadMsgs does its parsing in place. The server delimits messages with a final line feed character and it delimits fields within a message with commas. ReadMsgs replaces commas and line feeds with null characters as it finds them and notes where each field begins in a separate array of pointers to locations in the buffer. Now when ReadMsgs gets to the end of the region of the buffer that was last filled it sometimes finds an incomplete message. This is copied to the beginning of the buffer, expecting the next read to complete the message, and wsaBuf is modified accordingly. Thus the end of ReadMsgs looks like this:
wsaBuf.buf = pchar;
wsaBuf.len = remsize;
StartRecv();
where pchar points to the character just beyond the partial message and remsize is the size of the remaining buffer.
I know what messages have been sent to my application from detailed server logs. The replacement of delimiters with null characters also makes it easy to see what part of the buffer has been processed. By saving the buffer to file and examining it, I can tell that it was updated after ReadMsgs was called. Also, by log messages not shown in the code above, I know that in these cases ReadMsgs was called by the worker thread. It doesnt happen every time, but it does happen.
If anyone can tell me what my mistake is, I would be grateful.

Answer 1

I may have an answer. I have calls to ReadMsgs in both StartRecv and in Callback_Socket. In StartRecv, I call ReadMsgs when WSARecv returns 0, indicating that the transfer was completed by WSARecv. I was thinking that GetQueuedCompletion status would not get involved if the transfer was completed in WSARecv. However if GetQueuedCompletionStatus did return in response to the completed read, then I would have a duplicate call to ReadMsgs that would explain the logged data I collected, just as well as my previous hypothesis.
I've removed the call to ReadMsgs in StartRecv. The code is working properly now.
Thanks to the gentleman who gave me the negative vote. That suggested to me that the behavior I described has not been observed before and so is extremely unlikely. That sent me thinking in a new direction. Sometimes just a grunt from someone more experienced van speak volumes.