C++ TCP Socket Exponential Increasing Delay - c++

Question

I seem to have an issue with increasing latency on my packet transmission with my TCP server. Now, this server has to be TCP, since UDP is blocked by firewalls (this is a client-server-client type of communication). I'm also aware that the sending of a struct with floating point integers as I am is extremely non-portable, however, this system will operate Windows client to Windows server to Windows client for the foreseeable future.
The issue is this: the client begins receiving the data properly from the other client, however, there is a delay which gets exponentially worse (where, by about 3 minutes in, the packets are nearly 30 seconds behind - but correct, when they DO arrive). I researched it and found an answer on a Microsoft page explaining it is due to full send buffers, however, their syntax for the setsockopt doesn't match the documented examples, so perhaps I'm wrong.
Anyway, any advice would be appreciated:
The relevant part of the server:
(When accept() is called:)
int buff_size = 2048000;
int nodel = 1;
setsockopt(sock, SOL_SOCKET, SO_SNDBUF, (char*)&buff_size, sizeof(int));
setsockopt(sock, SOL_SOCKET, SO_RCVBUF, (char*)&buff_size, sizeof(int));
setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, (char*)&nodel, sizeof(nodel));
The message redirect loop:
if (gp->curr_pilot < sz && gp->users[gp->curr_pilot].pilot == TRUE) {
char* pbuf = new char[1024];
int recvd = recv(gp->users[gp->curr_pilot].sockfd_data, pbuf, 1024, NULL);
if (recvd > 0) {
for (int i = 0; i < sz; i++) {
if (i != gp->curr_pilot && gp->users[i].unioned == TRUE)
send(gp->users[i].sockfd_data, pbuf, recvd, NULL);
}
}
delete[] pbuf;
}
The client (master is set when it's sending, and it does get set properly by my code):
(data is my struct of doubles that gets written by the client, cdata is a copy of it that gets written into the client).
while (kill_dataproc == FALSE) {
if (master == TRUE) {
char* buff = new char[1024];
int packet_signer = 1192;
memcpy_s(buff, intsz, &packet_signer, intsz);
memcpy_s((void*)(buff + intsz), sz, data, sz);
send(server_sock, buff, buffsize, NULL);
delete[] buff;
}
else {
char* buffer = new char[1024];
int recvd = recv(server_sock, buffer, 1024, MSG_PEEK);
if (recvd > 0) {
int newpacketsigner = 0;
memcpy_s(&newpacketsigner, intsz, buffer, intsz);
if (newpacketsigner == 1192) {
if (recvd >= buffsize) {
char* nbuf = new char[buffsize];
int recvd2 = recv(server_sock, nbuf, buffsize, NULL);
int err = WSAGetLastError();
memcpy_s(&newpacketsigner, intsz, nbuf, intsz);
memcpy_s(cdata, sz, (void*)(nbuf + intsz), sz);
//do things w/ the struct
delete[] nbuf;
}
}
else
recv(server_sock, buffer, 1024, NULL);
}
delete[] buffer;
}
Sleep(10);
}
As well, identical calls to setsockopt and are called for the client's sockets, and all of the sockets, server and client, are nonblocking.

Answer 1

You're assuming that your reads are filling the buffer. They are only obliged to transfer at least one byte. You you need to loop.
So, you have unread data backing up and stalling the sender.
NB Those receive buffers are greater than 64k and so may be inoperative unless they are set before the socket is connected. In the case of the server you need to set the receive buffer size on the listening socket. Accepted sockets will inherit it. If you don't to it his way, window scaling won't be in effect so a window > 64k cannot be advertised (unless the platform has window scaling on by default).