I need to develop a TCP server capable of receiving periodic urgent out-of-bounds data. I am using the POCO C++ libraries to achieve this. In the (scarce) documentation of the StreamSocket class, I see that this should be a very easy task to achieve: it should be done by setting the SO_OOBINLINE flag when using the receiveBytes() method like so:
n = ss.receiveBytes( buffer, sizeof(buffer), SO_OOBINLINE );
To test this, I made a very simple TCP client (also using the POCO libraries) which uses the "sendUrgent()" method to send the OOB data (a single byte):
ss1.sendUrgent( 0xFF );
When I send the OOB data, nothing happens. The server doesn't seem to notice it.
On the other hand, when I send "normal" data, using the sendBytes() method, if the SO_OOBINLINE flag is set on the StreamSocket, I receive an infinite amount of the same sent data (it keeps reading the same data although nobody is resending it).
What am I missing?
Edit: After #JimR's suggestion and reading this, I have tried:
n = ss.receiveBytes( buffer, sizeof(buffer), MSG_OOB );
And even this:
n = ss.receiveBytes( buffer, sizeof(buffer), SO_OOBINLINE | MSG_OOB );
Both raise the same exception: Invalid argument.
I think (it's been a long time) you should change
n = ss.receiveBytes( buffer, sizeof(buffer), SO_OOBINLINE ); to
n = ss.receiveBytes( buffer, sizeof(buffer), MSG_OOB );
Edit: Along with the above changes, you will need to call setsockopt with the SO_OOBINLINE flag and the appropriate arguments so OOB data will be seen in the normal stream and not in a separate channel. In your case, as stated in the comments, call ss.setOOBInline(true);.
SO_OOBINLINE is a socket option used with setsockopt.
MSG_OOB is a flag for recv and send and company.
Related
This is the quickFix code in C++:
void SocketConnection::readFromSocket() EXCEPT ( SocketRecvFailed ) calls socket_recv()
ssize_t socket_recv(socket_handle s, char* buf, size_t length ) calls recv()
Question:
How to make sure that the FIX messages are always < BUFSIZ (for example 8192 bytes)? A message with many groups could be quite big.
Here I am not sure (but would like to learn) that recv can always read a complete message?
The QuickFix method SocketConnection::readFromSocket() does not just read just bytes from a low level operating system socket. It inspects the byte stream, buffering partial messages until the end-of-message indicator is received.
The 'readFromSocket' name is perhaps a little misleading from the low level point-of-view.
If you look at one example of the implementation from the QuickFix source of the method:
FIX::SocketConnection::readFromSocket() throw (SocketRecvFailed)
{
int size = recv( m_socket, m_buffer, sizeof(m_buffer), 0 );
if( size <= 0 ) throw SocketRecvFailed( size );
m_parser.addToStream( m_buffer, size );
}
You can see where it is injecting the bytes in reads from the socket as variable m_buffer into an object called m_parser which will in turn notify when a whole message has been received.
This is a very common pattern while serializing information.
QuickFix goes beyond ensuring a complete message, it can validate message contents and negotiate for re-transmissions of corrupted data.
I'm a bit confused by expected proper usage pattern of APIs such as SSL_connect(), SSL_write(), etc. I've read up on some other posts on SO and elsewhere, and those that I found are all centered around blocking or non-blocking sockets (i.e. where BIO is given a socket to use for underlying IO), and the return errors SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE from calls in such configurations are pretty clear how to handle.
However, I'm a bit puzzled as to what the proper handling would be when BIO is set up without underlying IO socket, and instead all IO is handled via memory buffers. (The reason for such a setup is because encrypted data stream is not immediately sent over a vanila socket, but rather may be enveloped over other protocols or delivery mechanisms, and cannot be written to some socket directly). E.g. the BIO is set up as
auto readBio = BIO_new(BIO_s_mem());
auto writeBio = BIO_new(BIO_s_mem());
auto ssl = SSL_new(...);
SSL_set_bio(ssl, readBio, writeBio);
My assumption - albeit it appears to be incorrect - that after making a call to say SSL_connect(), it would tell me when it's time to pick up its output from write buffer using BIO_read() call and deliver that buffer (by whatever custom underlying transport means) to the other end peer; and likewise when to feed it data from peer. In other words, something akin to:
while (true) {
auto ret = SSL_connect(ssl); // or SSL_read(), SSL_write(), SSL_shutdown() in other contexts...
if (ret <= 0) {
auto err = SSL_get_error(ssl, ret);
switch(err) {
case SSL_ERROR_WANT_READ:
auto buf = magicallyReadDataFromPeer();
BIO_write(buf, ...);
continue;
case SSL_ERROR_WANT_WRITE:
Buffer buf;
BIO_read(buf, ...);
magicallySendDataToPeer();
continue;
}
} else break;
}
But I'm noticing that the first call to SSL_connect() always results in SSL_EROR_WANT_READ with nothing sent to peer to actually initiate TLS handshake, and so it blocks indefinitely.
If after calling SSL_connect() I do flush the buffer by doing BIO_read() and sending it out, then things seem to proceed. Same seems for SSL_write() calls, but then it seems that if I always flush buffer after call, and then check for SSL_ERROR_WANT_WRITE, I'd be flushing the buffer twice (with second one probably being a no-op) and that seems nonsensical. It also seems strange that I should just always ignore SSL_ERROR_WANT_WRITE of every SSL_connect/accept/write/read/shutdown calls since I'd be flushing always after each call.
And so I'm puzzled about what's the proper and expected dance between SSL_connect/etc and BIO_read/write calls and their tying relationship of SSL_ERROR_WANT_* values, specifically when using mem buffer instead of socket or file descriptor for underlying IO.
I have been reading some socket guides such as Beej's guide to network programming. It is quite clear now that there is no guarantee on how many bytes are received in a single recv() call. Therefore a mechanism of e.g. first two bytes stating the message length should be sent and then the message. So the receiver receives the first two bytes and then receives in a loop until the whole message has been received. All good and dandy!?
I was asked by a colleague about messages going out of sync. E.g. what if, somehow, I receive two bytes in once recv() call that are actually in the middle of the message itself and it would appear as a integer of some value? Does that mean that the rest of the data sent will be out of sync? And what about receiving the header partially, i.e. one byte at a time?
Maybe this is overthinking, but I can't find this mentioned anywhere and I just want to be sure that I would handle this if it could be a possible threat to the integrity of the communication.
Thanks.
It is not overthinking. TCP presents a stream so you should treat it this way. A lot of problems concerning TCP are due to network issues and will probably not happen during development.
Start a message with a (4 byte) magic that you can look for followed by a (4 byte) length in an expected order (normally big endian). When receiving, read each byte of the header at the time, so you can handle it anyway the bytes were received. Based on that you can accept messages in a lasting TCP connection.
Mind you that when starting a new connection per message, you know the starting point. However, it doesn't hurt sending a magic either, if only to filter out some invalid messages.
A checksum is not necessary because TCP shows a reliable stream of bytes which was already checked by the receiving part of TCP, and syncing will only be needed if there was a coding issue with sending/receiving.
On the other hand, UDP sends packets, so you know what to expect, but then the delivery and order is not guaranteed.
Your colleague is mistaken. TCP data cannot arrive out of order. However you should investigate the MSG_WAITALL flag to recv() to overcome the possibility of the two length bytes arriving separately, and to eliminate the need for a loop when receiving the message body.
Its your responsibility to make you client and server syncing together, how ever in TCP there is no out of order delivery, if you got something by calling recv() you can think there isn't anything behind that that you doesn't received.
So the question is how to synchronize sender and receiver ? its easy, as stefaanv said, sender and receiver are knowing their starting point. so you can define a protocol for your network communication. for example a protocol could be defined this way :
4 bytes of header including message type and payload length
Rest of message is payload length
By this, you have to send 4 byte header before sending actual payload, then sending actual payload followed.
Because TCP has garauntied Inorder reliable delivery, you can make two recv() call for each pack. one recv() call with length of 4 bytes for getting next payload size, and another call to recv() with size specified in header. Its necessary to make both recv() blocking to getting synchronized all the time.
An example would be like this:
#define MAX_BUF_SIZE 1024 // something you know
char buf[MAX_BUF_SIZE];
int recvLen = recv(fd, buff, 4, MSG_PEEK);
if(recvLen==4){
recvLen = recv(fd, buff, 4);
if(recvLen != 4){
// fatal error
}
int payloadLen = extractPayloadLenFromHeader(buf);
recvLen = recv(fd, buff, payloadLen, MSG_PEEK);
if(recvLen == payloadLen){
recvLen = recv(fd, buff, payloadLen); // actual recv
if(recvLen != payloadLen){
// fatal error
}
// do something with received payload
}
}
As you can see, i have first called recv with MSG_PEEK flag to ensure is there really 4 bytes available or not, then received actual header. same for payload
Based on my understanding, each socket is associated with two buffers, a send buffer and a receive buffer, so when I call the send() function, what happens is that the data to send will be placed into the send buffer, and it is the responsibility of Windows now to send the content of this send buffer to the other end.
In a blocking socket, the send() function does not return until the entire data supplied to it has been placed into the send buffer.
So what is the size of the send buffer?
I performed the following test (sending 1 GB worth of data):
#include <stdio.h>
#include <WinSock2.h>
#pragma comment(lib, "ws2_32.lib")
#include <Windows.h>
int main()
{
// Initialize Winsock
WSADATA wsa;
WSAStartup(MAKEWORD(2, 2), &wsa);
// Create socket
SOCKET s = socket(AF_INET, SOCK_STREAM, 0);
//----------------------
// Connect to 192.168.1.7:12345
sockaddr_in address;
address.sin_family = AF_INET;
address.sin_addr.s_addr = inet_addr("192.168.1.7");
address.sin_port = htons(12345);
connect(s, (sockaddr*)&address, sizeof(address));
//----------------------
// Create 1 GB buffer ("AAAAAA...A")
char *buffer = new char[1073741824];
memset(buffer, 0x41, 1073741824);
// Send buffer
int i = send(s, buffer, 1073741824, 0);
printf("send() has returned\nReturn value: %d\nWSAGetLastError(): %d\n", i, WSAGetLastError());
//----------------------
getchar();
return 0;
}
Output:
send() has returned
Return value: 1073741824
WSAGetLastError(): 0
send() has returned immediately, does this means that the send buffer has a size of at least 1 GB?
This is some information about the test:
I am using a TCP blocking socket.
I have connected to a LAN machine.
Client Windows version: Windows 7 Ultimate 64-bit.
Server Windows version: Windows XP SP2 32-bit (installed on Virtual Box).
Edit: I have also attempted to connect to Google (173.194.116.18:80) and I got the same results.
Edit 2: I have discovered something strange, setting the send buffer to a value between 64 KB and 130 KB will make send() work as expected!
int send_buffer = 64 * 1024; // 64 KB
int send_buffer_sizeof = sizeof(int);
setsockopt(s, SOL_SOCKET, SO_SNDBUF, (char*)send_buffer, send_buffer_sizeof);
Edit 3: It turned out (thanks to Harry Johnston) that I have used setsockopt() in an incorrect way, this is how it is used:
setsockopt(s, SOL_SOCKET, SO_SNDBUF, (char*)&send_buffer, send_buffer_sizeof);
Setting the send buffer to a value between 64 KB and 130 KB does not make send() work as expected, but rather setting the send buffer to 0 makes it block (this is what I noticed anyway, I don't have any documentation for this behavior).
So my question now is: where can I find a documentation on how send() (and maybe other socket operations) work under Windows?
After investigating on this subject. This is what I believe to be the correct answer:
When calling send(), there are two things that could happen:
If there are pending data which are below SO_SNDBUF, then send() would return immediately (and it does not matter whether you are sending 5 KB or you are sending 500 MB).
If there are pending data which are above or equal SO_SNDBUF, then send() would block until enough data has been sent to restore the pending data to below SO_SNDBUF.
Note that this behavior is only applicable to Windows sockets, and not to POSIX sockets. I think that POSIX sockets only use one fixed sized send buffer (correct me if I'm wrong).
Now back to your main question "What is the size of a socket send buffer in Windows?". I guess if you have enough memory it could grow beyond 1 GB if necessary (not sure what is the maximum limit though).
I can reproduce this behaviour, and using Resource Monitor it is easy to see that Windows does indeed allocate 1GB of buffer space when the send() occurs.
An interesting feature is that if you do a second send immediately after the first one, that call does not return until both sends have completed. The buffer space from the first send is released once that send has completed, but the second send() continues to block until all the data has been transferred.
I suspect the difference in behaviour is because the second call to send() was already blocking when the first send completed. The third call to send() returns immediately (and 1GB of buffer space is allocated) just as the first one did, and so on, alternating.
So I conclude that the answer to the question ("how large are the send buffers?") is "as large as Windows sees fit". The upshot is that, in order to avoid exhausting the system memory, you should probably restrict blocking sends to no more than a few hundred megabytes.
Your call to setsockopt() is incorrect; the fourth argument is supposed to be a pointer to an integer, not an integer converted to a pointer. Once this is corrected, it turns out that setting the buffer size to zero causes send() to always block.
To summarize, the observed behaviour is that send() will return immediately provided:
there is enough memory to buffer all the provided data
there is not a send already in progress
the buffer size is not set to zero
Otherwise, it will return once the data has been sent.
KB214397 describes some of this - thanks Hans! In particular it describes that setting the buffer size to zero disables Winsock buffering, and comments that "If necessary, Winsock can buffer significantly more than the SO_SNDBUF buffer size."
(The completion notification described does not quite match up to the observed behaviour, depending I guess on how you interpret "previously buffered send". But it's close.)
Note that apart from the risk of inadvertently exhausting the system memory, none of this should matter. If you really need to know whether the code at the other end has received all your data yet, the only reliable way to do that is to get it to tell you.
In a blocking socket, the send() function does not return until the entire data supplied to it has been placed into the send buffer.
That is not guaranteed. If there is available buffer space, but not enough space for the entire data, the socket can (and usually will) accept whatever data it can and ignore the rest. The return value of send() tells you how many bytes were actually accepted. You have to call send() again to send the remaining data.
So what is the size of the send buffer?
Use getsockopt() with the SO_SNDBUF option to find out.
Use setsockopt() with the SO_SNDBUF option to specify your own buffer size. However, the socket may impose a max cap on the value you specify. Use getsockopt() to find out what size was actually assigned.
I am doing a simple TCP server in C++ for Windows to echo incoming data. I have a problem with it. Before I explain my problem, I have to say that Winsock is properly set up and this problem happens with any IP address as the source.
The general behaviour when a connection is established is this:
In the loop that runs while connection still alive, it must echo data, and precede it with REPLY word.
To do that, I'm currently using two send() calls:
One call sending "REPLY " alone.
Another call just sending back received data.
But using Putty Client, I'm getting this:
REPLY data_echoed REPLY.
Why REPLY is sent after the last send call if it was the first??? I'll post a little code where the problem happens:
//Reply to client
message = "HELLO CLIENT!! READY TO ECHO.\n";
send(new_socket, message, strlen(message), 0);
///Enter into a loop until connection is finished.
printf("%s \n\n", "Incoming data goes here: ");
do{
///Clear buffer and receive data.
memset(buffer, 0, sizeof(buffer));
ret = recv(new_socket, buffer, sizeof(buffer), 0);
printf("%s", buffer);
///Send a REPLY WORD and the data echoed.
send(new_socket, "REPLY\r\n", 7, 0);
send(new_socket, buffer, sizeof(buffer), 0);
}while(ret != SOCKET_ERROR);
What is wrong with that? I'll remove the first call, and the double effect disappears. Why I can't do two send calls one after other?
You ignore the return value of recv until after you send REPLY, so no matter what happens, you send REPLY followed by the contents of the buffer. After you echo the first time, something happens, and no matter what it is, you send REPLY.
Bluntly, it doesn't look like you understand the very basics of TCP. We used to have a rule that before anyone can do anything with TCP, they must memorize and completely understand the following statement: "TCP is a byte-stream protocol that does not preserve application message boundaries."
Your code pretends that it is receiving and echoing application-level messages. But there is no code to actually implement application-level messages. TCP has no support for application-level messages, so if you want them, you have to implement them. You clearly want them. You also have not implemented them.
Do newline characters delimit messages? If so, where's the code to find them in the data stream? If not, what does?