What's the difference between async_read and async_receive?
async_receive is a function that just receives into a buffer, but may not receive the amount you asked for. (It'll be equal or less, never more.)
async_read, however, will always receive the amount you asked for, as it states:
This function is used to asynchronously read a certain number of bytes of data from a stream. The function call always returns immediately. The asynchronous operation will continue until one of the following conditions is true:
The supplied buffers are full. That is, the bytes transferred is equal to the sum of the buffer sizes.
An error occurred.
The only thing the page is a bit vague on is what async_read does if it doesn't get that many bytes, and the connection closes gracefully. (Does that count as "error"?) This can probably be determined with a quick test. (async_receive, however, would just give you what it got.)
The first is a free function, the second is a member function.
Another difference is socket_base::message_flags flags parameter. See possible values, for example, in the recv(2) manual page.
Edit:
With async_receive you need to check how many bytes you got. Use it if you want to read at max N bytes, vs. exactly N bytes with async_read. Sorry, thought that was sort of obvious from boost docs.
Related
I have read all the MPI documentation and tutorials and Stack Overflow questions that I could find relevant to this, but I still do not fully understand how MPI_Wait behaves when "completing" an MPI_Isend. Can we summarize it succinctly? Does it
A. Return when the buffer used in the corresponding Isend is usable
again (I think yes, but this doesn't tell me everything I want to
know).
B. Return when the corresponding Recv completes (I think not
necessarily, but maybe sometimes if the message is large?)
C. Guarantee that the Recv can be completed by the receiving process at some later time after it returns
I ask because I am trying to implement a kind of non-blocking broadcast (since MPI_Ibcast is an MPI 3 thing and doesn't seem to exist in any of the implementations I actually encounter). The sequence I am currently following on each process is this:
MPI_Isend from every process to every other process
Do some other work
MPI_Wait for all the Isends to 'complete', whatever that means exactly
MPI_Recv all the messages
This seems to work fine in practice, but I don't know if it is guaranteed to work, or if I am just lucky because my messages are small (they are just one int each, so I suspect that they get rapidly shuffled off by MPI into some internal buffers or whatever). I don't know whether this would create deadlocks if the messages were bigger (I worry that in this case not all the MPI_Waits would return, because some might be deadlocked waiting for MPI_Recvs to happen on another process).
If this is not guaranteed to work in general, is it at least guaranteed to work for very small messages? Or is even this not necessarily true? I am really not sure what I can count on here.
If this is not guaranteed to work then how can a non-blocking broadcast be implemented? Perhaps there is some clever sequence of performing the Waits and Recvs that works out? Like first rank 0 Waits for rank 1 to Recv, then rank 1 Waits for rank 0 to Recv? Is some kind of switched pairing arrangement like that more correct?
Your conditions are met by the following:
MPI_Isend followed by MPI_Wait both complete:
A. The buffer used in the corresponding Isend is usable again.
If you were to use MPI_Issend and MPI_Wait:
almost B. Return when the corresponding Recv is posted.
The following is true right after MPI_Send:
C. Guarantee that a matching Recv can be completed by the receiving process at some later time after it returns.
In your suggested non-blocking broadcast, you would have to allow to swap 3. and 4., otherwise there is a deadlock. That means, there must not be a strict condition that 4. happens after 3. Since 3 happens on the root, and 4 happens on all other ranks, it may not even be an issue.
I would, however suggest to instead a cleaner approach:
MPI_Isend on the root to all processes
MPI_Irecv on **all88 processes from the root
Do some other work (on all processes)
MPI_Waitall for all posted requests (both send/recv) on all ranks
That should be clean to implement and work just fine. Of course that is not an optimized collective... but that is another topic.
The context for this: I'm working on implementing a simple stream adaptor on top of an RDMA wrapper library integrating with Boost ASIO.
The issue I'm running into is that the boost::asio::async_read aggregator being called by the client code is hanging even when the stream has returned enough data to fill the buffer, if there's no more pending data in the internal receive buffer. Debugging appears to show that it's calling my stream adaptor's async_read_some method with a single buffer of size 0.
The documentation I've found seems to be conflicting on whether that operation should complete immediately or not. On the one hand, the AsyncReadStream concept specification says:
If the total size of all buffers in the sequence mb is 0, the asynchronous read operation shall complete immediately and pass 0 as the argument to the handler that specifies the number of bytes read.
On the other hand, the overview of boost::asio::null_buffers says:
A null_buffers operation doesn't return until the I/O object is "ready" to perform the operation.
(And in fact, elsewhere I'm relying on that to register handlers to be called when the rdma_cm and ibverbs completion channel FDs indicate available events.) But looking at the implementation of null_buffers, it would appear that it's just a static object containing no buffers, so that would seem to satisfy the condition of a sequence with total size of all buffers being 0.
So, I'm confused as to how my async_read_some method should handle the case of trying to read 0 bytes. As a wild guess, maybe it should be something like: on a truly empty sequence like null_buffers it should complete only when there's data available in the receive buffer, while if it has a non-empty sequence with total length of the buffers equal to 0 then it should complete immediately regardless of the state of the receive buffer?
Note null_buffers has been deprecated since Boost 1.66.0: "(Deprecated: Use the socket/descriptor wait() and async_wait() member functions.)"
As a wild guess, maybe it should be something like: on a truly empty sequence like null_buffers it should complete only when there's data available in the receive buffer, while if it has a non-empty sequence with total length of the buffers equal to 0 then it should complete immediately regardless of the state of the receive buffer?
No. null_buffers are NOT "empty buffers" or "zero-length buffers". It's "no buffer", signalling to the ASIO routines that there are no buffers (so the size cannot be logically thought to be zero).
All of the completion notes pertaining to the buffer size are irrelevant, because no buffers are present. The relevant piece of documentation is Reactor Style Operations.
You said it correctly: passing null_buffers signals that you want a reactor-style operation to weave into the asio event subsystem (meaning, you can async wait for a socket to become ready for reading/writing, and then do the actual IO e.g. by passing the underlying socket handle to a third party API that doesn't support asynchronous IO natively.
I'm trying to make a simple tcp server according to this lesson. Everything works fine, and now I'm trying to encapsulate functions in the Socket class. I try to make a method that checks the amount of available bytes to read, and I can't find necessary the function. It could be some kind of ftell() or another method.
You should be respecting the communication protocol you are implementing on top of TCP. It dictates how you should be reading data and how you should be determining the sizes of the data to be read. If the protocol expects an integer, read an integer. If the protocol expects a length-preceded string, read the length value and then read how many bytes it specifies. If the protocol expects a delimited string, read until you encounter the delimiter. And so on.
That being said, to answer the actual question you asked - "[how to] check the amount of available bytes to read":
on Windows, you can use the ioctlsocket() function, where the cmd parameter is set to FIONREAD and the argp parameter is set to a pointer to a u_long variable.
u_long bytesAvailable;
ioctlsocket(socket, FIONREAD, &bytesAvailable);
on *Nix, you can use the ioctl() function, where the request parameter is set to FIONREAD and the third parameter is set to a pointer to a int variable.
int bytesAvailable;
ioctl(socket, FIONREAD, &bytesAvailable);
In either case, the output variable receives the number of unread bytes currently waiting on the socket. It is guaranteed that you can read at most this many bytes from the socket using recv() without blocking the calling thread (if the socket is running in a blocking mode). The socket may receive more bytes between the time you query FIONREAD and the time you call recv(), so if you try to read more than FIONREAD indicates, recv() may or may not have to block the calling thread waiting for more bytes.
Can you elaborate on why you would need to know that?
The preferred way to read data from a socket is to first make sure it has data you can read, via a call to select and then read into a predefined array, whose size is set to that of maximum size you and the client respect. Read will then report how many bytes it has read.
I might also add you should also select a socket before writing data to it.
Finally, you might want to look up circular buffers as they offer a way to append data to a buffer without risking to fill up your RAM. You would then read the data from the circular buffer once you have enough data.
As far as I know, it is not recommended to check how many bytes are available to be read on a socket.
I am reading data ~100 bytes at 100hz from a serial port. My buffer is 1024 bytes, so often my buffer doesn't get completely used. Sometimes however, I get hiccups from the serial port and the buffer gets filled up.
My data is organized as a [header]data[checksum]. When my buffer gets filled up, sometimes a message/data is split across two reads from the serial port.
This is a simple problem, and I'm sure there are a lot of different approaches. I am ahead of schedule so I would like to research different approaches. Could you guys name some paradigms that cover buffering in high speed data that might need to be put together from two reads? Note, the main difference I see in this problem from say other buffering I've done (image acquisition, tcp/ip), is that there we are guaranteed full packets/messages. Here a "packet" may be split between reads, which we will only know once we start parsing the data.
Oh yes, note that the data buffered in from the read has to be parsed, so to make things simple, the data should be contiguous when it reaches the parsing. (Plus I don't think that's the parser's responsibility)
Some Ideas I Had:
Carry over unused bytes to my original buffer, then fill it with the read after the left over bytes from the previous read. (For example, we read 1024 bytes, 24 bytes are left at the end, they're a partial message, memcpy to the beginning of the read_buffer_, pass the beginning + 24 to read and read in 1024 - 24)
Create my own class that just gets blocks of data. It has two pointers, read/write and a large chunk of memory (1024 * 4). When you pass in the data, the class updates the write pointer correctly, wraps around to the beginning of its buffer when it reaches the end. I guess like a ring buffer?
I was thinking maybe using a std::vector<unsigned char>. Dynamic memory allocation, guaranteed to be contiguous.
Thanks for the info guys!
Define some 'APU' application-protocol-unit class that will represent your '[header]data[checksum]'. Give it some 'add' function that takes a char parameter and returns a 'valid' bool. In your serial read thread, create an APU and read some data into your 1024-byte buffer. Iterate the data in the buffer, pushing it into the APU add() until either the APU add() function returns true or the iteration is complete. If the add() returns true, you have a complete APU - queue it off for handling, create another one and start add()-ing the remaining buffer bytes to it. If the iteration is complete, loop back round to read more serial data.
The add() method would use a state-machine, or other mechanism, to build up and check the incoming bytes, returning 'true' only in the case of a full sanity-checked set of data with the correct checksum. If some part of the checking fails, the APU is 'reset' and waits to detect a valid header.
The APU could maybe parse the data itself, either byte-by-byte during the add() data input, just before add() returns with 'true', or perhaps as a separate 'parse()' method called later, perhaps by some other APU-processing thread.
When reading from a serial port at speed, you typically need some kind of handshaking mechanism to control the flow of data. This can be hardware (e.g. RTS/CTS), software (Xon/Xoff), or controlled by a higher level protocol. If you're reading a large amount of data at speed without handshaking, your UART or serial controller needs to be able to read and buffer all the available data at that speed to ensure no data loss. On 16550 compatible UARTs that you see on Windows PCs, this buffer is just 14 bytes, hence the need for handshaking or a real time OS.
I am using berkeley sockets and TCP (SOCK_STREAM sockets).
The process is:
I connect to a remote address.
I send a message to it.
I receive a message from it.
Imagine I am using the following buffer:
char recv_buffer[3000];
recv(socket, recv_buffer, 3000, 0);
Questions are:
How can I know if after calling recv first time the read buffer is empty or not? If it's not empty I would have to call recv again, but if I do that when it's empty I would have it blocking for much time.
How can I know how many bytes I have readed into recv_buffer? I can't use strlen because the message I receive can contain null bytes.
Thanks.
How can I know if after calling recv
first time the read buffer is empty or
not? If it's not empty I would have to
call recv again, but if I do that when
it's empty I would have it blocking
for much time.
You can use the select or poll system calls along with your socket descriptor to tell if there is data waiting to be read from the socket.
However, usually there should be an agreed-upon protocol that both sender and receiver follow, so that both parties know how much data is to be transferred. For example, perhaps the sender first sends a 2-byte integer indicating the number of bytes it will send. The receiver then first reads this 2-byte integer, so that it knows how many more bytes to read from the socket.
Regardless, as Tony pointed out below, a robust application should use a combination of length-information in the header, combined with polling the socket for additional data before each call to recv, (or using a non-blocking socket). This will prevent your application from blocking in the event that, for example, you know (from the header) that there should still be 100 bytes remaining to read, but the peer fails to send the data for whatever reason (perhaps the peer computer was unexpectedly shut off), thus causing your recv call to block.
How can I know how many bytes I have
readed into recv_buffer? I can't use
strlen because the message I receive
can contain null bytes.
The recv system call will return the number of bytes read, or -1 if an error occurred.
From the man page for recv(2):
[recv] returns the number of bytes
received, or -1 if an error occurred.
The return value will be 0 when the
peer has performed an orderly
shutdown.
How can I know if after calling recv first time the read buffer is empty or not?
Even the first time (after accepting a client), the recv can block and fail if the client connection has been lost. You must either:
use select or poll (BSD sockets) or some OS-specific equivalent, which can tell you whether there is data available on specific socket descriptors (as well as exception conditions, and buffer space you can write more output to)
you can set the socket to be nonblocking, such that recv will only return whatever is immediately available (possibly nothing)
you can create a thread that you can afford to have block recv-ing data, knowing other threads will be doing the other work you're concerned to continue with
How can I know how many bytes I have readed into recv_buffer? I can't use strlen because the message I receive can contain null bytes.
recv() returns the number of bytes read, or -1 on error.
Note that TCP is a byte stream protocol, which means that you're only guaranteed to be able to read and write bytes from it in the correct order, but the message boundaries are not guaranteed to be preserved. So, even if the sender has made a large single write to their socket, it can be fragmented en route and arrive in several smaller blocks, or several smaller send()/write()s can be consolidated and retrieved by one recv()/read().
For that reason, make sure you loop calling recv until you either get all the data you need (i.e. a complete logical message you can process) or an error. You should be prepared/able to handle getting part/all of subsequent sends from your client (if you don't have a protocol where each side only sends after getting a complete message from the other, and are not using headers with message lengths). Note that doing recvs for the message header (with length) then the body can result in a lot more calls to recv(), with a potential adverse affect on performance.
These reliability issues are often ignored. They manifest less often when on a single host, a reliable and fast LAN, with less routers and switches involved, and fewer or non-concurrent messages. Then they may break under load and over more complex networks.
If the recv() returns fewer than 3000 bytes, then you can assume that the read buffer was empty. If it returns 3000 bytes in your 3000 byte buffer, then you'd better know whether to continue. Most protocols include some variation on TLV - type, length, value. Each message contains an indicator of the type of message, some length (possibly implied by the type if the length is fixed), and the value. If, on reading through the data you did receive, you find that the last unit is incomplete, you can assume there is more to be read. You can also make the socket into a non-blocking socket; then the recv() will fail with EAGAIN or EWOULDBLOCK if there is no data read for reading.
The recv() function returns the number of bytes read.
ioctl() with the FIONREAD option tells you how much data can currently be read without blocking.