I am reading data ~100 bytes at 100hz from a serial port. My buffer is 1024 bytes, so often my buffer doesn't get completely used. Sometimes however, I get hiccups from the serial port and the buffer gets filled up.
My data is organized as a [header]data[checksum]. When my buffer gets filled up, sometimes a message/data is split across two reads from the serial port.
This is a simple problem, and I'm sure there are a lot of different approaches. I am ahead of schedule so I would like to research different approaches. Could you guys name some paradigms that cover buffering in high speed data that might need to be put together from two reads? Note, the main difference I see in this problem from say other buffering I've done (image acquisition, tcp/ip), is that there we are guaranteed full packets/messages. Here a "packet" may be split between reads, which we will only know once we start parsing the data.
Oh yes, note that the data buffered in from the read has to be parsed, so to make things simple, the data should be contiguous when it reaches the parsing. (Plus I don't think that's the parser's responsibility)
Some Ideas I Had:
Carry over unused bytes to my original buffer, then fill it with the read after the left over bytes from the previous read. (For example, we read 1024 bytes, 24 bytes are left at the end, they're a partial message, memcpy to the beginning of the read_buffer_, pass the beginning + 24 to read and read in 1024 - 24)
Create my own class that just gets blocks of data. It has two pointers, read/write and a large chunk of memory (1024 * 4). When you pass in the data, the class updates the write pointer correctly, wraps around to the beginning of its buffer when it reaches the end. I guess like a ring buffer?
I was thinking maybe using a std::vector<unsigned char>. Dynamic memory allocation, guaranteed to be contiguous.
Thanks for the info guys!
Define some 'APU' application-protocol-unit class that will represent your '[header]data[checksum]'. Give it some 'add' function that takes a char parameter and returns a 'valid' bool. In your serial read thread, create an APU and read some data into your 1024-byte buffer. Iterate the data in the buffer, pushing it into the APU add() until either the APU add() function returns true or the iteration is complete. If the add() returns true, you have a complete APU - queue it off for handling, create another one and start add()-ing the remaining buffer bytes to it. If the iteration is complete, loop back round to read more serial data.
The add() method would use a state-machine, or other mechanism, to build up and check the incoming bytes, returning 'true' only in the case of a full sanity-checked set of data with the correct checksum. If some part of the checking fails, the APU is 'reset' and waits to detect a valid header.
The APU could maybe parse the data itself, either byte-by-byte during the add() data input, just before add() returns with 'true', or perhaps as a separate 'parse()' method called later, perhaps by some other APU-processing thread.
When reading from a serial port at speed, you typically need some kind of handshaking mechanism to control the flow of data. This can be hardware (e.g. RTS/CTS), software (Xon/Xoff), or controlled by a higher level protocol. If you're reading a large amount of data at speed without handshaking, your UART or serial controller needs to be able to read and buffer all the available data at that speed to ensure no data loss. On 16550 compatible UARTs that you see on Windows PCs, this buffer is just 14 bytes, hence the need for handshaking or a real time OS.
Related
I am trying to find a way to identify the start of a chunk of data sent via a TCP socket. The data chunk has the value of the integer 1192 written into it as the first four bytes, followed by the content length. How can I search the binary data (the char* received) for this value? I realize I can loop through and advance the pointer by one each time, copy out the first four bytes, and check it, but that isn't the most elegant or possibly efficient solution.
Is there also another way this could be done that I'm not thinking of?
Thanks in advance.
It sounds like linear scanning might be required, but you shouldn't really be losing your message positioning if the sending side of the connection is making its send()/write() calls in a sensible manner, you are reading in your buffers properly, and there isn't an indeterminate amount of "dead" space in the stream between messages.
If the protocol itself is sensible (there is at least a length field!), you should never lose track of message boundaries. Just read the marker/length pair, then read length payload bytes, and the next message should start immediately after this, so a linear scan shouldn't have to go anywhere ideally.
Also, don't bother copying explicitly, just cast:
// call htonl() to flip endianness if need be...
uint32_t x = *reinterpret_cast<uint32_t *>(charptr);
I'm writing a C++ program that simply receives data from another computer and writes the data into an SSD RAID with high throughput (about 100MB/s since GbEthernet).
I have set up 2 overlapped_io each, which are received from Ethernet and written to SSD.
When the receiving is done done, it'll post a message to the writer.
And I use FILE_NO_BUFFERING_FLAG when creating the file on disk.
On the side of network sender, I am using an overlapped IO to send data.
I got stuck in the problem: when received from the socket, the rv = recv() is not aligned with the disk (maybe 4096 times?).
What should I do?
recv and unbuffered writes are not really very compatible with each other. It is possible to get that working, but it will take a little extra work.
When doing unbuffered writes, both the start address of your buffer and the amount to write must be multiples of the sector size (see MSDN). Aligning the buffer is trivial, but dealing with the fact that recv can return pretty much every amount of data (up to the amount you ask for, but in theory it could be just 1 byte) is a bit of work.
Another problem is that while it is pretty much guaranteed that the sector size is a power of two (though at least there used to exist harddisks with non-power-of-two sectors in the 1990s, this fact was hidden by the controller) you do not know what it is. And even if you did know, it might be different on the next computer. It might be 512 or 1024 or something else.
How to handle this? Most programmers resort to simply using a function that allocates complete memory pages, such as VirtualAlloc, or an anonymous memory mapping. Since these operate on pages, they are necessarily page-size aligned, which (usually) means 4096 bytes1.
Since the amount of data to write must, too, be a multiple of the sector size (but the amount of data received probably isn't), you have round down, do a partial write, and save the rest for the next write.
Again, the problem is that you don't know the sector size, so the best thing you can do is round down to the same granularity that you're using for the buffer start (anything else would be nonsensical). In other words, you conceptually have to do something like this:
while(rv < 0xffff) // don't have enough yet
receive_more_and_append();
num_write = rv & ~0xffff;
rv -= num_write;
memcpy(other_buf, buf+num_write, rv);
WriteFileEx(...);
1That is only half the truth, since Windows has a minimum allocation granularity of 64kB. You can't allocate something smaller than 64k and it can't be aligned less than 64k. So in fact, you are good for sectors up to 64k, which is bigger than anything you are likely to ever encounter, realistically.
Also, as a small nitpick, Itanium has 8k pages, not 4k -- but that is no problem, it's actually better.
If I keep sending data to a receiver is it possible for the data sent to overlap such that they accumulate in the buffer and so the next read to the buffer reads also the data of another sent data?
I'm using Qt and readAll() to receive data and parse it. This data has some structure in it so I can know if the data is already complete or if it is valid data at all but I'm worried that other data will overlap with others when I call readAll() and so would invalidate this suppose-to-be valid data.
If it can happen, how do I prevent/control it? Or is that something the OS/API worries about instead? I'm worried partly because of how the method is called. lol
TCP is a stream based connection, not a packet based connection, so you may not assume that what is sent in one time will also be received in one time. You still need some kind of protocol to packetize your stream.
For sending strings, you could use the nul-character as separator or you could begin with a header which contains a magic and a length.
According to http://qt-project.org/doc/qt-4.8/qiodevice.html#readAll this function snarfs all the data and returns it as an array. I don't see how the API raises concerns about overlapping data. The array is returned by value, and given that it represents the entire stream, so what would it even overlap with? Are you worried that the returned object actually has reference semantics (i.e. that it just holds pointers to storage that is re-used in other calls to the same function?)
If send and receive buffers overlap in any system, that's a bug, unless special care is taken that the use is completely serialized. (I.e. a buffer is somehow used only for sending and only for receiving, without any mixup.)
Why dont you use a fixed length header followed by variable length packet with the header holding the information of length of packet.
This way you can avoid worrying about packet boundaries. Say for example instead of just sending the string send the length of the string followed by the string. In the receiver end always read the length and then based on the length read the string.
I'm really new at network-programming, so I hope this isn't a complete Newbie-question.
I read a tutorial at the Qt-Homepage how to build a little server, and I found this:
QByteArray block;
QDataStream out(&block, QIODevice::WriteOnly);
out << (quint16)0;
out << "..."; // just some text
out.device()->seek(0);
out << (quint16)(block.size() - sizeof(quint16));
At the start of our QByteArray, we reserve space for a 16 bit integer that will contain the total size of the data block we are sending. [We continue by streaming in a random fortune.] Then we seek back to the beginning of the QByteArray, and overwrite the reserved 16 bit integer value with the total size of the array. By doing this, we provide a way for clients to verify how much data they can expect before reading the whole packet.
So I want to know, what are the advantages of this procedure? What can happen if you don't do that? Maybe you also could add a little example.
It is standard stuff.
To the receiving program everything coming over the network is just a stream of bytes. The stream has no meaning beyond what the application imposes upon it, exactly the same way a file has no meaning beyond how its records, lines, etc., are defined by the application(s).
The only way the client and server can make sense of the stream is to establish a convention, or protocol, that they agree upon.
So some common ways to accomplish this are by:
have a delimiter that designates the end of a message (e.g. a carriage return)
pass a length field, as in your example, which tells the receiver how much data comprises the next message.
just establish a fixed convention (e.g. every message will be 20 bytes or type 'A' records will be one defined format, type 'B' records another...)
just treat it like a stream by having no convention at all (e.g. take whatever comes over the network and put it in a file w/o paying any attention to what it is)
One advantage of the length byte method is that the receiver knows exactly how much data to expect. With some added sanity checks this can help eliminate things like buffer overflows and such in your application.
Knowing the packet size before receiving it has a performance advantage.
You can then allocated exactly the needed number of bytes from the heap or whatever buffer management you use and receive all by few (ideally one) calls to the 'network receive function'.
If you don't know the size in advantage, you have to call the 'network receive function' for very small portion of the message.
Since the 'network receive function' (which may be recv() or whatever Qt offers to you) is a system call which also does TCP buffer handling and so on it should be assumed as being slow with a large per-call overhead. So you should call it as few as possible.
I'm trying to find concrete examples of how to manage breaking an incoming stream of data on a TCP/IP socket and aggregating this data in a buffer of some sort so that I could find the messages in it (variable length with header + delimiters) and extract them to reconstruct the messages for the receiving application.
Any good pointers/links/examples on an efficient way of doing this would be appreciated as I couldn't find good examples online and I'm sure this problem has been addressed by others in an efficient way in the past.
Efficient memory allocation of aggregation buffer
Quickly finding the message boundaries of a message to extract it from the buffer
Thanks
David
I've found that the simple method works pretty well.
Allocate a buffer of fixed size double the size of your biggest message. One buffer. Keep a pointer to the end of the data in the buffer.
Allocation happens once. The next part is the message loop:
If not using blocking sockets, then poll or select here.
Read data into the buffer at the end-data pointer. Only read what will fit into the buffer.
Scan the new data for your delimiters with strchr. If you found a message:
memcpy the message into its own buffer. (Note: I do this because I was using threading and you probably should too.)
memmove the remaining buffer data to the beginning of the buffer and update the end of data pointer.
Call the processing function for the message. (Send it to the thread pool.)
There are more complicated methods. I haven't found them worth the bother in the end but you might depending on circumstances.
You could use a circular buffer with beginning and end of data pointers. Lots of hassle keeping track and computing remaining space, etc.
You could allocate a new buffer after finding each message. You wouldn't have to copy so much data around. You do still have to move the excess data into a new message buffer after finding the delimiter.
Do not think that dumb tricks like reading one byte at a time out of the socket will improve performance. Every system call round-trip makes an 8 kB memmove look cheap.