Easiest way to get a line when data is coming in chunks? - c++

Question

I am using win32's ReadFile to read from a child process's pipe. That gives me chunks of characters at a time and the size of each chunk, but they may or may not have new lines. I want to process the output line by line. What is the simplest way of doing so?
I thought about appending each chunk to a string, and at the end, using a stringstream to process it line by line, but I'd like to do this as data is coming in. I suppose the trick is, how and where should I detect the new line ending? If only streamreader's getline returned nothing when no delimiter is found...

Answer 1

Append to a string until you encounter newline char or end of data. Then you have a line in the string, process it, empty string and repeat. Reason for using the string: that you don't know how long a line may be, and a string does the reallocation etc. for you as necessary.
Special case: end of data with nothing earlier on that line should probably be not-a-line, ignored.
Cheers & hth.

Answer 2

What I can think of is using a buffer where to store chunks coming to you, you know the size via lpNumberOfBytesRead, so on each chunk you check if it contains new line character/s and if it does, you output all the buffered characters till the new line character, then start buffering until you received another chunk with new line characters and so on.
Some pseudo code might look like:
w_char buffer[BUFFER_SIZE]; // enough for few chunks (use new to allocate dynamic memory)
ReadLine(hFile, lpBuffer, nNumberOfBytesToRead, lpNumberOfBytesRead, lpOverlapped);
if (w_char has new_line_character) {
process_whole_line(); // don't forget to clear up the buffer
}

Answer 3

You can extend std::basic_streambuf and implement xsputn so it stores data in an internal buffer, and detect newlines or do whatever processing you need. If you want to process only complete lines, upon detection you can push the buffer to a std::queue up to the newline, and erase the respective part from the buffer. You'll need to implement overflow too. For example:
template<typename CharT, typename Traits = std::char_traits<CharT> >
class line_streambuf : public std::basic_streambuf<CharT, Traits> {
public:
typedef CharT char_type;
typedef Traits traits_type;
typedef std::basic_string<char_type> string_type;
typedef typename string_type::size_type size_type;
typedef typename traits_type::int_type int_type;
line_streambuf(char_type separator = '\n') : _separator(separator) {}
virtual ~line_streambuf() {}
bool getline(string_type& output) { /* pop from the queue and return */ }
protected:
virtual int_type overflow(int_type v) {
if (v == _separator) {
_processed.push(_buffer);
_buffer.erase(_buffer.begin(), _buffer.end());
} else {
_buffer += v;
}
return v;
}
virtual std::streamsize xsputn(const char_type* p, std::streamsize n) {
_buffer.append(p, p + n);
while (true) {
// The 1st find could be smarter - finding only after old_length+p
size_type pos = _buffer.find(_separator);
if (pos == string_type::npos)
break;
_processed.push(string_type(_buffer.begin(), _buffer.begin()+pos));
_buffer.erase(_buffer.begin(), _buffer.begin() + pos + 1);
}
return n;
}
private:
char_type _separator;
string_type _buffer;
std::queue<string_type> _processed;
};
NOTE: highly untested code. Let me know if you find a problem, or feel free to edit.