Reading text file in blocks-c++ - c++

Question

string lineValue;
ifstream myFile("file.txt");
if (myFile.is_open()) {
//getline(myFile, lineValue);
//cout<<lineValue;
while (getline(myFile, lineValue)) {
cout << lineValue << '\n';
}
myFile.close();
}
else cout << "Unable to open file";
The txt file formate is like this
0 1
1 2
2 3
3 4
4 5
5 5
6 6
7 7
8 8
9 9
Above code is reading data from a text file line by line, but the text file size is quite large (10GB).
So how to read data from the file in chunks/blocks with less I/O and efficiently ?

Answer 1

If you are thinking of reading in large chunks of data then you will be using a technique called buffering. However, ifstream already provides buffering so my first step would be to see if you can get ifstream doing the job for you.
I would set a much larger buffer than the default in you're ifstream. Something like
const int BUFSIZE = 65536;
std::unique_ptr<char> buffer(new char[BUFSIZE]);
std::ifstream is;
is.rdbuf()->pubsetbuf(buffer.get(), BUFSIZE);
is.open(filename.c_str());
const int LINESIZE = 256;
char line[LINESIZE];
if (is) {
for (;;) {
is.getline(line, LINESIZE);
// check for errors and do other work here, (and end loop at some point!)
}
}
is.close();
Make sure your buffer lives as long as the ifstream object that uses it.
If you find the speed of this is still insufficient, then you can try reading chunks of data with ifstream::read. There is no guarantee it will be faster, you'll have to time and compare the options. You use ifstream::read something like this.
const int BUFSIZE = 65536;
std::unique_ptr<char> buffer(new char[BUFSIZE]);
is.read(buffer.get(), BUFSIZE);
You'll have to take care writing the code to call ifstream.read taking care to deal with the fact that a 'line' of input may get split across consecutive blocks (or even across more than two blocks depending upon your data and buffer size). That's why you want to modify ifstream's buffer as you're first option.

Answer 2

If and only if the text lines are the same length, you could simply read the file in using std::istream::read();.
The size of the block to read would be:
block_size = text_line_length * number_of_text_lines;
If you are brave enough to handle more complexity or your text lines are not equal lengths, you could read an arbitrary length of characters into a vector and process the text from the vector.
The complexities come into play when a text line overflows a block. Think of handling the case where only part of the sentence is available at the end of the block.