How to load large data from txt file in Qt - c++

Question

I need to load a txt file with 5 millions data (i.e. strings, just one word with 9 characters per word separated by new line.) into QVector as fast as possible. The code is now working just fine however, if the user hits upload, the application takes 3-5 seconds to load this data for further manipulation. I need to decrease the time of loading this data. What is the right approach to handle this issue? I'm Ok with Qt/STL/Boost. I prefer Qt though. The code that I'm using for this task is the one suggested in Qt documentation which is
QFile file("in.txt");
if (!file.open(QIODevice::ReadOnly | QIODevice::Text))
return;
QTextStream in(&file);
while (!in.atEnd()) {
QString line = in.readLine();
process_line(line);
}

Answer 1

Try this:
I tested it and read file in 2.1 seconds.
I reserve vector before reading and use QElapsedTimer to get reading time.
void MainWindow::readDataText()
{
QString filePath = "F:\\Qt\\Big_File\\Big_File\\data.txt";
QVector<qint64> *vector = new QVector<qint64>;
vector->reserve(5000000);
QElapsedTimer timer;
QFile readFile(filePath);
if(!readFile.open(QFile::ReadOnly | QFile::Text))
{
// Can't Open File.
}
else
{
QByteArray data;
timer.start();
for (int var = 0; var < 5000000; ++var)
{
vector->insert(var, (readFile.readLine()).toInt());
}
qint64 time = timer.elapsed();
ui->txtReadTimeText->setText(QString::number(time));
}
readFile.close();
}
Also it will better if your file being binary.
Another solution is to use readAll() function and read file in 116 miliseconds, and process(split by '\n') data later like this:
void MainWindow::readDataText()
{
QString filePath = "D:\\ProjectTest\\ProjectTest\\data.txt";
QByteArray data;
data.reserve(5000000);
QElapsedTimer timer;
QFile readFile(filePath);
if(!readFile.open(QFile::ReadOnly | QFile::Text))
{
// Can't Open File.
}
else
{
timer.start();
data = readFile.readAll();
qint64 time = timer.elapsed();
ui->txtReadTimeText->setText(QString::number(time));
}
readFile.close();
}

Answer 2

Your example code actually implicitly does decoding. It reads 8-bit encoded text from the file, and converts it to QString, which internally uses 16 bit Unicode encoding.
You will probably gain a big speedup, if instead of using QTextStream, you use just plain QFile directly, and read form it using this readLine method, which return QByteArray, in other words "raw" file contents. The purpose of doing it this way is to avoid creating QString objects for entire file contents.
If you have 5 million lines, then you will also get a significant memory footprint savings, if you store them in memory in QByteArray, instead of QString. Convert to QString only when you are actually going to display the text in the GUI.
Note: Be aware of text encoding! Any text in any file is always encoded, even if especially English-speakers might not realize it. The most straightforward encoding is 7-bit ASCII, a lot of pure English text is actually this, and almost every encoding including UTF-8 is actually superset of 7-bit ASCII, so 7-bit ASCII file can be loaded using almost any encoding. But for multilingual text, you need to know what encoding the file uses, or you will get the accented and other special characters, like ÄÅÁÀÃ, wrong. UTF8 is the only encoding which can store "everything", other encodings such as Latin1 are designed for specific language families.
Note 2: QByteArray actually corresponds to std::string for most purposes. QString is more like std::wstring. Not saying these are identical 1:1 matches, but it helps to think of them as similar.