Qt: How can I copy a big data using QT? - c++

Question

I want to read a big data, and then write it to a new file using Qt.
I have tried to read a big file. And the big file only have one line. I test with readAll() and readLine().
If the data file is about 600MB, my code can run although it is slow.
If the data file is about 6GB, my code will fail.
Can you give me some suggestions?
Update
My test code is as following:
#include <QApplication>
#include <QFile>
#include <QTextStream>
#include <QTime>
#include <QDebug>
#define qcout qDebug()
void testFile07()
{
QFile inFile("../03_testFile/file/bigdata03.txt");
if (!inFile.open(QIODevice::ReadOnly | QIODevice::Text))
{
qcout << inFile.errorString();
return ;
}
QFile outFile("../bigdata-read-02.txt");
if (!outFile.open(QIODevice::WriteOnly | QIODevice::Truncate))
return;
QTime time1, time2;
time1 = QTime::currentTime();
while(!inFile.atEnd())
{
QByteArray arr = inFile.read(3*1024);
outFile.write(arr);
}
time2 = QTime::currentTime();
qcout << time1.msecsTo(time2);
}
void testFile08()
{
QFile inFile("../03_testFile/file/bigdata03.txt");
if (!inFile.open(QIODevice::ReadOnly | QIODevice::Text))
return;
QFile outFile("../bigdata-readall-02.txt");
if (!outFile.open(QIODevice::WriteOnly | QIODevice::Truncate))
return;
QTime time1, time2, time3;
time1 = QTime::currentTime();
QByteArray arr = inFile.readAll();
qcout << arr.size();
time3 = QTime::currentTime();
outFile.write(inFile.readAll());
time2 = QTime::currentTime();
qcout << time1.msecsTo(time2);
}
int main(int argc, char *argv[])
{
testFile07();
testFile08();
return 0;
}
After my test, I share my experience about it.
read() and readAll() seem to be the same fast; more actually, read() is slightly faster.
The true difference is writing.
The size of file is 600MB:
Using read function, read and write the file cost about 2.1s, with 875ms for reading
Using readAll function, read and write the file cost about 10s, with 907ms for reading
The size of file is 6GB:
Using read function, read and write the file cost about 162s, with 58s for reading
Using readAll function, get the wrong answer 0. Fail to run well.

Answer 1

Open both files as QFiles. In a loop, read a fixed number of bytes, say 4K, into an array from the input file, then write that array into the output file. Continue until you run out of bytes.
However, if you just want to copy a file verbatim, you can use QFile::copy

Answer 2

You can use QFile::map and use the pointer to the mapped memory to write in a single shot to the target file:
void copymappedfile(QString in_filename, QString out_filename)
{
QFile in_file(in_filename);
if(in_file.open(QFile::ReadOnly))
{
QFile out_file(out_filename);
if(out_file.open(QFile::WriteOnly))
{
const qint64 filesize = in_file.size();
uchar * mem = in_file.map(0, filesize, QFileDevice::MapPrivateOption);
out_file.write(reinterpret_cast<const char *>(mem) , filesize);
in_file.unmap(mem);
out_file.close();
}
in_file.close();
}
}

Answer 3

One thing to keep in mind:
With read() you specify a maximum size for the currently read chunk (in your example 3*1024 bytes), with readAll() you tell the program to read the entire file at once.
In the first case you (repeatedly) put 3072 Bytes on the stack, write them and they get removed from the stack once the current loop iteration ends. In the second case you push the entire file on the stack. Pushing 600MB on the stack at once might be the reason for your performance issues. If you try to put 6GB on the stack at once you may just run out of memory/adress space - causing your program to crash.