Copying multiple file segments into a single file - Qt - c++

Question

I have a file split into many segments. I have to combine the files into a single file. Now the simple code I came up with is:
QFile file;
file.setFileName(fileUrl);
file.open(QIODevice::WriteOnly);
for(int j=0;j<totalSegments;j++)
{
Segment[j]->fileSegment.close();
if(!Segment[j]->fileSegment.open(QIODevice::ReadOnly))
{
qDebug()<<"Segment not found";
continue;
}
file.write(Segment[j]->fileSegment.readAll()); // is this really efficient and safe
Segment[j]->fileSegment.close();
Segment[j]->fileSegment.remove();
}
The above code snippet works fine on Windows as well as Linux. But I have some questions:
1- Is this method really efficient. If suppose the segment size is in GB's will this badly affect the performance of the system, or can even corrupt the file or fail due to less available RAM.
2- The above method fails in some Linux Distro's especially Fedora if total size is more than 2GB. I haven't tested myself but was reported to me by many.
3- In Linux can it fail if segments are on an EXT4 filesystem and target file into which the file will be written on NTFS system. It didn't fail on Ubuntu but many users are complaining that it does. I can't just replicate it. Am I doing something wrong.

Answer 1

Please avoid multiple sub-questions per question in general, but I will try to answer your questions regardless.
1- Is this method really efficient. If suppose the segment size is in GB's will this badly affect the performance of the system, or can even corrupt the file or fail due to less available RAM.
It is very bad idea for large files. I think you wish to establish chunk file read and write.
2- The above method fails in some Linux Distro's especially Fedora if total size is more than 2GB. I haven't tested myself but was reported to me by many.
2 GB < (or was it 4 GB?) counts as large file on 32 bit systems, so it is possible that they use the software without large file support build. It is necessary to make sure that support is enabled while building. There used to be a configure option for Qt as -largefile.
3- In Linux can it fail if segments are on an EXT4 filesystem and target file into which the file will be written on NTFS system. It didn't fail on Ubuntu but many users are complaining that it does. I can't just replicate it. Am I doing something wrong.
Yes, it can be the same issue, also you need to pay attention to memory fragmentation which means, you will not be able to allocate 2 GB in memory even if you have 2 GB available, but the memory is inappropriately fragmented. On Windows, you may wish to use the /LARGEADDRESSAWARE option for instance when using 32 bit process.
Overall, the best would be to establish the loop for reading and writing, and then you could forget the large address aware and so on issues. You would still need to make sure that Qt can handle large files though if you wish to support them for your clients. This is of course only necessary on 32 bit because there is no practical limit for 64 bit with the currently ongoing file sizes at this point.
Since you requested some code in the comment to get you going, here is a simple and untested version of chunk read and immediate write of the content from an input file into an output file. I am sure this will get you going so that you can figure out the rest.
QFileInfo fileInfo("/path/to/my/file");
qint64 size = fileInfo.size();
QByteArray data;
int chunkSize = 4096;
for (qint64 bytes = 0; bytes < size, bytes+=data.size()) {
data = myInputFile.read(chunkSize);
// Error check
myOutputFile.write(data);
}