How to multithread file processing in C++? - c++

Question

I'm working on one problem where I need to process 24 files (each size = 3 GB) and write the output into multiple files (24). Each file takes around 1 hour to process. Is it possible to write data into multiple files concurrently using multi-threading with below code?
int _tmain(int argc, _TCHAR* argv[])
{
std::string path;
cout << "Enter the folder of the logs: " << endl;
cin >> path;
WIN32_FIND_DATA FileInformation; // File information
memset(&FileInformation, 0, sizeof(WIN32_FIND_DATA));
std::string strExt = "\\*.txt";
std::string strEscape = "\\";
std::string strPattern = path + strExt;
HANDLE hFile = ::FindFirstFile(strPattern.c_str(), &FileInformation);
while(hFile != INVALID_HANDLE_VALUE)
{
int offset;
std::ifstream Myfile;
std::string strFileName = FileInformation.cFileName;
std:: string fullPath = path + strEscape + strFileName;
std::string outputFile = path + strEscape + strFileName.substr(0, strFileName.length()-3) + "processed"+".txt";
std::ofstream ofs(outputFile, std::ofstream::out);
Myfile.open (fullPath);
std::string line;
if(Myfile.is_open())
{
while(!Myfile.eof())
{
-------Processing--------
}
Myfile.close();
}
else
cout<<"Cannot open file."<<endl;
if(FindNextFile(hFile, &FileInformation) == FALSE)
break;
}
// Close handle
::FindClose(hFile);
return 0;
}

Answer 1

Looking into your code I assume you produce one output file from one input. In such case you do not need to write multithreaded code to check if processing multiple files at once will speed up the process. Just modify your program to accept file name as a parameter and run multiple of them in parallel. But unless you are reading/writing from/to SSD drive such parallel processing most probably would slow process down, as hard-drive will have to switch between reading/writing for multiple positions, and head positioning is slow.
It is not clear what you are doing on processing, but if it takes 100% CPU then you most probably will speed up process significantly by processing one file by multiple threads. You would have one thread reading, then thread pool processing, then one thread writing. Tricky part would be to synchronize data and make it not appear in output file in wrong order.

Answer 2

Don't write multithreaded code here, write multiprocess code. That is, have your program process one file (which is passed as an argument), and call it multiple times in parallel from a script.
Don't run your program 24 times concurrently (unless you have 24 cores and 72GB of memory available). Try running 2, 4 or 6 instances concurrently and see what's best. I guess it'll be the number of cores, maybe the number of cores * 2 - 1 (hyperthreading does help). Try it out.
Also, if your program reads the file at the start, then performs the calculations, then writes the result, measure the time it takes to read the 3GB of data. If it's, for example, 30 seconds, and you run 4 processes concurrently, have your run script start the first instance, then wait 45 seconds, then start the second one and so on until the fourth. Start the fifth instance once one of the first four is finished. Every time another instance finishes, run the next one until all 24 have been run.