How to speed up regex searching for large quantity of potentially large files in C++? - c++

Question

I'm trying to make a program to read user inputted wildcard files and wildcard strings using an excel document as a configuration file. For example the user may be able to enter in C:\Read*.txt, and any files in the C drive that start with Read and then any characters after read and are text files will be included in the search.
They could search for Message: * and all strings beginning with "Message: " and ending with any sequence of characters would get matched.
So far it is a working program but the problem is that the speed efficiency is quite terrible and I need it to be able to search very large files. I'm using a filestream and the regex class to do so and I'm not sure what is taking so much time.
The bulk of the time in my code is being spent in the following loop (I've only included the lines above the while loop so you can better understand what I'm trying to do):
smatch matches;
vector<regex> expressions;
for (int i = 0; i < regex_patterns.size(); i++){expressions.emplace_back(regex_patterns.at(i));}
auto startTimer = high_resolution_clock::now();
// Open file and begin reading
ifstream stream1(filePath);
if (stream1.is_open())
{
int count = 0;
while (getline(stream1, line))
{
// Continue to next step if line is empty, no point in searching it.
if (line.size() == 0)
{
// Continue to next step if line is empty, no point in searching it.
continue;
}
// Loop through each search string, if match, save line number and line text,
for (int i = 0; i < expressions.size(); i++)
{
size_t found = regex_search(line, matches, expressions.at(i));
if (found == 1)
{
lineNumb.push_back(count);
lineTextToSave.push_back(line);
}
}
count = count + 1;
}
}
auto stopTimer = high_resolution_clock::now();
auto duration2 = duration_cast<milliseconds>(stopTimer - startTimer);
cout << "Time to search file: " << duration2.count() << "\n";
Is there a better method of searching files than this? I tried looking up many things but haven't found a programmatic example that I've understood thus far.

Answer 1

Some ideas by order of priority:
You could join all the regex patterns together to form a single regex instead of matching r regexes on each line. This will speed up your program by a factor of r. Example: (R1)|(R2)|(...)|(Rr)
Ensure you are compiling the regex before usage.
Do not add the final .* to your regex pattern.
Some ideas but non-portable:
Memory map the file instead of reading through iostreams
Consider if it is worth reimplementing grep instead of calling to grep through popen()