How to read file or stream until string found - c++

Question

I am writing a dictionary program, the input is specified by a file and parsed as such:
std::string savedDictionary(std::istreambuf_iterator<char>(std::ifstream(DICTIONARY_SAVE_FILE)), {});
// entire file loaded into savedDictionary
for (size_t end = 0; ;)
{
size_t term = savedDictionary.find("|TERM|", end);
size_t definition = savedDictionary.find("|DEFINITION|", term);
if ((end = savedDictionary.find("|END|", definition)) == std::string::npos) break;
// store term and definition here...
}
This throws std::bad_alloc on some of my third world users' machines that don't have enough RAM to store the dictionary string + the dictionary as it's held inside my program.
If I could do this:
std::string term;
for (std::ifstream file(DICTIONARY_SAVE_FILE); file; std::getline(file, term, "|END|")
{
// same as above
}
then it would be great, but std::getline doesn't support a string as delimiter.
So, what's the most idiomatic way to read the file until I find "|END|" without allocating a crap ton of memory up front?

Answer 1

We can achieve the requested functionality by using a very simple proxy class. With that it is easy to use all std::algorithms and all std::iterators as usual.
So, we define a smal proxy class called LineUntilEnd. This can be used in conjunction with all streams like a std::ifstream or whatever you like. You can especially simply use the extractor operator to extract a value from the input stream and put it into the desired variable.
// Here we will store the lines until |END|
LineUntilEnd lue;
// Simply read the line until |END|
while (testInput >> lue) {
It works as expected.
An if we have such a string, we can parse it afterwords with easy regex operations.
I added a small example and put the resulting value in a std::multimap to build a demo dictionary.
Please see the following code
#include <iostream>
#include <string>
#include <iterator>
#include <regex>
#include <map>
#include <sstream>
#include <iterator>
// Ultra simple proxy class to read data until given word is found
struct LineUntilEnd
{
// Overload the extractor operator
friend std::istream& operator >>(std::istream& is, LineUntilEnd& lue);
// Intermediate storage for result
std::string data{};
};
// Read stream until "|END|" symbol has been found
std::istream& operator >>(std::istream& is, LineUntilEnd& lue)
{
// Clear destination string
lue.data.clear();
// We will count, how many bytes of the search string have been matched
size_t matchCounter{ 0U };
// Read characters from stream
char c{'\0'};
while (is.get(c))
{
// Add character to resulting string
lue.data += c;
// CHeck for a match. All characters must be matched
if (c == "|END|"[matchCounter]) {
// Check next matching character
++matchCounter;
// If there is a match for all characters in the searchstring
if (matchCounter >= (sizeof "|END|" -1)) {
// The stop reading
break;
}
}
else {
// Not all charcters could be matched. Start from the begining
matchCounter = 0U;
}
}
return is;
}
// Input Test Data
std::istringstream testInput{ "|TERM|bonjour|TERM|hola|TERM|hi|DEFINITION|hello|END||TERM|Adios|TERM|Ciao|DEFINITION|bye|END|" };
// Regex defintions. Used to build up a dictionary
std::regex reTerm(R"(\|TERM\|(\w+))");
std::regex reDefinition(R"(\|DEFINITION\|(\w+)\|END\|)");
// Test code
int main()
{
// We will store the found values in a dictionay
std::multimap<std::string, std::string> dictionary{};
// Here we will store the lines until |END|
LineUntilEnd lue;
// Simply read the line until |END|
while (testInput >> lue) {
// Search for the defintion string
std::smatch sm{};
if (std::regex_search(lue.data, sm, reDefinition)) {
// Definition string found
// Iterate over all terms
std::sregex_token_iterator tokenIter(lue.data.begin(), lue.data.end(), reTerm, 1);
while (tokenIter != std::sregex_token_iterator()) {
// STore values in dictionary
dictionary.insert({ sm[1],*tokenIter++ });
}
}
}
// And show some result to the user
for (const auto& d : dictionary) {
std::cout << d.first << " --> " << d.second << "\n";
}
return 0;
}

Answer 2

For those in the future, this is what I ended up writing:
std::optional<std::string> ReadEntry(std::istream& stream)
{
for (struct { char ch; int matched; std::string entry; } i{}; stream.get(i.ch); i.entry.push_back(i.ch))
if (i.ch == "|END|"[i.matched++]);
else if (i.matched == sizeof("|END|")) return i.entry;
else i.matched = 0;
return {};
}