Bug with Iterating over a string c++ - c++

Question

so I have a function called split_alpha() that takes in a std::string and splits the string into words, using any non-alphaneumeric character as a delimiter. It also maps the words to their lower-cased versions.
vector<string> split_alpha(string to_split) {
vector<string> results;
string::iterator start = to_split.begin();
string::iterator it = start;
++it;
//get rid of any non-alphaneumeric chars at the front of the string
while (!isalnum(*start)) {
++start;
++it;
}
while (it != to_split.end()) {
if (!isalnum(*it)) {
string to_add = string(start, it);
lower_alpha(to_add);
results.push_back(to_add);
++it;
if (it == to_split.end()) { break; }
while (!isalnum(*it)) {
++it;
if (it == to_split.end()) { break; }
}
start = it;
++it;
}
else {
++it;
if (it == to_split.end()) { break; }
}
}
//adds the last word
string to_add = string(start, it);
lower_alpha(to_add);
results.push_back(to_add);
return results;
}
The function works fine 99% of the time, but when I give it the string "Sending query: “SELECT * FROM users”" (not including the quotations around the whole string), it does something really weird. It essentially goes into an infinite loop (within that while loop) and never finds the end of the string. Instead it keeps reading random characters/strings from somewhere?? My vector ends up with a size of about 200 before it finally segfaults. Anyone know what could be causing this? I tried printing out the string and it seems perfectly fine. Once again, the code works on every other string I've tried.
Thanks!!

Answer 1

isn't the while loop doing that?
Yes, but you can have several ++it triggering before the while loop check, and in any one of those cases the iterator could already be at the end of the string. Most likely the other strings you tried did not cause a failure because they all end with an alphanumeric character.
Invert the order of the ++it and the check:
if (it == to_split.end()) { break; }
++it;
Explanation: the following assert will fail, as the iterator will no longer be pointing to the end of the string (but one character further):
if (it == to_split.end())
{
++it;
assert(it == to_split.end());
}

Answer 2

Since the origin of the bug in your function has been pointed out already, may I suggest slightly different approach to your word splitting, using regex:
#include <iostream>
#include <regex>
#include <vector>
#include <string>
#include <cctype>
std::vector<std::string> split_alpha(std::string str)
{
std::regex RE{ "([a-zA-Z0-9]+)" }; // isalnum equivalent
std::vector<std::string> result;
// find every word
for (std::smatch matches; std::regex_search(str, matches, RE); str = matches.suffix())
{
//push word to the vector
result.push_back(matches[1].str());
//transform to lower
for (char &c : result[result.size() - 1])
c = std::tolower(c);
}
return result;
}
int main()
{
// test the function
for (auto &word : split_alpha("Sending query: “SELECT * FROM users”"))
std::cout << word << std::endl;
return 0;
}
Result:
sending
query
select
from
users