Using strtok/strtok_r in a while loop in C++ - c++

Question

I'm getting unexpected behavior from the strtok and strtrok_r functions:
queue<string> tks;
char line[1024];
char *savePtr = 0;
while(true)
{
//get input from user store in line
tks.push(strtok_r(line, " \n", &savePtr)); //initial push only works right during first loop
char *p = nullptr;
for (...)
{
p = strtok_r(NULL, " \n", &savePtr);
if (p == NULL)
{
break;
}
tks.push(p);
}
delete p;
savePtr = NULL;
//do stuff, clear out tks before looping again
}
I've tried using strtok and realized that during the second loop, the initial push is not occurring. I attempted to use the reentrant version strtok_r in order to control what the saved pointer is pointing to during the second loop by making sure it is null before looping again.
tks is only correctly populated during the first time through the loop - subsequent loops give varying results depending on the length of line
What am I missing here?

Answer 1

Just focusing on the inner loop and chopping off all of the stuff I don't see as necessary.
#include <iostream>
#include <queue>
#include <string>
#include <cstring>
using namespace std;
int main()
{
std::queue<std::string> tks;
while(true)
{
char line[1024];
char *savePtr;
char *p;
cin.getline(line, sizeof(line));
p = strtok_r(line, " \n", &savePtr); // initial read. contents of savePtr ignored
while (p != NULL) // exit when no more data, which includes an emtpy line
{
tks.push(p); // got data, store it
p = strtok_r(NULL, " \n", &savePtr); // get next token
}
// consume tks
}
}
I prefer the while loop over the for loop used by Toby Speight in his answer because I think it is more transparent and easier to read. Your mileage may vary. By the time the compiler is done with it they will be identical.
There is no need to delete any memory. It is all statically allocated. There is no need to clear anything before the next round except for tks. savePtr will be reset by the first strtok_r.
There is a failure case if the user inputs more than 1024 characters on a line, but this will not crash. If this still doesn't work, look into how you're consuming tks. It's not posted so we can't troubleshoot that portion.
Wholeheartedly recommend changing to a string-based solution if possible. This is a really simple, easy to write, but slow, one:
#include <iostream>
#include <queue>
#include <string>
#include <sstream>
int main()
{
std::queue<std::string> tks;
while(true)
{
std::string line;
std::getline(std::cin, line);
std::stringstream linestream(line);
std::string word;
// parse only on ' ', not on the usual all whitespace of >>
while (std::getline(linestream, word, ' '))
{
tks.push(word);
}
// consume tks
}
}

Answer 2

Your code wouldn't compile for me, so I fixed it:
#include <iostream>
#include <queue>
#include <string>
#include <cstring>
std::queue<std::string> tks;
int main() {
char line[1024] = "one \ntwo \nthree\n";
char *savePtr = 0;
for (char *p = strtok_r(line, " \n", &savePtr); p;
p = strtok_r(nullptr, " \n", &savePtr))
tks.push(p);
// Did we read it correctly?
for (; tks.size() > 0; tks.pop())
std::cout << ">" << tks.front() << "<" << std::endl;
}
This produces the expected output:
>one<
>two<
>three<
So your problem isn't with the code you posted.

Answer 3

If you have the option to use boost, try this one out to tokenize a string. Of course by providing your own string and delimeters.
#include <vector>
#include <boost/algorithm/string.hpp>
int main()
{
std::string str = "Any\nString\nYou want";
std::vector< std::string > results;
boost::split( results, str, boost::is_any_of( "\n" ) );
}