Why strtok takes a char* and not a const char*? - c++

Question

As far as I can understand strtok() doesn't modify the underlying string, so why doesn't it not take a const char* pointer rather than a const char* pointer? Also while tokenizing you wouldn't want your string to change, right?
Updated:
https://godbolt.org/z/3SPvRB
It is clear that strtok() does modify the underlying string. What is the alternative for an non-mutating tokenizer?

Answer 1

But strtok DOES change the string.
Take the following code:
char sz[] = "The quick brown fox";
char* token = strtok(sz, " ");
It's going to alter the contents of the array into:
"The\0quick brown fox";
The first discovered delimiter gets replaced with a null char. Internally (via thread local storage or global variable), the pointer to the the next char past the discovered delimiter is stored such that a subsequent call to strtok(NULL, " ") will parse the next token from the original string.

Answer 2

It does modify the underlying string. See: http://www.cplusplus.com/reference/cstring/strtok/
This end of the token is automatically replaced by a null-character, and the beginning
of the token is returned by the function.
Proof:
/* strtok example */
#include <stdio.h>
#include <string.h>
int main ()
{
char str[] ="- This, a sample string.";
char * pch;
printf ("Splitting string \"%s\" into tokens:\n",str);
pch = strtok (str," ,.-");
while (pch != NULL)
{
printf ("%s\n",pch);
pch = strtok (NULL, " ,.-");
}
/* note this line... */
printf ("str = \"%s\"\n",str);
return 0;
}
Prints:
Splitting string "- This, a sample string." into tokens:
This
a
sample
string
str = "- This"

Answer 3

Updated: https://godbolt.org/z/3SPvRB It is clear that strtok() does
modify the underlying string. What is the alternative for an
non-mutating tokenizer?
As mentioned in the comments, you can either:
make a copy of the original string and then tokenize the copy with strtok(); or
write your own implementation that brackets the tokens and copies the tokens to new storage:
using C strspn to scan forward to the first non-delimiter character which will be the beginning of the token, then use strcspn to scan forward to the next delimiter marking the end of the token,
do the same thing manually with a pair of pointers; or
for C++11 or later, you can use .find_first_not_of() to scan forward to the first non-delimiter character, and then .find_first_of() to locate the delimiter that follows the token.
In each case you will then copy the token characters to a new string (using memcpy for C-type implementation -- don't forget to nul-terminate) or for C++11 simply using the .substr() member function.
A very-basic C++11 implementation would look similar to:
std::vector<std::string> stringtok (const std::string& s, const std::string& delim)
{
std::vector<std::string> v {}; /* vector of strings for tokens */
size_t beg = 0, end = 0; /* begin and end positons in str */
/* while non-delimiter char found */
while ((beg = s.find_first_not_of (delim, end)) != std::string::npos) {
end = s.find_first_of (delim, beg); /* find delim after non-delim */
v.push_back (s.substr (beg, end - beg)); /* add substr to vector */
if (end == std::string::npos) /* if last delim, break */
break;
}
return v; /* return vector of tokens */
}
If you follow the logic, it tracks exactly what is described above the function definition. Combining it into a short example, you would have:
#include <iostream>
#include <string>
#include <vector>
std::vector<std::string> stringtok (const std::string& s, const std::string& delim)
{
std::vector<std::string> v {}; /* vector of strings for tokens */
size_t beg = 0, end = 0; /* begin and end positons in str */
/* while non-delimiter char found */
while ((beg = s.find_first_not_of (delim, end)) != std::string::npos) {
end = s.find_first_of (delim, beg); /* find delim after non-delim */
v.push_back (s.substr (beg, end - beg)); /* add substr to vector */
if (end == std::string::npos) /* if last delim, break */
break;
}
return v; /* return vector of tokens */
}
int main (void) {
std::string str = " my dog has fleas ",
delim = " ";
std::vector<std::string> tokens;
tokens = stringtok (str, delim);
std::cout << "string: '" << str << "'\ntokens:\n";
for (auto s : tokens)
std::cout << " " << s << '\n';
}
Example Use/Output
$ ./bin/stringtok
string: ' my dog has fleas '
tokens:
my
dog
has
fleas
Note: this is only one of many ways to implement a string tokenization that does not modify the original. Look things over and let me know if you have further questions.