Strtok and Char* [duplicate] - c++

Question

This question already has answers here:
C's strtok() and read only string literals
(5 answers)
Closed 8 years ago.
I have a simple code where Iam trying to go through a char* and spit it into separate words. Here is the simple code I have.
#include <iostream>
#include <stdio.h>
int main ()
{
char * string1 = "- This is a test string";
char * character_pointer;
std::cout << "Splitting stringinto tokens:" << string1 << std::endl;
character_pointer = strtok (string1," ");
while (character_pointer != NULL)
{
printf ("%s\n", character_pointer);
character_pointer = strtok (NULL, " ");
}
return 0;
}
I am getting an error that will not allow me to do this.
So my question is, how do I go through and find each word in a char*. For my actual program I am working on, one of my libraries returns a paragraph of words as a const char* and I need to stem each word using a stemming algorithm (I know how to do this, I just do not know how to send each individual word to the stemmer). If someone could just solve how to get the example code to work, I will be able to figure it out. All of the examples online use a char[] for string1 instead of a char* and I cannot do that.

Answer 1

This is the simplest (codewise) way I know to split a string in c++:
std::string string1 = "- This is a test string";
std::string word;
std::istringstream iss(string1);
// by default this splits on any whitespace
while(iss >> word) {
std::cout << word << '\n';
}
or like this if you want to specify a delimiter.
while(std::getline(iss, word, ' ')) {
std::cout << word << '\n';
}

Answer 2

Here's a corrected version, try it out:
#include <iostream>
#include <stdio.h>
#include <cstring>
int main ()
{
char string1[] = "- This is a test string";
char * character_pointer;
std::cout << "Splitting stringinto tokens:" << string1 << std::endl;
character_pointer = strtok (string1," ");
while (character_pointer != NULL)
{
printf ("%s\n", character_pointer);
character_pointer = strtok (NULL, " ");
}
return 0;
}

Answer 3

There are different ways you could do this in C++.
If space is your delimited then you can get the tokens this way:
std::string text = "- This is a test string";
std::istringstream ss(text);
std::vector<std::string> tokens;
std::copy(std::istream_iterator<std::string>(ss),
std::istream_iterator<std::string>(),
std::back_inserter<std::vector<std::string>>(tokens));
You can also tokenize the string in C++ using regular expressions.
std::string text = "- This is a test string";
std::regex pattern("\\s+");
std::sregex_token_iterator it(std::begin(text), std::end(text), pattern, -1);
std::sregex_token_iterator end;
for(; it != end; ++it)
{
std::cout << it->str() << std::endl;
}

Answer 4

Forget about strtok. To get exactly what you seem to be
aiming for:
std::string const source = "- This is a test string";
std::vector<std::string> tokens;
std::string::const_iterator start = source.begin();
std::string::const_iterator end = source.end();
std::string::const_iterator next = std::find( start, end, ' ' );
while ( next != end ) {
tokens.push_back( std::string( start, next ) );
start = next + 1;
next = std::find( start, end, ' ' );
}
tokens.push_back( std::string( start, next ) );
Of course, this can be modified as much as you want: you can use
std::find_first_of is you want more than one separator, or
std::search if you want a multi-character separator, or even
std::find_if for an arbitrary test (with a lambda, if you have
C++11). And in most of the cases where you're parsing, you can
just pass around two iterators, rather than having to construct
a substring; you only need to construct a substring when you
want to save the extracted token somewhere.
Once you get used to using iterators and the standard
algorithms, you'll find it a lot more flexible than strtok,
and it doesn't have all of the drawbacks which the internal
state implies.