Counting the tokens when you tokenize the string in C++? - c++

Question

Java has this easy method to count the tokens that you tokenize:
import java.util.*;
public class Program
{
public static void main(String[] args)
{
String str =
"This is/some text/that I am/parsing/using StringTokenizer/.";
StringTokenizer strTok =
new StringTokenizer(str, "/", false);
System.out.println("Count...");
System.out.println(strTok.countTokens());
}
}
Output:Count...6
Is there any easy way to do in C++?

Answer 1

You could use std::istringstreamclass along with function std::getline. For example
#include <iostream>
#include <sstream>
#include <string>
int main()
{
char s[] = "This is/some text/that I am/parsing/using StringTokenizer/.";
std::istringstream is( s );
size_t count = 0;
std::string line;
while ( std::getline( is, line, '/' ) ) ++count;
std::cout << "There are " << count << " tokens" << std::endl;
}
The output is
There are 6 tokens
Or
#include <iostream>
#include <sstream>
#include <string>
#include <vector>
int main()
{
char s[] = "This is/some text/that I am/parsing/using StringTokenizer/.";
std::istringstream is( s );
std::vector<std::string> v;
std::string line;
while ( std::getline( is, line, '/' ) ) v.push_back( line );
std::cout << "There are " << v.size() << " tokens" << std::endl;
}
To build again the string from the vector you could use for example the following code
#include <iostream>
#include <sstream>
#include <string>
#include <vector>
int main()
{
char s[] = "This is/some text/that I am/parsing/using StringTokenizer/.";
std::istringstream is( s );
std::vector<std::string> v;
std::string line;
while ( std::getline( is, line, '/' ) ) v.push_back( line );
std::cout << "There are " << v.size() << " tokens" << std::endl;
std::string s1;
bool first = true;
for ( const std::string &t : v )
{
if ( first ) first = false;
else s1 += '/';
s1 += t;
}
std::cout << s1 << std::endl;
}
Or you could use standard algorithm std::replace declared in header <algorithm> to replace one delimeter to another in the original string.
If your compiler does not support the range based for loop then you can write instead
for ( std::vector<std::string>::size_type i = 0; i < v.size(); i++ )
{
if ( i != 0 ) s1 += '/';
s1 += v[i];
}

Answer 2

You could try this:
std::vector<std::string> v(std::istream_iterator<std::string>(std::cin), {});
std::cout << "Count..." << v.size() << "\n";
This will of course tokenize at spaces, not at arbitrary separators. To split on arbitary separators, we need std::getline, but now we don't have an easy istream_iterator. Thankfully, this is a solved problem. So we write:
#include <iostream>
#include <iterator>
#include <string>
#include <vector>
namespace detail
{
template <char Sep = '\n'>
class Line : public std::string
{
friend std::istream & operator>>(std::istream & is, Line & line)
{
return std::getline(is, line, Sep);
}
};
}
int main()
{
std::vector<std::string> v(std::istream_iterator<detail::Line<'/'>>(std::cin), {});
std::cout << "Count..." << v.size() << "\n";
for (auto const & s : v) std::cout << s << "\n";
}
If you want to tokenize an existing string rather than the standard input, use a string stream, i.e. replace std::cin with iss, where we have:
#include <sstream>
std::istringstream iss(my_input_string);