C++ : Interpret unicode white space - c++

Question

I have a file which contains text (ASCII + unicode) and I am trying to count total words in it using a C++ program. It is a requirement that I should read the file line by line (using getline) and then process each line to count the words within it.
So I have written the following simple program:
#include <iostream>
#include <fstream>
#include <sstream>
#include <string>
int main(int argc, char* argv[]) {
uint64_t ct = 0;
std::string line;
std::ifstream infile(argv[1]);
while(std::getline(infile, line)) {
std::stringstream inputStream(line);
std::string token;
while (inputStream >> token) {
++ct;
}
}
std::cout << ct << std::endl;
return 0;
}
However, the above program outputs a number that is lesser than what wc -w command gives. To narrow down the problem, I modified the program to simply output whatever it reads. So now the program becomes:
int main(int argc, char* argv[]) {
uint64_t ct = 0;
std::string line;
std::ifstream infile(argv[1]);
while(std::getline(infile, line)) {
std::stringstream inputStream(line);
std::string token;
while (inputStream >> token) {
std::cout << token << " ";
}
std::cout << std::endl;
}
return 0;
}
I redirected the output of this program to another file. Now, when I run wc -w on this new file, the number is same as running wc -w on the original file. This means, I am reading all the words (i.e., "words" defined by wc) in my program. And hence, a reasonable explanation would be that one of the values of token that is read using inputStream >> token consists of some unicode character that is interpreted as a white space by wc program. So how do I change my program to also support such interpretation of unicode white space characters?

Answer 1

You can go by either:
A. Java's definition of Unicode (not non-breaking) whitespace.
or
B. Wikipedia's list of all 25 Unicode code points defined as whitespace.