Splitting text into a list of words with ICU - c++

Question

I'm working on a text tokenizer. ICU is one of very few C++ libraries that have this feature, and probably the best maintained one, so I'd like to use it.
I've found the docs about BreakIterator, but there's one problem with it: how do I leave the punctuation out?
#include "unicode/brkiter.h"
#include <QFile>
#include <vector>
std::vector<QString> listWordBoundaries(const UnicodeString& s)
{
UErrorCode status = U_ZERO_ERROR;
BreakIterator* bi = BreakIterator::createWordInstance(Locale::getUS(), status);
std::vector<QString> words;
bi->setText(s);
for (int32_t p = bi->first(), prevBoundary = 0; p != BreakIterator::DONE; prevBoundary = p, p = bi->next())
{
const auto word = s.tempSubStringBetween(prevBoundary, p);
char buffer [16384];
word.toUTF8(CheckedArrayByteSink(buffer, 16384));
words.emplace_back(QString::fromUtf8(buffer));
}
delete bi;
return words;
}
int main(int /*argc*/, char * /*argv*/ [])
{
QFile f("E:\\words.TXT");
f.open(QFile::ReadOnly);
QFile result("E:\\words.TXT");
result.open(QFile::WriteOnly);
const QByteArray strData = f.readAll();
for (const QString& word: listWordBoundaries(UnicodeString::fromUTF8(StringPiece(strData.data(), strData.size()))))
{
result.write(word.toUtf8());
result.write("\n");
}
return 0;
}
Naturally, the resulting file looks like this:
“
Come
outside
.
Best
if
we
do
not
wake
him
.
”
What I need is just the words. How can this be done?

Answer 1

QT library include several useful methods for check the char's properties:
QChar.
Indeed, you could create the QString variable from the buffer
and check all properties you need before to insert into the output vector.
For example:
auto token = QString::fromUtf8(buffer);
if (token.length() > 0 && token.data()[0].isPunct() == false) {
words.push_back(std::move(token));
}
With that code I can access the first character of the string and check
whether it is a punctuation mark or not.
Something more robust, I express that as function:
bool isInBlackList(const QString& str) {
const auto len = str.lenght();
if (len == 0) return true;
for(int i = 0; i < len; ++i) {
const auto&& c = str.data()[i];
if (c.isPunct() == true || c.isSpace() == true) {
return true;
}
}
return false;
}
If that function returns true, the token hasn't to be inserted into the vector.