How to remove stopwords from a vector of sentences? - c++

Question

I am working on some code that requires stopwords to be removed from sentences. My current solution does not work.
I have a vector of two test sentences:
std::vector<std::string> sentences = {"this is a test", "another a test"};
I have an unordered set of strings containing stopwords:
std::unordered_set<std::string> stopwords;
Now I tried to loop over the sentences in the vector, check and compare each word with the stopwords, and if it is a stopword is should get removed.
sentences.erase(std::remove_if(sentences.begin(), sentences.end(),
[](const std::string &s){return stopwords.find(s) != stopwords.end();}),
sentences.end());
The idea is that my vector -after removing the stopwords- contains the sentences without the stopwords, but for now, I get the exact same sentences back. Any idea why?
My unordered set is filled with the following function:
void load() {
std::ifstream file;
file.open ("stopwords.txt");
if(!file.is_open()) {return;}
std::string stopword;
while (file >> stopword) {
stopwords.insert(stopword);
}
}

Answer 1

Your current code cannot work, since you are not deleting words from each individual string. Your erase/remove_if call takes an entire string and tries to match the word in the set with the entire string.
First, you should write a simple function that when given a std::string and a map of words to delete, return the string with the deleted words.
Here is a small function using std::istringstream that can do this:
#include <unordered_set>
#include <sstream>
#include <string>
#include <iostream>
std::string remove_stop_words(const std::string& src, const std::unordered_set<std::string>& stops)
{
std::string retval;
std::istringstream strm(src);
std::string word;
while (strm >> word)
{
if ( !stops.count(word) )
retval += word + " ";
}
if ( !retval.empty())
retval.pop_back();
return retval;
}
int main()
{
std::string test = "this is a test";
std::unordered_set<std::string> stops = {"is", "test"};
std::cout << "Changed word:\n" << remove_stop_words(test, stops) << "\n";
}
Output:
Changed word:
this a
So once you have this working correctly, the std::vector version is nothing more than looping through each item in the vector and calling the remove_stop_words function:
int main()
{
std::vector<std::string> test = {"this is a test", "another a test"};
std::unordered_set<std::string> stops = {"is", "test"};
for (size_t i = 0; i < test.size(); ++i)
test[i] = remove_stop_words(test[i], stops);
std::cout << "Changed words:\n";
for ( auto& s : test )
std::cout << s << "\n";
}
Output:
Changed words:
this a
another a
Note that you can utilize the std::transform function to remove the hand-rolled loop in the above example:
#include <algorithm>
//...
int main()
{
std::vector<std::string> test = {"this is a test", "another a test"};
std::unordered_set<std::string> stops = {"is", "test"};
// Use std::transform
std::transform(test.begin(), test.end(), test.begin(),
[&](const std::string& s){return remove_stop_words(s, stops);});
std::cout << "Changed words:\n";
for ( auto& s : test )
std::cout << s << "\n";
}