My program returning the set_intersection value of two text files containing 479k words each is really slow. Is it my code? - c++

Question

I wrote a program to compare two text files containing all of the words in the dictionary (one forwards and one backwards). The idea is that when the text file containing all of the backwards words is compared with the forwards words, any matches will indicate that those words can be spelled both forwards and backwards and will return all palindromes as well as any words that spell both a word backwards and forwards.
The program works and I've tested it on three different file sizes. The first set contain only two words, just for testing purposes. The second contains 10,000 English words (in each text file), and the third contains all English words (~479k words). When I run the program calling on the first set of text files, the result is almost instantaneous. When I run the program calling on the set of text files containing 10k words, it takes a few hours. However, when I run the program containing the largest files (479k words), it ran for a day and returned only about 30 words, when it should have returned thousands. It didn't even finish and was nowhere near finishing (and this was on a fairly decent gaming PC).
I have a feeling it has to do with my code. It must be inefficient.
There are two things that I've noticed:
When I run: cout << "token: " << *it << std::endl; it runs endlessly on a loop forever and never stops. Could this be eating up processing power?
I commented out sorting because all my data is already sorted. I noticed that the second I did this, the program running 10,000 word text files sped up.
However, even after doing these things there seemed to be no real change in speed in the program calling on the largest text files. Any advice? I'm kinda new at this. Thanks~
*Please let me know if you'd like a copy of the text files and I'd happily upload them. Thanks
#include <iostream>
#include <string>
#include <fstream>
#include <vector>
#include <iterator>
#include <algorithm>
#include <boost/tokenizer.hpp>
typedef boost::char_separator<char> separator_type;
using namespace std;
using namespace boost;
int main()
{
fstream file1; //fstream variable files
fstream file2; //fstream variable files
string dictionary1;
string dictionary2;
string words1;
string words2;
dictionary1 = "Dictionary.txt";
// dictionary1 = "Dictionarytenthousand.txt";
// dictionary1 = "Twoworddictionary.txt"; //this dictionary contains only two words separated by a comma as a test
dictionary2 = "Backwardsdictionary.txt";
// dictionary2 = "Backwardsdictionarytenthousand.txt";
// dictionary2 = "Backwardstwoworddictionary.txt"; //this dictionary contains only two words separated by a comma as a test
file1.open(dictionary1.c_str()); //opening Dictionary.txt
file2.open(dictionary2.c_str()); //opening Backwardsdictionary.txt
if (!file1)
{
cout << "Unable to open file1"; //terminate with error
exit(1);
}
if (!file2)
{
cout << "Unable to open file2"; //terminate with error
exit(1);
}
while (getline(file1, words1))
{
while (getline(file2, words2))
{
boost::tokenizer<separator_type> tokenizer1(words1, separator_type(",")); //separates string in Twoworddictionary.txt into individual words for compiler (comma as delimiter)
auto it = tokenizer1.begin();
while (it != tokenizer1.end())
{
std::cout << "token: " << *it << std::endl; //test to see if tokenizer works before program continues
vector<string> words1Vec; // vector to store Twoworddictionary.txt strings in
words1Vec.push_back(*it++); // adds elements dynamically onto the end of the vector
boost::tokenizer<separator_type> tokenizer2(words2, separator_type(",")); //separates string in Backwardstwoworddictionary.txt into individual words for compiler (comma as delimiter)
auto it2 = tokenizer2.begin();
while (it2 != tokenizer2.end())
{
std::cout << "token: " << *it2 << std::endl; //test to see if tokenizer works before program continues
vector<string> words2Vec; //vector to store Backwardstwoworddictionary.txt strings in
words2Vec.push_back(*it2++); //adds elements dynamically onto the end of the vector
vector<string> matchingwords(words1Vec.size() + words2Vec.size()); //vector to store elements from both dictionary text files (and ultimately to store the intersection of both, i.e. the matching words)
//sort(words1Vec.begin(), words1Vec.end()); //set intersection requires its inputs to be sorted
//sort(words2Vec.begin(), words2Vec.end()); //set intersection requires its inputs to be sorted
vector<string>::iterator it3 = set_intersection(words1Vec.begin(), words1Vec.end(), words2Vec.begin(), words2Vec.end(), matchingwords.begin()); //finds the matching words from both dictionaries
matchingwords.erase(it3, matchingwords.end());
for (vector<string>::iterator it4 = matchingwords.begin(); it4 < matchingwords.end(); ++it4) cout << *it4 << endl; // returns matching words
}
}
}
}
file1.close();
file2.close();
return 0;
}

Answer 1

Stop using namespace. Type the extra stuff.
Have code do one thing. Your code isn't doing what you claim it does, probably becuase you are doing 4 things at once and getting confused.
Then glue the code together.
Getline supports arbitrary delimiters. Use it with ','.
Write code that converts a file into a vector of strings.
std::vector<std::string> getWords(std::string filename);
then test it works. You are doing this wrong in your code posted above, in that you are making length 1 vectors and tossing them.
That will remove about half of your code.
Next, for set_intersection, use std::back_inserter and an empty vector as your output. Like (blah begin, blah end, foo begin, foo end, std::back_inserter(vec3)). It will call push_back with each result.
In pseudo code:
std::vec<std::string> loadWords(std::string filename)
auto file=open(filename)
std::vec<std::string> retval
while(std::readline(file, str, ','))
retval.push_back(str)
return retval
std::vec<string> intersect(std::string file1, std::string file2)
auto v1=loadWords(file1)
auto v2=loadWords(file2)
std::vec<string> v3;
std::set_intersect(begin(v1),end(v1),begin(v2),end(v2),std::back_inserter(v3))
return v3
and done.
Also stop it with the C++03 loops.
for(auto& elem:vec)
std::cout<<elem<<'\n';
is far clearer and less error prone than manually futzing with iterators.