STL Container For Best Performance? - c++

Question

I have a project where I need to read a text file and record how many occurrences there are of each string, character, or number that is read until the EoF.
I then need to print the top 10 most used words.
For example, the file would contain "This is a test for this project". I would read this and store each word in a container as well as its current count.
Now, we are graded on how our efficient our time complexity is as input grows. So, I need some help on choosing which STL container would be the most efficient.
It seems order is not important, I can forever insert at the end, and I will never have to make insertions. I will, however, have to search through the container for the top 10 most used words. Which STL container has the best time complexity for requirements like this?
Also, if you could explain your reasoning so I know more going forward, that would be great!

Answer 1

Let's assume that you decide to use a std::unordered_map<std::string, int> to get a frequency count of the items. That is a good start, but the other part of the question that needs to be addressed is to get the top 10 items.
Whenever a question asks "get the top N" or "get the smallest N", or similar, there are various methods of getting this information.
One way is to sort the data and get the first N items. Using std::sort or a good sorting routine, that operation should take O(N*log(N)) in time complexity.
The other method is to use a min-heap or max-heap of N items, depending on whether you want to get the top N or bottom N, respectively.
Assume you have working code using the unordered_set to get the frequency count. Here is a routine that uses the STL heap functions to get the top N items. It has not been fully tested, but should demonstrate how to handle the heap.
#include <vector>
#include <algorithm>
#include <iostream>
#include <unordered_map>
void print_top_n(const std::unordered_map<std::string, int>& theMap, size_t n)
{
// This is the heap
std::vector<std::pair<std::string, int>> vHeap;
// This lambda is the predicate to build and perform the heapify
auto heapfn =
[](std::pair<std::string, int>& p1, std::pair<std::string, int>& p2) -> bool
{ return p1.second > p2.second; };
// Go through each entry in the map
for (auto& m : theMap)
{
if (vHeap.size() < n)
{
// Add item to the heap, since we haven't reached n items yet
vHeap.push_back(m);
// if we have reached n items, now is the time to build the heap
if (vHeap.size() == n)
// make the min-heap of the N elements
std::make_heap(vHeap.begin(), vHeap.end(), heapfn);
continue;
}
else
// Heap has been built. Check if the next element is larger than the
// top of the heap
if (vHeap.front().second <= m.second)
{
// adjust the heap
// remove the front of the heap by placing it at the end of the vector
std::pop_heap(vHeap.begin(), vHeap.end(), heapfn);
// get rid of that item now
vHeap.pop_back();
// add the new item
vHeap.push_back(m);
// heapify
std::push_heap(vHeap.begin(), vHeap.end(), heapfn);
}
}
// sort the heap
std::sort_heap(vHeap.begin(), vHeap.end(), heapfn);
// Output the results
for (auto& v : vHeap)
std::cout << v.first << " " << v.second << "\n";
}
int main()
{
std::unordered_map<std::string, int> test = { {"abc", 10},
{ "123",5 },
{ "456",1 },
{ "xyz",15 },
{ "go",8 },
{ "text1",7 },
{ "text2",17 },
{ "text3",27 },
{ "text4",37 },
{ "text5",47 },
{ "text6",9 },
{ "text7",7 },
{ "text8", 22 },
{ "text9", 8 },
{ "text10", 2 } };
print_top_n(test, 10);
}
Output:
text5 47
text4 37
text3 27
text8 22
text2 17
xyz 15
abc 10
text6 9
text9 8
go 8
The advantage of using a heap is that:
The complexity of heapifying is O(log(N)), and not the usual O(N*log(N)) that a sorting routine will give you.
Note that we only need to heapify if we detect that the top item on the min-heap is going to be discarded.
We don't need to store an entire (multi)map of frequency counts to strings in addition to the original map of strings to frequency counts.
The heap will only store N elements, regardless of the number of items are in the original map.

Answer 2

I have used two containers for such task: std::unordered_map<std::string, int> to store words frequencies, and std::map<int, std::string> to track most frequent words.
While updating the firs map with the new word, you also update the second one. To keep it neat, erase the least-frequent word if the size of that second map gets over 10.
UPDATE
In response to the comments below, I did some benchmarking.
First, #PaulMcKenzie - you are correct: to keep the ties, I need std::map<int, std::set<std::string>> (that became obvious as soon as I started to implement this).
Second, #dratenik - turns out you are correct, too. While constantly cleaning up the frequency map keeps it small, the overhead doesn't pay for the benefits. Also, this would only be needed if the client want to see the "running total" (as I was asked for in my project). It makes no sense at all in post-processing, when all words are loaded.
For the test, I used alice29.txt (available online), pre-processed - I removed all punctuation marks and converted to upper case. Here is my code:
int main()
{
auto t1 = std::chrono::high_resolution_clock::now();
std::ifstream src("c:\\temp\\alice29-clean.txt");
std::string str;
std::unordered_map<std::string, int> words;
std::map<int, std::set<std::string>> freq;
int i(0);
while (src >> str)
{
words[str]++;
i++;
}
for (auto& w : words)
{
freq[w.second].insert(w.first);
}
int count(0);
for (auto it = freq.rbegin(); it != freq.rend(); ++it)
{
for (auto& w : it->second)
{
std::cout << w << " - " << it->first << std::endl;
++count;
}
if (count >= 10)
break;
}
auto t2 = std::chrono::high_resolution_clock::now();
std::cout << std::chrono::duration_cast<std::chrono::microseconds>(t2 - t1).count() << std::endl;
return i;
}