Keeping only N smallest elements with STL (with duplicates) - c++

Question

The MVCE below attempts to output only the 5 smallest elements in ascending order from a large incoming input stream of random elements (which contains duplicates).
int main(int argc, char *argv[])
{
std::set<int> s; //EDIT: std::multiset is an answer to Q1
for (int i : {6, 6, 5, 8, 3, 4, 0, 2, 8, 9, 7, 2}) //Billions of elements in reality
{
if ( (s.size() < 5) || (i <= *(--s.end())) ) //Insert only if not full or when the element to be inserted is smaller than the greatest one already in the set
{
if (s.size() >= 5) //Limit the number of smallest elements that are kept. In reality ~8000
s.erase(*(--s.end())); //Erase the largest element
s.insert(i);
}
}
for (int d: s)
std::cout << d << " "; //print the 5 smallest elements in ascending order
std::cout << '\n';
return 0;
}
The output is:
0 2 3 4
The output should be:
0 2 2 3 4
Q1: What must be changed to allow duplicates ?
Q2: How can this code be made faster while not wasting GBs of memory for storing all of the input elements? (the code is way too slow now, as it is).

Answer 1

This sounds like the classical interview question "how to store the smallest N items, without knowledge of the size of the data that will be processed?".
One answer is to use a max-heap of N items, and then adjust the heap (remove the top element, add the new element, heapify) if the subsequent item is less than or equal to the top most item in the heap.
This can be easily done using the C++ library functions std::make_heap, std::pop_heap, and std::push_heap.
Here is an example:
#include <vector>
#include <algorithm>
#include <iostream>
int main(int argc, char *argv[])
{
std::vector<int> s;
for (int i : {6, 6, 5, 8, 3, 4, 0, 2, 8, 9, 7, 2})
{
// add the first 5 elements to the vector
if (s.size() < 5)
{
s.push_back(i);
if ( s.size() == 5 )
// make the max-heap of the 5 elements
std::make_heap(s.begin(), s.end());
continue;
}
// now check if the next element is smaller than the top of the heap
if (s.front() >= i)
{
// remove the front of the heap by placing it at the end of the vector
std::pop_heap(s.begin(), s.end());
// get rid of that item now
s.pop_back();
// add the new item
s.push_back(i);
// heapify
std::push_heap(s.begin(), s.end());
}
}
// sort the heap
std::sort_heap(s.begin(), s.end());
for (int d : s)
std::cout << d << " "; //print the 5 smallest elements in ascending order
std::cout << '\n';
return 0;
}
Output:
0 2 2 3 4
Of course you can make this a function and replace the hard-coded 5 with N.
If there are billions of elements, i.e. many more elements than N, the only thing that will be kept in the heap are N elements.
The max-heap is only manipulated if it is detected that the new item satisfies being one of the smallest N elements, and that is easily done by inspecting the top item in the heap and comparing it with the new item that is being processed.

Answer 2

try this, no need to sort everything. this only sorts until the first 5 elements are on the front of the vector, does not need any additional memory (sort is in-place), vector is fast regarding inserts
#include <iostream>
#include <vector>
#include <algorithm>
int main()
{
std::vector<int> vec{ 6, 6, 5, 8, 3, 4, 0, 2, 8, 9, 7, 2 };
int numElements = 5;
std::partial_sort(vec.begin(), vec.begin() + numElements, vec.end());
for (int i = 0; i < numElements; ++i)
{
std::cout << vec[i] << "\n";
}
return 0;
}
if you do not want to store all inputs, it depends a bit on how you read the input, but the solution will be a bit different. For example read chunks, take smallest 5 of each chunck and in the end just execute once more on the combined "smallest 5" of each chunk.

Answer 3

Q2 ans: Go trought first N elements of the multiset(i am not sure if it is sorted highest to lowest or lowest to highest, so adjust that), and push_back() them to a std::vector.