c++ concatenate many std::vectors and remove duplicates - c++

Question

I am receiving some integers from an external API (std::vector).
The API usually needs to be called multiple times, so I need to accumulate all the integers from the consecutive API calls to a local vector. At the end every element of the array must be unique (does not need to be sorted).
My code is below (uses the getNextVector to "mock" data and simulate the API call).
The code works, however I want maximum performance for this operation. Is my approach the right one?
#include <vector>
#include <iostream>
#include <iterator>
#include <algorithm>
std::vector<int> getNextVector(int i) {
if ( i == 0 ) {
std::vector<int> v = { 1,2,3 };
return v;
} else if ( i == 1 ) {
std::vector<int> v = { 3,4,5 };
return v;
} else if ( i == 2 ) {
std::vector<int> v = { 5,6,7 };
return v;
} else if ( i == 3 ) {
std::vector<int> v = { 7,8,9 };
return v;
}
}
int count() { return 4; } //we have four vectors
int main(int argc, char** argv) {
std::vector<int> dest;
dest.reserve(20); // we can find this, for simplicity hardcode...
for( int i = 0; i < count(); i++ ) {
std::vector<int> src = getNextVector(i);
dest.insert(
dest.end(),
std::make_move_iterator(src.begin()),
std::make_move_iterator(src.end())
);
}
std::sort(dest.begin(), dest.end());
dest.erase(unique(dest.begin(), dest.end()), dest.end());
/*
std::copy(
dest.begin(),
dest.end(),
std::ostream_iterator<int>(std::cout, "\n")
);
*/
return 0;
}

Answer 1

I think you can store the elements of the vector in a set. If ordering is not needed you can use unordered_set. Simply do the following -
std::unordered_set<int> integers;
for (int i = 0; i < count; i++) {
std::vector<int> src = getNextVector(i);
for (int j = 0; j < src.size(); j++) {
integers.insert(src[i]);
}
}
Or as suggested by #StoryTeller, you can use an appropriate function instead of the loop. For example -
std::unordered_set<int> integers;
for (int i = 0; i < count; i++) {
std::vector<int> src = getNextVector(i);
integers.insert(src.begin(), src.end());
}

Answer 2

My first thought was about "It can be done fast and easly with unordered_set", later I realised that it will not help too much with ints (hash of int is still int, so I don't see here performance increase). So, lastly I decided to benchmark it and my results are:
N = 4 Set implementation 304703 miliseconds
N = 4 Unordered set implementation 404469 miliseconds
N = 4 Vect implementation 91769 miliseconds
N = 20 Set implementation 563320 miliseconds
N = 20 Unordered set implementation 398049 miliseconds
N = 20 Vect implementation 176558 miliseconds
N = 40 Set implementation 569628 miliseconds
N = 40 Unordered set implementation 420496 miliseconds
N = 40 Vect implementation 207368 miliseconds
N = 200 Set implementation 639829 miliseconds
N = 200 Unordered set implementation 456763 miliseconds
N = 200 Vect implementation 245343 miliseconds
N = 2000 Set implementation 728753 miliseconds
N = 2000 Unordered set implementation 499716 miliseconds
N = 2000 Vect implementation 303813 miliseconds
N = 20000 Set implementation 760176 miliseconds
N = 20000 Unordered set implementation 480219 miliseconds
N = 20000 Vect implementation 331941 miliseconds
So, apperently, for samples you gave us here you implementation is the fastest one. This is case when your API returns only few possible vector combinations and number of iterations is small. I've decided to verify what happends when you have more different values via rand() for N > 4 (*). And it keeps it that way. Unordered set is the slowest one (hash calculation cost).
So, to answer your question: benchmark you case on your own - this is the best way to determine which is the fastest one.
(*) Bad randomness of rand() is not bug, but a feature here.
EDIT:
My answer does not provide not says there are no faster algorithms - I've benchmarked STL ones, which at first glance seems to be behave differently than results provide. But for sure there is a way of doing unique concatetion faster, maybe some combination of set of vectors or different container and I hope someone will provide one.
#include <vector>
#include <iostream>
#include <iterator>
#include <algorithm>
#include <set>
#include <unordered_set>
#include <chrono>
std::vector<int> getNextVector(int i) {
if (i == 0) {
std::vector<int> v = { 1,2,3 };
return v;
}
else if (i == 1) {
std::vector<int> v = { 3,4,5 };
return v;
}
else if (i == 2) {
std::vector<int> v = { 5,6,7 };
return v;
}
else if (i == 3) {
std::vector<int> v = { 7,8,9 };
return v;
}
return {rand() % 10000,rand() % 10000,rand() % 10000 };
}
void set_impl(std::set<int>& dest, int N)
{
// dest.reserve(20); // we can find this, for simplicity hardcode...
for (int i = 0; i < N; i++) {
std::vector<int> src = getNextVector(i);
dest.insert(
std::make_move_iterator(src.begin()),
std::make_move_iterator(src.end())
);
}
}
void uset_impl(std::unordered_set<int>& dest, int N)
{
// dest.reserve(20); // we can find this, for simplicity hardcode...
for (int i = 0; i < N; i++) {
std::vector<int> src = getNextVector(i);
dest.insert(
std::make_move_iterator(src.begin()),
std::make_move_iterator(src.end())
);
}
}
void vect_impl(std::vector<int>& dest, int N)
{
for (int i = 0; i < N; i++) {
std::vector<int> src = getNextVector(i);
dest.insert(
dest.end(),
std::make_move_iterator(src.begin()),
std::make_move_iterator(src.end())
);
}
std::sort(dest.begin(), dest.end());
dest.erase(unique(dest.begin(), dest.end()), dest.end());
}
int main(int argc, char** argv) {
for (int N : { 4, 20, 40, 200, 2000, 20000 })
{
const int K = 1000000 / N;
using clock = std::chrono::high_resolution_clock;
std::set<int> sdest;
auto start = clock::now();
for (int i = 0; i < K; i++)
{
sdest.clear();
set_impl(sdest, N);
}
auto set_ms = std::chrono::duration_cast<std::chrono::microseconds>(clock::now() - start).count();
std::unordered_set<int> usdest;
start = clock::now();
for (int i = 0; i < K; i++)
{
usdest.clear();
uset_impl(usdest, N);
}
auto uset_ms = std::chrono::duration_cast<std::chrono::microseconds>(clock::now() - start).count();
std::vector<int> dest;
dest.reserve(N); // we can find this, for simplicity hardcode...
start = clock::now();
for (int i = 0; i < K; i++)
{
dest.clear();
vect_impl(dest, N);
}
auto vect_ms = std::chrono::duration_cast<std::chrono::microseconds>(clock::now() - start).count();
std::cout << "N = " << N << " Set implementation " << set_ms << " miliseconds\n";
std::cout << "N = " << N << " Unordered set implementation " << uset_ms << " miliseconds\n";
std::cout << "N = " << N << " Vect implementation " << vect_ms << " miliseconds\n";
}
return 0;
}

Answer 3

If you want to preserve the order of elements recieved from the external API and they are not sorted then I recommend you create a second vector which you keep sorted. Then do a lower_bound on the sorted vector and if the returned iterator is not the value insert in both the target and sorted vectors (using the returned iterator as the insert position in the sorted vector). Using set or unordered set for integers is likely to be very much slower (probably orders of magnitude slower). If you don't care about the order then use a single sorted vector.
vector<int> sorted;
....
vector<int> src = getNextVector(i);
for( int i : src ) {
auto itr = std::lower_bound( sorted.begin(), sorted.end(), i );
if( *itr != i ) {
sorted.insert(itr, i);
integers.push_back(i);
}
}
If you know the values from each call to getNextVector are unique then you could do something like the following (which might be faster.)
vector<int> sorted;
....
vector<int> src = getNextVector(i);
vector<int> usrc;
for( int i : src ) {
auto itr = std::lower_bound( sorted.begin(), sorted.end(), i );
if( *itr != i ) {
usrc.push_back(i);
integers.push_back(i);
}
}
sorted.insert(sorted.end(), usrc.begin(), usrc.end());
std::sort( sorted.begin(), sorted.end() );

Answer 4

[C++20 solution]
In order to merge multiples containers into a vector,
while removing duplicates, I'd use something simplier like that :
namespace mp {
template <typename T>
concept Container = requires (T value) { // very naive container requirements
typename T::value_type;
std::begin(value), std::end(value);
std::cbegin(value), std::cend(value);
};
}
template <mp::Container ... Ts>
requires
requires { typename std::common_type_t<typename Ts::value_type...>; }
auto merge_uniques(const Ts & ... names) {
using value_type = typename std::common_type_t<typename Ts::value_type...>;
auto value = std::unordered_set<value_type>{};
(value.insert(std::cbegin(names), std::cend(names)), ...);
return std::vector<value_type> {
std::move_iterator{std::begin(value)},
std::move_iterator{std::end(value)}
};
}