Hash function for sequences of tuples - Duplicate elimination - c++

Question

I want to eliminate duplicates of sequences of tuples. These sequences look like this:
1. (1,1)(2,5,9)(2,3,10)(2,1)
2. (1,2)(3,2,1)(2,5,9)(2,1)
3. (1,1)(2,5,9)(2,3,10)(2,1)
4. (2,1)(2,3,10)(2,5,9)(1,1)
5. (2,1)(2,3,10)(1,1)
6. (1,1)(2,5,9)(2,3,10)(2,2)
The number of entries per tuple varies as does the number of tuple per sequence. Since I have lots of sequences which I ultimately want to deal with in parallel using CUDA, I thought that calculating a hash per sequence would be an efficient way to identify duplicate sequences.
How would such a hash function be implemented?
And: How big is the collision probability of two different sequences producing the same final hash value?
I have two requirements which I am not sure if they can be fulfilled:
a) Can such a hash be calculated on the fly?
I want to avoid the storage of the full sequences, therefore I'd like to do something like this:
h = 0; // init hash
...
h = h + hash(1,1);
...
h = h + hash(2,5,9);
...
h = h + hash(2,3,10)
...
h = h + hash(2,1)
where + is any operator which combines hashes of tuples.
b) Can such a hash be independent of the "direction" of the sequence?
In the above example sequences 1. and 4. consist of the same tuples but the order is reversed, but I like to identify them as duplicates.

Answer 1

For hashing you can use std::hash<std::size_t> or whatever (unsigned) integer type you use. The collision probability is somewhere around 1.0/std::numeric_limits<std::size_t>::max(), which is very small. To make the usability a bit better you can write your own tuple hasher:
namespace hash_tuple
{
std::size_t hash_combine(std::size_t l, std::size_t r) noexcept
{
constexpr static const double phi = 1.6180339887498949025257388711906969547271728515625;
static const double val = std::pow(2ULL, 4ULL * sizeof(std::size_t));
static const std::size_t magic_number = val / phi;
l ^= r + magic_number + (l << 6) + (l >> 2);
return l;
}
template <typename TT>
struct hash
{
std::size_t operator()(TT const& tt) const noexcept
{
return std::hash<TT>()(tt);
}
};
namespace
{
template <class TupleT, std::size_t Index = std::tuple_size<TupleT>::value - 1ULL>
struct HashValueImpl
{
static std::size_t apply(std::size_t seed, TupleT const& tuple) noexcept
{
seed = HashValueImpl<TupleT, Index - 1ULL>::apply(seed, tuple);
seed = hash_combine(seed, std::get<Index>(tuple));
return seed;
}
};
template <class TupleT>
struct HashValueImpl<TupleT, 0ULL>
{
static std::size_t apply(size_t seed, TupleT const& tuple) noexcept
{
seed = hash_combine(seed, std::get<0>(tuple));
return seed;
}
};
}
template <typename ... TT>
struct hash<std::tuple<TT...>>
{
std::size_t operator()(std::tuple<TT...> const& tt) const noexcept
{
std::size_t seed = 0;
seed = HashValueImpl<std::tuple<TT...> >::apply(seed, tt);
return seed;
}
};
}
Thus you can write code like
using hash_tuple::hash;
auto mytuple = std::make_tuple(3, 2, 1, 0);
auto hasher = hash<decltype(mytuple)>();
std::size_t mytuple_hash = hasher(mytuple);
To fulfill your constraint b we need for each tuple 2 hashes, the normal hash and the hash of the reversed tuple.
So at first we need to deal with how to reverse one:
template<typename T, typename TT = typename std::remove_reference<T>::type, size_t... I>
auto reverse_impl(T&& t, std::index_sequence<I...>)
-> std::tuple<typename std::tuple_element<sizeof...(I) - 1 - I, TT>::type...>
{
return std::make_tuple(std::get<sizeof...(I) - 1 - I>(std::forward<T>(t))...);
}
template<typename T, typename TT = typename std::remove_reference<T>::type>
auto reverse(T&& t)
-> decltype(reverse_impl(std::forward<T>(t),
std::make_index_sequence<std::tuple_size<TT>::value>()))
{
return reverse_impl(std::forward<T>(t),
std::make_index_sequence<std::tuple_size<TT>::value>());
}
Then we can calculate our hashes
auto t0 = std::make_tuple(1, 2, 3, 4, 5, 6);
auto t1 = std::make_tuple(6, 5, 4, 3, 2, 1);
using hash_tuple::hash;
auto hasher = hash<decltype(t0)>();
std::size_t t0hash = hasher(t0);
std::size_t t1hash = hasher(t1);
std::size_t t0hsah = hasher(reverse(t0));
std::size_t t1hsah = hasher(reverse(t1));
And if hash_combine(t0hash, t1hash) == hash_combine(t1hsah, t0hsah) you found what you want. You can apply this "inner-tuple-hashing-mechanic" to the hashes of many tuples pretty easily. Play around with this online!