How to fill a stl::map by iterating to maximum value? - c++

Question

I have an std::map, and I use the following method for filling up to the maximum value of the supplied data type. In this case, if K is int then the maximum value is 2,147,483,647. I want my map to have 2,147,483,647 keys with the same value.
The below loop is very inefficient. Is there any method to reduce the time consumption?
for (auto i = keyBegin; i!=numeric_limits<K>::max(); i++) {
m_map[i] = val;
}

Answer 1

The problem with the code above is that you're inserting 2 billion numbers, all at the end of the map. But operator[] has no idea that you'll be inserting a new item there!
std::map::insert(hint_before, value) is what you need. You've got a perfect hint - all values will be inserted directly before m_map.end()

Answer 2

To supplement the existing answers, this is really not a good use of std::map.
Maps are designed for quick lookup in a collection of keys & values, where the keys are "sparse". They're generally implemented as trees, requiring lots of dynamic allocation, tree rebalancing, and the sacrifice of cache locality. This is worth it for the general map use case.
But your keys are far from sparse! You literally have a value for every possible number in the key's type's range. This is what arrays are for.
Use an array and you will benefit from cache, you will benefit from constant-time lookups, and you will not need any dynamic allocation inside the container. You will of course need to dynamically allocate the container itself because it is huge, so, you're looking for std::vector.
And that's only if you really need to precalculate all these values. If you don't necessarily need them all multiple times, consider generating them on-demand instead. Because, regardless how much RAM a contemporary PC can provide you, this feels a bit like an abuse of the technology.

Answer 3

To supplement MSalters's answer, that suggests to use map::insert(hint, {key, value}) construct, I suggest to use a non-default allocator. A specialized allocater can further speed up the insertion twofold. Consider the following trivial allocator:
template <class T>
class chunk_allocator
{
private:
struct node
{
union
{
node *next;
char element[sizeof(T)];
};
};
public:
using value_type = T;
using size_type = std::size_t;
using difference_type = std::ptrdiff_t;
using is_always_equal = std::false_type;
using propagate_on_container_move_assignment = std::true_type;
T*allocate(std::size_t n)
{
if (n > 1)
return reinterpret_cast<T*>(::operator new(sizeof(T) * n));
if (!m_free_head) populate();
node * head = m_free_head;
m_free_head = head->next;
using node_ptr = node*;
head->next.~node_ptr();
return reinterpret_cast<T*>(&head->element);
}
void deallocate(T* p, std::size_t n) {
if (!p)
return;
if (n > 1) {
::operator delete((void*)p);
return;
}
node * new_head = new ((void*)p) node;
new_head->next = m_free_head;
m_free_head = new_head->next;
}
private:
static constexpr unsigned NODES_IN_CHUNK = 1000;
void populate()
{
if (m_free_head) return;
m_chunks.emplace_back();
for (node & entry : m_chunks.back()) {
entry.next = m_free_head;
m_free_head = &entry;
}
}
std::list<std::array<node, NODES_IN_CHUNK>> m_chunks;
node * m_free_head = nullptr;
};
template< class T1, class T2 >
bool operator==( const chunk_allocator<T1>& a, const chunk_allocator<T2>& b )
{
return (void*)&a == (void*)&b;
}
template< class T1, class T2 >
bool operator!=( const chunk_allocator<T1>& a, const chunk_allocator<T2>& b )
{
return !(a == b);
}
And its use:
std::map<int, int, std::less<int>,
chunk_allocator<std::pair<const int, int >>> m_map;
Working 100,000,000 elements takes (on windows with cygwin):
std::allocator: Insertion: 7.987, destruction: 7.238
chunk_allocator: Insertion: 2.745, destruction: 1.357
On Linux the differences are not that big, but still 2x improvements are possible.
Bonus points - the chunk_allocator takes less memory, since it does not use operator new for individual map nodes. Each call to operator new has to maintain memory management bookkeeping, which chunk_allocator does not have to.