How best to fill a vector of vectors (avoiding wasting memory and unnecessary allocations & de-allocations)? - c++

Question

I want to fill a vector of vectors when individual vectors can have different size(), e.g.
std::vector<std::vector<big_data_type> > table;
std::vector<big_data_type> tmp;
for(auto i=0; i!=4242; ++i) {
tmp = make_vector(i); // copy elison; calls new[] only for i=0
table.push_back(tmp); // copy calls new[] each time
}
My main issue is to avoid wasting memory on unused capacity. So my first question is:
Q1 Will the copy (made inside push_back) have capacity() == size() (what I want), or preserve whatever tmp had, or is this implementation dependent / undefined?
I was considering to move the individual vectors into the table
table.push_back(std::move(tmp)); // move
but that would surely preserve the capacity and hence waste memory. Moreover, this doesn't avoid the allocation of each individual vector, it only moves it into another place (inside make_vector instead of push_back).
Q2 I was wondering what difference it makes to omit the variable tmp, resulting in the more elegant looking code (2 instead of 5 lines):
for(auto i=0; i!=4242; ++i)
table.push_back(make_vector(i)); // move!
My initial thought is that this will construct and destruct another temporary at each iteration and hence generate many calls to new[] and delete[] (which will essentially re-use the same memory). However, in addition this will call the moving version of push_back and hence waste memory (see above). Correct?
Q3 Is it possible that the compiler "optimizes" my former code into this latter form and thus uses moving instead of copying (resulting in wasting memory)?
Q4 If I'm correct, it seems to me that all this implies that moving data automatically for temporary objects is a mixed blessing (as it prevents compacting). Is there are any way to explicitly suppress moving in the last code snipped, i.e. something like
for(auto i=0; i!=4242; ++i)
table.push_back(std::copy(make_vector(i))); // don't move!

Answer 1

Q1 Will the copy (made inside push_back) have capacity() == size() (what I want), or preserve whatever tmp had, or is this implementation dependent / undefined?
The standard never sets maximums for capacity, only minimums. That said, most implementations will have capacity() == size() for a fresh vector copy or capacity slightly rounded up to the blocksize of the allocator implementation.
Q2 I was wondering what difference it makes to omit the variable tmp, resulting in the more elegant looking code.
The result is to move into table instead of copying.
Q3 Is it possible that the compiler "optimizes" my former code into this latter form and thus uses moving instead of copying (resulting in wasting memory)?
It's possible but very unlikely. The compiler would have to prove that moving isn't observably different from copying, which is challenging enough that to my knowledge no current compiler tries.
Q4 If I'm correct, it seems to me that all this implies that moving data automatically for temporary objects is a mixed blessing (as it prevents compacting).
Moving is a speed optimization, not necessarily a space optimization. Copying may reduce the space, but it definitely will increase the processing time.
If you want to optimize space, your best bet is to use shrink_to_fit:
std::vector<std::vector<big_data_type> > table;
for(auto i=0; i!=4242; ++i) {
std::vector<big_data_type> tmp = make_vector(i); // copy elison
tmp.shrink_to_fit(); // shrink
table.push_back(std::move(tmp)); // move
}
EDIT: In-depth analysis.
Assumptions:
table will have its space reserved in advance since its size is known, we
thus focus on allocations and deallocations of the vector<big_data_type>s
that are returned from make_vector, stored temporarily in tmp,
and finally in table.
The return value of make_vector(i) may or may not have capacity == size.
This analysis treats make_vector as opaque and ignores any allocations
necessary to build the returned vector.
A default-constructed vector has 0 capacity.
reserve(n) sets capacity to exactly n if and only if n > capacity().
shrink_to_fit() sets capacity == size. It may or may not be implemented
to require a copy of the entire vector contents.
The vector copy constructor sets capacity == size.
std::vector may or may not provide the strong exception guarantee for
copy assignment.
I'll parameterize the analysis on two positive integers: N, the number of
vectors that will be in table at the end of the algorithm (4242 in the OP),
and K: the total number of big_data_type objects contained in all vectors
produced by make_vector during the course of the algorithm.
Your Technique
std::vector<std::vector<big_data_type> > table;
table.reserve(N);
std::vector<big_data_type> tmp;
for(auto i=0; i!=N; ++i) {
tmp = make_vector(i); // #1
table.push_back(tmp); // #2
}
// #3
For C++11
At #1, since tmp is already constructed, RVO/copy elision cannot occur. On
every pass through the loop the return value is assigned to tmp. The
assignment is a move: old data in tmp will be destroyed (except on the
first iteration when tmp is empty) and the contents of the return value from
make_vector moved into tmp with no copying taking place. tmp has capacity == size
if and only if make_vector's return value has that property.
At #2, tmp is copied into table. The newly constructed copy in table has
capacity == size as desired. At #3 tmp presumably leaves scope and its
storage is deallocated.
Total allocations/deallocations: N. All allocations at #2, N - 1 deallocations at #1, and one at #3.
Total copies (of big_data_type objects): K.
For Pre-C++11
At #1, since tmp is already constructed, RVO/copy elision cannot occur. On
every pass through the loop the return value is assigned to tmp. This
assignment requires an allocation and a deallocation if either (a) the
implementation provides the strong guarantee, or (b) tmp is too small to
contain all the elements in the returned vector. In any case the elements must
be copied individually. At the end of the full expression, the temporary object
that holds the return value from make_vector is destroyed, resulting in a
deallocation.
At #2, tmp is copied into table. The newly constructed copy in table has
capacity == size as desired. At #3 tmp presumably leaves scope and its
storage is deallocated.
Total allocation/deallocations: N + 1 to 2 * N. 1 to N allocations at #1, N at #2;
N to 2 * N - 1 deallocations at #1, and one at #3.
Total copies: 2 * K. K at #1 and K at #2.
My Technique (C++11-only)
std::vector<std::vector<big_data_type> > table;
table.reserve(N);
for(auto i=0; i!=N; ++i) {
auto tmp = make_vector(i); // #1
tmp.shrink_to_fit(); // #2
table.emplace_back(std::move(tmp)); // #3
}
At #1 tmp is freshly constructed from the return value of make_vector, so
RVO/copy elision is possible. Even if the implementation of make_vector
impedes RVO, tmp will be move-constructed resulting in no allocations,
deallocations, or copies.
At #2 shrink_to_fit may or may not require a single allocation and
deallocation, depending on whether the return value from make_vector already
has the capacity == size property. If allocation/deallocation occurs, the
elements may or may not be copied depending on quality of implementation.
At #3 the contents of tmp are moved into a freshly constructed vector in
table. No allocations/deallocations/copies are performed.
Total allocations/deallocations: 0 or N, all at #2 if and only if make_vector does not return vectors with capacity == size.
Total copies: 0 or K, all at #2 if and only if shrink_to_fit is implemented as a copy.
If the implementor of make_vector produces vectors with the capacity == size
property and the standard library implements shrink_to_fit optimally, there
are no news/deletes and no copies.
Conclusions
Worst case performance of My Technique is the same as expected case performance
of Your Technique. My technique is conditionally optimal.

Answer 2

Here are some run time tests with a helper type that counts creation, moving and copying:
#include <vector>
#include <iostream>
struct big_data_type {
double state;
big_data_type( double d ):state(d) { ++counter; ++create_counter; }
big_data_type():state(0.) { ++counter; }
big_data_type( big_data_type const& o ): state(o.state) { ++counter; }
big_data_type( big_data_type && o ): state(o.state) { ++move_counter; }
big_data_type& operator=( big_data_type const& o ) {
state = o.state;
++counter;
return *this;
}
big_data_type& operator=( big_data_type && o ) {
state = o.state;
++move_counter;
return *this;
}
static int counter;
static int create_counter;
static int move_counter;
};
int big_data_type::move_counter = 0;
int big_data_type::create_counter = 0;
int big_data_type::counter = 0;
std::vector<big_data_type>& make_vector( int i, std::vector<big_data_type>& tmp ) {
tmp.resize(0);
tmp.reserve(1000);
for( int j = 0; j < 10+i/100; ++j ) {
tmp.emplace_back( 100. - j/10. );
}
return tmp;
}
std::vector<big_data_type> make_vector2( int i ) {
std::vector<big_data_type> tmp;
tmp.resize(0);
tmp.reserve(1000);
for( int j = 0; j < 10+i/100; ++j ) {
tmp.emplace_back( 100. - j/10. );
}
return tmp;
}
enum option { a, b, c, d, e };
void test(option op) {
std::vector<std::vector<big_data_type> > table;
std::vector<big_data_type> tmp;
for(int i=0; i!=10; ++i) {
switch(op) {
case a:
table.emplace_back(make_vector(i, tmp));
break;
case b:
tmp = make_vector2(i);
table.emplace_back(tmp);
break;
case c:
tmp = make_vector2(i);
table.emplace_back(std::move(tmp));
break;
case d:
table.emplace_back(make_vector2(i));
break;
case e:
std::vector<big_data_type> result;
make_vector(i, tmp);
result.reserve( tmp.size() );
result.insert( result.end(), std::make_move_iterator( tmp.begin() ),std::make_move_iterator( tmp.end() ) );
table.emplace_back(std::move(result));
break;
}
}
std::cout << "Big data copied or created:" << big_data_type::counter << "\n";
big_data_type::counter = 0;
std::cout << "Big data created:" << big_data_type::create_counter << "\n";
big_data_type::create_counter = 0;
std::cout << "Big data moved:" << big_data_type::move_counter << "\n";
big_data_type::move_counter = 0;
std::size_t cap = 0;
for (auto&& v:table)
cap += v.capacity();
std::cout << "Total capacity at end:" << cap << "\n";
}
int main() {
std::cout << "A\n";
test(a);
std::cout << "B\n";
test(b);
std::cout << "C\n";
test(c);
std::cout << "D\n";
test(d);
std::cout << "E\n";
test(e);
}
Live example
Output:
+ g++ -O4 -Wall -pedantic -pthread -std=c++11 main.cpp
+ ./a.out
A
Big data copied or created:200
Big data created:100
Big data moved:0
Total capacity at end:100
B
Big data copied or created:200
Big data created:100
Big data moved:0
Total capacity at end:100
C
Big data copied or created:100
Big data created:100
Big data moved:0
Total capacity at end:10000
D
Big data copied or created:100
Big data created:100
Big data moved:0
Total capacity at end:10000
E
Big data copied or created:100
Big data created:100
Big data moved:100
Total capacity at end:100
E is an example when your big data can be moved, which often doesn't work.
created refers to only explicitly created data (ie, from the double) -- data "created on purpose". Copied or created refers to any time that any big data is duplicated in a way that the source big data cannot be "discarded". And moved refers to any situation where big data is moved in a way that the source big data can be "discarded".
Case a and b, which are identical in result, are probably want you are looking for. Note the explicit use of the tmp vector as an argument to make_vector: elision won't let you reuse the buffer, you have to be explicit about it.

Answer 3

In addition to Casey's post, I have the following remarks.
As jrok said in a comment here, shrink_to_fit is not guaranteed to do anything. However, if shrink_to_fit allocates the memory for exaclty size() number of elements, copy/move the elements, and deallocate the original buffer, then this is exactly what the OP's asked.
My exact answer to Q4, that is,
Is there are any way to explicitly suppress moving in the last code snipped [...]?
is: Yes, you can do
for(auto i=0; i!=4242; ++i)
table.push_back(static_cast<const std::vector<big_data_type>&>(make_vector(i)));
The copy function suggested by the OP could be written as follow.
template <typename T>
const T& copy(const T& x) {
return x;
}
and the code becomes
for(auto i=0; i!=4242; ++i)
table.push_back(copy(make_vector(i)));
But, honestly, I don't think this is a sensible thing to do.
The best place to make each element v of table such that v.size() == v.capacity() is in make_vector(), if possible. (As Casey said the standard doesn't set any upper bound on capacity.) Then moving the result of make_vector() to table would be optimal in both senses (memory and speed). The OP's snipped should probably take care of table.size() instead.
In summary, the standard doesn't provide any way to force capacity to match size. There was a (sensible, IMHO) suggestion by Jon Kalb to make std::vector::shrink_to_fit at least as efficient (with respect to memory usage) as the shrink_to_fit idiom (which also doesn't guarantee anything). However, some members of the committee were not very keen on it and suggested that people should rather complain with their vendors or implement their own containers and allocation functions.

Answer 4

The vector of vectors construct brings a lot of unnecessary overhead in cases where data is only ever added at the end of the top level vector (as appears to be the case here).
The main issue is the separate buffer allocations and management for each individual entry in the top level vector.
It's much better to concatenate all the sub-entries together into a single contiguous buffer, if possible, with a separate buffer to index into this for each top level entry.
See this article (on my blog), for more discussion about this, and for an example implementation of a 'collapsed vector vector' class to wrap this kind of indexed buffer setup up in a generic container object.
As I said before, this only applies if data only ever gets added at the end of your data structure, i.e. you don't come back later and push entries into arbitrary top level sub vectors, but in cases where this technique applied it can be quite a significant optimisation..

Answer 5

In general, if you want capacity to equal size you can use vector::shrink_to_fit()
http://www.cplusplus.com/reference/vector/vector/shrink_to_fit/

Answer 6

Okay, I think I learned a bit, but couldn't really find a complete answer. So let's first clarify the task:
We have a function filling in a vector. In order to avoid arguments whether copy elision is possible or not, let's just assume that its definition is
void fill_vector(std::vector<big_data_type>& v, int i)
{
v.clear();
v.reserve(large_number); // allocates unless v.capacity() >= large_number
for(int done=0,k=0; k<large_number && !done; ++k)
v.push_back(get_more_big_data(i,done));
// v.capacity() == v.size() is highly unlikely at this point.
}
Further we want to fill the table
std::vector<std::vector<big_data_type>> table;
with N entries, each generated by fill_vector() in such a way that (1) minimises memory usage in the table but (2) avoids unnecessary allocations/de-allocations. In a simple C code, there would be N+2 allocations and 1 de-allocation and only the total number K of big_data_type actually provided by fill_vector() would be allocated. We should not need more with C++. Here is a possible C++ answer
table.reserve(N); // allocates enough space for N vectors
size_t K=0; // count big_data_types in table
std::vector<big_data_type> tmp;
for(int n=0; n!=N; ++n) {
fill_vector(tmp,i); // allocates at first iteration only
K += tmp.size();
table.push_back(tmp.begin(),tmp.end()); // allocates tmp.size() big_data_type
}
// de-allocates tmp
Thus we have N+2 allocations and 1 de-allocation as required and no memory wasted (not more than K big_data_type allocated in table). The push_back calls a constructor of std::vector (without passing information about the capacity of tmp) and implies a copy of each big_data_type. (If big_data_type can be moved, we could use make_move_iterator(tmp.begin()) etc.)
Note that no matter how we code this, we must do at least N+1 allocations (for table and each of its elements). This implies that usage of shrink_to_fit cannot be helpful, because at best it does one allocation and one de-allocation (unless capacity==size which we don't expect to happen with any probability), cancelling each other (so that the allocation cannot contribute towards the required sum of N+1). This is why some other answers were unacceptable.