Related
I think this is probably a trivial problem to solve but I have been struggling with this for past few days.
I have the following vector: v = [7,3,16,4,2,1]. I was able to implement with some help from google simple minheap algorithm to get the smallest element in each iteration. After extraction of the minimum element, I need to decrease the values of some of the elements and then bubble them up.
The issue I am having is that I want find the elements whose value has to be reduced in the heap in constant time, then reduce that value and then bubble it up.
After the heapify operation, the heap_vector v_h looks like this: v_h = [1,2,7,4,3,16]. When I remove the min element 1, then the heap vector becomes, [2,3,7,4,16]. But before we do the swap and bubble up, say I want to change the values of 7 to 4, 16 to 4 and 4 to 3.5 . But I am not sure where they will be in the heap. The indices of values of the elements that have to be decreased will be given with respect to the original vector v. I figured out that I need to have an auxiliary data structure that can keep track of the heap indices in relation to the original order of the elements (the heap index vector should look like h_iv = [2,4,5,3,1,0] after all the elements have been inserted into the minheap. And whenever an element is deleted from the minheap, the heap_index should be -1. I created a vector to try to update the heap indices whenever there is a change but I am unable to do it.
I am pasting my work here and also at https://onlinegdb.com/SJR4LqQO4
Some of the work I had tried is commented out. I am unable to map the heap indices when there is a swap in the bubble up or bubble down operations. I will be very grateful to anyone who can lead me in a direction to solve my problem. Please also let me know if I have to rethink some of my logic.
The .hpp file
#ifndef minheap_hpp
#define minheap_hpp
#include <stdio.h>
// #include "helper.h"
#include <vector>
class minheap
{
public:
std::vector<int> vect;
std::vector<int> heap_index;
void bubble_down(int index);
void bubble_up(int index);
void Heapify();
public:
minheap(const std::vector<int>& input_vector);
minheap();
void insert(int value);
int get_min();
void delete_min();
void print_heap_vector();
};
#endif /* minheap_hpp */
The .cpp file
#include "minheap.hpp"
minheap::minheap(const std::vector<int>& input_vector) : vect(input_vector)
{
Heapify();
}
void minheap::Heapify()
{
int length = static_cast<int>(vect.size());
// auto start = 0;
// for (auto i = 0; i < vect.size(); i++){
// heap_index.push_back(start);
// start++;
// }
for(int i=length/2-1; i>=0; --i)
{
bubble_down(i);
}
}
void minheap::bubble_down(int index)
{
int length = static_cast<int>(vect.size());
int leftChildIndex = 2*index + 1;
int rightChildIndex = 2*index + 2;
if(leftChildIndex >= length){
return;
}
int minIndex = index;
if(vect[index] > vect[leftChildIndex])
{
minIndex = leftChildIndex;
}
if((rightChildIndex < length) && (vect[minIndex] > vect[rightChildIndex]))
{
minIndex = rightChildIndex;
}
if(minIndex != index)
{
std::swap(vect[index], vect[minIndex]);
// std::cout << "swap " << index << " - " << minIndex << "\n";
// auto a = heap_index[heap_index[index]];
// auto b = heap_index[heap_index[minIndex]];
// heap_index[a] = b;
// heap_index[b] = a;
// print_vector(heap_index);
bubble_down(minIndex);
}
}
void minheap::bubble_up(int index)
{
if(index == 0)
return;
int par_index = (index-1)/2;
if(vect[par_index] > vect[index])
{
std::swap(vect[index], vect[par_index]);
bubble_up(par_index);
}
}
void minheap::insert(int value)
{
int length = static_cast<int>(vect.size());
vect.push_back(value);
bubble_up(length);
}
int minheap::get_min()
{
return vect[0];
}
void minheap::delete_min()
{
int length = static_cast<int>(vect.size());
if(length == 0)
{
return;
}
vect[0] = vect[length-1];
vect.pop_back();
bubble_down(0);
}
void minheap::print_heap_vector(){
// print_vector(vect);
}
and the main file
#include <iostream>
#include <iostream>
#include "minheap.hpp"
int main(int argc, const char * argv[]) {
std::vector<int> vec {7, 3, 16, 4, 2, 1};
minheap mh(vec);
// mh.print_heap_vector();
for(int i=0; i<3; ++i)
{
auto a = mh.get_min();
mh.delete_min();
// mh.print_heap_vector();
std::cout << a << "\n";
}
// std::cout << "\n";
return 0;
}
"I want to change the values of 7 to 4, 16 to 4 and 4 to 3.5 . But I am not sure where they will be in the heap. The indices of values of the elements that have to be decreased will be given with respect to the original vector v. ... Please also let me know if I have to rethink some of my logic."
Rather than manipulate the values inside the heap, I would suggest keeping the values that need changing inside a vector (possibly v itself). The heap could be based on elements that are a struct (or class) that holds an index into the corresponding position in the vector with the values, rather than hold the (changing) value itself.
The struct (or class) would implement an operator< function that compares the values retrieved from the two vector locations for the respective index values. So, instead of storing the comparison value in the heap elements and comparing a < b, you would store index positions i and j and so on and compare v[i] < v[j] for the purpose of heap ordering.
In this way, the positions of the numerical values you need to update will never change from their original positions. The position information will never go stale (as I understand it from your description).
Of course, when you make changes to those stored values in the vector, that could easily invalidate any ordering that might have existed in the heap itself. As I understand your description, that much was necessarily true in any case. Therefore, depending on how you change the values, you might need to do a fresh make_heap to restore proper heap ordering. (That isn't clear, since it depends on whether your intended changes violate heap assumptions, but it would be a safe thing to assume unless there are strong assurances otherwise.)
I think the rest is pretty straight forward. You can still operate the heap as you intended before. For ease you might even give the struct (or class) a lookup function to return the current value at it's corresponding position in the vector, if you need that (rather than the index) as you pop out minimum values.
p.s. Here is a variation on the same idea.
In the original version above, one would likely need to also store a pointer to the location of the vector that held the vector of values, possibly as a shared static pointer of that struct (or class) so that all the members could dereference the pointer to that vector in combination with the index values to look up the particular member associated with that element.
If you prefer, instead of storing that shared vector pointer and an index in each member, each struct (or class) instance could more simply store a pointer (or iterator) directly to the corresponding value's location. If the values are integers, the heap element struct's member value could be int pointer. While each pointer might be larger than an index value, this does have the advantage that it eliminates any assumption about the data structure that holds the compared values and it is even simpler/faster to dereference vs. lookup with an index into the vector. (Both are constant time.)
One caution: In this alternate approach, the pointer values would be invalidated if you were to cause the vector's storage positions to change, e.g. by pushing in new values and expanding it in a way that forces it to reallocate it's space. I'm assuming you only need to change values, not expand the number of values after you've begun to use the heap. But if you did need to do that, that would be one reason to prefer index values, since they remain valid after expanding the vector (unlike pointers).
p.p.s. This technique is also valuable when the objects that you want to compare in the heap are large. Rather than have the heap perform many copy operations on large objects as it reorders the positions of the heap elements, by storing only pointers (or index values) the copying is much more efficient. In fact, this makes it possible to use heaps on objects that you might not want to copy at all.
Here is a quick idea of one version of the comparison function (with some class context now added).
class YourHeapElementClassName
{
public:
// constructor
explicit YourHeapElementClassName(theTypeOfYourComparableValueOrObject & val)
: m_valPointer(&val)
{
}
bool operator<(const YourHeapElementClassName & other) const
{
return *m_valPointer < *(other.m_valPointer);
}
...
private:
theTypeOfYourComparableValueOrObject * m_valPointer;
}; // YourHeapElementClassName
// and later instead of making a heap of int or double,
// you make a heap of YourHeapElementClassName objects
// that you initialize so each points to a value in v
// by using the constructor above with each v member.
// If you (probably) don't need to change the v values
// through these heap objects, the member value could be
// a pointer to a const value and the constructor could
// have a const reference argument for the original value.
If you had need to do this with different types of values or objects, the pointer approach could be implemented with a template that generalizes on the type of value or object and holds a pointer to that general type.
This question already has an answer here:
C++ sizeof Vector is 24?
(1 answer)
Closed 4 years ago.
here is the class
class A{
public:
vector<int> vvv1{1};
};
class B{
public:
vector<int> vvv2{1,2,3,4,5};
};
and the main
int main(){
A a;/*sizeof(a) == 24;*/
B b;/*sizeof(b) == 24;*/
return 0;
}
why the size of a and the size of b are both 24?
sizeof() gives you the number of bytes in memory that the object is occupying. The class std::vector is a container that has its own member variables to manage the internal array that it is representing, and they are counted as well as part of the size. Both a and b in your case are too small in number of elements to make it reallocate its internal array to hold more than what it initially uses to hold a single element.
For illustration, my compiler returns 32 for both these cases:
#include <vector>
int main()
{
std::vector<int> a{ 1 };
std::vector<int> b{ 1,2,3,4,5 };
int sizeA = sizeof(a); // Returns 32
int sizeB = sizeof(b); // Returns 32
return 0;
}
The size of the contents of the vector does not change the size of the class. A vector object will always take up 24 bytes with the std lib implementation that you’re currently using. To be a dynamic array, the vector allocates memory on the fly to hold its contents. What’s actually in the vector class is the capacity of the vector, it’s current size, and a pointer to the data. Class sizes are always static
I have defined a struct which contains a vector of integer. Then I insert 10 integers in the vector and check for the size of struct. But I see no difference.
Here is my code:
struct data
{
vector<int> points;
}
int main()
{
data d;
cout << sizeof(d) << endl;
for (int i=0; i< 10; ++i)
d.points.push_back(i)
cout << sizeof(d) << endl;
In both the cases I am getting the same result : 16
Why is it so? Shouldn't the size of struct grow?
A vector will store its elements in dynamically allocated memory (on the heap). Internally, this might be represented as:
T* elems; // Pointer memory.
size_t count; // Current number of elements.
size_t capacity; // Total number of elements that can be held.
so the sizeof(std::vector) is unaffected by the number of elements it contains as it calculating the sizeof its contained members (in this simple example roughly sizeof(T*) + (2 * sizeof(size_t))).
The sizeof operator is a compile time operation that gives you the size of the data structure used to maintain the container, not including the size of the stored elements.
While this might not seem too intuitive at first, consider that when you use a std::vector you are using a small amount of local storage (where the std::vector is created) which maintains pointers to a different region holding the actual data. When the vector grows the data block will grow, but the control structure is still the same.
The fact that sizeof will not change during it's lifetime is important, as it is the only way of making sure that the compiler can allocate space for points inside data without interfering with other possible members:
struct data2 {
int x;
std::vector<int> points;
int y;
};
If the size of the object (std::vector in this case) was allowed to grow it would expand over the space allocated for y breaking any code that might depend on its location:
data2 d;
int *p = &d.y;
d.points.push_back(5);
// does `p` still point to `&d.y`? or did the vector grow over `y`?
I have an STL vector My_Partition_Vector of Partition objects, defined as
struct Partition // the event log data structure
{
int key;
std::vector<std::vector<char> > partitions;
float modularity;
};
The actual nested structure of Partition.partitions varies from object to object but in the total number of chars stored in Partition.partitions is always 16.
I assumed therefore that the total size of the object should be more or less 24 bytes (16 + 4 + 4). However for every 100,000 items I add to My_Partition_Vector, memory consumption (found using ps -aux) increases by around 20 MB indicating around 209 bytes for each Partition Object.
This is a nearly 9 Fold increase!? Where is all this extra memory usage coming from? Some kind of padding in the STL vector, or the struct? How can I resolve this (and stop it reaching into swap)?
For one thing std::vector models a dynamic array so if you know that you'll always have 16 chars in partitions using std::vector is overkill. Use a good old C style array/matrix, boost::array or boost::multi_array.
To reduce the number of re-allocations needed for inserting/adding elements due to it's memory layout constrains std::vector is allowed to preallocate memory for a certain number of elements upfront (and it's capacity() member function will tell you how much).
While I think he may be overstating the situation just a tad, I'm in general agreement with DeadMG's conclusion that what you're doing is asking for trouble.
Although I'm generally the one looking at (whatever mess somebody has made) and saying "don't do that, just use a vector", this case might well be an exception. You're creating a huge number of objects that should be tiny. Unfortunately, a vector typically looks something like this:
template <class T>
class vector {
T *data;
size_t allocated;
size_t valid;
public:
// ...
};
On a typical 32-bit machine, that's twelve bytes already. Since you're using a vector<vector<char> >, you're going to have 12 bytes for the outer vector, plus twelve more for each vector it holds. Then, when you actually store any data in your vectors, each of those needs to allocate a block of memory from the free store. Depending on how your free store is implemented, you'll typically have a minimum block size -- frequently 32 or even 64 bytes. Worse, the heap typically has some overhead of its own, so it'll add some more memory onto each block, for its own book-keeping (e.g., it might use a linked list of blocks, adding another pointer worth of data to each allocation).
Just for grins, let's assume you average four vectors of four bytes apiece, and that your heap manager has a 32-byte minimum block size and one extra pointer (or int) for its bookkeeping (giving a real minimum of 36 bytes per block). Multiplying that out, I get 204 bytes apiece -- close enough to your 209 to believe that's reasonably close to what you're dealing with.
The question at that point is how to deal with the problem. One possibility is to try to work behind the scenes. All the containers in the standard library use allocators to get their memory. While they default allocator gets memory directly from the free store, you can substitute a different one if you choose. If you do some looking around, you can find any number of alternative allocators, many/most of which are to help with exactly the situation you're in -- reducing wasted memory when allocating lots of small objects. A couple to look at would be the Boost Pool Allocator and the Loki small object allocator.
Another possibility (that can be combined with the first) would be to quit using a vector<vector<char> > at all, and replace it with something like:
char partitions[16];
struct parts {
int part0 : 4;
int part1 : 4;
int part2 : 4;
int part3 : 4;
int part4 : 4;
int part5 : 4;
int part6 : 4
int part7 : 4;
};
For the moment, I'm assuming a maximum of 8 partitions -- if it could be 16, you can add more to parts. This should probably reduce memory usage quite a bit more, but (as-is) will affect your other code. You could also wrap this up into a small class of its own that provides 2D-style addressing to minimize impact on the rest of your code.
If you store a near constant amount of objects, then I suggest to use a 2-dimensional array.
The most likely reason for the memory consumption is debug data. STL implementations usually store A LOT of debug data. Never profile an application with debug flags on.
...This is a bit of a side conversation, but boost::multi_array was suggested as an alternative to the OP's use of nested vectors. My finding was that multi_array was using a similar amount of memory when applied to the OP's operating parameters.
I derived this code from the example at Boost.MultiArray. On my machine, this showed multi_array using about 10x more memory than ideally required assuming that the 16 bytes are arranged in a simple rectangular geometry.
To evaluate the memory usage, I checked the system monitor while the program was running and I compiled with
( export CXXFLAGS="-Wall -DNDEBUG -O3" ; make main && ./main )
Here's the code...
#include <iostream>
#include <vector>
#include "boost/multi_array.hpp"
#include <tr1/array>
#include <cassert>
#define USE_CUSTOM_ARRAY 0 // compare memory usage of my custom array vs. boost::multi_array
using std::cerr;
using std::vector;
#ifdef USE_CUSTOM_ARRAY
template< typename T, int YSIZE, int XSIZE >
class array_2D
{
std::tr1::array<char,YSIZE*XSIZE> data;
public:
T & operator () ( int y, int x ) { return data[y*XSIZE+x]; } // preferred accessor (avoid pointers)
T * operator [] ( int index ) { return &data[index*XSIZE]; } // alternative accessor (mimics boost::multi_array syntax)
};
#endif
int main ()
{
int COUNT = 1024*1024;
#if USE_CUSTOM_ARRAY
vector< array_2D<char,4,4> > A( COUNT );
typedef int index;
#else
typedef boost::multi_array<char,2> array_type;
typedef array_type::index index;
vector<array_type> A( COUNT, array_type(boost::extents[4][4]) );
#endif
// Assign values to the elements
int values = 0;
for ( int n=0; n<COUNT; n++ )
for(index i = 0; i != 4; ++i)
for(index j = 0; j != 4; ++j)
A[n][i][j] = values++;
// Verify values
int verify = 0;
for ( int n=0; n<COUNT; n++ )
for(index i = 0; i != 4; ++i)
for(index j = 0; j != 4; ++j)
{
assert( A[n][i][j] == (char)((verify++)&0xFF) );
#if USE_CUSTOM_ARRAY
assert( A[n][i][j] == A[n](i,j) ); // testing accessors
#endif
}
cerr <<"spinning...\n";
while ( 1 ) {} // wait here (so you can check memory usage in the system monitor)
return 0;
}
On my system, sizeof(vector) is 24. This probably corresponds to 3 8-byte members: capacity, size, and pointer. Additionally, you need to consider the actual allocations which would be between 1 and 16 bytes (plus allocation overhead) for the inner vector and between 24 and 384 bytes for the outer vector ( sizeof(vector) * partitions.capacity() ).
I wrote a program to sum this up...
for ( int Y=1; Y<=16; Y++ )
{
const int X = 16/Y;
if ( X*Y != 16 ) continue; // ignore imperfect geometries
Partition a;
a.partitions = vector< vector<char> >( Y, vector<char>(X) );
int sum = sizeof(a); // main structure
sum += sizeof(vector<char>) * a.partitions.capacity(); // outer vector
for ( int i=0; i<(int)a.partitions.size(); i++ )
sum += sizeof(char) * a.partitions[i].capacity(); // inner vector
cerr <<"X="<<X<<", Y="<<Y<<", size = "<<sum<<"\n";
}
The results show how much memory (not including allocation overhead) is need for each simple geometry...
X=16, Y=1, size = 80
X=8, Y=2, size = 104
X=4, Y=4, size = 152
X=2, Y=8, size = 248
X=1, Y=16, size = 440
Look at the how the "sum" is calculated to see what all of the components are.
The results posted are based on my 64-bit architecture. If you have a 32-bit architecture the sizes would be almost half as much -- but still a lot more than what you had expected.
In conclusion, std::vector<> is not very space efficient for doing a whole bunch of very small allocations. If your application is required to be efficient, then you should use a different container.
My approach to solving this would probably be to allocate the 16 chars with
std::tr1::array<char,16>
and wrap that with a custom class that maps 2D coordinates onto the array allocation.
Below is a very crude way of doing this, just as an example to get you started. You would have to change this to meet your specific needs -- especially the ability to specify the geometry dynamically.
template< typename T, int YSIZE, int XSIZE >
class array_2D
{
std::tr1::array<char,YSIZE*XSIZE> data;
public:
T & operator () ( int y, int x ) { return data[y*XSIZE+x]; } // preferred accessor (avoid pointers)
T * operator [] ( int index ) { return &data[index*XSIZE]; } // alternative accessor (mimics boost::multi_array syntax)
};
16 bytes is a complete and total waste. You're storing a hell of a lot of data about very small objects. A vector of vector is the wrong solution to use. You should log sizeof(vector) - it's not insignificant, as it performs a substantial function. On my compiler, sizeof(vector) is 20. So each Partition is 4 + 4 + 16 + 20 + 20*number of inner partitions + memory overheads like the vectors not being the perfect size.
You're only storing 16 bytes of data, and wasting ridiculous amounts of memory allocating them in the most segregated, highest overhead way you could possibly think of. The vector doesn't use a lot of memory - you have a terrible design.
I'm using C++, and I have the following structures:
struct ArrayOfThese {
int a;
int b;
};
struct DataPoint {
int a;
int b;
int c;
};
In memory, I want to have 1 or more ArrayOfThese elements at the end of each DataPoint. There are not always the same number of ArrayOfThese elements per DataPoint.
Because I have a ridiculous number of DataPoints to assemble and then stream across a network, I want all my DataPoints and their ArrayOfThese elements to be contiguous. Wasting space for a fixed number of the ArrayOfThese elements is unacceptable.
In C, I would have made an element at the end of DataPoint that was declared as ArrayOfThese d[0];, allocated a DataPoint plus enough extra bytes for however many ArrayOfThese elements I had, and used the dummy array to index into them. (Of course, the number of ArrayOfThese elements would have to be in a field of DataPoint.)
In C++, is using placement new and the same 0-length array hack the correct approach? If so, does placement new guarantee that subsequent calls to new from the same memory pool will allocate contiguously?
Since you are dealing with plain structures that have no constructors, you could revert to C memory management:
void *ptr = malloc(sizeof(DataPoint) + n * sizeof(ArrayOfThese));
DataPoint *dp = reinterpret_cast<DataPoint *>(ptr));
ArrayOfThese *aotp = reinterpet_cast<ArrayOfThese *>(reintepret_cast<char *>(ptr) + sizeof(DataPoint));
Since your structs are PODs you might as well do it just as you would in C. The only thing you'll need is a cast. Assuming n is the number of things to allocate:
DataPoint *p=static_cast<DataPoint *>(malloc(sizeof(DataPoint)+n*sizeof(ArrayOfThese)));
Placement new does come into this sort of thing, if your objects have a a non-trivial constructor. It guarantees nothing about any allocations though, for it does no allocating itself and requires the memory to have been already allocated somehow. Instead, it treats the block of memory passed in as space for the as-yet-unconstructed object, then calls the right constructor to construct it. If you were to use it, the code might go like this. Assume DataPoint has the ArrayOfThese arr[0] member you suggest:
void *p=malloc(sizeof(DataPoint)+n*sizeof(ArrayOfThese));
DataPoint *dp=new(p) DataPoint;
for(size_t i=0;i<n;++i)
new(&dp->arr[i]) ArrayOfThese;
What gets constructed must get destructed so if you do this you should sort out the call of the destructor too.
(Personally I recommend using PODs in this sort of situation, because it removes any need to call constructors and destructors, but this sort of thing can be done reasonably safely if you are careful.)
As Adrian said in his answer, what you do in memory doesn't have to be the same as what you stream over the network. In fact, it might even be good to clearly divide this, because having a communication protocol relying on your data being designed in a specific way makes huge problem if you later need to refactor your data.
The C++ way to store an arbitrary number of elements contiguously is of course to std::vector. Since you didn't even consider this, I assume that there's something that makes this undesirable. (Do you only have small numbers of ArrayOfThese and fear the space overhead associated with std::vector?)
While the trick with over-allocating a zero-length array probably isn't guaranteed to work and might, technically, invoke the dreaded undefined behavior, it's a widely spread one. What platform are you on? On Windows, this is done in the Windows API, so it's hard to imagine a vendor shipping a C++ compiler which wouldn't support this.
If there's a limited number of possible ArrayOfThese element counts, you could also use fnieto's trick to specify those few numbers and then new one of the resulting template instances, depending on the run-time number:
struct DataPoint {
int a;
int b;
int c;
};
template <std::size_t sz>
struct DataPointWithArray : DataPoint {
ArrayOfThese array[sz];
};
DataPoint* create(std::size_t n)
{
switch(n) {
case 1: return new DataPointWithArray[1];
case 2: return new DataPointWithArray[2];
case 5: return new DataPointWithArray[5];
case 7: return new DataPointWithArray[7];
case 27: return new DataPointWithArray[27];
default: assert(false);
}
return NULL;
}
Prior to C++0X, the language had no memory model to speak of. And with the new standard, I don't recall any talk of guarantees of contiguity.
Regarding this particular question, it sounds as if what you want is a pool allocator, many examples of which exist. Consider, for instance, Modern C++ Design, by Alexandrescu. The small object allocator discussion is what you should look at.
I think boost::variant might accomplish this. I haven't had an opportunity to use it, but I believe it's a wrapper around unions, and so a std::vector of them should be contiguous, but of course each item will take up the larger of the two sizes, you can't have a vector with differently-sized elements.
Take a look at the comparison of boost::variant and boost::any.
If you want the offset of each element to be dependent on the composition of the previous elements, you will have to write your own allocator and accessors.
Seems like it would be simpler to allocate an array of pointers and work with that rather than using placement new. That way you could just reallocate the whole array to the new size with little runtime cost. Also if you use placement new, you have to explicitly call destructors, which means mixing non-placement and placement in a single array is dangerous. Read http://www.parashift.com/c++-faq-lite/dtors.html before you do anything.
don't confuse data organisation inside your program and data organisation for serialization: they do not have the same goal.
for streaming across a network, you have to consider both side of the channel, the sending and the receiving side: how does the receiving side differentiate between a DataPoint and an ArrayOfThese ? how does the receiving side know how many ArrayOfThese are appended after a DataPoint ? (also to consider: what is the byte ordering of each side ? does data types have the same size in memory ?)
personally, i think you need a different structure for streaming your data, in which you add the number of DataPoint you are sending as well as the number of ArrayOfThese after each DataPoint. i would also not care about the way data is already organized in my program and reorganize/reformat to suit my protocol and not my program. after that writing a function for sending and another for receiving is not a big deal.
Why not have DataPoint contain a variable-length array of ArrayOfThese items? This will work in C or C++. There are some concerns if either struct contains non-primitive types
But use free() rather than delete on the result:
struct ArrayOfThese {
int a;
int b;
};
struct DataPoint {
int a;
int b;
int c;
int length;
ArrayOfThese those[0];
};
DataPoint* allocDP(int a, int b, int c, size_t length)
{
// There might be alignment issues, but not for most compilers:
size_t sz = sizeof(DataPoint) + length * sizeof(ArrayOfThese);
DataPoint dp = (DataPoint*)calloc( sz );
// (Check for out of memory)
dp->a = a; dp->b = b; tp->c = c; dp->length = length;
}
Then you can use it "normally" in a loop where the DataPoint knows its length:
DataPoint *dp = allocDP( 5, 8, 3, 20 );
for(int i=0; i < dp->length; ++i)
{
// Initialize or access: dp->those[i]
}
Could you make those into classes with the same superclass and then use your favourite stl container of choice, using the superclass as the template?
Two questions: Is the similarity between ArrayOfThese and DataPoint real, or a simplification for posting? I.e. is the real difference just one int (or some arbitrary number of the same type of items)?
Is the number of ArrayOfThese associated with a particular DataPoint known at compile time?
If the first is true, I'd think hard about simply allocating an array of as many items as necessary for one DataPoint+N ArrayOfThese. I'd then build a quick bit of code to overload operator[] for that to return item N+3, and overload a(), b() and c() to return the first three items.
If the second is true, I was going to suggest essentially what I see fnieto has just posted, so I won't go into more detail.
As far as placement new goes, it doesn't really guarantee anything about allocation -- in fact, the whole idea about placement new is that it's completely unrelated to memory allocation. Rather, it allows you to create an object at an arbitrary address (subject to alignment restrictions) in a block of memory that's already allocated.
Here's the code I ended up writing:
#include <iostream>
#include <cstdlib>
#include <cassert>
using namespace std;
struct ArrayOfThese {
int e;
int f;
};
struct DataPoint {
int a;
int b;
int c;
int numDPars;
ArrayOfThese d[0];
DataPoint(int numDPars) : numDPars(numDPars) {}
DataPoint* next() {
return reinterpret_cast<DataPoint*>(reinterpret_cast<char*>(this) + sizeof(DataPoint) + numDPars * sizeof(ArrayOfThese));
}
const DataPoint* next() const {
return reinterpret_cast<const DataPoint*>(reinterpret_cast<const char*>(this) + sizeof(DataPoint) + numDPars * sizeof(ArrayOfThese));
}
};
int main() {
const size_t BUF_SIZE = 1024*1024*200;
char* const buffer = new char[BUF_SIZE];
char* bufPtr = buffer;
const int numDataPoints = 1024*1024*2;
for (int i = 0; i < numDataPoints; ++i) {
// This wouldn't really be random.
const int numArrayOfTheses = random() % 10 + 1;
DataPoint* dp = new(bufPtr) DataPoint(numArrayOfTheses);
// Here, do some stuff to fill in the fields.
dp->a = i;
bufPtr += sizeof(DataPoint) + numArrayOfTheses * sizeof(ArrayOfThese);
}
DataPoint* dp = reinterpret_cast<DataPoint*>(buffer);
for (int i = 0; i < numDataPoints; ++i) {
assert(dp->a == i);
dp = dp->next();
}
// Here, send it out.
delete[] buffer;
return 0;
}