Handling Huge Multidimensional Arrays in C++ - c++

Question

I'm designing a game in C++ similar to Minecraft that holds an enormous amount of terrain data in memory. In general, I want to store an array in memory that is [5][4][5][50][50][50]. This isn't bad since it amounts to about 100mb of virtual memory since my structure will only be about 8 bytes.
However, I'm having trouble figuring out the best way to handle this. I do want this to be in virtual memory, but obviously not on the stack. And I keep making the mistake some how of creating this array on the stack an causing a stack overflow. What I would like to do is below. This is just code that I threw together to give you an example of what I'm doing, I have code with correct syntax on my machine, I just didn't want to clutter the post.
typedef struct modelBlock
{
// Information about the blocks
} BLOCK;
typedef struct modelGrid
{
bool empty;
BLOCK blocksArray[50][50][50];
} GRID;
class Parent
{
Child* child;
Parent(void);
}
Parent::Parent()
{
Child c;
child = &c;
}
class Child
{
GRID grids[5][4][5];
}
However, every time I do this, I cause a stack overflow (appropriate web site choice right?). I played with using pointer based arrays, but I had a lot of trouble with data being lost outside of its scope.
If anyone could give me some insight on how to get my data to store on the heap instead of the stack, or if I should use some other way of creating my array, I'd really appreciate the help. I'd like to avoid using vectors because of overhead, though I'm not sure how substantial it is.

Answer 1

Use boost::multi_array

Answer 2

If you want to allocate something on the heap, use new.
#include <memory>
class Parent
{
std::auto_ptr<Child> child; // use auto_ptr for dynamically-allocated members
Parent(const Parent&); // You probably don't want to copy this giant thing
public:
Parent();
};
Parent::Parent()
: child(new Child) // initialize members with an initializer list
{
}
Also, avoid mixing C and C++ styles. There's no reason to do
typedef struct blah{ ... } BLAH;
in C++. A struct is just a class with all of the members public by default; just like a class, you can refer to the struct type's name without using the struct tag. There's also no need to specify void for a function that takes no parameters.
boost::multi_array (linked in PigBen's answer) is a good choice over raw arrays.

Answer 3

If you want the class created on the heap, create it with new:
Child * c = new Child;
and then of course delete it, or better still use a smart pointer.

Answer 4

In order to do exactly what you're trying to do you have to declare everything as pointers (and pointers to pointers to pointers to pointers) and then allocate each one individually.
Teh sux!
A better option is to simply allocate the ginormous block in one chunk and use multiple variable along with pointer arithmetic to arrive at the correct location.
Edit: Wasn't paying attention and didn't notice your constructor. That's not only not the way to get your Child allocated on the free-store, it's a great way to create situations eliciting undefined behavior. Your Child will be gone when the constructor is through and the pointer to it will then be invalid. I wonder if you shouldn't run through some basic tutorials before trying to write a game.

Answer 5

Here's something that works and can be built upon without the boost dependency. One downside is it removes use of [][][] style of referencing elements, but it's a small cost and can be added.
template<class T>
class Matrix {
unsigned char* _data;
const size_t _depth;
const size_t _cols;
const size_t _rows;
public:
Matrix(const size_t& depth, const size_t& rows, const size_t& cols):
_depth(depth),
_rows(rows),
_cols(cols) {
_data = new unsigned char [depth * rows * cols * sizeof(T)];
}
~Matrix() {
delete[] _data;
}
T& at(const size_t& depthIndex, const size_t& rowIndex, const size_t& colIndex) const {
return *reinterpret_cast<T*>(_data + ((((depthIndex * _cols + colIndex) * _rows) + rowIndex) * sizeof(T)));
}
const size_t& getDepth() const {
return _depth;
}
const size_t& getRows() const {
return _rows;
}
const size_t& getCols() const {
return _cols;
}
};
int _tmain(int argc, _TCHAR* argv[])
{
Matrix<int> block(50, 50, 50);
size_t d, r, c;
for (d = 0; d < block.getDepth(); d++) {
for (r = 0; r < block.getRows(); r++) {
for (c = 0; c < block.getCols(); c++) {
block.at(d, r, c) = d * 10000000 + r * 10000 + c;
}
}
}
for (d = 0; d < block.getDepth(); d++) {
for (r = 0; r < block.getRows(); r++) {
for (c = 0; c < block.getCols(); c++) {
assert(block.at(d, r, c) == d * 10000000 + r * 10000 + c);
}
}
}
return 0;
}

Answer 6

A smaller example (with changed names for all the structs, to make the general principle clearer). The 'Bloe' struct is the one you want to allocate on the heap, and this is accomplished using 'new'.
struct Bla {
int arr[4][4];
};
struct Bloe {
Bla bla[2][2];
};
int main()
{
Bloe* bloe = new Bloe();
bloe->bla[1][1].arr[1][1] = 1;
return 0;
}

Answer 7

I did this by putting all the data in a binary file. I calculated the offset of the data and used seek() and read() to get the data when needed. The open() call is very slow so you should leave the file open during the lifetime of the program.

Answer 8

Below is how I understood what you showed you were trying to do in your example. I tried to keep it straightforward. Each Array of [50][50][50] is allocated in one memory chunk on the heap, and only allocated when used. There is also an exemple of access code. No fancy boost or anything special, just basic C++.
#include <iostream>
class Block
{
public:
// Information about the blocks
int data;
};
class Grid
{
public:
bool empty;
Block (*blocks)[50][50];
Grid() : empty(true) {
}
void makeRoom(){
this->blocks = new Block[50][50][50];
this->empty = false;
}
~Grid(){
if (!this->empty){
delete [] this->blocks;
}
}
};
class Parent
{
public:
Grid (* child)[4][5];
Parent()
{
this->child = new Grid[5][4][5];
}
~Parent()
{
delete [] this->child;
}
};
main(){
Parent p;
p.child[0][0][0].makeRoom();
if (!p.child[0][0][0].empty){
Block (* grid)[50][50] = p.child[0][0][0].blocks;
grid[49][49][49].data = 17;
}
std::cout << "item = "
<< p.child[0][0][0].blocks[49][49][49].data
<< std::endl;
}
This could still be more simple and straightfoward and just use one bug array of [50][50][50][5][4][5] blocks in one memory chunk on the heap, but I'll let you figure out how if this is what you want.
Also, usind dynamic allocation in class Parent only has the sole purpose to use heap instaed of stack, but for such a small array (5*4*5 pointers), allocating it on stack should not be a problem, hence it could be written.
class Parent
{
public:
Grid child[5][4][5];
};
without changing anything in the way it is used.