creating large 2d array of size int arr[1000000][1000000] - c++

Question

I want to create a two-dimensional integer array of size 106 × 106 elements. For this I'm using the boost library:
boost::multi_array<int, 2> x(boost::extents[1000000][1000000]);
But it throws the following exception:
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Please tell me how to solve the problem.

Answer 1

You seriously don't want to allocate an array that huge. It's about 4 terabytes in memory.
Depending on what you want to do with that array you should consider two options:
External data structure. The array will be written on a hard drive. The most recently accessed parts is also in RAM, so depending on how you access it it can be pretty fast, but of course never as fast as if it would be fully in RAM. Have a look at STXXL for external data structures.
This method has the advantage that you can access all of the elements in the array (in contrast to the second method as you'll see). However, the problem still remains: 4 terabytes are very huge even on a hard drive, at least if you are talking about a general desktop application.
Sparse data structure. If you only actually need a couple of items from that array, but you want to address these items in a space of size 10⁶ ⨯ 10⁶, don't use an array but something like a map or a combination of both: Allocate the array in "blocks" of, let's say 1024 x 1024 elements. Put these blocks into a map while referencing the block index (coordinate divided by 1024) as the key in the map.
This method has the advantage that you don't have to link against another library, since it can be written easily by yourself. However, it has the disadvantage that if you access elements distributed over the whole coordinate space of 10⁶ ⨯ 10⁶ or even need all of the values, it also uses around 4TB (even a bit more) memory. It only works if you actually access only a smart part of this huge "virtual" array.
The following (untested) C++ code should demonstrate this:
class Sparse2DArray
{
struct Coord {
int x, y;
Coord(int x, int y) : x(x), y(y) {}
bool operator<(const Coord &o) const { return x < o.x || (x == o.x && y < o,y); } // required for std::map
};
static const int BLOCKSIZE = 1024;
std::map<Coord, std::array<std::array<int,BLOCKSIZE>,BLOCKSIZE> blocks;
static Coord block(Coord c) {
return coord(c.x / BLOCKSIZE, c.y / BLOCKSIZE);
}
static Coord blockSubCoord(Coord c) {
return coord(c.x % BLOCKSIZE, c.y % BLOCKSIZE);
}
public:
int & operator[](int x, int y) {
Coord c(x, y);
Coord b = block(c);
Coord s = blockSubCoord(c);
return blocks[b][s.x][s.y];
}
};
Instead of a std::map you can also use a std::unordered_map (hash map) but have to define a hash function instead of operator< for the Coord type (or use std::pair instead).

Answer 2

When you create an array that way, it is created on the stack and the stack has a limited size. Therefore, your program will crash because it doesn't have enough room to allocate that big of an array.
There are two ways you can solve this, you can create the array on the heap using the new keyword But you have to delete it afterword or else you have a memory leak, and also be careful because while the heap has a larger memory size then the stack it is still finite.
The other way is for you to use std::vector inside std::vector and let it handle the memory for you.

Answer 3

What do you intend by creating a 106×106 matrix? If you're trying to create a sparse matrix (i.e. a diffusion matrix for a heat transfer problem with 106 finite elements), then you should look at using an existing linear algebra library. For example, the trilinos project has support for solving large sparse matrices like the one you may be trying to create.