c++ 2d array leads to Segmentation fault (core dumped) - c++

Question

I have a program (below) to build a matrix of distances between points (in my test file there are ~8000 points in 3D space). So I want a (roughly) 8000x8000 matrix, but when I try to build it using an array of doubles (or floats), I always get a 'Segmentation fault (core dumped)' error. Any ideas why? I have 16GB of RAM, so this should be feasible, since 8000 * 8000 * 8 is only roughly 0.5GB. Also (commented out in code below), I can build the matrix as a vector of vectors, but this is slow - takes around 30 seconds. As it happens, I only need to record distances less than 1.5, so the matrix is very sparse - no doubt there are better ways of implementation, but it's bugging me that this doesn't work. Any advice gratefully received!
//Get distance matrix from .dms file
#include <iostream>
#include <stdlib.h>
#include <fstream>
#include <string>
#include <cmath>
#include <vector>
using namespace std;
double dist(vector<double> a, vector<double> b) {
if (a.size() != b.size()) return -1;
else
{
double dist = 0;
for (int i = 0; i < a.size(); i++) dist += pow(a[i] - b[i], 2);
return dist;
}
}
int main() {
ifstream infile;
ofstream outfile;
vector<vector<double> > points;
string line;
infile.open("1dwr.dms");
outfile.open("1dwr.mat");
while (getline(infile,line))
{
if ((line.at( line.length() - 1)) != 'A')
{
double x[3] = {atof((line.substr(13,8)).c_str()), atof((line.substr(21,9)).c_str()), atof((line.substr(30,9)).c_str())};
vector<double> point;
for (int i=0; i<3; i++)
{
point.push_back(x[i]);
}
points.push_back (point);
}
}
infile.close();
int len = points.size();
double dist_matrix[len][len];
for (int i=0; i<len; i++)
{
for(int j=i; j<len; j++)
{
double d = dist(points[i], points[j]);
if(d < 2.25)
{
dist_matrix[i][j] = sqrt(d);
dist_matrix[j][i] = sqrt(d);
}
}
}
// vector<vector<double> > dist_matrix;
// for (int i=0; i<len; i++)
// {
// vector< double> row;
// for (int j=0; j<len; j++)
// {
// double d = dist(points[i], points[j]);
// if (d < 2.25) row.push_back (sqrt(d));
// else row.push_back (0);
// }
// dist_matrix.push_back (row);
// }
outfile.close();
return 0;
}

Answer 1

The problem is that statements int len = points.size(); double dist_matrix[len][len] lets the program create a 8000 x 8000-array of doubles on the stack, and the size of the stack is - compared to the heap - rather limited. So it is very likely that you get a "stack overflow", indicated probably by a "Bad access"-error. If you try your code with 80x80, it will probably work.
So for len==8000, you'd have to create the array on the heap; yet there is - to my knowledge - no way to do a statement like double x[len][len] = new double... if len is not known at compile time.
You could come around this by allocating a 1D-array of size len * len and calculate the "2D"-index manually, as shown in the code below:
int main() {
int len = 8000;
double *dist_matrix = new double[len*len];
for (int i=0; i<len; i++)
{
for(int j=i; j<len; j++)
{
size_t idx = len*i + j;
dist_matrix[idx] = 5.0;
}
}
return 0;
}