C++ program stability after millions of executions - c++

Question

I have a program in C++ that performs mainly matrix multiplcations, additions and so on.
The problem is, a EXC_BAD_ACCESS happens when the calculation performs for about 3 million times.
Is there any possible problems that can arise when a problem is executed for millions of times and for several hours?
Details of the program:
The program is simply calculations on different ranges of values, so it is executing on 6 threads at the same time. There is no resource sharing between the threads.
There seems be no evident problem in the program since:
there is no memory leak, I've confirmed this using Instruments, and the memory size of the program is stable.
the program can execute for at least 2 million times on each thread without any problem, but it is almost guaranteed that the EXC_BAD_ACCESS exception arises some time, on some thread. (the exception happens in my 2 tries of the program (2/2) )
About the matrix multiplication:
Sometimes the size of the matrices is about 2*2 multiply 2*1000.
The elements of the matrix is a custom Complex Number class.
the values of the elements are randomly generated by rand() and converted to float.
the structure is like this:
class Complex
{
private:
float _real, _imag;
public:
// getters, setters and overloaded operators
};
class Matrix
{
private:
Complex **_values;
int _row,_col;
public:
getters, setters and overloaded operators
};
Thank you very much!
Any possible reason for the crash is greatly welcomed!

Answer 1

EXC_BAD_ACCESS means that you dereferenced a pointer which doesn't point into your process's current memory space. This is a bug in your code. Run it under a debugger until it fails and then have a look at the variable values in the statement where it fails. It could be simple or exceedingly subtle.

Answer 2

There's too little information in your post to make a decisive answer. However, it might be that no information available to you now would change it, and you need to debug the case more carefully. Here's what I'd do.
To debug, you want repeatability. But… you say that you're using random numbers. It seems though, that what your program does is some scientific-ish computations. In most cases you don't actually need “true” randomness, but “repeatable” randomness—randomness which passes statistical tests, but where you have enough data to reset the random number generator so that it will produce the exactly the same results as in a previous run. For that, you can just write down the current RNG state (e.g. seed) every time you start a new block of computation.
Now, write some piece of code that will store all the state necessary to restart computations (including RNG) once every few minutes, and run the program. This way, if your code crashes, you will be able to restart the computations with the same exact state and get to the point where it crashed without waiting for millions of iterations. I am putting a strong assumption here, that except for RNG your code does not depend on any other kind of external state (like, network activity, IO, process scheduler making certain choices when scheduling your threads…)
With this kind of data it will be easier to test if the problem is due to a machine fault (overheating, bad memory, etc.). Simply restart the computation with the last state before crashing—preferably after letting the machine cool down, maybe restarting it… if you'll encounter another crash (and it will happen every time you try to restart code), it's quite certain it's due to a bug in your code.
If not, we still cannot say that it's machine fault—your code might (by pure accident/mistake in code) crash due to an undefined behavior which depends on factors out of your control. Examples include using an uninitialized pointer in a rarely-taken code path: it might throw bad access sometimes, and go unnoticed if by pure luck the pointer points to memory you allocated. Try valgrind, this is probably the best tool to check for memory problems… except that it slows down execution so much that you'll again prefer to rerun the computations from a state known to be suspicious (the last state before crash) instead of waiting for millions of iterations. I've seen slowdowns of 5x to 100x.
In the meantime, try running your code on another machine. If you'll also get crashes after a similar number of iterations (to be sure wait for at least 3 times more iterations than it took to crash on the original machine), then it's quite probable that it's a bug in your code.
Happy hacking!

Answer 3

Calculations with finite precision that fail after a few million iterations? That could be accumulated round-off error. Problem is, those usually exhibit themselves as division by zero or other mathematical errors. EXC_BAD_ACCESS is not. However, there's one case in which this can happen: when you use the mathematical result as an array index.