Random crashes with short c openmp program on RHEL5 - c++

Question

I have a real puzzler for you folks.
Below is a small, self-contained, simple 40-line program that calculates partial sums of a bunch of numbers and routinely (but stochastically) crashes nodes on a distributed memory cluster that I'm using. If I spawn 50 PBS jobs that run this code, between 0 and 4 of them will crash their nodes. It will happen on a different repeat of the main loop each time and on different nodes each time, there is no discernible pattern. The nodes just go "down" on the ganglia report and I can't ssh to them ("no route to host"). If instead of submitting jobs I ssh onto one of the nodes and run my program there, if I'm unlucky and it crashes then I just stop seeing text and then see that that node is dead on ganglia.
The program is threaded with openmp and the crashes only happen when a large number of threads are spawned (like 12).
The cluster it's killing is a RHEL 5 cluster with nodes that have 2 6-core x5650 processors:
[jamelang#hooke ~]$ tail /etc/redhat-release
Red Hat Enterprise Linux Server release 5.7 (Tikanga)
I have tried enabling core dumps ulimit -c unlimited but no files show up. This is the code, with comments:
#include <cstdlib>
#include <cstdio>
#include <omp.h>
int main() {
const unsigned int numberOfThreads = 12;
const unsigned int numberOfPartialSums = 30000;
const unsigned int numbersPerPartialSum = 40;
// make some numbers
srand(0); // every instance of program should get same results
const unsigned int totalNumbersToSum = numbersPerPartialSum * numberOfPartialSums;
double * inputData = new double[totalNumbersToSum];
for (unsigned int index = 0; index < totalNumbersToSum; ++index) {
inputData[index] = rand()/double(RAND_MAX);
}
omp_set_num_threads(numberOfThreads);
// prepare a place to dump output
double * partialSums = new double[numberOfPartialSums];
// do the following algorithm many times to induce a problem
for (unsigned int repeatIndex = 0; repeatIndex < 100000; ++repeatIndex) {
if (repeatIndex % 1000 == 0) {
printf("Absurd testing is on repeat %06u\n", repeatIndex);
}
#pragma omp parallel for
for (unsigned int partialSumIndex = 0; partialSumIndex < numberOfPartialSums;
++partialSumIndex) {
// get this partial sum's limits
const unsigned int beginIndex = numbersPerPartialSum * partialSumIndex;
const unsigned int endIndex = numbersPerPartialSum * (partialSumIndex + 1);
// we just sum the 40 numbers, can't get much simpler
double sumOfNumbers = 0;
for (unsigned int index = beginIndex; index < endIndex; ++index) {
// only reading, thread-safe
sumOfNumbers += inputData[index];
}
// writing to non-overlapping indices (guaranteed by omp),
// should be thread-safe.
// at worst we would have false sharing, but that would just affect
// performance, not throw sigabrts.
partialSums[partialSumIndex] = sumOfNumbers;
}
}
delete[] inputData;
delete[] partialSums;
return 0;
}
I compile it with the following:
/home/jamelang/gcc-4.8.1/bin/g++ -O3 -Wall -fopenmp Killer.cc -o Killer
It seems to be linking against the right shared objects:
[jamelang#hooke Killer]$ ldd Killer
linux-vdso.so.1 => (0x00007fffc0599000)
libstdc++.so.6 => /home/jamelang/gcc-4.8.1/lib64/libstdc++.so.6 (0x00002b155b636000)
libm.so.6 => /lib64/libm.so.6 (0x0000003293600000)
libgomp.so.1 => /home/jamelang/gcc-4.8.1/lib64/libgomp.so.1 (0x00002b155b983000)
libgcc_s.so.1 => /home/jamelang/gcc-4.8.1/lib64/libgcc_s.so.1 (0x00002b155bb92000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003293a00000)
libc.so.6 => /lib64/libc.so.6 (0x0000003292e00000)
/lib64/ld-linux-x86-64.so.2 (0x0000003292a00000)
librt.so.1 => /lib64/librt.so.1 (0x0000003298600000)
Some Notes:
1. On osx lion with gcc 4.7, this code will throw a SIGABRT, similar to this question: Why is this code giving SIGABRT with openMP?. Using gcc 4.8 seems to fix the issue on OSX. However, using gcc 4.8 on the RHEL5 machine does not fix it. The RHEL5 machine has GLIBC version 2.5, and it seems that yum doesn't provide a later one, so the admins are sticking with 2.5.
2. If I define a SIGABRT signal handler, it doesn't catch the problem on the RHEL5 machine, but it does catch it on OSX with gcc47.
3. I believe that no variables should need to be shared in the omp clause because they can all have private copies, but adding them as shared does not change the behavior.
4. The killing of nodes occurs regardless of the level of optimization used.
5. The killing of nodes occurs even if I run the program from within gdb (i.e. put "gdb -batch -x gdbCommands Killer" in the pbs file) where "gdbCommands" is a file with one line: "run"
6. This example spawns threads on every repeat. One strategy would be to make a parallel block that contains the repeats loop in order to prevent this. However, that does not help me - this example is only representative of a much larger research code in which I cannot use that strategy.
I'm all out of ideas, at my last straw, at my wit's end, ready to pull my hair out, etc with this. Does anyone have suggestions or ideas?

Answer 1

You are trying to parallelize nested for loops, in this case you need to make the variables in the inner loop private, so that each thread has its own variable. It can be done using private clause, as in the example below.
#pragma omp parallel for private(j)
for (i = 0; i < height; i++)
for (j = 0; j < width; j++)
c[i][j] = 2;
In your case, index and sumOfNumbers need to be private.