MPI_Scatter Segfaulting - c++

Question

I'm working on a parallel sort program to learn MPI, and I've been having problems with MPI_Scatter. Every time I attempt to run, I get the following:
reading input
Scattering input
_pmii_daemon(SIGCHLD): [NID 00012] PE 0 exit signal Segmentation fault
[NID 00012] 2011-03-28 10:12:56 Apid 23655: initiated application termination
A basic look at other questions didn't really answer why I'm having troubles - The arrays are contiguous, so I shouldn't have problems with non-contiguous memory access, and I'm passing the correct pointers in the correct order. Does anyone have any ideas?
Source code is below - It's specified for a specific number because I don't want to deal with variable input and rank size just yet.
#include <mpi.h>
#include <iostream>
using std::endl;
using std::cout;
#include <fstream>
using std::ifstream;
using std::ofstream;
#include <algorithm>
using std::sort;
#define SIZEOF_INPUT 10000000
#define NUMTHREADS 100
#define SIZEOF_SUBARRAY SIZEOF_INPUT/NUMTHREADS
int main(int argc, char** argv){
MPI_Init(&argc, &argv);
int input[SIZEOF_INPUT];
int tempbuf[SIZEOF_SUBARRAY];
int myRank;
MPI_Comm_rank(MPI_COMM_WORLD, &myRank);
/*
Read input from file
*/
if(myRank == 0){
cout << "reading input" << endl;
ifstream in(argv[1]);
for(int i = 0; i < SIZEOF_INPUT; ++i)
in >> input[i];
cout << "Scattering input" << endl;
}
// Scatter, Sort, and Gather again
MPI_Scatter(input,SIZEOF_INPUT,MPI_INT,tempbuf,SIZEOF_SUBARRAY,MPI_INT,0,MPI_COMM_WORLD);
cout << "Rank " << myRank << "Sorting" << endl;
sort(tempbuf,tempbuf+SIZEOF_SUBARRAY);
MPI_Gather(tempbuf,SIZEOF_SUBARRAY,MPI_INT,input,SIZEOF_INPUT,MPI_INT,0,MPI_COMM_WORLD);
if(myRank == 0){
cout << "Sorting final output" << endl;
// I'm doing a multi-queue merge here using tricky pointer games
//list of iterators representing things in the queue
int* iterators[NUMTHREADS];
//The ends of those iterators
int* ends[NUMTHREADS];
//Set up iterators and ends
for(int i = 0; i < NUMTHREADS; ++i){
iterators[i] = input + (i*SIZEOF_SUBARRAY);
ends[i] = iterators[i] + SIZEOF_SUBARRAY;
}
ofstream out(argv[2]);
int ULTRA_MAX = SIZEOF_INPUT + 1;
int* ULTRA_MAX_POINTER = &ULTRA_MAX;
while(true){
int** curr_min = &ULTRA_MAX_POINTER;
for(int i = 0 ; i < NUMTHREADS; ++i)
if(iterators[i] < ends[i] && *iterators[i] < **curr_min)
curr_min = &iterators[i];
if(curr_min == &ULTRA_MAX_POINTER) break;
out << **curr_min << endl;
++(*curr_min);
}
}
MPI_Finalize();
}
Any help would be much appreciated.
Regards,
Zach

Answer 1

Hah! Took me a while to see this one.
The trick is, in MPI_Scatter, the sendcount is the amount to send to each process, not in total. Same with gather; it's the amount to receive from each. That is, it's like MPI_Scatterv with counts; the count is to each process, but in this case, it's assumed to be the same.
so this
MPI_Scatter(input,SIZEOF_SUBARRAY,MPI_INT,tempbuf,SIZEOF_SUBARRAY,MPI_INT,0,MPI_COMM_WORLD);
cout << "Rank " << myRank << "Sorting" << endl;
MPI_Gather(tempbuf,SIZEOF_SUBARRAY,MPI_INT,input,SIZEOF_SUBARRAY,MPI_INT,0,MPI_COMM_WORLD);
works for me.
Also, be careful of allocating big arrays like that on the stack; I know this is just an example problem, but for me this was causing crashes right away. Doing it dynamically
int *input = new int[SIZEOF_INPUT];
int *tempbuf = new int[SIZEOF_SUBARRAY];
//....
delete [] input;
delete [] tempbuf;
solved that problem.

Answer 2

int* iterators[NUMTHREADS];
//The ends of those iterators
int* ends[NUMTHREADS];
//Set up iterators and ends
for(int i = 0; i < NUMTHREADS; ++i){
iterators[i] = input + (i*SIZEOF_SUBARRAY); // problem
ends[i] = iterators[i] + SIZEOF_SUBARRAY; // problem
}
Both iterators and ends are array of integer pointers pointing no where or garbage. But in the for loop trying to keep values as if they are pointing to some location, which results segmentation fault. Program should should first allocate memory, iterators can point to and then should keep the values at locations pointed by them.
for( int i=0 ; i < NUMTHREADS; ++i )
{
iterators[i] = new int;
end[i] = new int ;
}
// Now do the earlier operation which caused problem
Since the program manages resources( i.e., acquired from new ), it should return resources to free store using delete[] when no longer needed. Use std::vector instead of managing resources your self, which is very easy.