MPI_Scatter segmentation fault dependent on node number - c++

Question

I get a strange behavior when running a test code for MPI_Scatter. The program seems to work fine, but it returns a segmentation fault if the number of nodes is larger than 4. I compile with mpicxx and run with mpirun -n N ./a.o.
#include <mpi.h>
#include <vector>
#include <stdio.h>
using std::vector;
int main(void){
MPI_Init(NULL,NULL);
int num_PE;
MPI_Comm_size(MPI_COMM_WORLD, &num_PE);
int my_PE;
MPI_Comm_rank(MPI_COMM_WORLD, &my_PE);
int data_per_PE=2;
int remainder=0; //conceptually should be less than data_per_PE but shouldn't matter from code perspective
vector<int> elem_count(num_PE,data_per_PE); //number of elements to scatter
elem_count[num_PE-1]=data_per_PE+remainder; //let last PE take extra load
vector<int> start_send(num_PE); //the offset to send from main buffer
vector<double> small_vec(data_per_PE+remainder); //small place to store values
vector<double> bigVec; //the big list to distribute to processes
if (my_PE==0){
bigVec.reserve(data_per_PE*num_PE+remainder); //make room
for(int i=0; i<data_per_PE*num_PE+remainder; i++){
bigVec.push_back(static_cast<double>(i)+1.0); //1,2,3...
start_send[i]=i*data_per_PE; //the stride
}
}
// MPI_Scatterv(&bigVec[0],&elem_count[0],&start_send[0],MPI_DOUBLE,&small_vec[0],data_per_PE+remainder,MPI_DOUBLE,0,MPI_COMM_WORLD);
MPI_Scatter(&bigVec[0],data_per_PE,MPI_DOUBLE,&small_vec[0],data_per_PE,MPI_DOUBLE,0,MPI_COMM_WORLD); //scatter
if (my_PE==0){
printf("Proc \t elems \n");
}
MPI_Barrier(MPI_COMM_WORLD); //let everything catch up before printing
for (int i=0;i<data_per_PE+remainder;i++){
printf("%d \t %f \n", my_PE, small_vec[i]); //print the values scattered to each processor
}
MPI_Barrier(MPI_COMM_WORLD); //don't think this is necessary but won't hurt
MPI_Finalize(); //finish
return 0;
}

Answer 1

The issue has nothing to do with the scatter, but rather this line:
start_send[i]=i*data_per_PE;
Since i can go beyond num_PE, you write outside of the bounds of start_send - overwriting some memory that probably belongs to small_vec.
This could have easily been found by creating a truly minimal example.
You have another issue in your code: &bigVec[0] is a problem for my_PE!=0. While the parameter to MPI_Scatter is ignored by non-root ranks, the statement involves dereferencing in std::vector::operator[] the first element. As the vector is empty, this is undefined behavior on it's own. Here is an explanation as to why that can create subtle problems. Use bigVec.data() instead.

Answer 2

You are writing past the end of start_send's internal storage, thus corrupting the heap and any other objects contained in it:
if (my_PE==0){
bigVec.reserve(data_per_PE*num_PE+remainder); //make room
for(int i=0; i<data_per_PE*num_PE+remainder; i++){
bigVec.push_back(static_cast<double>(i)+1.0); //1,2,3...
start_send[i]=i*data_per_PE; //the stride <--- HERE
}
}
i runs until data_per_PE*num_PE+remainder - 1, but start_send has storage for num_PE elements only. Writing past the end corrupts the linked list of heap objects and the program likely segfaults when a destructor tries to free a corrupted heap block or when some other heap object is accessed.