Deadlock in MPI_Reduce() when run on multiple nodes - c++

Question

I have a problem with my MPI code, it hangs when the code is run on multiple nodes. It successfully completes when run on a single node. I am not sure how to debug this. Can someone help me debug this issue?
Program Usage:
mpicc -o string strin.cpp
mpirun -np 4 -npernode 2 -hostfile hosts ./string 12 0.1 0.9 10 2
My Code:
#include <iostream>
#include <vector>
#include <stdio.h>
#include <stdlib.h>
#include "mpi.h"
int main ( int argc, char **argv )
{
float *y, *yold;
float *v, *vold;
int nprocs, myid;
FILE *f = NULL;
MPI_Status status;
int namelen;
char processor_name[MPI_MAX_PROCESSOR_NAME];
// const int NUM_MASSES = 1000;
// const float Ktension = 0.1;
// const float Kdamping = 0.9;
// const float duration = 10.0;
#if 0
if ( argc != 5 ) {
std::cout << "usage: " << argv[0] << " NUM_MASSES durationInSecs Ktension Kdamping\n";
return 2;
}
#endif
int NUM_MASSES = atoi ( argv[1] );
float duration = atof ( argv[2] );
float Ktension = atof ( argv[3] );
float Kdamping = atof ( argv[4] );
const int PICKUP_POS = NUM_MASSES / 7; // change this for diff harmonics
const int OVERSAMPLING = 16; // run sim at this multiple of audio sampling rate
MPI_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD,&nprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
MPI_Get_processor_name(processor_name, &namelen);
// open output file
if (myid == 0) {
f = fopen ( "rstring.raw", "wb" );
if (!f) {
std::cout << "can't open output file\n";
return 1;
}
}
// allocate displacement and velocity arrays
y = new float[NUM_MASSES];
yold = new float[NUM_MASSES];
v = new float[NUM_MASSES];
// initialize displacements (pluck it!) and velocities
for (int i = 0; i < NUM_MASSES; i++ ) {
v[i] = 0.0f;
yold[i] = y[i] = 0.0f;
if (i == NUM_MASSES/2 )
yold[i] = 1.0; // impulse at string center
}
// Broadcast data
//MPI_Bcast(y, NUM_MASSES, MPI_FLOAT, 0, MPI_COMM_WORLD);
//MPI_Bcast(yold, NUM_MASSES, MPI_FLOAT, 0, MPI_COMM_WORLD);
//MPI_Bcast(v, NUM_MASSES, MPI_FLOAT, 0, MPI_COMM_WORLD);
//int numIters = duration * 44100 * OVERSAMPLING;
int numIters = atoi( argv[5] );
for ( int t = 0; t < numIters; t++ ) {
// for each mass element
float sum = 0;
float gsum = 0;
int i_start;
int i_end ;
i_start = myid * (NUM_MASSES/nprocs);
i_end = i_start + (NUM_MASSES/nprocs);
for ( int i = i_start; i < i_end; i++ ) {
if ( i == 0 || i == NUM_MASSES-1 ) {
} else {
float accel = Ktension * (yold[i+1] + yold[i-1] - 2*yold[i]);
v[i] += accel;
v[i] *= Kdamping;
y[i] = yold[i] + v[i];
sum += y[i];
}
}
MPI_Reduce(&sum, &gsum, 1, MPI_FLOAT, MPI_SUM, 0, MPI_COMM_WORLD);
float *tmp = y;
y = yold;
yold = tmp;
if (myid == 0) {
//printf("%f\n", gsum);
if ( t % OVERSAMPLING == 0 ) {
fwrite ( &gsum, sizeof(float), 1, f );
}
}
}
if (myid == 0) {
fclose ( f );
}
MPI_Finalize();
}

Answer 1

If you have the possibility to do it, you may try to run your application inside a parallel debugger (like Totalview).
Otherwise, when the program hangs, you can attach a freely available serial debugger (like GDB) to one process at a time so to see where the potential problem may be located.

Answer 2

I guess you're receiving message, which isn't sent by any node. If every node first try to receive message, which node will send it?
You can modify program for example if id == 0 send(msg) else receive(&msg) and try use timeouts.
Write on a piece of paper how it works and how nodes interact and you will see, where is the problem.

Answer 3

I finally found the answer from OpenMPI mailing list. I think the problem is because of the way my hosts are setup.
I guess the TCP BTL gets confused by virtual interfaces (vmnet?) when run on multiple nodes. I limited the used interfaces using the "--mca btl_tcp_if_include eth0" argument. and this solved my issue.