why is mpi_bcast so much slower than mpi_reduce? - c++

Question

Using MPI, we can do a broadcast to send an array to many nodes, or a reduce to combine arrays from many nodes onto one node.
I guess that the fastest way to implement these will be using a binary tree, where each node either sends to two nodes (bcast) or reduces over two nodes (reduce), which will give a time logarithmic in the number of nodes.
There doesn't seem to be any reason for which broadcast would be particularly slower than reduce?
I ran the following test program on a 4-computer cluster, where each computer has 12 cores. The strange thing is that broadcast was quite a lot slower than reduce. Why? Is there anything I can do about it?
The results were:
inited mpi: 0.472943 seconds
N: 200000 1.52588MB
P = 48
did alloc: 0.000147641 seconds
bcast: 0.349956 seconds
reduce: 0.0478526 seconds
bcast: 0.369131 seconds
reduce: 0.0472673 seconds
bcast: 0.516606 seconds
reduce: 0.0448555 seconds
The code was:
#include <iostream>
#include <cstdlib>
#include <cstdio>
#include <ctime>
#include <sys/time.h>
using namespace std;
#include <mpi.h>
class NanoTimer {
public:
struct timespec start;
NanoTimer() {
clock_gettime(CLOCK_MONOTONIC, &start);
}
double elapsedSeconds() {
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now);
double time = (now.tv_sec - start.tv_sec) + (double) (now.tv_nsec - start.tv_nsec) * 1e-9;
start = now;
return time;
}
void toc(string label) {
double elapsed = elapsedSeconds();
cout << label << ": " << elapsed << " seconds" << endl;
}
};
int main( int argc, char *argv[] ) {
if( argc < 2 ) {
cout << "Usage: " << argv[0] << " [N]" << endl;
return -1;
}
int N = atoi( argv[1] );
NanoTimer timer;
MPI_Init( &argc, &argv );
int p, P;
MPI_Comm_rank( MPI_COMM_WORLD, &p );
MPI_Comm_size( MPI_COMM_WORLD, &P );
MPI_Barrier(MPI_COMM_WORLD);
if( p == 0 ) timer.toc("inited mpi");
if( p == 0 ) {
cout << "N: " << N << " " << (N*sizeof(double)/1024.0/1024) << "MB" << endl;
cout << "P = " << P << endl;
}
double *src = new double[N];
double *dst = new double[N];
MPI_Barrier(MPI_COMM_WORLD);
if( p == 0 ) timer.toc("did alloc");
for( int it = 0; it < 3; it++ ) {
MPI_Bcast( src, N, MPI_DOUBLE, 0, MPI_COMM_WORLD );
MPI_Barrier(MPI_COMM_WORLD);
if( p == 0 ) timer.toc("bcast");
MPI_Reduce( src, dst, N, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD );
MPI_Barrier(MPI_COMM_WORLD);
if( p == 0 ) timer.toc("reduce");
}
delete[] src;
MPI_Finalize();
return 0;
}
The cluster nodes were running 64-bit ubuntu 12.04. I tried both openmpi and mpich2, and got very similar results. The network is gigabit ethernet, which is not the fastest, but what I'm most curious about is not the absolute speed, so much as the disparity between broadcast and reduce.

Answer 1

I don't think this quite answers your question, but I hope it provides some insight.
MPI is just a standard. It doesn't define how every function should be implemented. Therefore the performance of certain tasks in MPI (in your case MPI_Bcast and MPI_Reduce) are based strictly on the implementation you are using. It is possible that you could design a broadcast using point-to-point communication methods that performs better than the given MPI_Bcast.
Anyways, you have to consider what each of these functions is doing. Broadcast is taking information from one process and sending it to all other processes; reduce is taking information from each process and reducing it onto one process. According to the (most recent) standard, MPI_Bcast is considered a One-to-All collective operation and MPI_Reduce is considered an All-to-One collective operation. Therefore your intuition about using binary trees for MPI_Reduce is probably found in both implementations. However, it most likely not found in MPI_Bcast. It might be the case that MPI_Bcast is implemented using non-blocking point-to-point communication (sending from the process containing the information to all other processes) with a wait-all after the communication. In any case, in order to figure out how both functions work, I would suggest delving into the source code of your implementations of OpenMPI and MPICH2.

Answer 2

As Hristo mentioned, it depends on the size of your buffer. If you're sending a large buffer, the broadcast will have to do lots of large sends, while a receive does some local operation on the buffer to reduce it down to a single value and then only transmits that one value instead of the full buffer.