OpenMPI cross nodes Allreduce operator failed and connection timed out - c++

Question

I am new to MPI programming and I write two simple programs to check if I can do the cross nodes communication. I first run the following program by using my own hostfile
int main(int argc, char** argv) {
// Initialize the MPI environment
MPI_Init(NULL, NULL);
// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
// Print off a hello world message
printf("Hello world from processor %s, rank %d out of %d processors\n",
processor_name, world_rank, world_size);
// Finalize the MPI environment.
MPI_Finalize();
}
And then I got the follow results. It seems that everything works will and I can do the communication cross nodes. However, when I tried to do the Allreduce operation, I got the error message.
Hello world from processor lifs1.math.ust.hk, rank 4 out of 8 processors
Hello world from processor lifs1.math.ust.hk, rank 6 out of 8 processors
Hello world from processor lifs1.math.ust.hk, rank 5 out of 8 processors
Hello world from processor lifs1.math.ust.hk, rank 7 out of 8 processors
Hello world from processor lifs2.math.ust.hk, rank 0 out of 8 processors
Hello world from processor lifs2.math.ust.hk, rank 1 out of 8 processors
Hello world from processor lifs2.math.ust.hk, rank 2 out of 8 processors
Hello world from processor lifs2.math.ust.hk, rank 3 out of 8 processors
[lifs1.math.ust.hk:29387] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[lifs1.math.ust.hk:29387] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
The allreduce programming is
int main(int argc, char** argv) {
MPI_Init(&argc, &argv);
int total_process;
int id;
MPI_Comm_size(MPI_COMM_WORLD, &total_process);
MPI_Comm_rank(MPI_COMM_WORLD, &id);
if (id == 0) {
std::cout << "========== Testing MPI Across Nodes ==========" << std::endl;
}
int N = 1000;
float* vec = new float[N];
for (int i = 0; i < N; ++i) {
vec[i] = 12.345;
}
float* global_sum = new float[N];
MPI_Allreduce(vec, global_sum, N, MPI_FLOAT, MPI_SUM, MPI_COMM_WORLD);
std::cout << "Result " << global_sum[0] << std::endl;
delete [] vec;
delete [] global_sum;
MPI_Finalize();
return 0;
}
I think 1000 single-precision floating point should not take too much time to finish the communication. The error message is
[lifs1.math.ust.hk:30146] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[lifs1.math.ust.hk:30146] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[lifs2][[13004,1],0][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.207 failed: Connection timed out (110)
[lifs2][[13004,1],2][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.207 failed: Connection timed out (110)
[lifs2][[13004,1],1][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.207 failed: Connection timed out (110)
[lifs2][[13004,1],3][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.207 failed: Connection timed out (110)
[lifs1][[13004,1],4][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.208 failed: Connection timed out (110)
[lifs1][[13004,1],5][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] [lifs1][[13004,1],6][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.208 failed: Connection timed out (110)
[lifs1][[13004,1],7][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.208 failed: Connection timed out (110)
connect() to 143.89.16.208 failed: Connection timed out (110)
My questions are
Why can't I do the allreduce operation cross nodes?
My program is very simple, but I still get this message 7 more processes have sent help message help-mpi-btl-base.txt. What's this?