In MPI how to get communicator when using several executables? - fortran

Question

In MPI we work with multiple process that do not share anything but communicate with recv/send operations. The recv/send operations are done with respect to a communicator which can be the whole set of processors or a subset of them. The basic commands are:
call MPI_Comm_size ( MPI_COMM_WORLD, nproc, ierr )
call MPI_Comm_rank ( MPI_COMM_WORLD, myrank, ierr )
with MPI_COMM_WORLD the communicator associated to the set of all processors. One interesting feature of MPI is that we can run several executables together with the command:
mpirun -n 3 prog1 : -n 2 prog2
with 3 nodes assigned to the first executable and 2 to the second. However for practical work, one would like to have a communicator associated to prog1 or prog2. Is there a way to get thi directly without using the command MPI_COMM_SPLIT?

Answer 1

There is no such predefined communicator specified by the standard.
The great philosopher Jagger once said “you can’t always get what you want” and your best bet here is indeed to use MPI_Comm_split() and the value of MPI_COMM_WORLD's MPI_APPNUM attribute as the color argument.
From the MPI 3.1 standard chapter 10.5.3
10.5.3 MPI_APPNUM
There is a predefined attribute MPI_APPNUM of MPI_COMM_WORLD. In Fortran, the attribute is an integer value. In C,
the attribute is a pointer to an integer value. If a process was
spawned with MPI_COMM_SPAWN_MULTIPLE, MPI_APPNUM is the command number
that generated the current process. Numbering starts from zero. If a
process was spawned with MPI_COMM_SPAWN, it will have MPI_APPNUM equal
to zero. Additionally, if the process was not started by a spawn call,
but by an implementation specific startup mechanism that can handle
multiple process specifications, MPI_APPNUM should be set to the number
of the corresponding process specification. In particular, if it is
started with
mpiexec spec0 [: spec1 : spec2 : ...]
MPI_APPNUM should be set to the number of the corresponding specification.
If an application was not spawned with MPI_COMM_SPAWN or
MPI_COMM_SPAWN_MULTIPLE, and MPI_APPNUM does not make sense in the
context of the implementation-specific startup mechanism, MPI_APPNUM is
not set.
MPI implementations may optionally provide a mechanism to
override the value of MPI_APPNUM through the info argument. MPI
reserves the following key for all SPAWN calls.
appnum Value contains
an integer that overrides the default value for MPI_APPNUM in the
child.
Rationale.
When a single application is started, it is able to
figure out how many processes there are by looking at the size of
MPI_COMM_WORLD. An application consisting of multiple SPMD
sub-applications has no way to find out how many sub-applications there
are and to which sub-application the process belongs. While there are
ways to figure it out in special cases, there is no general mechanism.
MPI_APPNUM provides such a general mechanism. (End of rationale.)

Answer 2

For anybody interested in splitting the MPI_COMM_WORLD, I wrote a small utility library (one header). It splits the communicator into local communicators and establishes intercommunicators between them. It takes care of many of the technicalities:
You can find the header at: https://github.com/cfd-go/MPI_MPMD
It also gives the same syntax for running multiple programs at once, and spawning programs. If you have any questions or ideas to extend this, open an issue at github.
For example, in your first program you do:
MPMDHelper MPMD;
MPI_Init(&argc, &argv);
MPMD.Init(MPI_COMM_WORLD, "programA");
MPMD.local //<-- this is your local Comm.
In the other program you do:
MPMDHelper MPMD;
MPI_Init(&argc, &argv);
MPMD.Init(MPI_COMM_WORLD, "programB");
MPMD.local //<-- this is your local Comm.
MPMD["programA"].local //<-- intercommunicator for communication with programA
Hope it helps.

Answer 3

An alternative option is to use the client-server mechanism that is described by MPI standard (the chapter on this can be found here). The idea there is that you compile two independent MPI applications. One of them eventually becomes a server and opens a port for connections. Then the other one is the client that has to connect to that port. The code will look something like this:
Server:
program server
use mpi_f08
implicit none
integer :: error
type(MPI_Comm) :: intercomm
real, dimension(5) :: data = [1,2,3,4,5]
character(len=MPI_MAX_PORT_NAME) :: port_name
! Normal MPI initialization
call MPI_Init(error)
! Here we open a port for incoming connections
call MPI_Open_port(MPI_INFO_NULL, port_name, error)
! Copy it in order to pass the address to a client
print*, "PORT NAME:", port_name
! Accept the incoming connection creating the intercommunicator
call MPI_Comm_accept(port_name, MPI_INFO_NULL, 0, MPI_COMM_WORLD, intercomm, error)
! Send test data
call MPI_Send(data, 5, MPI_FLOAT, 0, 0, intercomm)
print*, "DATA SENT"
! Close connection
call MPI_Comm_disconnect(intercomm, error)
call MPI_Finalize(error)
end program server
Client:
program client
use mpi_f08
implicit none
integer :: error
type(MPI_Comm) :: intercomm
type(MPI_Status) :: status
real, dimension(5) :: data = [0,0,0,0,0]
character(len=MPI_MAX_PORT_NAME) :: port_name
call MPI_Init(error)
! Here we copy the port name obtained from the server
print*, "Type in port name"
read(*,*) port_name
! Establish a connection creating the intercommunicator
call MPI_Comm_connect(port_name, MPI_INFO_NULL, 0, MPI_COMM_WORLD, intercomm, error)
! Receive test data
call MPI_Recv(data, 5, MPI_FLOAT, 0, 0, intercomm, status, error)
print*, "DATA RECEIVED", data
! Close connection
call MPI_Comm_disconnect(intercomm, error)
call MPI_Finalize(error)
end program client
In a real program you may find some other way of transferring the information about the port address to the client (e.g. name publishing or file system transfer). Then you can just run your code like this:
mpirun -n <N1> server &
mpirun -n <N2> client
Few things to note about this approach. If you create only one intercommunicator - you will have only one MPI task from each side communicating to the partner code. If you need to send a lot of data - you may want to consider creating multiple intercommunicators. In addition, the implementation of this part of the MPI standard may be somewhat finicky (for instance in Open MPI 2.x there was a bug preventing its usage completely).