Currently I am working on an MPI code in Fortran. After using mpi_cart_create then mpi_group_excl to create a new group with half of the nodes in it, I am trying to perform a reduction using this communicator but I am obviously doing something wrong.
With the code
call MPI_cart_create(MPI_comm_world, 2, dims, (/.false.,.false./), reorder, comm_cart, ierr)
if (ierr/=0) stop 'Error with MPI_cart_create'
call MPI_group_excl(group_world, dims(2), excl_a, division_comm_a, ierr)
if (ierr/=0) stop 'Error with MPI_group_excl - division_comm_a'
call MPI_group_excl(group_world, dims(2), excl_b, division_comm_b, ierr)
if (ierr/=0) stop 'Error with MPI_group_excl - division_comm_b'
if (div_a_rank .gt. 0) then
call MPI_reduce(division_a(1), division_a(1), L_outer_y, MPI_DOUBLE_PRECISION, MPI_SUM, &
& 0, division_comm_a, ierr)
if (ierr/=0) stop 'Error with MPI_reduce on division_comm_a'
end if
the error I am getting is :
*** An error occurred in MPI_Reduce
*** on communicator MPI_COMM_WORLD
*** MPI_ERR_ARG: invalid argument of some other kind
*** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
Following an answer I have used MPI_comm_create_group, however I am still getting
*** An error occurred in MPI_Reduce
*** reported by process [140046521663489,140045998620672]
*** on communicator MPI_COMM_WORLD
*** MPI_ERR_ARG: invalid argument of some other kind
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
The problem is that you're mixing groups and communicators. In MPI, a group is just a logical collection of processes. It can't be used to communicate.
If you want to create a new communicator from your new group, you should use the function MPI_COMM_CREATE_GROUP. You can pass your new group into that function to create a new communicator that you can use for your reduction.
Related
In MPI we work with multiple process that do not share anything but communicate with recv/send operations. The recv/send operations are done with respect to a communicator which can be the whole set of processors or a subset of them. The basic commands are:
call MPI_Comm_size ( MPI_COMM_WORLD, nproc, ierr )
call MPI_Comm_rank ( MPI_COMM_WORLD, myrank, ierr )
with MPI_COMM_WORLD the communicator associated to the set of all processors. One interesting feature of MPI is that we can run several executables together with the command:
mpirun -n 3 prog1 : -n 2 prog2
with 3 nodes assigned to the first executable and 2 to the second. However for practical work, one would like to have a communicator associated to prog1 or prog2. Is there a way to get thi directly without using the command MPI_COMM_SPLIT?
There is no such predefined communicator specified by the standard.
The great philosopher Jagger once said “you can’t always get what you want” and your best bet here is indeed to use MPI_Comm_split() and the value of MPI_COMM_WORLD's MPI_APPNUM attribute as the color argument.
From the MPI 3.1 standard chapter 10.5.3
10.5.3 MPI_APPNUM
There is a predefined attribute MPI_APPNUM of MPI_COMM_WORLD. In Fortran, the attribute is an integer value. In C,
the attribute is a pointer to an integer value. If a process was
spawned with MPI_COMM_SPAWN_MULTIPLE, MPI_APPNUM is the command number
that generated the current process. Numbering starts from zero. If a
process was spawned with MPI_COMM_SPAWN, it will have MPI_APPNUM equal
to zero. Additionally, if the process was not started by a spawn call,
but by an implementation specific startup mechanism that can handle
multiple process specifications, MPI_APPNUM should be set to the number
of the corresponding process specification. In particular, if it is
started with
mpiexec spec0 [: spec1 : spec2 : ...]
MPI_APPNUM should be set to the number of the corresponding specification.
If an application was not spawned with MPI_COMM_SPAWN or
MPI_COMM_SPAWN_MULTIPLE, and MPI_APPNUM does not make sense in the
context of the implementation-specific startup mechanism, MPI_APPNUM is
not set.
MPI implementations may optionally provide a mechanism to
override the value of MPI_APPNUM through the info argument. MPI
reserves the following key for all SPAWN calls.
appnum Value contains
an integer that overrides the default value for MPI_APPNUM in the
child.
Rationale.
When a single application is started, it is able to
figure out how many processes there are by looking at the size of
MPI_COMM_WORLD. An application consisting of multiple SPMD
sub-applications has no way to find out how many sub-applications there
are and to which sub-application the process belongs. While there are
ways to figure it out in special cases, there is no general mechanism.
MPI_APPNUM provides such a general mechanism. (End of rationale.)
For anybody interested in splitting the MPI_COMM_WORLD, I wrote a small utility library (one header). It splits the communicator into local communicators and establishes intercommunicators between them. It takes care of many of the technicalities:
You can find the header at: https://github.com/cfd-go/MPI_MPMD
It also gives the same syntax for running multiple programs at once, and spawning programs. If you have any questions or ideas to extend this, open an issue at github.
For example, in your first program you do:
MPMDHelper MPMD;
MPI_Init(&argc, &argv);
MPMD.Init(MPI_COMM_WORLD, "programA");
MPMD.local //<-- this is your local Comm.
In the other program you do:
MPMDHelper MPMD;
MPI_Init(&argc, &argv);
MPMD.Init(MPI_COMM_WORLD, "programB");
MPMD.local //<-- this is your local Comm.
MPMD["programA"].local //<-- intercommunicator for communication with programA
Hope it helps.
An alternative option is to use the client-server mechanism that is described by MPI standard (the chapter on this can be found here). The idea there is that you compile two independent MPI applications. One of them eventually becomes a server and opens a port for connections. Then the other one is the client that has to connect to that port. The code will look something like this:
Server:
program server
use mpi_f08
implicit none
integer :: error
type(MPI_Comm) :: intercomm
real, dimension(5) :: data = [1,2,3,4,5]
character(len=MPI_MAX_PORT_NAME) :: port_name
! Normal MPI initialization
call MPI_Init(error)
! Here we open a port for incoming connections
call MPI_Open_port(MPI_INFO_NULL, port_name, error)
! Copy it in order to pass the address to a client
print*, "PORT NAME:", port_name
! Accept the incoming connection creating the intercommunicator
call MPI_Comm_accept(port_name, MPI_INFO_NULL, 0, MPI_COMM_WORLD, intercomm, error)
! Send test data
call MPI_Send(data, 5, MPI_FLOAT, 0, 0, intercomm)
print*, "DATA SENT"
! Close connection
call MPI_Comm_disconnect(intercomm, error)
call MPI_Finalize(error)
end program server
Client:
program client
use mpi_f08
implicit none
integer :: error
type(MPI_Comm) :: intercomm
type(MPI_Status) :: status
real, dimension(5) :: data = [0,0,0,0,0]
character(len=MPI_MAX_PORT_NAME) :: port_name
call MPI_Init(error)
! Here we copy the port name obtained from the server
print*, "Type in port name"
read(*,*) port_name
! Establish a connection creating the intercommunicator
call MPI_Comm_connect(port_name, MPI_INFO_NULL, 0, MPI_COMM_WORLD, intercomm, error)
! Receive test data
call MPI_Recv(data, 5, MPI_FLOAT, 0, 0, intercomm, status, error)
print*, "DATA RECEIVED", data
! Close connection
call MPI_Comm_disconnect(intercomm, error)
call MPI_Finalize(error)
end program client
In a real program you may find some other way of transferring the information about the port address to the client (e.g. name publishing or file system transfer). Then you can just run your code like this:
mpirun -n <N1> server &
mpirun -n <N2> client
Few things to note about this approach. If you create only one intercommunicator - you will have only one MPI task from each side communicating to the partner code. If you need to send a lot of data - you may want to consider creating multiple intercommunicators. In addition, the implementation of this part of the MPI standard may be somewhat finicky (for instance in Open MPI 2.x there was a bug preventing its usage completely).
The background of this question is in some computational areas such as Computational Fluid Dynamics (CFD). We often need finer mesh/grid in some critical regions while the background mesh can be coarser. For example the adaptive refine mesh to track shock waves and nesting domains in meteorology.
The Cartesian topology is used and domain decomposition is shown in the following sketch. In this case, 4*2=8 processors are used. The single number means the processor's rank and (x,y) means its topological coordinate.
Assume the mesh is refined in the regions with ranks 2, 3, 4, 5 (in the middle) and the local refinement ratio is defined as R=D_coarse/D_fine=2 in this case. Since the mesh is refined, so the time advancement should also be refined as well. This needs in the refined region the time steps t, t+1/2*dt, t+dt should be computed while only time steps t and t+dt are computed in global regions. This requires a smaller communicator which only includes ranks in the middle for extra computation. A global rank + coordinate and correspondent local ones (in red) sketch is shown as following:
However, I have some errors in implementation of this scheme and a snippet code in Fortran (not complete) is shown:
integer :: global_comm, local_comm ! global and local communicators
integer :: global_rank, local_rank !
integer :: global_grp, local_grp ! global and local groups
integer :: ranks(4) ! ranks in the refined region
integer :: dim ! dimension
integer :: left(-2:2), right(-2:2) ! ranks of neighbouring processors in 2 directions
ranks=[2,3,4,5]
!---- Make global communicator and their topological relationship
call mpi_init(ierr)
call mpi_cart_create(MPI_COMM_WORLD, 2, [4,2], [.false., .false.], .true., global_comm, ierr)
call mpi_comm_rank(global_comm, global_rank, ierr)
do dim=1, 2
call mpi_cart_shift(global_comm, dim-1, 1, left(-dim), right(dim), ierr)
end do
!---- make local communicator and its topological relationship
! Here I use group and create communicator
! create global group
call mpi_comm_group(MPI_COMM_WORLD, global_grp, ierr)
! extract 4 ranks from global group to make a local group
call mpi_group_incl(global_grp, 4, ranks, local_grp, ierr)
! make new communicator based on local group
call mpi_comm_create(MPI_COMM_WORLD, local_grp, local_comm, ierr)
! make topology for local communicator
call mpi_cart_create(global_comm, 2, [2,2], [.false., .false.], .true., local_comm, ierr)
! **** get rank for local communicator
call mpi_comm_rank(local_comm, local_rank, ierr)
! Do the same thing to make topological relationship as before in local communicator.
...
When I run the program, the problem comes from ' **** get rank for local communicator' step. My idea is to build two communicators: global and local communicators and local one is embedded in the global one. Then create their correspondent topological relationship in global and local communicators respectively. I do not if my concept is wrong or some syntax is wrong. And thank you very much if you can give me some suggestions.
The error message is
*** An error occurred in MPI_Comm_rank
*** reported by process [817692673,4]
*** on communicator MPI_COMM_WORLD
*** MPI_ERR_COMM: invalid communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
You are creating a 2x2 Cartesian topology from the group of the global communicator, which contains eight ranks. Therefore, in four of them the value of local_comm as returned by MPI_Cart_create will be MPI_COMM_NULL. Calling MPI_Comm_rank on the null communicator results in the error.
If I understand your logic correctly, you should instead do something like:
if (local_comm /= MPI_COMM_NULL) then
! make topology for local communicator
call mpi_cart_create(local_comm, 2, [2,2], [.false., .false.], .true., &
local_cart_comm, ierr)
! **** get rank for local communicator
call mpi_comm_rank(local_cart_comm, local_rank, ierr)
...
end if
I have some questions on mpi_comm_split in Fortran.
Question I)
How can I create a single one communicator with mpi_comm_split? For example, I want to create a communicator based only on processors which are on the top of my domain (Cartesian). I know that I have to use MPI_UNDEFINED for process that I don't want to be part of my new communicator, but my code below didn't make want I expect.
do k=1,size(proc_up)
if(rank==proc_up(k)) then
color_up=1
else
color_up=MPI_UNDEFINED
end if
call MPI_COMM_SPLIT(comm2d ,color_up ,coords(2) ,comm_up ,code)
Why it didn't work?
Question II)
When I want to make several MPI_COMM_SPLIT (new comm for up, down, side1, side2), it returns an error:
[nin:30039] *** An error occurred in MPI_Comm_split
[nin:30039] *** on communicator MPI_COMM_WORLD
[nin:30039] *** MPI_ERR_ARG: invalid argument of some other kind
[nin:30039] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
Does anyone know why?
QUESTION III)
I can also use MPI_Cart_sub, but it will returns me many groups of process. How to be sure to use only the group I just want?
I am currently having problems with a MPI Application.
I am sporadically receiving MPI errors of the form:
Fatal error in MPI_Allreduce: Message truncated, error stack:
MPI_Allreduce(1339)...............: MPI_Allreduce(sbuf=0x7ffa87ffcb98, rbuf=0x7ffa87ffcba8, count=2, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(1180).........:
MPIR_Allreduce_intra(755).........:
MPIDI_CH3U_Receive_data_found(129): Message from rank 0 and tag 14 truncated; 384 bytes received but buffer size is 16
rank 1 in job 1 l1442_42561 caused collective abort of all ranks
exit status of rank 1: killed by signal 9
However I do not know at where to look. I know that the error is happening in an Allreduce function call however there are multiple ones.
How do I know which function call produces the error? Simple printf debugging does not help as the function could be called a million times before the error occurs the first time.
It might also not occur at all or immediately after the start of the program.
I have been able to track down the origin of the error by calling
MPI_Errhandler_set(MPI_COMM_WORLD, MPI_ERRORS_RETURN)
and then checking the return value of each of the Allreduce functions for not being equal to MPI_SUCCESS. This is a location where an error occurs
I am having trouble with MPI_BCAST in Fortran. I create a new communicator using MPI_CART_CREATE (say 'COMM_NEW'). When I broadcast data from root using old communicator (i.e. MPI_COMM_WORLD) it works fine. But, when i use new communicator that i just created it gives the error:
[compute-4-15.local:15298] *** An error occurred in MPI_Bcast
[compute-4-15.local:15298] *** on communicator MPI_COMM_WORLD
[compute-4-15.local:15298] *** MPI_ERR_COMM: invalid communicator
[compute-4-15.local:15298] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
It do get the result from the processors involved in COMM_NEW, and also the above error, think the problem is with other processors which are not included in COMM_NEW, but are present in MPI_COMM_WORLD. Any help will be greatly appreciated. Is it because the number of processors in COMM_NEW is less than total processors. If so how do i broadcast among a set of processors which are less than the total. Thanks.
My sample code is:
!PROGRAM TO BROADCAST THE DATA FROM ROOT TO DEST PROCESSORS
PROGRAM MAIN
IMPLICIT NONE
INCLUDE 'mpif.h'
!____________________________________________________________________________________
!-------------------------------DECLARE VARIABLES------------------------------------
INTEGER :: ERROR, RANK, NPROCS, I
INTEGER :: SOURCE, TAG, COUNT, NDIMS, COMM_NEW
INTEGER :: A(10), DIMS(1)
LOGICAL :: PERIODS(1), REORDER
!____________________________________________________________________________________
!-------------------------------DEFINE VARIABLES-------------------------------------
SOURCE = 0; TAG = 1; COUNT = 10
PERIODS(1) = .FALSE.
REORDER = .FALSE.
NDIMS = 1
DIMS(1) = 6
!____________________________________________________________________________________
!--------------------INITIALIZE MPI, DETERMINE SIZE AND RANK-------------------------
CALL MPI_INIT(ERROR)
CALL MPI_COMM_SIZE(MPI_COMM_WORLD, NPROCS, ERROR)
CALL MPI_COMM_RANK(MPI_COMM_WORLD, RANK, ERROR)
!
CALL MPI_CART_CREATE(MPI_COMM_WORLD, NDIMS, DIMS, PERIODS, REORDER, COMM_NEW, ERROR)
IF(RANK==SOURCE)THEN
DO I=1,10
A(I) = I
END DO
END IF
!____________________________________________________________________________________
!----------------BROADCAST VECTOR A FROM ROOT TO DESTINATIONS------------------------
CALL MPI_BCAST(A,10,MPI_INTEGER,SOURCE,COMM_NEW,ERROR)
!PRINT*, RANK
!WRITE(*, "(10I5)") A
CALL MPI_FINALIZE(ERROR)
END PROGRAM
I think the error you give at the top of your question doesn't match up with the code at the bottom since it's complaining about a Bcast on MPI_COMM_WORLD and you don't actually do one in your code.
Anyway, if you're running with more processes than dimensions, some of the processes won't be included in COMM_NEW. Instead, when the call to MPI_CART_CREATE returns, they'll get MPI_COMM_NULL for COMM_NEW instead of the new communicator with the topology. You just need to do a check to make sure you have a real communicator instead of MPI_COMM_NULL before doing the Bcast (or just have all of the ranks above DIMS(1) not enter the Bcast.
To elaborate on Wesley Bland's answer and to clarify the apparent discrepancy in the error message. When the number of MPI processes in MPI_COMM_WORLD is larger than the number of processes in the created Cartesian grid, some of the processes won't become members of the new Cartesian communicator and will get MPI_COMM_NULL -- the invalid communicator handle -- as a result. Calling a collective communication operation requires a valid inter- or intra-communicator handle. Unlike the allowed usage of MPI_PROC_NULL in point-to-point operations, using the invalid communicator handle in collective calls is erroneous. The last statement is not explicitly written in the MPI standard - instead, the language used is:
If comm is an intracommunicator, then ... If comm is an intercommunicator, then ...
Since MPI_COMM_NULL is neither an intra-, nor an inter-communicator, it doesn't fall in any of the two categories of defined behaviour and hence leads to an error condition.
Since communication errors have to occur in some context (i.e. in a valid communicator), Open MPI substitutes MPI_COMM_WORLD in the call to the error handler and hence the error message says "*** on communicator MPI_COMM_WORLD". This is the relevant code section from ompi/mpi/c/bcast.c, where MPI_Bcast is implemented:
if (ompi_comm_invalid(comm)) {
return OMPI_ERRHANDLER_INVOKE(MPI_COMM_WORLD, MPI_ERR_COMM,
FUNC_NAME);
}
...
if (MPI_IN_PLACE == buffer) {
return OMPI_ERRHANDLER_INVOKE(comm, MPI_ERR_ARG, FUNC_NAME);
}
Your code triggers the error handler inside the first check. In all other error checks comm is used instead (since it is determined to be a valid communicator handle) and the error message will state something like "*** on communicator MPI COMMUNICATOR 5 SPLIT FROM 0".