The background of this question is in some computational areas such as Computational Fluid Dynamics (CFD). We often need finer mesh/grid in some critical regions while the background mesh can be coarser. For example the adaptive refine mesh to track shock waves and nesting domains in meteorology.
The Cartesian topology is used and domain decomposition is shown in the following sketch. In this case, 4*2=8 processors are used. The single number means the processor's rank and (x,y) means its topological coordinate.
Assume the mesh is refined in the regions with ranks 2, 3, 4, 5 (in the middle) and the local refinement ratio is defined as R=D_coarse/D_fine=2 in this case. Since the mesh is refined, so the time advancement should also be refined as well. This needs in the refined region the time steps t, t+1/2*dt, t+dt should be computed while only time steps t and t+dt are computed in global regions. This requires a smaller communicator which only includes ranks in the middle for extra computation. A global rank + coordinate and correspondent local ones (in red) sketch is shown as following:
However, I have some errors in implementation of this scheme and a snippet code in Fortran (not complete) is shown:
integer :: global_comm, local_comm ! global and local communicators
integer :: global_rank, local_rank !
integer :: global_grp, local_grp ! global and local groups
integer :: ranks(4) ! ranks in the refined region
integer :: dim ! dimension
integer :: left(-2:2), right(-2:2) ! ranks of neighbouring processors in 2 directions
ranks=[2,3,4,5]
!---- Make global communicator and their topological relationship
call mpi_init(ierr)
call mpi_cart_create(MPI_COMM_WORLD, 2, [4,2], [.false., .false.], .true., global_comm, ierr)
call mpi_comm_rank(global_comm, global_rank, ierr)
do dim=1, 2
call mpi_cart_shift(global_comm, dim-1, 1, left(-dim), right(dim), ierr)
end do
!---- make local communicator and its topological relationship
! Here I use group and create communicator
! create global group
call mpi_comm_group(MPI_COMM_WORLD, global_grp, ierr)
! extract 4 ranks from global group to make a local group
call mpi_group_incl(global_grp, 4, ranks, local_grp, ierr)
! make new communicator based on local group
call mpi_comm_create(MPI_COMM_WORLD, local_grp, local_comm, ierr)
! make topology for local communicator
call mpi_cart_create(global_comm, 2, [2,2], [.false., .false.], .true., local_comm, ierr)
! **** get rank for local communicator
call mpi_comm_rank(local_comm, local_rank, ierr)
! Do the same thing to make topological relationship as before in local communicator.
...
When I run the program, the problem comes from ' **** get rank for local communicator' step. My idea is to build two communicators: global and local communicators and local one is embedded in the global one. Then create their correspondent topological relationship in global and local communicators respectively. I do not if my concept is wrong or some syntax is wrong. And thank you very much if you can give me some suggestions.
The error message is
*** An error occurred in MPI_Comm_rank
*** reported by process [817692673,4]
*** on communicator MPI_COMM_WORLD
*** MPI_ERR_COMM: invalid communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
You are creating a 2x2 Cartesian topology from the group of the global communicator, which contains eight ranks. Therefore, in four of them the value of local_comm as returned by MPI_Cart_create will be MPI_COMM_NULL. Calling MPI_Comm_rank on the null communicator results in the error.
If I understand your logic correctly, you should instead do something like:
if (local_comm /= MPI_COMM_NULL) then
! make topology for local communicator
call mpi_cart_create(local_comm, 2, [2,2], [.false., .false.], .true., &
local_cart_comm, ierr)
! **** get rank for local communicator
call mpi_comm_rank(local_cart_comm, local_rank, ierr)
...
end if
Related
In MPI we work with multiple process that do not share anything but communicate with recv/send operations. The recv/send operations are done with respect to a communicator which can be the whole set of processors or a subset of them. The basic commands are:
call MPI_Comm_size ( MPI_COMM_WORLD, nproc, ierr )
call MPI_Comm_rank ( MPI_COMM_WORLD, myrank, ierr )
with MPI_COMM_WORLD the communicator associated to the set of all processors. One interesting feature of MPI is that we can run several executables together with the command:
mpirun -n 3 prog1 : -n 2 prog2
with 3 nodes assigned to the first executable and 2 to the second. However for practical work, one would like to have a communicator associated to prog1 or prog2. Is there a way to get thi directly without using the command MPI_COMM_SPLIT?
There is no such predefined communicator specified by the standard.
The great philosopher Jagger once said “you can’t always get what you want” and your best bet here is indeed to use MPI_Comm_split() and the value of MPI_COMM_WORLD's MPI_APPNUM attribute as the color argument.
From the MPI 3.1 standard chapter 10.5.3
10.5.3 MPI_APPNUM
There is a predefined attribute MPI_APPNUM of MPI_COMM_WORLD. In Fortran, the attribute is an integer value. In C,
the attribute is a pointer to an integer value. If a process was
spawned with MPI_COMM_SPAWN_MULTIPLE, MPI_APPNUM is the command number
that generated the current process. Numbering starts from zero. If a
process was spawned with MPI_COMM_SPAWN, it will have MPI_APPNUM equal
to zero. Additionally, if the process was not started by a spawn call,
but by an implementation specific startup mechanism that can handle
multiple process specifications, MPI_APPNUM should be set to the number
of the corresponding process specification. In particular, if it is
started with
mpiexec spec0 [: spec1 : spec2 : ...]
MPI_APPNUM should be set to the number of the corresponding specification.
If an application was not spawned with MPI_COMM_SPAWN or
MPI_COMM_SPAWN_MULTIPLE, and MPI_APPNUM does not make sense in the
context of the implementation-specific startup mechanism, MPI_APPNUM is
not set.
MPI implementations may optionally provide a mechanism to
override the value of MPI_APPNUM through the info argument. MPI
reserves the following key for all SPAWN calls.
appnum Value contains
an integer that overrides the default value for MPI_APPNUM in the
child.
Rationale.
When a single application is started, it is able to
figure out how many processes there are by looking at the size of
MPI_COMM_WORLD. An application consisting of multiple SPMD
sub-applications has no way to find out how many sub-applications there
are and to which sub-application the process belongs. While there are
ways to figure it out in special cases, there is no general mechanism.
MPI_APPNUM provides such a general mechanism. (End of rationale.)
For anybody interested in splitting the MPI_COMM_WORLD, I wrote a small utility library (one header). It splits the communicator into local communicators and establishes intercommunicators between them. It takes care of many of the technicalities:
You can find the header at: https://github.com/cfd-go/MPI_MPMD
It also gives the same syntax for running multiple programs at once, and spawning programs. If you have any questions or ideas to extend this, open an issue at github.
For example, in your first program you do:
MPMDHelper MPMD;
MPI_Init(&argc, &argv);
MPMD.Init(MPI_COMM_WORLD, "programA");
MPMD.local //<-- this is your local Comm.
In the other program you do:
MPMDHelper MPMD;
MPI_Init(&argc, &argv);
MPMD.Init(MPI_COMM_WORLD, "programB");
MPMD.local //<-- this is your local Comm.
MPMD["programA"].local //<-- intercommunicator for communication with programA
Hope it helps.
An alternative option is to use the client-server mechanism that is described by MPI standard (the chapter on this can be found here). The idea there is that you compile two independent MPI applications. One of them eventually becomes a server and opens a port for connections. Then the other one is the client that has to connect to that port. The code will look something like this:
Server:
program server
use mpi_f08
implicit none
integer :: error
type(MPI_Comm) :: intercomm
real, dimension(5) :: data = [1,2,3,4,5]
character(len=MPI_MAX_PORT_NAME) :: port_name
! Normal MPI initialization
call MPI_Init(error)
! Here we open a port for incoming connections
call MPI_Open_port(MPI_INFO_NULL, port_name, error)
! Copy it in order to pass the address to a client
print*, "PORT NAME:", port_name
! Accept the incoming connection creating the intercommunicator
call MPI_Comm_accept(port_name, MPI_INFO_NULL, 0, MPI_COMM_WORLD, intercomm, error)
! Send test data
call MPI_Send(data, 5, MPI_FLOAT, 0, 0, intercomm)
print*, "DATA SENT"
! Close connection
call MPI_Comm_disconnect(intercomm, error)
call MPI_Finalize(error)
end program server
Client:
program client
use mpi_f08
implicit none
integer :: error
type(MPI_Comm) :: intercomm
type(MPI_Status) :: status
real, dimension(5) :: data = [0,0,0,0,0]
character(len=MPI_MAX_PORT_NAME) :: port_name
call MPI_Init(error)
! Here we copy the port name obtained from the server
print*, "Type in port name"
read(*,*) port_name
! Establish a connection creating the intercommunicator
call MPI_Comm_connect(port_name, MPI_INFO_NULL, 0, MPI_COMM_WORLD, intercomm, error)
! Receive test data
call MPI_Recv(data, 5, MPI_FLOAT, 0, 0, intercomm, status, error)
print*, "DATA RECEIVED", data
! Close connection
call MPI_Comm_disconnect(intercomm, error)
call MPI_Finalize(error)
end program client
In a real program you may find some other way of transferring the information about the port address to the client (e.g. name publishing or file system transfer). Then you can just run your code like this:
mpirun -n <N1> server &
mpirun -n <N2> client
Few things to note about this approach. If you create only one intercommunicator - you will have only one MPI task from each side communicating to the partner code. If you need to send a lot of data - you may want to consider creating multiple intercommunicators. In addition, the implementation of this part of the MPI standard may be somewhat finicky (for instance in Open MPI 2.x there was a bug preventing its usage completely).
I'm having a segmentation fault that I can not really understand in a simple code, that just:
calls the MPI_INIT
duplicates the global communicator, via MPI_COMM_DUP
creates a group with half of processes of the global communicator, via MPI_COMM_GROUP
finally from this group creates a new communicator via MPI_COMM_CREATE_GROUP
Specifically I use this last call, instead of just using MPI_COMM_CREATE, because it's only collective over the group of processes contained in group, while MPI_COMM_CREATE is collective over every process in COMM.
The code is the following
program mpi_comm_create_grp
use mpi
IMPLICIT NONE
INTEGER :: mpi_size, mpi_err_code
INTEGER :: my_comm_dup, mpi_new_comm, mpi_group_world, mpi_new_group
INTEGER :: rank_index
INTEGER, DIMENSION(:), ALLOCATABLE :: rank_vec
CALL mpi_init(mpi_err_code)
CALL mpi_comm_size(mpi_comm_world, mpi_size, mpi_err_code)
!! allocate and fill the vector for the new group
allocate(rank_vec(mpi_size/2))
rank_vec(:) = (/ (rank_index , rank_index=0, mpi_size/2) /)
!! create the group directly from the comm_world: this way works
! CALL mpi_comm_group(mpi_comm_world, mpi_group_world, mpi_err_code)
!! duplicating the comm_world creating the group form the dup: this ways fails
CALL mpi_comm_dup(mpi_comm_world, my_comm_dup, mpi_err_code)
!! creatig the group of all processes from the duplicated comm_world
CALL mpi_comm_group(my_comm_dup, mpi_group_world, mpi_err_code)
!! create a new group with just half of processes in comm_world
CALL mpi_group_incl(mpi_group_world, mpi_size/2, rank_vec,mpi_new_group, mpi_err_code)
!! create a new comm from the comm_world using the new group created
CALL mpi_comm_create_group(mpi_comm_world, mpi_new_group, 0, mpi_new_comm, mpi_err_code)
!! deallocate and finalize mpi
if(ALLOCATED(rank_vec)) DEALLOCATE(rank_vec)
CALL mpi_finalize(mpi_err_code)
end program !mpi_comm_create_grp
If instead of duplicating the COMM_WORLD, I directly create the group from the global communicator (commented line), everything works just fine.
The parallel debugger I'm using traces back the seg fault to a call to MPI_GROUP_TRANSLATE_RANKS, but, as far as I know, the MPI_COMM_DUP duplicates all the attributes of the copied communicator, ranks numbering included.
I am using the ifort version 18.0.5, but I also tried with the 17.0.4, and 19.0.2 with no better results.
Well the thing is a little tricky, at least for me, but after some tests and help, the root of the problem was found.
In the code
CALL mpi_comm_create_group(mpi_comm_world, mpi_new_group, 0, mpi_new_comm, mpi_err_code)
Creates a new communicator for the group mpi_new_group, previously
created. However the mpi_comm_world, which is used as first argument, is not in the same context as mpi_new_group, as explained in the mpich reference:
MPI_COMM_DUP will create a new communicator over the same group as
comm but with a new context
So the correct call would be:
CALL mpi_comm_create_group(my_comm_copy, mpi_new_group, 0, mpi_new_comm, mpi_err_code)
I.e. , replacing the mpi_comm_world for my_comm_copy, that is the one from which the mpi_group_world was created.
I am still not sure why it is working with OpenMPI, but it is generally more tolerant
with this sort of things.
Like suggested in the comments I wrote to openmpi user list, and they replied
That is perfectly valid. The MPI processes that make up the group are all part of comm world. I would file a bug with Intel MPI.
So I try and post a question on Intel forum.
It is a bug they solved in the last version of the libray, 19.3.
Can someone explain and tell me more about MPI_Comm_split communicator?
MPI_Comm_split(MPI_COMM_WORLD, my_row, my_rank,&my_row_comm);
This is just example i met by reading some basic documentations. Maybe someone could tell me how this communicator is working?
Just to begin with, let's have a look at the man page:
MPI_Comm_split(3) MPI MPI_Comm_split(3)
NAME
MPI_Comm_split - Creates new communicators based on colors and keys
SYNOPSIS
int MPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm *newcomm)
INPUT PARAMETERS
comm - communicator (handle)
color - control of subset assignment (nonnegative integer). Processes
with the same color are in the same new communicator
key - control of rank assignment (integer)
OUTPUT PARAMETERS
newcomm
- new communicator (handle)
So what does that do?
Well, as the name suggests, it will split the communicator comm into disjoint sub-communicators newcomm. Each process of comm will be into one unique of these sub-communicators, hence the fact that the output newcomm is only one single communicator (for the current process). However, globally speaking, you have to understand that the many versions of newcomm are different sub-communicators, partitioning the input comm.
So that is what the function does. But how does it do it?
Well, that's where the two parameters color and key come into play:
color is an integer value that permits to decide in which of the sub-communicators the current process will fall. More specifically, all processes of comm for which color will have the same numerical value will be part of the same sub-communicator newcomm. For example, if you were to define color = rank%2; (with rank the rank of the process in comm), then you would create (globally) two new communicators: one for the processes of odd ranks, and one for the processes of even ranks. However, keep in mind that each processes will only be seeing the one of these new communicators they are part of... So in summary, color permits to tell apart the various "teams" you will create, like the colour of the jersey football teams will wear to distinguish themselves during a match (hence the naming I presume).
key will just permit to optionally decide how the processes will be ranked into the new communicators they are part of. For example, if you set key = rank;, then the order of ranking (not the ranking itself) in each new communicators newcomm will follow the order of ranking in the original communicator comm. But if you don't care about the ordering, you can as well set key=0; and the ranking in each of the new communicators will be whatever the library decides...
Finally, two trivial examples:
MPI_Comm_split(comm, 0, rank, &newcomm) will just duplicate comm into newcomm (just as MPI_Comm_dup())
MPI_Comm_split(comm, rank, rank, &newcomm) will just return an equivalent of MPI_COMM_SELF for each of the processes
I am writing a parallel VTK file (pvti) from my fortran CFD solver. The file is really just a list of all the individual files for each piece of the data. Running MPI, if I have each process write the name of its individual file to standard output
print *, name
then I get a nice list of each file, ie
block0.vti
block1.vti
block2.vti
This is exactly the sort of list I want. But if I write to a file
write(9,*) name
then I only get one output in the file. Is there a simple way to replicate the standard output version of this without transferring data?
You could try adapting the following which uses MPI-IO, which is really the only way to ensure ordered files from multiple processes. It does assume an end of line character and that all the lines are the same length (padded with blanks if required) but I think that's about it.
Program ascii_mpiio
! simple example to show MPI-IO "emulating" Fortran
! formatted direct access files. Note can not use the latter
! in parallel with multiple processes writing to one file
! is behaviour is not defined (and DOES go wrong on certain
! machines)
Use mpi
Implicit None
! All the "lines" in the file will be this length
Integer, Parameter :: max_line_length = 30
! We also need to explicitly write a carriage return.
! here I am assuming ASCII
Character, Parameter :: lf = Achar( 10 )
! Buffer to hold a line
Character( Len = max_line_length + 1 ) :: line
Integer :: me, nproc
Integer :: fh
Integer :: record
Integer :: error
Integer :: i
! Initialise MPI
Call mpi_init( error )
Call mpi_comm_rank( mpi_comm_world, me , error )
Call mpi_comm_size( mpi_comm_world, nproc, error )
! Create a MPI derived type that will contain a line of the final
! output just before we write it using MPI-IO. Note this also
! includes the carriage return at the end of the line.
Call mpi_type_contiguous( max_line_length + 1, mpi_character, record, error )
Call mpi_type_commit( record, error )
! Open the file. prob want to change the path and name
Call mpi_file_open( mpi_comm_world, '/home/ian/test/mpiio/stuff.dat', &
mpi_mode_wronly + mpi_mode_create, &
mpi_info_null, fh, error )
! Set the view for the file. Note the etype and ftype are both RECORD,
! the derived type used to represent a whole line, and the displacement
! is zero. Thus
! a) Each process can "see" all of the file
! b) The unit of displacement in subsequent calls is a line.
! Thus if we have a displacement of zero we write to the first line,
! 1 means we write to the second line, and in general i means
! we write to the (i+1)th line
Call mpi_file_set_view( fh, 0_mpi_offset_kind, record, record, &
'native', mpi_info_null, error )
! Make each process write to a different part of the file
Do i = me, 50, nproc
! Use an internal write to transfer the data into the
! character buffer
Write( line, '( "This is line ", i0, " from ", i0 )' ) i, me
!Remember the line feed at the end of the line
line( Len( line ):Len( line ) ) = lf
! Write with a displacement of i, and thus to line i+1
! in the file
Call mpi_file_write_at( fh, Int( i, mpi_offset_kind ), &
line, 1, record, mpi_status_ignore, error )
End Do
! Close the file
Call mpi_file_close( fh, error )
! Tidy up
Call mpi_type_free( record, error )
Call mpi_finalize( error )
End Program ascii_mpii
Also please note you're just getting lucky with your standard output "solution", you're not guaranteed to get it all nice sorted.
Apart from having the writes from different ranks well mixed, your problem is that the Fortran OPEN statement probably truncates the file to zero length, thus obliterating the previous content instead of appending to it. I'm with Vladimir F on this and would write this file only in rank 0. There are several possible cases, some of which are listed here:
each rank writes a separate VTK file and the order follows the ranks or the actual order is not significant. In that case you could simply use a DO loop in rank 0 from 0 to #ranks-1 to generate the whole list.
each rank writes a separate VTK file, but the order does not follow the ranks, e.g. rank 0 writes block3.vti, rank 1 writes block12.vti, etc. In that case you can use MPI_GATHER to collect the block number from each process into an array at rank 0 and then loop over the elements of the array.
some ranks write a VTK file, some don't, and the block order does not follow the ranks. It's similar to the previous case - just have the ranks that do not write a block send a negative block number and then rank 0 would skip the negative array elements.
block numbering follows ranks order but not all ranks write a block. In that case you can use MPI_GATHER to collect one LOGICAL value from each rank that indicates if it has written a block or not.
If you are not in a hurry, you can force the output from different tasks to be in order:
! Loop over processes in order
DO n = 0,numProcesses-1
! Write to file if it is my turn
IF(nproc == n)THEN
! Write output here
ENDIF
! This call ensures that all processes wait for each other
#ifdef MPI
CALL MPI_Barrier(mpi_comm_world,ierr)
#endif
ENDDO
This solution is simple, but not efficient for very large output. This does not seem to be your case. Make sure you flush the output buffer after each write. If using this method, make sure to do tests before implementing, as success is not guaranteed on all architectures. This method works for me for outputting large NetCDF files without the need to pass the data around.
I am having trouble with MPI_BCAST in Fortran. I create a new communicator using MPI_CART_CREATE (say 'COMM_NEW'). When I broadcast data from root using old communicator (i.e. MPI_COMM_WORLD) it works fine. But, when i use new communicator that i just created it gives the error:
[compute-4-15.local:15298] *** An error occurred in MPI_Bcast
[compute-4-15.local:15298] *** on communicator MPI_COMM_WORLD
[compute-4-15.local:15298] *** MPI_ERR_COMM: invalid communicator
[compute-4-15.local:15298] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
It do get the result from the processors involved in COMM_NEW, and also the above error, think the problem is with other processors which are not included in COMM_NEW, but are present in MPI_COMM_WORLD. Any help will be greatly appreciated. Is it because the number of processors in COMM_NEW is less than total processors. If so how do i broadcast among a set of processors which are less than the total. Thanks.
My sample code is:
!PROGRAM TO BROADCAST THE DATA FROM ROOT TO DEST PROCESSORS
PROGRAM MAIN
IMPLICIT NONE
INCLUDE 'mpif.h'
!____________________________________________________________________________________
!-------------------------------DECLARE VARIABLES------------------------------------
INTEGER :: ERROR, RANK, NPROCS, I
INTEGER :: SOURCE, TAG, COUNT, NDIMS, COMM_NEW
INTEGER :: A(10), DIMS(1)
LOGICAL :: PERIODS(1), REORDER
!____________________________________________________________________________________
!-------------------------------DEFINE VARIABLES-------------------------------------
SOURCE = 0; TAG = 1; COUNT = 10
PERIODS(1) = .FALSE.
REORDER = .FALSE.
NDIMS = 1
DIMS(1) = 6
!____________________________________________________________________________________
!--------------------INITIALIZE MPI, DETERMINE SIZE AND RANK-------------------------
CALL MPI_INIT(ERROR)
CALL MPI_COMM_SIZE(MPI_COMM_WORLD, NPROCS, ERROR)
CALL MPI_COMM_RANK(MPI_COMM_WORLD, RANK, ERROR)
!
CALL MPI_CART_CREATE(MPI_COMM_WORLD, NDIMS, DIMS, PERIODS, REORDER, COMM_NEW, ERROR)
IF(RANK==SOURCE)THEN
DO I=1,10
A(I) = I
END DO
END IF
!____________________________________________________________________________________
!----------------BROADCAST VECTOR A FROM ROOT TO DESTINATIONS------------------------
CALL MPI_BCAST(A,10,MPI_INTEGER,SOURCE,COMM_NEW,ERROR)
!PRINT*, RANK
!WRITE(*, "(10I5)") A
CALL MPI_FINALIZE(ERROR)
END PROGRAM
I think the error you give at the top of your question doesn't match up with the code at the bottom since it's complaining about a Bcast on MPI_COMM_WORLD and you don't actually do one in your code.
Anyway, if you're running with more processes than dimensions, some of the processes won't be included in COMM_NEW. Instead, when the call to MPI_CART_CREATE returns, they'll get MPI_COMM_NULL for COMM_NEW instead of the new communicator with the topology. You just need to do a check to make sure you have a real communicator instead of MPI_COMM_NULL before doing the Bcast (or just have all of the ranks above DIMS(1) not enter the Bcast.
To elaborate on Wesley Bland's answer and to clarify the apparent discrepancy in the error message. When the number of MPI processes in MPI_COMM_WORLD is larger than the number of processes in the created Cartesian grid, some of the processes won't become members of the new Cartesian communicator and will get MPI_COMM_NULL -- the invalid communicator handle -- as a result. Calling a collective communication operation requires a valid inter- or intra-communicator handle. Unlike the allowed usage of MPI_PROC_NULL in point-to-point operations, using the invalid communicator handle in collective calls is erroneous. The last statement is not explicitly written in the MPI standard - instead, the language used is:
If comm is an intracommunicator, then ... If comm is an intercommunicator, then ...
Since MPI_COMM_NULL is neither an intra-, nor an inter-communicator, it doesn't fall in any of the two categories of defined behaviour and hence leads to an error condition.
Since communication errors have to occur in some context (i.e. in a valid communicator), Open MPI substitutes MPI_COMM_WORLD in the call to the error handler and hence the error message says "*** on communicator MPI_COMM_WORLD". This is the relevant code section from ompi/mpi/c/bcast.c, where MPI_Bcast is implemented:
if (ompi_comm_invalid(comm)) {
return OMPI_ERRHANDLER_INVOKE(MPI_COMM_WORLD, MPI_ERR_COMM,
FUNC_NAME);
}
...
if (MPI_IN_PLACE == buffer) {
return OMPI_ERRHANDLER_INVOKE(comm, MPI_ERR_ARG, FUNC_NAME);
}
Your code triggers the error handler inside the first check. In all other error checks comm is used instead (since it is determined to be a valid communicator handle) and the error message will state something like "*** on communicator MPI COMMUNICATOR 5 SPLIT FROM 0".