mpiexec fails as MPI init aborts - mpich

Question

I am trying to install MPICH 2 on a 64-bit machine running on Ubuntu 11.04 (Natty Narwhal). I used
sudo apt-get install mpich2
First I was surprised to see that mpd was not installed. On looking up on Google, I saw that Hydra is the new default package manager.
So I tried to run my MPI code. I got the following error.
> -------------------------------------------------------------------------------------------
> [ip-10-99-75-58:02212] [[INVALID],INVALID] ORTE_ERROR_LOG: A
> system-required executable either could not be found or was not
> executable by this user in file
> ../../../../../../orte/mca/ess/singleton/ess_singleton_module.c at
> line 357 [ip-10-99-75-58:02212] [[INVALID],INVALID] ORTE_ERROR_LOG: A
> system-required executable either could not be found or was not
> executable by this user in file
> ../../../../../../orte/mca/ess/singleton/ess_singleton_module.c at
> line 230 [ip-10-99-75-58:02212] [[INVALID],INVALID] ORTE_ERROR_LOG: A
> system-required executable either could not be found or was not
> executable by this user in file ../../../orte/runtime/orte_init.c at
> line 132
> --------------------------------------------------------------------------
> It looks like orte_init failed for some reason; your parallel process
> is likely to abort. There are many reasons that a parallel process
> can fail during orte_init; some of which are due to configuration or
> environment problems. This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
>
> orte_ess_set_name failed --> Returned value A system-required
> executable either could not be found or was not executable by this
> user (-127) instead of ORTE_SUCCESS
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> It looks like MPI_INIT failed for some reason; your parallel process
> is likely to abort. There are many reasons that a parallel process
> can fail during MPI_INIT; some of which are due to configuration or
> environment problems. This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
>
> ompi_mpi_init: orte_init failed --> Returned "A system-required
> executable either could not be found or was not executable by this
> user" (-127) instead of "Success" (0)
> --------------------------------------------------------------------------
> *** The MPI_Init() function was called before MPI_INIT was invoked.
> *** This is disallowed by the MPI standard.
> *** Your MPI job will now abort.
> -------------------------------------------------------------------------------------------
First of all, it looks to me as an Open MPI error. But I installed MPICH 2 and not Open MPI.
Secondly, I am at a fix on how to handle this as all help seems to be directed to Open MPI users. Am I missing something?

Answer 1

I have the same problem on Ubuntu 12.04. I find my problem is because I have both open-mpi and mpich2 on my computer. When I compile my program using mpicc, it will be linked to open-mpi not mpich2. To fix this problem, you can use "mpicc.mpich2" to compile your program and then use "mpiexec.mpich2" to execute your code.

Answer 2

Indeed, these error messages are all Open MPI errors. For some reason, you appear to also have a (badly configured?) copy of Open MPI installed somewhere. You can check which particular file you are executing when you type mpiexec by running which mpiexec. I believe that you can compare this with the result of:
dpkg --listfiles mpich2
(or similar) in order to figure out where the MPICH2 package was installed.

Answer 3

I had this happen to me, and I found the issue. Somewhere on your system during startup LD_PRELOAD was set to point to libmpi.so in OpenMPI.
Example:
export LD_PRELOAD=<some_directory>/openmpi/1.4.4/lib/libmpi.so
The result is that MPICH2 fails. Just do 'unset LD_PRELOAD' before running MPICH2 and the issue goes away.
Note that setting LD_PRELOAD to OpenMPI's libmpi.so is actually required sometimes for OpenMPI to work, so unsetting may break OpenMPI for you. Just remember to reset it if you need to use OpenMPI.