I'm in the process of adding an option to a Fortran program to run it using multiple processors using MPI. If the user is going to run it in parallel, the user needs to specify different input files---one file for each domain (processor) of the problem. The program will look for a specific filename by default (a file called "serial.inp"). So I need the program to know when it is being run in parallel so that it can instead look for the other filenames instead (e.g. "parallel_1.inp", "parallel_2.inp", "parallel_3.inp", etc.). My first thought is to have the user pass an argument to the program when they execute it, e.g.:
mpiexec -n 4 myprogram.exe -parallel
This way, it will look for the parallel files when that argument is present. But it seems kind of redundant. If the program is being called with mpiexec, there is no question that the user is attempting to run it in parallel. Is there any way that my program will know it was started using mpiexec? Or is the command line argument my best bet?
Alex Leach is right in that you can do this with MPI-implementation-specific environment variable lookups, but there's no portable way to do this.
But as I understand, I don't think you really need to; you can get most of what you want with just checking to see if it was run with one rank:
program filenames
use mpi
implicit none
integer :: comsize, rank, ierr
character(len=128) :: inputfilename
call MPI_Init(ierr)
call MPI_Comm_size(MPI_COMM_WORLD,comsize,ierr)
call MPI_Comm_rank(MPI_COMM_WORLD,rank,ierr)
if (comsize == 1) then
inputfilename = 'serial.inp'
else
write(inputfilename, '(A,I0,A)'), 'parallel_',rank,'.imp'
endif
write(*,'(I,1X,A)'), rank, trim(inputfilename)
call MPI_Finalize(ierr)
end program filenames
Running gives
$ mpirun -np 4 ./filenames
0 parallel_0.imp
1 parallel_1.imp
2 parallel_2.imp
3 parallel_3.imp
$ ./filenames
0 serial.inp
That's not perfect; it'll give the serial result if you run using mpirun -np 1 filenames, but depending on your use case that may not be a terrible thing in exchange for having something portable.
Processes run with mpiexec will have various environment variables set, indicating to the subprocesses whether they are the master process or slaves, amongst other things.
Look in your mpiexec's documentation for specific details. Microsoft have some documentation online too.
Why not do it programmatically? This is how I do it in my program:
#ifdef MPI
CALL MPI_Init(ierr) ! Initialize MPI
CALL MPI_Comm_rank(mpicomm,nproc,ierr) ! Who am I?
CALL MPI_Comm_size(mpicomm,size,ierr) ! How many processes?
#else
nproc = 0
size = 1
#endif
After this point in the program, you can inquire whether the program is serial or parallel by inquiring the value of size.
Related
Is there any possibility to launch an external program from Fortran and write something to this programs standard input?
I know e.g. of gfortran's SYSTEM but there is no such option.
As you have noticed, GFortran does not have anything like the pipe()/fork()/exec() functions builtin.
If you're on Linux or some other Unix-like system you could do something like
execute_command_line("mkfifo /path/to/fifo")
The mkfifo command creates a named pipe, that is, a pipe that also has a name in the filesystem.
open(newunit=plot_unit, file="/path/to/fifo", access="stream", format="formatted")
execute_command_line("gnuplot < /path/to/fifo")
So the idea is that you can then open the FIFO like a normal external unit in GFortran, then execute gnuplot with standard input connected to the FIFO.
You might need to exchange the order of #2 and #3 in case this deadlocks. But some minor variation of the above should work (I've used it to connect to one Fortran program from another).
Firstly, if you're using a relatively recent compiler you should be able to use execute_command_line (part of the f2008 spec) instead of system (compiler extension). This launches a command using the C library's system call which uses the sh shell on nix and cmd.exe on Windows (see here). As such you can use standard input redirection type approaches to connect to stdin of the launched program, but it may not be suitable for more complicated use.
The following example shows a simple example
program stdIn
implicit none
character(len=20) :: cmd, args
character(len=50) :: fullcmd
cmd = "bc"
args = "1+2"
fullcmd = cmd//" <<< "//args
print*,"Running ",fullcmd
call execute_command_line(fullcmd)
end program stdIn
Which should output
Running bc <<< 1+2
3
This one's going to take a bit of explaining. Please bear with me.
What I Have
I have in my possession some Fortran source code and some binaries that have been compiled from that code. I did not do the compilation, but there is a build script that suggests G77 was used to do it.
As well as the Fortran stuff, there is also some Java code that provides users with a GUI "wrapper" around the binaries. It passes information between itself and the binaries via their input/output/error pipes. The Java code is very messy, and this way of doing things adds a lot of boilerplate and redundancy, but it does the job and I know it works.
What I Need
Unfortunately, I'd like to make some changes:
I want to create a new Python wrapper for the binaries (or, more precisely, extend an existing Python program to become the new wrapper).
I want to be able to compile the Fortran code as part of this existing program's build process. I would like to use gfortran for this, since MinGW is used elsewhere in the build and so it will be readily available.
The Problem
When I compile the Fortran code myself using gfortran, I cannot get the resulting binaries to "talk" to either the current Java wrapper or my new Python wrapper.
Here are the various ways of printing to the console that I have tried in the Fortran code:
subroutine printA(message)
write(6,*) message
end
subroutine printB(message)
write(*,*) message
end
subroutine printC(message)
use iso_fortran_env
write(output_unit,*) message
end
There are also read commands as well, but the code doesn't even get a change to execute that part so I'm not worrying about it yet.
Extra Info
I have to call gfortran with the -ffixed-line-length-132 flag so that the code compiles, but apart from that I don't use anything else. I have tried using the -ff2c flag in the vague hope that it will make a difference. It doesn't.
This stackoverflow post is informative, but doesn't offer me anything that works.
The relavant manual page suggests that printA should work just fine.
I'm working on Windows, but will need this to be multi-platform.
Juse in case you're intested, the Java code uses Runtime.getRuntime().exec("prog.exe") to call the binaries and then the various "stream" methods of the resulting Process object to communicate with them. The Python code uses equivalents of this provided by the Popen object of the subprocess module.
I should also say that I am aware there are alternatives. Rewriting the code in Python (or something else like C++), or making amendments so that is it can be called via F2Py have been ruled out as options. Using g77 is also a no-go; we have enough dependencies as it is. I'd like to be able to write to / read from the console properly with gfortran, or know that it's just not possible.
Hard to say without seeing more details from your Fortran and Python codes. The following pair of code works for me (at least under Linux):
Fortran program repeating its input line by line prefixed with line number:
program test_communication
use iso_fortran_env, stdout => output_unit, stdin => input_unit
implicit none
character(100) :: buffer
integer :: ii
ii = 1
do while (.true.)
read(stdin, *) buffer
write(stdout, "(I0,A,A)") ii, "|", trim(buffer)
flush(stdout)
ii = ii + 1
end do
end program test_communication
Python program invoking the Fortran binary. You can feed it with arbitrary strings from the console.
import subprocess as sub
print "Starting child"
proc = sub.Popen("./a.out", stdin=sub.PIPE, stdout=sub.PIPE)
while True:
send = raw_input("Enter a string: ")
if not send:
print "Exiting loop"
break
proc.stdin.write(send)
proc.stdin.write("\n")
proc.stdin.flush()
print "Sent:", send
recv = proc.stdout.readline()
print "Received:", recv.rstrip()
print "Killing child"
proc.kill()
I am an MPI and Fortran 77 noob. I have a fortran 77 code FKRPRO.f which I wanted to parallelize using OpenMPI. The code requires a lot of parameters which are fed into it during run time from a separate file. Compilation and running is something like this
gfortran -o FKRPRO FKRPRO.f
./FKRPRO < Modelfile.txt
the equivalent lines in the code (not my code) are
PARAMETER(LIN=5)
INTEGER ERROR
LOGICAL PRNTA
PRNTA=.FALSE.
READ(LIN,'(L3)') PRNTA
READ(LIN,21) M1,M2
21 FORMAT(11I5)
and so on. Can someone please explain to me what READ(LIN,'(L3)') PRNTA means. The input in the input file Modelfile.txt is something like this
.F.
0 64
and so on..
I put the necessary MPI statements in the code.
INCLUDE 'MPIF.H'
...
CALL MPI_INIT(ERROR)
CALL MPI_COMM_SIZE(MPI_COMM_WORLD,NPROCS,ERROR)
CALL MPI_COMM_RANK(MPI_COMM_WORLD,PRANK,ERROR)
...
CALL MPI_TYPE_FREE(NEWMATRIX,ERROR)
CALL MPI_FINALIZE(ERROR)
All processes are not being able to read the input file. I have compiled and run the code like this
mpif77 -o bc3 FKRPROG5.f
mpirun -np 4 bc3 < Modelfile.txt
This is not working. I get the following errors. Only the first process or rank 0 can read the file.
At line 50 of file FKRPROG5.f (unit = 5, file = 'stdin')
Fortran runtime error: End of file
At line 50 of file FKRPROG5.f (unit = 5, file = 'stdin')
Fortran runtime error: End of file
At line 50 of file FKRPROG5.f (unit = 5, file = 'stdin')
Fortran runtime error: End of file
mpirun has exited due to process rank 3 with PID 866 on
node Avinash-rMBP.local exiting improperly. There are two reasons this could occur:
1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.
2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"
This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
50th line is READ(LIN,'(L3)') PRNTA.Someone kindly point out where I am going wrong :(
So, how can I make all processes read from this input file < Modelfile.txt ?? Thanks.
The statement
READ(LIN,'(L3)') PRNTA
causes the program to read, from the unit attached to the channel with id LIN, a 3-character sequence which represents a logical value and assigns the value read to the variable PRNTA. From the fragments you've shown us the program will read .F. and set PRNTA to .false..
LIN is set to the constant value 5, which usually means stdin. This use of 5 to denote stdin is not a de jure standard, it is more of a de facto standard.
The straightforward way to read a parameter file into an MPI program is to ensure that only one process reads the file and then sends out the values to the other processes which need them.
You seem to have written a program in which all processes try to read the same input file but, at run-time, the redirection you've used to pass Modelfile.txt is only working for one process (presumably the process with rank 0). The other processes are not getting an input file at all and are complaining, then bringing the program crashing down. The error message you show is typical of a Fortran program which doesn't find an input file at all when it tries to read.
Far better to write code along the lines:
call mpi_init ...
...
if (myrank==0) then
open(...) inputfile
read(...) parameters
close(...)
end if
...
call mpi_bcast(parameters from 0 to all)
...
In general, don't expect the run-time environment for MPI processes to be identical copies of the run-time environment for a sequential program. I think that you are seeing evidence that your run-time directs the input only to the first process created when your program runs. Since mpirun is not standardised (though mpiexec is) I don't think you can rely on this run-time behaviour being the same for all MPI implementations. For portability and compatibility you're better off handling I/O explicitly within your program than using o/s features such as redirection.
You could, rather than have process 0 read the parameters and distribute them to other processes, write your code such that each process reads the same file. If you do write your code this way take care to ensure that the processes aren't fighting over access to the I/O channels; having multiple processes trying to (nearly-)simultaneously read across a single input channel is a sure way to slow things down.
i have been using MPI for a while, but i'm not experienced. So
i'm here to ask an advice on general structure of the following implementation.
Say, I have the main C++ file with
MPI_Init(&narg,&arg);
int me,nprocs;
MPI_Comm_rank(MPI_COMM_WORLD,&me);
MPI_Comm_size(MPI_COMM_WORLD,&nprocs);
int N = 10;
for (int i=0;i<N;i++) {
//(1)do some stuff in parallel...
//(2)gather results and write an input file for executable
MPI_Barrier(MPI_COMM_WORLD);
//(3)run executable in parallel.
// which is usually run from command line as:
//
// mpirun -np 6 external.exe < input.file
//
MPI_Barrier(MPI_COMM_WORLD);
//(4)gather output from executable, distribute info among processors and keep running
}
MPI_Finalize();
it's the (3) where i have a problem understanding how to do it and tell how many processors can it use. My confusion is also that some kind of "run" command should probably be executed from a single processor/instance. So how do i make it work and let parallel executable use all processors which were provided to the main program? If it is possible.
p/s/ i saw similar questions here in stackoverflow, but no definite answer on if it is possible or not.
Do you have control over the exe, i.e. can you change its code? If so, I'd suggest re-developing it so that the exe is simply a wrapper around the behavior you need, and then you can link in the actual action into your application.
If that is not an option, I suggest just calling the executable from your master (rank 0) process, and let the others wait. Not super efficient, but it'll do the job:
if (me == 0) {
system("mpirun -np 6 external.exe < input.file")
}
You'll have to figure out a way to wait until the command is finished, but according to the docs of system and mpirun it should be as simple as checking if the return value from system(...) is zero, and then continue (after a barrier, as in your example).
I wrote a subroutine for a Fortran program and I want to execute a command (delete file in the program directory and open a new one) the first, and only first time, the subroutine is called. I know that I can achieve this by creating some flag outside of the subroutine, in the main program, that I set to false at program startup and then set to true upon entering the subroutine. Then I could use this flag in an if statement to figure if the commands I want to execute on the initial call should be executed or not. But this requires me modifying the existing program and I didn't want to do that if I could avoid it. Is there some other way to do what I want to do?
An example might be:
subroutine test(a)
implicit none
integer, intent(inout) :: a
logical, save :: first_time=.true.
if(first_time) then
first_time=.false.
a = a + 12345
else
a = a - 67890
end if
end subroutine test
How about using some characteristic of the output file to determine whether or not to delete it? Time stamp, file lock, a particular file extension, etc.