I hope someone can help me. I am coding in Fortran 90 with MPI libraries. I am trying to read files in parallel (with function Mpi_File_Read) but am unable to. Here is a simple code which demonstrates the problem:
program Read_And_Write
implicit none
include "mpif.h"
integer, parameter :: N = 16
integer :: i, this_proc, file_handle, error
integer :: read_data(N)
integer :: status(MPI_STATUS_SIZE)
character(len=13) :: file_name = 'one_array.dat'
! Start MPI run
call Mpi_Init(error)
call Mpi_Comm_Rank(MPI_COMM_WORLD, & ! integer comm
this_proc, & ! integer rank
error) ! integer error
! Create a file from one processor, hence "sequentially"
if(this_proc .eq. 0) then
open(9, file=file_name, form='unformatted')
do i = 1, N
write(9) i*N
end do
close(9)
end if
! Open the same file in MPI mode
call Mpi_File_Open(MPI_COMM_WORLD, & ! integer comm
file_name, & ! character filename(*)
MPI_MODE_RDONLY, & ! integer amode
MPI_INFO_NULL, & ! integer info
file_handle, & ! integer file handle
error) ! integer error
! Read the file in MPI mode
call Mpi_File_Read_All(file_handle, & ! integer file handle
read_data, & ! buffer
N, & ! integer count
MPI_INTEGER, & ! integer datatype
status, & ! integer status
error) ! integer error
! Write out what you got
do i = 1, N
print *, 'read: ', read_data(i)
end do
! End MPI run
call Mpi_Finalize(error)
end program
Any hints or ideas what am I doing wrong here?
I am using gfortran 5.4.0 with MPICH 3.2 on Xubuntu 16.04.09
Thanks
Fortran unformatted sequential files are not compatible with MPI I/O. MPI I/O files are compatible with stream access files which work similarly as C language I/O.
Just change
open(9, file=file_name, form='unformatted')
to
open(9, file=file_name, access="stream", form='unformatted')
And the result will be:
read: 16
read: 32
read: 48
read: 64
read: 80
read: 96
read: 112
read: 128
read: 144
read: 160
read: 176
read: 192
read: 208
read: 224
read: 240
read: 256
Sequential Fortran files use additional record markers which MPI I/O read does not expect (e.g., Fortran unformatted file format). You would have to make a compiler specific MPI datatype to avoid the record markers. Not worth doing it if you don't have to.
Related
When performing reductions on logical values in Fortran with MPI (MPI_LAND operation), MPICH2 returns invalid values of the logical type, which return .true. to both value and .not.value.
For example, this is the result of an MPI_ALLREDUCE of the logical my_value = rank>=0 in the sample program below:
on CPU 0, the local boolean value is 00000000000000000000000000000001
on CPU 0, the logical and of all values is T; its negation is T
on CPU 0, logical AND bits = 11111111111111111111111111111111 negation bits = 11111111111111111111111111111110
MPI seems to have performed a bitwise AND instead of an operation on the logical variables. MPI bug or am I just doing something wrong here? BTW - note that:
using old-style mpif.h for portability
MPICH2 v1.4.1 + gfortran 10.2.0
program test_mpi_land
implicit none
include 'mpif.h'
! Local variables
integer :: ierror,ncpu,cpuid
logical :: my_bool,all_ok
integer, parameter :: master_node = 0
! Init MPI; get basic info
CALL MPI_Init(ierror)
CALL MPI_Comm_size(MPI_COMM_WORLD, ncpu, ierror)
CALL MPI_Comm_rank(MPI_COMM_WORLD, cpuid, ierror)
! Each MPI process sends .true. to reduction
my_bool = cpuid>=master_node
CALL MPI_AllReduce(my_bool,all_ok, 1, MPI_LOGICAL, MPI_LAND, MPI_COMM_WORLD, ierror)
if (cpuid==master_node) then
print '(A,I0,A,B32.32)', 'on CPU ',cpuid,', the local boolean value is ',my_bool
print '(A,I0,2(A,L0))', 'on CPU ',cpuid,', the logical and of all values is ', all_ok, '; its negation is ',.not.all_ok
print '(A,I0,2(A,B32.32))', 'on CPU ',cpuid,', logical AND bits = ',all_ok,' negation bits = ',.not.all_ok
endif
CALL MPI_Finalize(ierror)
end program test_mpi_land
I am trying to read a CSV dataset in Fortran 95.
I have originally asked this question which has been marked as duplicate, two posts have been suggested. However, when I incorporate the posts check for iostat I still get Segmentation Overflow. The posts are related to read a file while my problem is related to standard input (that might not be fully clear in the original SO request, so I make it more clear now).
The CSV data has the following structure:
12,30,2010,23,00,01,125550,1,10643,125550,125575,4665142,0,0
12,30,2010,23,00,44,125550,1,10644,125525,125550,4665188,0,0
12,30,2010,23,01,55,125575,1,10645,125550,125575,4665287,0,0
12,30,2010,23,02,20,125550,1,10646,125550,125575,4665299,0,0
The data is presented in the standard input.
I have been suggested to look at these SO posts:
Using do loop in a Fortran 90 program to read different number of lines for n frames?
Read a file with an unknown number rows in Fortran
Both uses files, not the standard input.
I have modified my code and added the iostat check:
program file_parser
implicit none
! -------------------
! TYPE DEFINITION
! -------------------
type :: type1_record
integer :: month
integer :: day
integer :: year
integer :: hour
integer :: minute
integer :: second
integer :: field1
integer :: field2
integer :: field3
integer :: field4
integer :: field5
integer :: field6
integer :: field7
integer :: field8
end type
! -------------------
! VARIABLE DEFINITION
! -------------------
integer :: i, io_result
type(type1_record), dimension(10000) :: input_data
i = 0
do
read(*,*,iostat=io_result) input_data(i)
if (io_result /= 0) exit
i = i + 1
end do
do i = 1, 3
write(*,*) input_data(i)
end do
end program
However, when I run the program I still get Segmentation fault (core dumped):
$cat test_data.txt | ./a.out
12 30 2010 23 0 44
125550 1 10644 125525 125550 4665188
0 0
12 30 2010 23 1 55
125575 1 10645 125550 125575 4665287
0 0
12 30 2010 23 2 20
125550 1 10646 125550 125575 4665299
0 0
Segmentation fault (core dumped)
Note: test_data.txt contains the data presented at the beginning.
What I am doing wrong with respect of the SO posts that have been previously suggested?
Edit: Apparently it reads the standard input until end of data and then outputs the three registers, but the program ends with Segmentation fault.
In Fortran, is it possible to print data in a tabular manner, without losing information, when more space is needed than specified?
For instance consider the program
! format.f90
program main
real(8) :: arr(5)
arr = [0.0, 1.111, 22.22, 333.3, 444444444444444444.44]
print '(F10.3)', arr
end program main
Then by default the output for the last entry will be replaced by stars, indicating the lack of space.
>> ifort format.f90 -o format.bin
>> ./format.bin
0.000
1.111
22.220
333.300
**********
By comparison, C-style format specifiers automatically increase the column width when required, e.g.
// format.c
#include <stdio.h>
int main () {
double arr[5] = {0.0, 1.111, 22.22, 333.3, 444444444444444444.44};
for(int i=0; i<5; i++) {
printf("%10.3f\n", arr[i]);
}
}
>> gcc format.c -o format.bin
>> ./format.bin
0.000
1.111
22.220
333.300
444444444444444416.000
Is it possible to obtain such behavior in Fortran with built-in features?
Options, that don't fulfill the requirements
G descriptor. The G descriptor allows reliably outputting data in a tabular well-readable format and automatically adds exponentials when needed. However, it also wastes space if the exponentials are not needed and it doesn't line up the comma. For example, when switching F10.3 for G11.4,"¶" (paragraph sign added for emphasis):
>> ifort format.f90 -o format.bin
>> ./format.bin
0.000 ¶
1.111 ¶
22.22 ¶
333.3 ¶
0.4444E+18¶
Building a formatting API based on the F0 specifier. The specifier F0.3 would allow variable-width output, but doesn't allow specifying a minimum width. This could be solved using a wrapper function akin to leftpad, but a built-in or widely-used solution would be preferable for a better chance of actually being used in a codebase. As an example:
! format.f90
program main
real(8) :: arr(5)
integer :: i
arr = [0.0, 1.111, 22.22, 333.3, 444444444444444444.44]
! more complicated print statement, because 'float2char'
! cannot be 'elemental' due to needing the 'alloctable' property.
print '(A)', (float2char('(F0.3)', 10, arr(i)), i=1,5)
contains
function float2char(format, width, value) result(r)
character(:), allocatable :: r
character(*), intent(in) :: format
integer, intent(in) :: width
real(8), intent(in) :: value
character(64) :: buffer ! better: calculate size from value?
write(buffer, format) value
allocate(character(max(width, len_trim(buffer))) :: r)
r(:) = trim(buffer) ! (:) needed to prevent reallocation in recent compilers
r(:) = adjustr(r)
end function float2char
end program main
>> ifort format.f90 -o format.bin
>> ./format.bin
.000
1.111
22.220
333.300
444444452740661248.000
Yes, so in Fortran the fixed-width edit descriptors really are FIXED width. Sometimes useful, often annoying.
One thing you can do is to use the G edit descriptor, which is similar to %g in C, namely that it switches to scientific format when the number is large or small. That allows very large or small values to fit in a fixed width field. Note however that with G editing the d is the number of significant digits, not the number of digits after the decimal point as with F editing. Also it leaves space at the end for the exponent even if the number is in the range that no exponent is needed.
Your example could look like
! format.f90
program main
real(8) :: arr(4)
arr = [0.0, 1.111, 222222222222.222, 3.333]
print '(F10.3)', arr
print *, 'With G edit'
print '(G10.4)', arr
end program main
with output
0.000
1.111
**********
3.333
With G edit
0.000
1.111
0.2222E+12
3.333
Following this thread, I want to cast a single / double precision real number "AA" into an integer "II" to compute the checksum of a distributed variable.
Following comments, I have used the intrinsic 'transfer' and rewritten completely this post. Below is a small fortran module that can be used to compute checksums of distributed arrays which depends on the library 2DECOMP&FFT. The module seems to work on my workstation (gfortran 4.9.2, openmpi 1.6.5, 4 processors). Any comment / remark that may improve the portability of the code will be highly appreciated. Main question regarding portability is: do fortran and MPI_reduce deal with integer overflow in the same way according to standards?
module checksum
use MPI
use decomp_2d, only : mytype, nrank, &
xsize, ysize, zsize, &
transpose_x_to_y, transpose_y_to_z, &
transpose_z_to_y, transpose_y_to_x
implicit none
private ! Make everything private unless declared public
real(mytype), parameter :: xx=1.
integer, parameter, public :: chksum_size = size(transfer(xx,(/0,0/)))
integer, dimension(chksum_size) :: chkr1, chkr2, chkr3
logical, save :: chksum_is_working
! Temporary work variables / arrays
integer :: code
integer, dimension(chksum_size) :: tmprchk
public :: init_chksum, chksum, equal_chksum
contains
!
! Function to compute the checksum of a real 3D array var
!
function chksum(var,nx,ny,nz)
integer, intent(in) :: nx, ny, nz
real(mytype), dimension(nx,ny,nz), intent(in) :: var
integer, dimension(chksum_size) :: chksum
tmprchk = sum(transfer(var,(/0,0/)))
call MPI_ALLREDUCE(tmprchk,chksum,chksum_size,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,code)
end function chksum
!
! Subroutine to make sure input arrays have the same checksum
! First / second / third array are in X / Y / Z pencil
! If switch is provided, reference array is var3.
! Otherwise, reference array is var1
!
subroutine equal_chksum(var1, var2, var3, switch)
real(mytype), dimension(xsize(1),xsize(2),xsize(3)), intent(inout) :: var1
real(mytype), dimension(ysize(1),ysize(2),ysize(3)), intent(inout) :: var2
real(mytype), dimension(zsize(1),zsize(2),zsize(3)), intent(inout) :: var3
logical, optional, intent(in) :: switch
if (chksum_is_working) then ! compute checksums
chkr1 = chksum(var1,xsize(1),xsize(2),xsize(3))
chkr2 = chksum(var2,ysize(1),ysize(2),ysize(3))
chkr3 = chksum(var3,zsize(1),zsize(2),zsize(3))
else ! generate checksums
chkr1 = 1
chkr2 = 2
chkr3 = 3
endif
if (present(switch)) then
if (any(chkr3.ne.chkr2)) call transpose_z_to_y(var3,var2)
if (any(chkr3.ne.chkr1)) call transpose_y_to_x(var2,var1)
else
if (any(chkr1.ne.chkr2)) call transpose_x_to_y(var1,var2)
if (any(chkr1.ne.chkr3)) call transpose_y_to_z(var2,var3)
endif
end subroutine equal_chksum
!
! Subroutine used to check we have a working checksum
!
subroutine init_chksum(var1,var2,var3)
real(mytype), dimension(xsize(1),xsize(2),xsize(3)), intent(out) :: var1
real(mytype), dimension(ysize(1),ysize(2),ysize(3)), intent(out) :: var2
real(mytype), dimension(zsize(1),zsize(2),zsize(3)), intent(out) :: var3
! Same random data inside all arrays
call random_number(var1)
call transpose_x_to_y(var1,var2)
call transpose_y_to_z(var2,var3)
! Compute checksums
chkr1 = chksum(var1,xsize(1),xsize(2),xsize(3))
chkr2 = chksum(var2,ysize(1),ysize(2),ysize(3))
chkr3 = chksum(var3,zsize(1),zsize(2),zsize(3))
! Check checksums
if (any(chkr1.ne.chkr2).or.any(chkr1.ne.chkr3)) then
chksum_is_working = .false.
if (nrank.eq.0) print *,'Checksums based on integer overflow do not work'
else
chksum_is_working = .true.
endif
end subroutine init_chksum
end module checksum
do fortran and MPI_reduce deal with integer overflow in the same way according to standards?
Neither the Fortran standard nor the MPI 3.0 standard even mentions integer overflow so you are at the mercy of the implementers.
However, I see you are only using integers of default kind, use a larger integer kind for intermediate results and you can implement your own overflow detection.
Integer overflow is not defined by the Fortran standard. In C signed integer overflow is undefined behaviour.
If you enable undefined behaviour santizations in gfortran, your program will be stopped with an error message. (Happened to me when I was using a 3rd party random number generator.)
You can perform the operation using larger integers and crop the result or call a C function which uses unsigned integers. Integer overflow is only well defined for signed integers.
An odd issue here. I am converting legacy F77 code that packed 8 a1 characters into a 64 bit integer, and the code worked back in the day. But in today's world to get 8 characters in a word I need to move to integer*8 variables, and the code is failing in that case. It works until it's packed 4 characters, but character 5 simply rotates around and overwrites the first packed character. Here is a sample output for "hello world”, of note is that the integer*8 variable OUTBUF suddenly shrinks and appears to convert to an integer*4 variable too:
hello world
in='hello world '
cn= 8,k=1,j='h',outbuf(k)='h '
cn=16,k=1,j='e',outbuf(k)='he '
cn=24,k=1,j='l',outbuf(k)='hel '
cn=32,k=1,j='l',outbuf(k)='hell'
cn=40,k=1,j='o',outbuf(k)='oell'
# SPACE
cn= 8,k=2,j='w',outbuf(k)='w '
cn=16,k=2,j='o',outbuf(k)='wo '
cn=24,k=2,j='r',outbuf(k)='wor '
cn=32,k=2,j='l',outbuf(k)='worl'
cn=40,k=2,j='d',outbuf(k)='dorl'
# SPACE
I've distilled the code to this snippet, anyone know F77 anymore and see what’s happening? thanks.
program cow
implicit integer*8 (a-z)
integer inbuf(72)
integer*8 outbuf(40)
c changing the integer*1 to integer*8 fixes the problem !!!!!!!!!
integer*1 j
read 99, inbuf
99 FORMAT(BZ,72a1)
print 1, inbuf
1 format("in='",72a1,"'")
IP=0
k = 1
DO 100 I=1,40
OUTBUF(I)= 8H
100 CONTINUE
200 IP=IP+1
IF(IP.GT.72) GO TO 6000
J= INBUF(IP)
IF(J.EQ." ") GO TO 6000
CN = (MOD(CP,8)+1) * 8
outbuf(k) = outbuf(k) .and. (.not. lshift('377'O, (cn-8)))
OUTBUF(K) = OUTBUF(K) .OR. lSHiFT( j,(CN-8))
print 4301, cn, k, j, outbuf(k)
4301 format("cn=",i2,",k=",i1,",j='",a1,"',outbuf(k)='",a8,"'")
CP=CP+1
GO TO 200
6000 continue
print *,'# SPACE'
call exit
end
Note that this code is for little endian machines.
" So ignore the code, the task at hand - in FORTRAN 77 - is to pack 8
a1 characters into an integer*8 variable. "
Ok, but I will have to wash my hands after this:
INTEGER*8 i
CHARACTER*8 ch
EQUIVALENCE (i, ch)
ch = "abcdefgh"
Of course, transfer is nicer, but certainly not FORTRAN 77.
This is quite horrible code to do something trivial in newer Fortran
(since F77...).
You can replace the whole routine by something like
integer, parameter :: llen=72
character(len=llen) inbuf
character(len=llen) output
integer :: i,k
read (*,'(BZ,A)') outbuf
k = 1
do i=1,llen
if (inbuf(i) /= ' ') then
outbuf(k) = inbuf(i)
if (mod(k,8) == 0) then
! Some output goes here
endif
k = k + 1
end if
end do
OK, here is the second answer; not preferred, it is really better to rewrite the program using characters.
Caveat: integer*8 is non-standard.
program main
character(len=8) :: c
integer*8 output(10)
c = "12345678"
call foo(output,10,c)
write (*,'(Z16)') output(1)
end program main
subroutine foo(a,n,b)
integer*8 :: a(1)
character(len=8) :: b
a(1) = transfer(b,mold=a(1))
end subroutine foo
As I mentioned, this input parser took Hollerith data and compared it against Hollerith data in a database, in a case-insensitive manner. The code was written for 60-64 bit, big-endian machines, and used hardware specific shifts, masks, ANDs and ORs to pack the A1 input characters into an A8 INTEGER word for comparison against the database.
And although I did manage to adjust all those shifts, masks and bitwise operations to work on 64-bit little-endian machines (x86_64), it was a PITA, and only worked with pgf77 and ifort, gfortran was having none of that crap! So this is what I came up with, much nicer, hardware independent, and works on all the compilers available to me:
The parser still accepts A1 input characters stored in INTEGERs (80a1), and returns packed A8 characters in INTEGER*8 words, for backwards compatibility.
It converts 80a1 input data to CHARACTER*80 variable IN0:
write( in0, '80a1' ) (inbuf(i), i=1,80)
Since this is all ASCII data, I used this code to lower-case everything in IN0 to a new character*80 variable IN:
do i = 1, len(in0)
#if (! defined __GFORTRAN__)
j = ichar(in0(i:i))
if (j>= ichar("A") .and. j<=ichar("Z") ) then
in(i:i) = char(ichar(in0(i:i))+32)
else
in(i:i) = in0(i:i)
end if
#else
j = iachar(in0(i:i))
if (j>= iachar("A") .and. j<=iachar("Z") ) then
in(i:i) = achar(iachar(in0(i:i))+32)
else
in(i:i) = in0(i:i)
end if
#endif
end do
It was then a simple matter to loop through the CHARACTER*80 variable IN, collect non-space characters into the character variable WORD, and then encode them into INTEGER*8 array OUTBUF:
read( word(1:8), ‘a8' ) outbuf(outbufc)
Thanks all.