Here is a snippet of simple code that reads line from file, then returns to previous position and re-reads same line:
program main
implicit none
integer :: unit, pos, stat
character(128) :: buffer
! Open file as formatted stream
open( NEWUNIT=unit, FILE="data.txt", ACCESS="stream", FORM="formatted", STATUS="old", ACTION="read", IOSTAT=stat )
if ( stat /= 0 ) error stop
! Skip 2 lines
read (unit,*) buffer
read (unit,*) buffer
! Store position
pos = ftell(unit)
! Read & write next line
read (unit,*) buffer
write (*,*) "buffer=", trim(buffer)
! Return to previous position
call fseek(unit,pos,0)
! pos = ftell(unit) ! <-- ?!
! Read & write next line (should be same output)
read (unit,*) buffer
write (*,*) "buffer=", trim(buffer)
! Close file stream
close (UNIT=unit)
end program main
The "data.txt" is just a dummy file with 4 lines:
1
2
3
4
Now when I compile the snippet (gfortran 9.3.0) and run it, I get an answer:
buffer=3
buffer=4
which is wrong, as they should be same. More interestingly when I add an additional ftell (commented line in the snippet) after 'fseek' I get correct answer:
buffer=3
buffer=3
Any idea why it does that? or am I using ftell and fseek incorrectly?
gfortran's documentation for FTELL and FSEEK clearly states that these routines are provided for backwards compatibility with g77. As your code is using NEWUNIT, ERROR STOP, and STREAM access, you are not compiling old moldy code. You ought to use standard conforming methods as pointed out by #Vladimir.
A quick debugging session shows that FTELL and FSEEK are using a 0-based reference for the file position while the inquire method of modern Fortran is 1 based. There could be an off-by-one type bug in gfortran, but as FTELL and FSEEK are for backwards compatibility with g77 (an unmaintained 15+ year old compiler), someone would need to do some code spelunking to determine the intended behavior. I suspect none of the current, active, gfortran developers care enough to explore the problem. So, to fix your problem
program main
implicit none
integer pos, stat, unit
character(128) buffer
! Open file as formatted stream
open(NEWUNIT=unit, FILE="data.txt", ACCESS="stream", FORM="formatted", &
& STATUS="old", ACTION="read", IOSTAT=stat)
if (stat /= 0) stop
! Skip 2 lines
read (unit,*) buffer
read (unit,*) buffer
! Store position
inquire(unit, pos=pos)
! Read & write next line
read (unit,*) buffer
write (*,*) "buffer=", trim(buffer)
! Reread & write line (should be same output)
read (unit,*,pos=pos) buffer
write (*,*) "buffer=", trim(buffer)
! Close file stream
close (UNIT=unit)
end program main
I'm running a distributed model stripped to its bare minimum below:
integer, parameter :: &
nx = 1200,& ! Number of columns in grid
ny = 1200,& ! Number of rows in grid
nt = 6000 ! Number of timesteps
integer :: it ! Loop counter
real :: var1(nx,ny), var2(nx,ny), var3(nx,ny), etc(nx,ny)
! Create netcdf to write model output
call check( nf90_create(path="out.nc",cmode=nf90_clobber, ncid=nc_out_id) )
! Loop over time
do it = 1,nt
! Calculate a lot of variables
...
! Write some variables in out.nc at each timestep
CALL check( nf90_put_var(ncid=nc_out_id, varid=var1_varid, values=var1, &
start = (/ 1, 1, it /), count = (/ nx, ny, 1 /)) )
! Close the netcdf otherwise it is not readable:
if (it == nt) call check( nf90_close(nc_out_id) )
enddo
I'm in the development stage of the model so, it inevitably crashes at unexpected points (usually at the Calculate a lot of variables stage), which means that, if the model crashes at timestep it =3000, 2999 timesteps will be written to the netcdf output file, but I will not be able to read the file because the file has not been closed. Still, the data have been written: I currently have a 2GB out.nc file that I can't read. When I ncdump the file it shows
netcdf out.nc {
dimensions:
x = 1400 ;
y = 1200 ;
time = UNLIMITED ; // (0 currently)
variables:
float var1 (time, y, x) ;
data:
}
My questions are: (1) Is there a way to close the file retrospectively, even outside Fortran, to be able to read the data that have already been written? (2) Alternatively, is there another way to write the file in Fortran that would make the file readable even without closing it?
When nf90_close is called, buffered output is written to disk and the file ID is relinquished so it can be reused. The problem is most likely due to buffered output not having been written to the disk when the program terminates due to a crash, meaning that only the changes you made in "define mode" are present in the file (as shown by ncdump).
You therefore need to force the data to be written to the disk more often. There are three ways of doing this (as far as I am aware).
nf90_sync - which synchronises the buffered data to disk when called. This gives you the most control over when to output data (every loop step, or every n loop steps, for example), which can allow you to optimize for speed vs robustness, but introduces more programming and checking overhead for you.
Thanks to #RussF for this idea. Creating or opening the file using the nf90_share flag. This is the recommended approach if the netCDF file is intended to be used by multiple readers/writers simultaneously. It is essentially the same as an automatic implementation of nf90_sync for writing data. It gives less control, but also less programming overhead. Note that:
This only applies to netCDF-3 classic or 64-bit offset files.
Finally, an option I wouldn't recommend, but am including for completeness (and I guess there may be situations where this is the best option, although none spring to mind) - closing and reopening the file. I don't recommend this, because it will slow down your program, and adds greater possibility of causing errors.
I'm fairly new to Fortran so this might be a naive question. I would like to read a huge .txt file with a # of rows=540001. The data are from a sonic anemometer with measurements of velocity and temperature as u,v,w,T at five heights. Here are the first few lines of the file:
"2011-07-10 09:30:00",9838,1.132,2.30225,-0.5635,29.18585,0.30275,0.689,-0.01125,29.67004,0.2165,-0.25475,0.12725,29.8923,0.51425,3.0405,-0.58375,29.5242,-0.0085,3.6235,-0.65175,29.61972,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
"2011-07-10 09:30:00.05",9839,-0.21325,3.22775,-0.17,29.10953,0.33925,0.6867501,-0.0015,29.67874,0.1715,-0.196,0.1235,29.8923,0.035,2.6915,-0.3845,29.82806,-0.102,3.5505,-0.15825,29.61795,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
"2011-07-10 09:30:00.1",9840,0.403,3.1195,-0.37175,29.22574,0.06550001,0.6655,0.1275,29.76208,0.196,-0.2,0.1,29.901,0.16225,2.31525,-0.5975,29.69263,0.24175,3.11925,-0.3725,29.57977,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
etc...
I would like to save/write the matrices u(5,540001),v(5,540001),w(5,540001), and T(5,540001) so that I can do some calculations and plots. Here is what I am using:
PROGRAM READ_MAIZE
IMPLICIT NONE
REAL,DIMENSION(:,:),Allocatable :: u_r, v_r, w_r, T_r
CHARACTER(len=*) :: fname
fname='FINALDATA.txt'
open(unit=1,file=fname,status='old',action='read')
do i=1,540001
READ(1,*)timestamp(i),count(i),u_r(5,i),v_r(5,i),w_r(5,i), &
T_r(5,i),u_r(2,i),v_r(2,i),w_r(2,i),T_r(2,i), &
u_r(1,i),v_r(1,i),w_r(1,i),T_r(1,i), &
u_r(3,i),v_r(3,i),w_r(3,i),T_r(3,i), &
u_r(4,i),v_r(4,i),w_r(4,i),T_r(4,i),flags(1:20)
end do
close(1)
WRITE(U_maize,'(A,I7.7,A,I7.7,A)'), &
'.txt'
open(11,file=U_maize,status='unknown',action='write')
write(11,'(F20.14)')(u_r)
end
Never mind the order in u_r(5,i) followed by u_r(2,i)... (they just correspond to different heights that are out of order). This is not working.
There's quite a lot going on in your code which makes it hard to understand what you're trying to do in the first place. I have annotated your code below and turned it into something that compiles and produces output. Maybe it'll help.
PROGRAM READ_MAIZE
IMPLICIT NONE ! This means that every variable has to be declared
! explicitly. You don't do that. So I did it for you
REAL,DIMENSION(:,:),Allocatable :: u_r, v_r, w_r, T_r
integer, dimension(:), allocatable :: data_count ! using fortran keywords
! (such as "count") as variables is dangerous and should be avoided
character(len=len("2011-07-10 09:30:00.05")), allocatable :: timestamp(:)
CHARACTER(len=*), parameter :: fname='FINALDATA.txt'
character(len=len("U_XXXXXXX_XXXXXXX.txt")) :: U_maize
integer :: in_unit, out_unit ! Use integer variables for the unit.
! together with newunit keyword, this is
! safer
integer, parameter :: num_records = 3 ! you need to up this number to
! 540001 again
integer :: i
! If you have allocatable arrays, you need to allocate them before you
! can use them
allocate(u_r(5, num_records))
allocate(v_r(5, num_records))
allocate(w_r(5, num_records))
allocate(T_r(5, num_records))
allocate(data_count(num_records))
allocate(timestamp(num_records))
! the "newunit" keyword is a safe way to create a unique unit
! identifier. You should really use this.
open(newunit=in_unit,file=fname,status='old',action='read')
do i=1,num_records
READ(in_unit,*) timestamp(i), data_count(i), &
u_r(5,i),v_r(5,i),w_r(5,i),T_r(5,i), &
u_r(2,i),v_r(2,i),w_r(2,i),T_r(2,i), &
u_r(1,i),v_r(1,i),w_r(1,i),T_r(1,i), &
u_r(3,i),v_r(3,i),w_r(3,i),T_r(3,i), &
u_r(4,i),v_r(4,i),w_r(4,i),T_r(4,i) ! I left out the flags
! since I didn't know what
! that was.
end do
close(in_unit)
! I don't know how the file name should be constructed, except
! that it should end in a .txt, and the format. So I made something up.
write(U_maize, '(A, I7.7, A, I7.7, A)') 'U_', 35, '_', 6, '.txt'
open(newunit=out_unit,file=U_maize,status='unknown',action='write')
! To make it more readable, I tell write to write 5 numbers per row,
! Not sure whether this is what you want.
write(out_unit,'(5(X, F20.14))') u_r
close(out_unit) ! I know it isn't technically needed, but please always
! close files when finished, even if the program terminates anyway.
end program READ_MAIZE ! tell the compiler what you want to end here.
I get the following error with MPICH-2.1.5 and PGI compiler;
Fatal error in PMPI_Waitall: Invalid MPI_Request, error stack:
PMPI_Waitall(311): MPI_Waitall(count=4, req_array=0x2ca0ae0, status_array=0x2c8d220) failed
PMPI_Waitall(288): The supplied request in array element 0 was invalid (kind=0)
in the following example Fortran code for a stencil based algorithm,
Subroutine data_exchange
! data declaration
integer request(2*neighbor),status(MPI_STATUS_SIZE,2*neighbor)
integer n(neighbor),iflag(neighbor)
integer itag(neighbor),neigh(neighbor)
! Data initialization
request = 0; n = 0; iflag = 0;
! Create data buffers to send and recv
! Define values of n,iflag,itag,neigh based on boundary values
! Isend/Irecv look like this
ir=0
do i=1,neighbor
if(iflag(i).eq.1) then
ir=ir+1
call MPI_Isend(buf_send(i),n(i),MPI_REAL,neigh(i),itag(i),MPI_COMM_WORLD,request(ir),ierr)
ir=ir+1
call MPI_Irecv(buf_recv(i),nsize,MPI_REAL,neigh(i),MPI_ANY_TAG,MPI_COMM_WORLD,request(ir),ierr)
endif
enddo
! Calculations
call MPI_Waitall(2*neighbor,request,status,ierr)
end subroutine
The error occurs when the array_of_request in mpi_waitall gets a null value (request(i)=0). The null value in array_of_request comes up when the conditional iflag(i)=1 is not satisfied. The straight forward solution is to comment out the conditional but then that would introduce overheads of sending and receiving messages of 0 sizes which is not feasible for large scale systems (1000s of cores).
As per the MPI-forum link, the array_of_requests list may contain null or inactive handles.
I have tried following,
not initializing array_of_requests,
resizing array_of_request to match the MPI_isend + MPI_irecv count,
assigning dummy values to array_of_request
I also tested the very same code with MPICH-1 as wells as OpenMPI 1.4 and the code works without any issue.
Any insights would be really appreciated!
You could just move the first increment of ir into the conditional as well. Then you would have all handles in request(1:ir) at the and of the loop and issue:
call MPI_Waitall(ir,request(1:ir),status(:,1:ir),ierr)
This would make sure all requests are initialized properly.
Another thing: does n(i) in MPI_Isend hold the same value as nsize in the corresponding MPI_Irecv?
EDIT:
After consulting the MPI Standard (3.0, Ch. 3.7.3) I think you need to initialize the request array to MPI_REQUEST_NULL if you want give the whole request array to MPI_Waitall.
I am trying to write a series of values for time (real values) into a dat file in FORTRAN. This is a part of an MPI code and the code runs for a long time. So I would like to extract data at every time step and print it into a file and read the file any time during the execution of the program. Currently, the problem I am facing is, the values of time are not written into the file until the program ends. I have put the open statement before the do loop and the close statement after the end of do loop.
The parts of my code look like:
open(unit=57,file='inst.dat')
do loop starts
.
.
.
write(57,*) time
.
.
.
end do
close(57)
try call flush(unit). Check your compiler docs as this is i think an extension.
You mention MPI: For parallel codes I think you need to give each thread its own file/unit,
or take other measures to avoid conflicts.
From Gfortran manual:
Beginning with the Fortran 2003 standard, there is a FLUSH statement that should be preferred over the FLUSH intrinsic.
The FLUSH intrinsic and the Fortran 2003 FLUSH statement have identical effect: they flush the runtime library's I/O buffer so that the data becomes visible to other processes. This does not guarantee that the data is committed to disk.
On POSIX systems, you can request that all data is transferred to the storage device by calling the fsync function, with the POSIX file descriptor of the I/O unit as argument (retrieved with GNU intrinsic FNUM). The following example shows how:
! Declare the interface for POSIX fsync function
interface
function fsync (fd) bind(c,name="fsync")
use iso_c_binding, only: c_int
integer(c_int), value :: fd
integer(c_int) :: fsync
end function fsync
end interface
! Variable declaration
integer :: ret
! Opening unit 10
open (10,file="foo")
! ...
! Perform I/O on unit 10
! ...
! Flush and sync
flush(10)
ret = fsync(fnum(10))
! Handle possible error
if (ret /= 0) stop "Error calling FSYNC"
How about closing the file after every time step (assuming a reasonable amount of time elapses between time steps)?
do loop starts
.
.
!Note: an if statement should wrap the following so that it is
!only called by one processor.
open(unit=57,file='inst.dat')
write(57,*) time
close(57)
.
.
end do
Alternatively if the time between time steps is short, writing the data after blocks of 10, 100, ... iterations may be more efficient.