I would like to open 10,000 files with file names starting from abc25000 until abc35000 and copy some information into each file. The code I have written is as below:
PROGRAM puppy
IMPLICIT NONE
integer :: i
CHARACTER(len=3) :: n1
CHARACTER(len=5) :: cnum
CHARACTER(len=8) :: n2
loop1: do i = 25000 ,35000 !in one frame
n1='abc'
write(cnum,'(i5)') i
n2=n1//cnum
print*, n2
open(unit=i ,file=n2)
enddo loop1
end
This code is supposed to generate files starting from abc24000 until abc35000 but it stops about half way saying that
At line 17 of file test-openFile.f90 (unit = 26021, file = '')
Fortran runtime error: Too many open files
What do I need to do to fix the above code?
This limit is set by your OS. If you're using a Unix/Linux variant, you can check the limit using from the command line using ulimit -n, and raise it using ulimit -n 16384. You'll need to set a limit greater than 10000 to allow for all the other files that the shell will have open. You may also need admin privileges to do this.
I regularly bump the limit up to 2048 to run Fortran programs, but never as high as 10000. However, I echo the other answers that, if possible, it's better to restructure your program to close each file before opening the next.
You need to work on the files one at a time (or in small groups that do not exceed the limitation imposed by the operating system).
for each file:
open file
write
close file
Operating systems tend to have limits on resources. Typically on, for instance, Linux, there is by default a limit of 1024 file descriptors per process. The error message you're getting is just the Fortran runtime library passing information upwards that it was unable to open yet another file due to an OS error.
Related
I have a Fortran program that needs to read ASCII files, however the list of files sometimes includes a file of size 0. The program then crashes when trying to read this file. I have not find any way so far that will allow me to flag such a file.
I have following READ statement in my code
read(10,220,END=320,ERR=195)parm(1:)
although I expect code to go to statement 195, or to statement 320, without crashing, it crashes
this is where the code crashes when the file size is zero, with the following messages
...
fmt: end of file
apparent state: unit 10 named junko.con
last format: (A)
lately reading sequential formatted external IO
I tried using the INQUIRE statement
inquire (unit=10,SIZE=nsize), but the program would not compile
the OPEN statement did not give any error when opening the zero size file, and the values of IOSTAT was the same, irrespective of the file size
As Ian noted, any modern Fortran compiler should have INQUIRE. A simple test of
program foo
integer sz
inquire(file='tmp.dat',size=sz)
print *, sz
end program foo
with an empty tmp.dat file sets sz=0.
I have a Fortran routine that opens a lot of text files write data from a time loop. This routine uses open with the newunit option, this unit is stored in an object in order to write things in files later. This works fine most of the time but when the program needs to open a large number N of files at the same time I get the following error:
**forrtl: severe (104): incorrect STATUS= specifier value for connected file, unit -1, file CONOUT$**
reffering to the first open function in createFiles subroutine. This error occurs whether the file already exists or not. I don't know if this might help but at this stage the new unit that should be generated would be -32768.
I include a minimal code sample with a "timeSeries" class including a routine that creates two files:
the first file fileName1 is opened and closed directy after writing stuff inside
the second file fileName2 is kept open in order to write things comùputed in a time loop later and closed at the end of the time loop
The example is composed of the two following files. It breaks for i=32639.
main.f90 :
program writeFiles
use TS
logical :: stat
integer :: i, istep, N, NtimeSteps
character(len=16) :: fileName1, fileName2
character(len=300) :: path
type(timeSeries), dimension(:), allocatable :: myTS
call getcwd( path )
path = trim(path) // '\Output_files'
inquire(directory = trim(path), exist = stat )
if (.not. stat) call system("mkdir " // '"' // trim(path) // '"' )
N = 50000
NtimeSteps = 100
allocate(myTS(N))
do i = 1, N
write(fileName1,'(a6,i6.6,a4)') 'file1_', i, '.txt'
write(fileName2,'(a6,i6.6,a4)') 'file2_', i, '.txt'
call myTS(i)%createFiles(trim(path),fileName1,fileName2)
end do
do istep = 1, NtimeSteps
#
#compute stuff
#
do i = 1, N
write(myTS(i)%fileUnit,*) 'stuff'
end do
end do
do i = 1, N
close(myTS(i)%fileUnit)
end do
end program writeFiles
module.f90 :
module TS
type timeSeries
integer :: fileUnit
contains
procedure :: createFiles => timeSeries_createFiles
end type timeSeries
contains
subroutine timeSeries_createFiles(this,dir,fileName1,fileName2)
class(timeSeries) :: this
character(*) :: dir, fileName1, fileName2
open(newunit = this%fileUnit , file = dir // '\' // fileName1, status = 'replace') !error occurs here after multiple function calls
write(this%fileUnit,*) 'Write stuff'
close(this%fileUnit)
open(newunit = this%fileUnit , file = dir // '\' // fileName2, status = 'replace')
end subroutine timeSeries_createFiles
end module
Any idea about the reason for this error? Is there a limitation for the number of files opened at the same time? Could it be related to a memory issue?
I'm using Intel(R) Visual Fortran Compiler 17.0.4.210
Windows has this interesting habit of not releasing all the resources for a closed file for a short time after you do a close. I have seen this sort of problem on and off for decades. My usual recommendation is to put a call to SLEEPQQ with a duration of half a second after a CLOSE when you intend to do another OPEN soon after on the same file. But you're not doing that here.
There's more here that is puzzling. The error message referring to unit -1 and CONOUT$ should not occur when opening an explicit file and using NEWUNIT. In Intel's implementation, NEWUNIT numbers start at -129 and go more negative from there. Unit -1 is used for PRINT or WRITE(*), and CONOUT$ is the console. STATUS='REPLACE' would not be valid for a unit connected to the console. That the newunit number would be -32768 is telling and suggests an internal limit for NEWUNIT in the Intel libraries.
I did a test of my own and see that if you use NEWUNIT and close the unit, the unit numbers go as low as -16384 before cycling back to -129. That's ok if indeed you're closing the units, but you're never closing the second file you open, so you're at least hitting a maximum number of NEWUNIT files open. I would recommend figuring out a different way of approaching the problem that didn't require leaving thousands of files open.
I have a little Problem reading a file in Fortran. As you can see I am lopping over a file reading certain records with a specific length.
What happens is, when it comes to a certain record I'm getting an IOSTAT Error 5002. Now my question is what does this error mean: is it end of file or there is no record left or something else? Can I ignore it?
I am using MinGW GFortran 4.8.0.
Here's the code:
PROGRAM test_read
INTEGER*4 HCM_error
DOUBLE PRECISION N_Record(22)
CHARACTER*8 C_Record(22)
EQUIVALENCE (N_Record,C_Record)
OPEN (UNIT=11, FILE='C:/BORDER/D__HOL.000',STATUS='OLD', ACCESS='DIRECT',RECL=176, ACTION='READ', IOSTAT=IOS)
HCM_error=0
DO N_rec = 1, 2000
READ (11, REC=N_rec, IOSTAT=IOS) C_Record
WRITE(*,*) "|",IOS,' ',N_rec,' ',N_record(21),' ',N_record(22),"|"
!End of file reached (or non existing record) ?
IF ((IOS .LT. 0) .OR. (IOS .EQ. 36)) EXIT
IF (IOS .NE. 0) THEN
!Error in (border-) line data
HCM_Error = 1049
EXIT
END IF
END DO
CLOSE(UNIT=11)
WRITE (*,*) HCM_error
END PROGRAM
The non-zero values returned by an iostat= specifier are not portable across compilers. If you wish to determine what a particular code means then you have two options:
read the compiler's documentation (if it exists)
use the iomsg= specifier with a character variable
In this case, when you tried iomsg= you got the message "Non-existing record number". So, problem solved.
Well, almost. There's more to say.
You may be surprised that you are going through records in turn in direct access, but are reaching a "no record" state without first reaching an "end of file" state. You are testing (IOS .LT. 0) with a comment "!End of file reached".
When reading a file connected for direct access, the end of file condition doesn't arise.
What can you do to detect that the record isn't a valid number, beyond the end of the file? Not much, portably, but any positive number from iostat= indicates an error condition. You know now, though, what this particular 5002 means.
I should probably also add that the character variable for iomsg= is defined by the transfer statement only if there isn't success. Consider it only if you know the transfer failed.
I'm sorry if this has been asked before, but I couldn't find an answer that seemed to work for me. I'm working with an old program, but have made a few modifications to it.
I can include the whole 2500 line program, but it seems like that is a lot.
I've successfully compiled the a program, but it fails when I try and run it. I'm getting a "Fortran runtime error: End of file" at the line which reads the .dat file. I've tried to compile a test segment, using the same .dat file and same variables. It results in the same problem.
PROGRAM OPEN
INTEGER (KIND=1), PARAMETER :: dy=3 ! number of income states
INTEGER (KIND=2) :: OpenStatus
REAL, DIMENSION(dy) :: grid,wt
OPEN(1,file='cygdrive/user/mk.dat',status='old',form='formatted',IOSTAT=OpenStatus)
READ (1,*) grid, wt
IF(OpenStatus>0) STOP 'cannot open mk.dat'
CLOSE(1)
PRINT*, grid(1)
END PROGRAM
The data file referenced is:
-1.7320508e+000
0.0000000e+000
1.7320508e+000
4.1777138e-001
1.6710855e+000
4.1777138e-001
Where each of these numbers is on its own line and preceeded by a space
This generates the same end of file runtime error. I'd really appreciate any help here.
I should add that I compiled with gfortran.
EDIT:
As per High Performance Mark's suggestion below, I've modified it to include an inquire test.
PROGRAM TEST
CHARACTER :: fnm, seq, fort
Logical :: lex
INTEGER (KIND=1), PARAMETER :: dy=3 ! number of income states
INTEGER (KIND=2) :: j,j0,j1,j2,j4,j5,j6,j7,k,jjj,jj,dyy,OpenStatus
REAL, DIMENSION(dy) :: grid,wt
OPEN(1,file='cygdrive/user/mk.dat',status='old',form='formatted',IOSTAT=OpenStatus)
INQUIRE (1, EXIST=lex, NAME=fnm, SEQUENTIAL=seq, FORMATTED=fort)
PRINT*, 'Exists=',lex, ' Name=',fnm, ' Sequential=', seq, 'Formatted=', fort
READ (1,*) grid, wt
IF(OpenStatus>0) STOP 'cannot open mk.dat'
CLOSE(1)
PRINT*, grid(1)
END PROGRAM
The results of the inquire statement are:
Exists= T Name= Sequential=U Formatted=U
My understanding is that the File is found (i.e. exists is returned as true), is un-named and the format and sequential access are returned as unknown (as is direct which I included later). I've also checked the delimiter and padding which are coming back as unknown.
My beginner intuition is telling me that I should try and create a new data file by writing to it with a fortran program and that should solve the problem? Is that correct? If so is there a fundamental misunderstanding at play here i.e. is this a problem with data files from other sources?
Thanks for all your patience.
(Answered in the comments. See Question with no answers, but issue solved in the comments (or extended in chat) )
The OP wrote:
I've fixed this problem-the program was looking for it in a different place, I've now corrected that.
I have a shell script from which I pass a binary file to a fortran program such that
Mth=$1
loop=1
it=1
while test $it -le 12
do
Mth=`expr $Mth + $loop`
file="DataFile"$Mth".bin"
./fort_exe ${Yr} ${nt} ${it}
# Increment loop
it=`expr $it + 1`
done
This script is used to pass 12 files within a do loop to the fortran program. In the fortran program, I read the binary file passed from the shell script and I am trying to write a 2nd file which would compile in a single file all the data that was read from the consecutive files e.g.
!Open binary file passed from shell script
open(1,file='Datafile'//TRIM{Mth)//.bin',action='read',form='unformatted',access='direct', &
recl=4*x*y, status='old')
! Open write file for t 1. The status is different in t 1 and t > 1 so I open it twice: I guess there is a more elegant way to do this...
open(2,file='Newfile.bin',action='write',form='unformatted', &
access='stream', position='append', status='replace')
irec = 0
do t = 1, nt
! Read input file
irec = irec + 1
read(1,rec=irec) val(:,:)
! write output file
irecW= irec + (imonth-1)*nt
if ( t .eq. 1) write(2,pos=irecW) val(:,:)
! Close file after t = 1, update the status to old and reopen.
if ( t .eq. 2) then
close (2)
open(2,file='Newfile.bin',action='write',form='unformatted', &
access='stream', position='append',status='old')
endif
if ( t .ge. 2) write(2,pos=irecW) val(:,:)
enddo
I can read the binary data from the first file no problem but when I try and read from another program the binary data from the file that I wrote in the first program such that
open(1,file='Newfile.bin',action='read',form='unformatted', &
access='stream', status='old')
irec=0
do t = 1, nt
! Read input file
irec = irec + 1
read(1,pos=irec) val(:,:)
write(*,*) val(:,:)
enddo
val(:,:) is nothing but a list of zeros. This is the first time I use access=stream which I believe is the only way I can use position='append'. I have tried compiling with gfortran and ifort but I do not get any error messages.
Does anyone have any idea why this is happening?
Firstly, I do not think you need to close and reopen your output file as you are doing. The status specifier is only relevant to the open statement in which it appears: replace will delete Newfile.bin if it exists at that time, before opening a new file with the same name. The status is implicitly changed to old, but this does not affect any operations done to the file.
However, since your Fortran code does not know you run it 12 times, you should have a way of making sure the file is only replaced the first time and opened as old afterwards; otherwise, Newfile.bin will only contain the information from the last file processed.
As for reading in the wrong values, this most likely occurs because of the difference between direct access (where you can choose a record length) and stream access (where you cannot). With stream access, data is stored as a sequence of "file storage units". Their size is in general compiler-dependent, but is available through the module iso_fortran_env as file_storage_size; it is usually 8 bits. This means that each entry will usually occupy multiple storage units, so you have to take care that a read or write with the pos = specifier does not access the wrong storage units.
Edit:
Some example code writing and reading with stream access:
program stream
use, intrinsic :: iso_fortran_env
implicit none
integer :: i, offset
real(real32), dimension(4,6) :: val, nval
open(unit=2, file='Newfile.bin', action='readwrite', form='unformatted', &
access='stream', status='replace')
do i = 1,2
call random_number(val)
write(2) val
enddo
! The file now contains two sequences of 24 reals, each element of which
! occupies the following number of storage units:
offset = storage_size(val) / file_storage_size
! Retrieve the second sequence and compare:
read(2, pos = 1 + offset*size(val)) nval
print*, all(nval == val)
close(2)
end program
The value true should be printed to the screen.
Note also that it's not strictly necessary to specify a pos while writing your data to the file, because the file will automatically be positioned beyond the last record read or written.
That said, direct or stream access is most beneficial if you need to access the data in a non-sequential manner. If you only need to combine input files into one, it could be easier to write the output file with sequential access, for which you can also specify recl and position = 'append'.
You can check for the existence of a file in standard Fortran, by using the inquire statement:
logical :: exist
inquire(file="test.dat", exist=exist)
if (exist) then
print *, "File test.dat exists"
else
print *, "File test.dat does not exist"
end if
Alternatively you can have a look at the modFileSys library which provides libc like file manipulation routines.
As for appending and streams: Appending files is also possible when you use "classical" record based fortran files, you do not have to use streams for that.