I want to read out some output data from a Fortran code to data files for postprocessing. The code given to me uses MPI to write to those data files. For the most basic case, when the number of processors should be 1 and also the Nx, Ny, Nz are all set to 1 and l is 3, the output of the written to the data file should be all 1. However, the output written to the data file is in some non-readable format, as below:
^#^#^#^#^#^#^#^#^#^#^#^#
The relevant portion of the code which writes to the data file is appended below.
do rst=1,1
fname='phi'
fname = trim('ufs')//":" // trim(fname)
write(buffer,"(i3.3)") 2*rst-1
fname = trim(adjustl(fname))//'.'//trim(adjustl(buffer))
print *, 'The output file to which all the data is written is ', fname
call MPI_FILE_OPEN(MPI_COMM_WORLD,fname,MPI_MODE_WRONLY+MPI_MODE_CREATE,MPI_INFO_NULL,ifile,ierr)
do l=1,numvar
disp = Nx_MOK*Ny_MOK*Nz_MOK*WP_MOK*(l-1)
call MPI_FILE_SET_VIEW(ifile,disp,MPI_REAL_SP,view,"native",MPI_INFO_NULL,ierr)
call MPI_FILE_WRITE_ALL(ifile,phi_xyz(1:Nxp,1:Nyp,1:Nzp,l,2*rst 1),Nxp*Nzp*Nyp, MPI_REAL_SP,status,ierr)
end do
call MPI_FILE_CLOSE(ifile,ierr)
call MPI_BARRIER(MPI_COMM_WORLD, ierr)
end do
Related
I want to read text of unknown lenght from a file into a vector like character(len=1) a(3000) but the input file does not have an "end of file marker"
I tried things like:
open(2,file=surnames.txt,status='old')
read(2,15, IOSTAT=ios,END=20,ERR=20)(a(i),i=1,3000)
isize=0
do while (ios == 0)
isize=isize+1
read(2,15, iostat=ios, end=20) a(isize)
end do
15 format(a1)
20 continue
but nothing works
Thanks for your attention
I want to count the total number of rows in a csv file/.txt, output/write it to a new csv file, then clean the file and write a 2nd column to the new file with total number of rows after cleaning. ( I currently have the code for cleaning, I only need help with accepting a file and writing the total rows to a new file before and after cleaning) I have attached the code below which writes only the column name to a new csv file and doesn't print the result.
import csv
data = open ('/anusha.csv','r')
#numline = len(file.readlines(data))
#print(numline)
before_clean = []
with open('out_anusha.csv', 'w') as f1:
for row in data:
f1 = len(file.readlines(data))
before_clean.append(f1)
writer = csv.writer(f1)
f1.write("Before_clean")
Any help is appreciated!
One way to count number of lines in file without going through whole reading process is to use wc utility if this program is supposed to run on *nix system.
You can refer Running "wc -l <filename>" within Python Code
I have a list.dat file that contains the names, in order, of about 1000 hdf files. I need to read these into MATLAB one by one in order and input the data contained in them into a matrix. How do I make MATLAB read in the hdf files? I know how to make MATLAB read one file, but when it's only the filenames in a list (in the same directory as the actual files), I don't know how to make it read in the variable.
Here's what I have so far:
% Read in sea ice concentrations
% AMSR-E data format: 'asi-s6250-20110101-v5.hdf';
% AMSR2 data format: 'asi-AMSR2-s6250-20120724-v5.hdf';
% SSMI data format: 'asi-SSMIS17-s6250-20111001-v5.hdf';
fname = 'list.dat';
data = double(hdfread(fname, 'ASI Ice Concentration'));
This currently does not work. It throws an error saying,
??? Error using ==> hdfquickinfo>findInsideVgroup at 156
HDF file '/home/AMSR_SeaIceData_Antarctic/list.dat' may be invalid or corrupt.
Error in ==> hdfquickinfo at 34
[found, hinfo] = findInsideVgroup ( filename, dataname );
Error in ==> hdfread>dataSetInfo at 363
hinfo = hdfquickinfo(filename,dataname);
Error in ==> hdfread at 210
[hinfo,subsets] = dataSetInfo(varargin{:});
The code works when I just put in the actual filename of the hdf file for fnames.
Thanks.
I am just learning how to code in Python and have not been able to find a solution or answer as to why when I attempt to read a file that has just been written to it bears additional characters.
Code
#-*-coding:utf-8-*-
from sys import argv
from os.path import exists
script, source, copy = argv
print "We'll be opening, reading, writing to and closing a file"
opensource = open(source)
readsource = opensource.read()
print readsource
print "Great. We opened and read file"
opencopy = open(copy, 'w+') #we want to write and read file
opencopy.write(readsource) #copy the contents of the source file
opencopy.read()
opensource.close()
opencopy.close()
Output
Contents
test °D ΃ ø U ø U ` 6 ` 6 0M Ð
I am running version 2.7 of Python on Windows 7 Professional 64bit.
This seems to be a Windows issue with reading a file opened with "w+" directly after a write.
Start with adding two print statements like this:
opencopy.write(readsource) #copy the contents of the source file
print opencopy.tell()
opencopy.read()
print opencopy.tell()
And run this on a file with only as contents the words 'test' + CR + LF, you get as output:
We'll be opening, reading, writing to and closing a file
test
Great. We opened and read file
6
4098
(If you do the same under Linux, the read does not work beyond the end of the file (and you get two times the value 6 from opencopy.tell().)
What you probably want to do is:
print opencopy.tell()
opencopy.seek(0)
print opencopy.tell()
opencopy.read()
print opencopy.tell()
You then get 6 and 6 as output from tell(). Now this results in reading the word 'test', that
you just wrote.
If you do not want to read what you just wrote, put opencopy.flush() between the read and the write statement:
opencopy.write(readsource) #copy the contents of the source file
print opencopy.tell()
opencopy.flush()
opencopy.read()
print opencopy.tell()
I'm working with large CSV. How can I take a random sample of rows—say, 200 total—and recombine them into a CSV with the same structure as the original?
The procedure I would use is as follows:
Generate 200 unique numbers between 0 and the number of lines in the CSV file.
Read each line of the CSV file and keep a track of which line number your are reading. If its line number matches one of the numbers above, then output it.
Use the Resevoir Sampling random sampling technique that does not require all records be in memory or the actual number of records be known. With it, you stream in you records one-by-one and probabilistically select them into the sample. Once the stream is exhausted, output the final sample records. The technique guarantees each record in the stream has the same probability of being in the final sample. That is to say, it generates a simple random sample.
You can use random module's random.sample method to randomize a list of line offsets as shown below.
import random
# Fetching line offsets.
# Courtesy: Adam Rosenfield's tip about how to read a HUGE text file.
# http://stackoverflow.com/questions/620367/
# Read in the file once and build a list of line offsets
line_offset = []
offset = 0
for line in file:
line_offset.append(offset)
offset += len(line)
file.seek(0)
# Part where you pick the random lines and copy to your new file
# My 2 cents.
randoffsets = random.sample(line_offset, 200)
with open('your_file') as f:
for k in randoffsets:
f.seek(k)
f.readline() # and append to your new file
You could try to use linecache if it works for you but since linecache reads the entire file into memory I'm not sure how well it would work for a 6GB file.