reading hexadecimal data from file in fortran - fortran

Question

In Fortran, I'm trying to read a file with data in 8-bit (hexadecimal) bytes, on Linux.
In 'hexedit' the first line looks as it should, for the tiff-file it is.
49 49 2A 00 08 00 20 00 00 00 0B 02 00 00 00 00 II*... .........
I declare a two-byte character variable (character(len=2) :: tifhead(8))
and read like this:
open(1,file=filename,access='stream')
read(1) tifhead,greyvalue
I get the first two (49 49), which print out as II in a formatted write
(format (2Z2), but not the other ones.
How can I get all these hex values out? I should see 49 49 2A 00 08 .......
.

Answer 1

Your read statement will simply read 2 characters for tifhead(1), the next 2 characters for tifhead(2), etc, including spaces. Therefore you end up with tifhead(1)="49", tifhead(2)=" 4", tifhead(3)="9 ", and so on. You think you read the first 2 bytes correctly only because you print the strings "49", " 4", "9 ",... one after the other, so it looks like "49 49 " in the output. The compiler has no way to know there is a single blank space separating strings and 2 spaces every four data.
To read your data properly you must use formatted reading which implies you must also declare your stream as 'formatted' in the open statement. The following example shows how this can be done:
program example
implicit none
character(len=2) :: tifhead(8), greyscale(8)
open(1, file="example.txt", access='stream', form='formatted')
read(1, "(4(a2,tr1),tr1,3(a2,tr1),a2)", advance='no') tifhead
read(1, "(tr2,4(a2,tr1),tr1,3(a2,tr1),a2)", advance='no') greyscale
close(1)
print "(a,7(a2,tr1),a2,a)", " tifhead = (", tifhead, ")"
print "(a,7(a2,tr1),a2,a)", "greyscale = (", greyscale, ")"
end program example
Perhaps some explanation is needed: a2,tr1 means read a string of 2 characters, then advance the reading pointer once (this skips the space between your hexadecimal "numbers" - actually, they are treated as just strings). 4(a2,tr1) means do that 4 times. This reads the first 4 bytes plus one space. Now, there is one more space before the next data to be read so we add tr1 to skip it, and our format is 4(a2,tr1),tr1 so far; then we read 3 more bytes with 3(a2,tr1), then the last byte alone with just a2 (not skipping the space after it). So the format string is (4(a2,tr1),tr1,3(a2,tr1),a2), which will read the first 8 bytes correctly, leaving the reading pointer right after the 8th byte. Note that advance='no' is necessary, otherwise Fortran will assume carriage return and will skip the rest of the data in the same record (line).
Now, to read the next 8 bytes we use the same format, except we add tr2 in the beginning to skip the two blank spaces. I added formatted printing in the program to check if data were read correctly. Running the program gives:
tifhead = (49 49 2A 00 08 00 20 00)
greyscale = (00 00 0B 02 00 00 00 00)
which verifies data were read correctly.
Last but not least, I would recommend to avoid old-fashion Fortran used in your code and the example above. This means use newunit to let the program find the first free unit instead of explicitly giving a unit number, have some way to check if the file you are trying to open actually exists or if you reached end of file, avoid unnamed arguments, use the dimension attribute to declare arrays, etc. None of those is strictly necessary, and it might look like unnecessary verbosity at first. But in the long run being strict (as modern Fortran encourages) will save you a lot of time while debugging larger programs. So the above example could (arguably should) be written as follows.
program example2
implicit none
integer :: unt, status
character(len=2), dimension(8) :: tifhead, greyscale
open(newunit=unt, file="example.txt", access='stream', form='formatted',&
action='read', status='old', iostat=status)
if (status /= 0) then
print "(a)","Error reading file."; stop
end if
! More sophisticated reading is probably needed to check for end of file.
read(unit=unt, fmt="(4(a2,tr1),tr1,3(a2,tr1),a2)", advance='no') tifhead
read(unit=unt, fmt="(tr2,4(a2,tr1),tr1,3(a2,tr1),a2)") greyscale
close(unit=unt)
print "(a,7(a2,tr1),a2,a)", " tifhead = (", tifhead, ")"
print "(a,7(a2,tr1),a2,a)", "greyscale = (", greyscale, ")"
end program example2

Answer 2

I wasn't sure if I had to massively modify my previous answers (since I believe they still serve a purpose), so I decided to just add yet another answer, hopefully the last one. I apologize for the verbosity.
The following Fortran 90 module provides a subroutine named tiff_reader_16bit which reads any TIFF data file and returns its 16-bit content in an array of integers:
module tiff_reader
implicit none
private
public :: tiff_reader_16bit
contains
subroutine tiff_reader_16bit(filename, tifdata, ndata)
character(len=*), intent(in) :: filename
integer, allocatable, intent(out) :: tifdata(:)
integer, intent(out) :: ndata
integer, parameter :: max_integers=10000000
integer :: unt, status, record_length, i, records, lsb, msb
character ch;
integer, dimension(max_integers) :: temp
ndata=0
inquire(iolength=record_length) ch
open(newunit=unt, file=filename, access='direct', form='unformatted',&
action='read', status='old', iostat=status, recl=record_length)
if (status /= 0) then
print "(3a)","Error reading file """,filename,""": File not found."; return
end if
records=1
do i=1,max_integers
read(unit=unt, rec=records, iostat=status) ch; msb=ichar(ch)
if (status /= 0) then; records=records-1; ndata=i-1; exit; end if
read(unit=unt, rec=records+1, iostat=status) ch; lsb=ichar(ch)
if (status /= 0) then; ndata=i; temp(ndata)=msb; exit; end if
temp(i)=lsb+256*msb; records=records+2
end do
close(unit=unt)
if (ndata==0) then
print "(a)","File partially read."; records=records-1; ndata=max_integers
end if
allocate(tifdata(ndata), stat=status); tifdata=temp(:ndata)
print "(2(i0,a),/)",records," records read, ",ndata," 16-bit integers returned."
end subroutine tiff_reader_16bit
end module tiff_reader
The subroutine gets the TIFF file name and returns an array of integers, together with the the total number of integers read. Internally, the subroutine uses a fixed-size array temp to temporarily store the data. To save memory, the subroutine returns an allocatable array tifdata which is part of temp, containing the data that was read only. The maximum number of data read is set in the parameter max_integers to 10 million, but can go up to huge(0) if necessary and if memory allows (in my system that's about 2.14 billion integers); it can go even further if you use "higher" kind of integers. Now, there are other ways to do that, avoiding the use of a temporary fixed-size array, but this usually comes at the cost of additional computation time, and I wouldn't go that way. More sophisticated implementations can also be done, but that would add more complexity to the code, and I don't think it fits here.
Since you need the results in the form of 16-bit data, two consecutive bytes from the file must be read, then you treat them as most significant byte first, less significant byte next. This is why the first byte read in each iteration is multiplied by 256. Note that this is NOT always the case in binary files (but it is in TIFF). Some binary files come with less significant byte first.
The subroutine is lengthier than the previous examples I posted, but that's because I added error checking, which is actually necessary. You should always check if the file exists and if end of file has been reached while reading it. Special care must also be taken for TIFF images with an "orphan" last byte (this is indeed the case for the sample file "FLAG_T24.TIF" which I found here - but not the case for the sample image "MARBLES.TIF" found at the same webpage).
An example driver program using the module above would be:
program tiff_reader_example
use tiff_reader
implicit none
integer :: n
integer, allocatable :: tifdata(:)
call tiff_reader_16bit("FLAG_T24.TIF", tifdata, n);
if (n > 0) then
print "(a,7(z4.4,tr1),z4.4,a)", "First 8 integers read: (", tifdata(:8), ")"
print "(a,7(z4.4,tr1),z4.4,a)", " Last 8 integers read: (", tifdata(n-7:), ")"
deallocate(tifdata)
end if
end program tiff_reader_example
Running the program gives:
46371 records read, 23186 16-bit integers returned.
First 8 integers read: (4949 2A00 0800 0000 0E00 FE00 0400 0100)
Last 8 integers read: (F800 F8F8 00F8 F800 F8F8 00F8 F800 00F8)
which is correct. Note that in this case the number of records (= bytes, since the file is opened as unformatted) is not double the number of integers returned. That's because this particular sample image has that "orphaned" last byte I mentioned earlier. Also note that I used another format to print 16-bit hexadecimals, including leading zeroes if needed.
There are more detailed explanations that can be given but this thread is already quite long. Feel free to ask in the comments if something is not clear.
EDIT: By default intel Fortran treats direct access records as 4-byte words, which doesn't seem quite right to me. This unusual behavior can be fixed with a compiler flag, but to avoid the lack of portability in case someone uses that specific compiler without such a flag, I slightly modified the module tiff_reader to take care of this.

Answer 3

Assuming your data are actually stored in binary format (in fact it seems to be a tiff image data file), my first answer is valid only if you convert data to plain text. If you prefer to read the binary file directly, the simplest way I can think of is to open the file with access='direct', and read data byte-by-byte. Each byte is read as a character, then it is converted to an integer, which I guess is more useful than a string supposed to represent a hexadecimal number.
As an example, the following program will read the header (first 8 bytes) from a tiff data file. The example reads data from a sample tiff image I found here, but it works for any binary file.
program read_tiff_data
implicit none
integer :: unt, status, i
character :: ch
integer, dimension(8) :: tifhead
open(newunit=unt, file="flag_t24.tif", access='direct', form='unformatted',
action='read', status='old', iostat=status, recl=1)
if (status /= 0) then
print "(a)","Error reading file."; stop
end if
do i=1,8
read(unit=unt, rec=i) ch; tifhead(i)=ichar(ch)
end do
close(unit=unt)
print "(a,7(i0,tr1),i0,a)", "tifhead = (", tifhead, ")"
end program read_tiff_data
The program gives this output:
tifhead = (73 73 42 0 8 0 0 0)
which is correct. You can easily expand the program to read more data from the file.
If you still need the hexadecimal representation, just replace i0 with z0 in the print statement so that it reads
print "(a,7(z0,tr1),z0,a)", "tifhead = (", tifhead, ")"
This will print the result in hexadecimals, in this case:
tifhead = (49 49 2A 0 8 0 0 0)

Answer 4

Here is the code that works for me. Most of this is comments. Any remarks you may have on the fortran style are most welcome. Please note that I've been familiar with fortran 77 in the past, and learned a little more modern fortran in the process of writing this piece of code
program putiff
c This program is solely intended to read the data from the .tif files made by the CCD camera
c PIXIS 1024F at beamline 1-BM at the Advanced Photon Source, so that they can be manipulated
c in fortran. It is not a general .tif reader.
c A little bit extra work may make this a reader for baseline .tif files,: some of the
c information below may help with such an implementation.
c
c The PIXIS .tif file is written in hex with the little-endian convention.
c The hex numbers have two 8-bit bytes. They are read with an integer(kind=2) declaration.
c When describing an unsigned integer these cover numbers from 0 to 65535 (or 2**16-1).
c For the PIXIS files the first two bytes are the decimal number 18761. The TIFF6 specification
c gives them as a hexadecimal number (0x4949 for a little-endian convention, 4D4D for the
c big-endian convention. The PIXIS files are little-endian.
c
c The next two bytes should be 42 decimal, and 0x2A.
c
c The next 4 bytes give the byte offset for the first image file directory (IFD) that contains
c all the other information needed to understand how the .tif files are put together.
c This number should be read together as a 4 byte integer (kind=4). These (unsigned) integers
c go from 0 to 2**32-1, or 4294967295: this is the maximum file length for a .tif file.
c For the PIXIS this number is 2097160, or 0x200008: in between are the image date for the
c PIXIS's 1024x1024 pixels, each with a two-byte gray range from 0 to 2**16-1 (or 65535 decimal).
c Therefore the PIXIS image can be read without understanding the IFD.
c
c The line right below the hex representation gives the byte order, for the
c little-endian convention indicated by two first bytes. It's 4949 for little-endian,
c in both the first and in the second byte separately. The byte order is then least importan
c part first; with two bytes together, it is byte by byte. For big-endian it is 4D4D.
c
c One way to confirm all this information is to look at the files
c with a binary editor (linux has xxd) or a binary editor (linux has hexedit).
c For the PIXIS image .tif file, the first 8 bytes in hexedit are indeed:
c 49 49 2A 00 08 00 20 00
c For a little-endian file, the bytes are read from the least important to the
c most important within the two-byte number, like this:
c 49 49 2A 00 08 00 20 00
c (34 12) (34 12) (78 56 34 12)
c Here the byte order is indicated below the numbers. The second two-byte number is
c therefore 2+2*16+0*256+0*4096, or 42. Likewise, the last 4-byte number is 0x00200008.
c
c (When the individual byte are read in binary (with 'xxd -b -l 100') this gives
c for the hexadecimals 49 49 2A 00 08 00 20 00
c binary 01001001 01001001 00101010 00000000 00001000 00000000 00100000 00000000
c in ASCII I I * . . . . . )
c After the PIXIS data comes the so-called IFD (Image File Directory).
c These contain 209 bytes. They mean something, but what I do not know. I printed them
c out one by one at the end of the program. Perhaps they are better read in two-byte units
c (right now they are read as 'integer(kind=1); integer(kind=2) may be better). But, then
c there's an odd number so you have to read one separately.
c I want to know these only because I want to use the same .tif format to
c write the results of rctopo (the max, the COM, the FWHM, and the spread).
c I know what's in the first 8 bytes, and what the data are, so I can just
c copy the ifd at the end and count on getting a good .tif file back.
c It's sort of stupid, but it should work.
use iso_fortran_env
implicit logical (A-Z)
integer :: j,jmin,jmax
integer :: k,kmin,kmax
integer :: ifdlength
data jmin,kmin/1,1,/
parameter(jmax=1024,kmax=1024)
parameter(ifdlength=209)
c 8-byte header that starts the PIXIS data file
integer (kind=2) :: tifh12,tifh34 ! each two (8-bit) bytes
integer (kind=4) :: tifh5678 ! 4 bytes
c open and read the file now that you have the correct file name in the sequence
open(newunit=unt,file='tiff_file,access='stream',iostat=ios)
if (ios /= 0) then ; call problem(ios,'read_in_samples'); end if
read (unt) tifh12,tifh34,tifh5678,greyread,ifd
close (unt)
stop
end