Everything I'm finding via google is garbage... Note that I want the answer in C, however if you supplement your answer with a C++ solution as well then you get bonus points!
I just want to be able to read some floats into an array from a binary file
EDIT: Yes I know about Endian-ness... and no I don't care how it was stored.
How you have to read the floats from the file completely depends on how the values were saved there in the first place. One common way could be:
void writefloat(float v, FILE *f) {
fwrite((void*)(&v), sizeof(v), 1, f);
}
float readfloat(FILE *f) {
float v;
fread((void*)(&v), sizeof(v), 1, f);
return v;
}
float f;
if(read(fd,&f,sizeof(f))==sizeof(f))
printf("%f\n",f);
else
printf("oops\n");
Provided that it's written as compatible binary representation.
read for file descriptors, fread for FILE*s and istream::read for c++ iostreams. Pick whatever pleases you:
read(fd,&f,sizeof(f))==sizeof(f)
fread(&f,sizeof(f),1,fp)==1
fin.read((char*)&f,sizeof(f)).gcount()==sizeof(f)
You could use fread. (Note the the API is for C, even though the website says C++ reference :))
Use fread() from <stdio.h>. The assertions should be replaced with actual error handling code.
#include <stdio.h>
#include <assert.h>
#define countof(ARRAY) (sizeof (ARRAY) / sizeof *(ARRAY))
float data[5];
FILE *file = fopen("foo.bin", "rb");
assert(file);
size_t n = fread(data, sizeof(float), countof(data), file);
assert(n == countof(data));
Keep in mind that you might run into endian issues if you transfer files between different architectures.
If the file is all "float" and you wanted to read it X number of times, all you have to do is this:
FILE *fp;
if((fp=fopen("filename.whatever", "rb"))==NULL)
return 0;
fseek(fp, 0, SEEK_END);
long size = ftell(fp);
fseek(fp, 0, SEEK_SET);
float *f = (float *)malloc(sizeof(float)*size);
if(f==NULL)
{
fclose(fp);
return 0;
}
if(fread(f, sizeof(float), size, fp)!=size)
{
fclose(fp);
return 0;
}
fclose(fp);
// do something with f
FILE *thisFile=fopen("filename","fb");
float myFloat;
fscanf(thisFile,"%f",&myFloat);
fclose(thisFile);
This works if the data is written using fprintf (implementation specific)
However, you can also typecast your float to int32 and save , load and typecast.
std::fstream thisFile;
thisFile.open("filename",ios::read|ios::binary);
float myFloat;
thisFile>>myFloat;
thisFile.close();
May be wrong (I haven't used the C++ F.IO functions for a loooong loooong time)
If these values are sequentially placed into a binary file you can do a read of sizeof(float) bytes per float value into a character array. You can then cast these into a float value.
Related
I want to write three characters to a file, then a struct, then one more character.
Finally I would like to read the character before the struct, the struct itself, the character after the struct and display them on the screen.
struct stuff{
int a;
int b;
};
int main(){
FILE * fp = fopen("input.txt", "w+");
char charA = 'z';
char charB = 's';
char charC = 'q';
char charD = 'e';
//create a struct of type stuff
stuff s;
s.a = 123;
s.b = 2111;
//fwrite three first chars
fwrite(&charA, 1, sizeof(char), fp);
fwrite(&charB, 1, sizeof(char), fp);
fwrite(&charC, 1, sizeof(char), fp);
//fwrite the struct
fwrite(&s, 1, sizeof(struct stuff), fp);
//fwrite the last char
fwrite(&charD, 1, sizeof(char), fp);
//read the char before the struct, the struct itself,
// and the char after the struct
char expectedCharC;
stuff expectedStructS;
char expectedCharD;
fseek(fp, sizeof(struct stuff) + sizeof(char), SEEK_END);
fread(&expectedCharC, 1, sizeof(char), fp);
fread(&expectedStructS, 1, sizeof(struct stuff), fp);
fseek(fp, sizeof(char)*3 + sizeof(struct stuff), SEEK_SET);
fread(&expectedCharD, 1, sizeof(char), fp);
cout<<expectedCharC<<" "<<expectedStructS.a<<" ";
cout<<expectedStructS.b<<" "<<expectedCharD<<endl;
fclose(fp);
return 0;
}
Instead of this result:
q 123 2111 e
I get this result:
4197174 0 e
I don't know what I'm doing wrong. I'm writing bytes to the file, reading them back and displaying them on the screen. What goes wrong?
thank you in advance
Wow, lots of problems in your code. Let's tackle them one by one.
As mentioned by unwind, the mode you're using to open the file seems to be incorrect as to what you're trying to do. For one, you're trying to read from a file that is opened for write-only.
You're using fwrite wrongly. It goes fwrite(pointer to data, size of each data, number of data, FILE pointer);.
You're using fseek wrongly. I see you're confused with the offset parameter. This offset defines a signed distance from the origin specified as the last argument to fseek. Therefore, if you're at SEEK_END, you should be moving backwards by having your offset be a negative number.
I've done these changes myself and now it works. Output: q 123 2111 e
Here's a nice little website for you too. Helped me with your problem.
Thank you for reading.
First, as has been pointed out, you must open the file in binary
mode. Even then, just dumping the bytes of a struct means
that you won't be able to read it correctly some time in the
future. But as long as you're reading from the same process, it
should be OK.
The real problem is what you are doing with all of the fseek:
before the first fread, you do an fseek beyond the end of
the file. Any read from that position is guaranteed to fail.
You really should check the status of the file, and ensure that
the fread has succeeded before accessing any of the values you
read. If it failed, accessing the variables (at least those in
stuff) is undefined behavior; most likely, you'll get some
random garbage.
Your first fseek should probably be to the beginning of the file, or
else:
fseek( fp, -(sizeof( stuff ) + 4), SEEK_BEG);
If you've just read the struct, then the second fseek is
unnecessary as well. (In your case, it means that the final
'e' is correctly read.)
You must open your file in binary mode for this to work.
FILE * fp = fopen("input.txt", "wb+");
^
|
blam!
Your wanted result is also a bit unclear, shouldn't it start with the three characters 'z', 's' and 'q', and then have the integers? Note that the integers are likely to appear byte-swapped if you're on a little-endian machine.
To help debug the code, you should add return-value checking to all I/O calls, since I/O can fail. Also note that sizeof (char) is always 1, so it's not very beneficial to write it like that.
I have a program which uses stdio for reading and writing a binary file. It caches the current stream position and will not seek if the read/write offset is already at the desired position.
However, an interesting problem appears, that when a byte is read and the following byte is written, it doesn't actually get written!
Here is a program to reproduce the problem:
#include <cstdio>
int main() {
FILE *f = fopen("test.bin", "wb");
unsigned char d[1024] = { 0 };
fwrite(d, 1, 1024, f);
fclose(f);
f = fopen("test.bin", "rb+");
for (size_t i = 0; i < 1024; i++) {
unsigned char a[1] = { 255 - (unsigned char)(i) };
fflush(f);
fwrite(a, 1, 1, f);
fflush(f);
fseek(f, i, SEEK_SET);
fread(a, 1, 1, f);
printf("%02X ", a[0]);
}
fclose(f);
return 0;
}
You are supposed to see it write the bytes FF down to 00, however only the first byte is written because it does not follow a fread immediately.
If it seeks before fwrite, it acts correctly.
The problem happens on Visual Studio 2010/2012 and TDM-GCC 4.7.1 (Windows), however it works on codepad which I guess is due to it being executed on Linux.
Any idea why this happens?
C99 §7.18.5.3/6 (quoted from N869 final draft):
“When a file is opened with update mode (’+’ as the second or third character in the
above list of mode argument values) […] input shall not be directly followed by output without an
intervening call to a file positioning function, unless the input operation encounters end-
of-file.”
I'm trying to write-to-disk an array containing 11.26 million uint16_t values. The total memory size should be ~22 MB. However, the size of my file is 52MB. I'm using fprintf to write the array to disk. I thought maybe the values were being promoted. I tried to be explicit but it seems to make no difference. The size of my file is stubbornly unchanged.
What am I doing wrong? Code follows.
#define __STDC_FORMAT_MACROS
...
uint32_t dbsize = 11262336;
uint16_t* db_ = new uint16_t[dbsize_];
...
char fname[256] = "foo";
FILE* f = fopen(fname, "wb");
if(f == NULL)
{
return;
}
fprintf(f, "%i\t", dbsize_);
for(uint32_t i = 0; i < dbsize_; i++)
{
fprintf(f, "%" SCNu16 "", db_[i]);
}
fclose(f);
You're writing ASCII to your file, not binary.
Try writing your array like this instead of using fprintf in a loop.
fwrite(db_, sizeof(db_[0]), dbsize, f);
fprintf always formats numbers and other types to text, whether you've opened the file in binary mode or not. Binary mode just keeps the runtime from doing things like converting \n to \r\n.
fprintf will convert you number to a series of ASCII characters and write them to a file. Depending on its value, a 32-bit int will be from 1 to 10 characters long when expressed as a string. You need to use fwrite to write raw binary values to a file.
The source of confusion is likely to be that the "b" in FILE* f = fopen(fname, "wb"); does not do what you think it does.
Most significantly, it doesn't change any of the print or scan statements to use binary values instead of ASCII values. Like others have said - use fwrite instead.
I'm starting to write a c++ program that is supposed to read a large binary file (several gigabytes). As a start, my program just does fread() inside a loop to read into an array of doubles:
double *arr = (double*)malloc(sizeof(double) * ARRSIZE);
FILE *fp = fopen(argv[1], "r");
int num_read;
do {
num_read = fread(arr, sizeof(double), ARRSIZE, fp);
} while (num_read > 0);
free(arr);
I want to see the throughput of my program, but I find that I can read two gigabytes in about half a second. Are there any compiler optimizations that may have ignored the fread() calls if I'm not doing anything with the data?
If there are, how can I be sure I'm seeing the actual maximum throughput of my program?
Thanks!
If I have a large binary file (say it has 100,000,000 floats), is there a way in C (or C++) to open the file and read a specific float, without having to load the whole file into memory (i.e. how can I quickly find what the 62,821,214th float is)? A second question, is there a way to change that specific float in the file without having to rewrite the entire file?
I'm envisioning functions like:
float readFloatFromFile(const char* fileName, int idx) {
FILE* f = fopen(fileName,"rb");
// What goes here?
}
void writeFloatToFile(const char* fileName, int idx, float f) {
// How do I open the file? fopen can only append or start a new file, right?
// What goes here?
}
You know the size of a float is sizeof(float), so multiplication can get you to the correct position:
FILE *f = fopen(fileName, "rb");
fseek(f, idx * sizeof(float), SEEK_SET);
float result;
fread(&result, sizeof(float), 1, f);
Similarly, you can write to a specific position using this method.
fopen allows to open a file for modification (not just to append) by using either the rb+ or wb+ mode on fopen. See here: http://www.cplusplus.com/reference/clibrary/cstdio/fopen/
To position the file to a specific float, you can use the fseek by using index*sizeof(float) as the offset ad SEEK_SET as the orign. See here: http://www.cplusplus.com/reference/clibrary/cstdio/fseek/
Here is an example if you would like to use C++ streams:
#include <fstream>
using namespace std;
int main()
{
fstream file("floats.bin", ios::binary);
float number;
file.seekp(62821214*sizeof(float), ios::beg);
file.read(reinterpret_cast<char*>(&number), sizeof(float));
file.seekp(0, ios::beg); // move to the beginning of the file
number = 3.2;
// write number at the beginning of the file
file.write(reinterpret_cast<char*>(&number), sizeof(float));
}
One way would be to call mmap() on the file. Once you've done that, you can read/modify the file as if it was an in-memory array.
Of course that method only works if the file is small enough to fit in your process's address space... if you're running in 64-bit mode, you'll be fine; in 32-bit mode, a file with 100,000,000 floats should fit, but another order or two of magnitude above that and you might run into trouble.
I know this question has been answered already, but Linux/Unix provides easy system calls to read/write(pread/pwrite) in the middle of a file. If you look at the kernel source code for the system calls 'read' & 'pread', both eventually calls the vfs_read().And vfs_read requires a OFFSET, i.e it requires a POSITION to read from the file. In pread,this offset is given by us and in read() the offset is calculated internally in the kernel and maintained for the file descriptor. pread() offers exceptional performance compared to read() and using pread ,you can read/write in the same file descriptor simultaneously in multiple threads in different parts of the file. My Humble opionion, never use read() or other file streams, use pread(). Hope the filestream libraries have wrapped the read() calls, the streams perform well by making fewer system calls.
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
int main()
{
char* buf; off_t offToStart = id * sizeof(float); size_t sizeToRead = sizeof(float);
int fd = open("fileName", O_RDONLY);
ret = pread(fd, buf, sizeToRead, offToStart);
//processs from the read 'buf'
close(fd);
}