RESOLVED
I'm trying to make a simple file loader.
I aim to get the text from a shader file (plain text file) into a char* that I will compile later.
I've tried this function:
char* load_shader(char* pURL)
{
FILE *shaderFile;
char* pShader;
// File opening
fopen_s( &shaderFile, pURL, "r" );
if ( shaderFile == NULL )
return "FILE_ER";
// File size
fseek (shaderFile , 0 , SEEK_END);
int lSize = ftell (shaderFile);
rewind (shaderFile);
// Allocating size to store the content
pShader = (char*) malloc (sizeof(char) * lSize);
if (pShader == NULL)
{
fputs ("Memory error", stderr);
return "MEM_ER";
}
// copy the file into the buffer:
int result = fread (pShader, sizeof(char), lSize, shaderFile);
if (result != lSize)
{
// size of file 106/113
cout << "size of file " << result << "/" << lSize << endl;
fputs ("Reading error", stderr);
return "READ_ER";
}
// Terminate
fclose (shaderFile);
return 0;
}
But as you can see in the code I have a strange size difference at the end of the process which makes my function crash.
I must say I'm quite a beginner in C so I might have missed some subtilities regarding the memory allocation, types, pointers...
How can I solve this size issue?
*EDIT 1:
First, I shouldn't return 0 at the end but pShader; that seemed to be what crashed the program.
Then, I change the type of reult to size_t, and added a end character to pShader, adding pShdaer[result] = '/0'; after its declaration so I can display it correctly.
Finally, as #JamesKanze suggested, I turned fopen_s into fopen as the previous was not usefull in my case.
First, for this sort of raw access, you're probably better off
using the system level functions: CreateFile or open,
ReadFile or read and CloseHandle or close, with
GetFileSize or stat to get the size. Using FILE* or
std::filebuf will only introduce an additional level of
buffering and processing, for no gain in your case.
As to what you are seeing: there is no guarantee that an ftell
will return anything exploitable as a numeric value; it could
very well be just a magic cookie. On most current systems, it
is a byte offset into the physical file, but on any non-Unix
system, the offset into the physical file will not map directly
to the logical file you are reading unless you open the file in
binary mode. If you use "rb" to open the file, you'll
probably see the same values. (Theoretically, you could get
extra 0's at the end of the file, but practically, the OS's
where that happened are either extinct, or only used on legacy
mainframes.)
EDIT:
Since the answer stating this has been deleted: you should loop
on the fread until it returns 0 (setting errno to 0 before
each call, and checking it after the return to see whether the
function returned because of an error or because it reached the
end of file). Having said this: if you're on one of the usual
Windows or Unix systems, and the file is local to the machine,
and not too big, fread will read it all in one go. The
difference in size you are seeing (given the numerical values
you posted) is almost certainly due to the fact that the two
byte Windows line endings are being mapped to a single '\n'
character. To avoid this, you must open in binary mode;
alternatively, if you really are dealing with text (and want
this mapping), you can just ignore the extra bytes in your
buffer, setting the '\0' terminator after the last byte
actually read.
Related
I am porting over c++ code from linux to windows. I am currently using Visual Studio 2013 to port my code.
I need to read a binary file and am using this portion of c++ code:
// Open the stream
std::ifstream is("myfile.bin");
// Determine the file length
is.seekg(0, std::ios_base::end);
std::size_t size=is.tellg();
is.seekg(0, std::ios_base::begin);
// Create a vector to store the data
int* Data = new int[size/sizeof(int)];
// Load the data
is.read((char*) &Data[0], size);
// Close the file
is.close();
In linux, the size of my binary file is correctly found to be 744mb. However, in windows, the size of my binary file is incorrectly found to be >4GB. How can I correct this issue?
Change std::ifstream is("myfile.bin"); to std::ifstream is("myfile.bin", std::ios::binary);
With your current default open mode, the compiler choses "char" mode. In Linux chars in files are UTF8, first 128 positions are 1-byte char. But for memory UTF32, 4-bytes per char, is used. In Windows chars are "wide-chars", 2-bytes per char.
I finally had the time to actually run this myself, though I had to fix a couple of things, like ios_base::beg instead of begin (different function) Also, as mentioned, the array allocation should be this int* Data = new int[size / sizeof(int) + 1]; // At most one extra int
I found your problem: you're not in the right directory. Check if you successfully opened the file or not. If you don't, then you get a huge garbage value (probably -1, but unsigned, so massive) for size.
Try this to find your directory in Windows: (probably need Windows.h or something that I "just had" already)
char dirBuf[256];
GetCurrentDirectory(256, dirBuf);
cout << "Current directory is: " << dirBuf << endl;
See if that's where your file is and move it accordingly. Or specify the ENTIRE path in the constructor to ifstream.
Also, it has nothing to do with ios::binary or not. Works fine both ways, or fails if the file isn't there.
std::size_t size=is.tellg();
The standard doesn't require tellg to return the byte offset from the beginning of the file. In general, this may not be a reliable way to get the size of the file, though it probably does what you expect on Linux and Windows.
The return type of the tellg method is std::basic_stream::pos_type, so you're starting with an implicit conversion to std::size_t which may or may not be appropriate. In a 32-bit build, for example, it's conceivable that the size of a file could be larger than a std::size_t can represent.
But the root problem is that you're not checking for errors. If you have exceptions disabled, then tellg reports an error by returning pos_type(-1). When you cast that to an unsigned type (which std::size_t is), then you get a very large value. I suspect you failed to open the file, and since you didn't detect that error, the seekg and the tellg failed. You then coerced pos_type(-1) to a std::size_t, which made it look like the file was huge.
You also have the problems others have noted: failing to open the file in binary mode and computing the wrong size for the buffer when the file isn't a multiple of the size of an int.
The most reliable to get the file size is to use the OS's API. On Windows, you can do this instead:
// Open the file. [TODO: Get the file name in wide characters and use
// CreateFileW instead. If the file name contains characters not
// representable by the user's ANSI codepage, then CreateFileA will fail.]
HANDLE hfile = CreateFileA("myfile.bin", GENERIC_READ, FILE_SHARE_READ,
nullptr, OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL | FILE_FLAG_SEQUENTIAL_SCAN,
nullptr);
if (hfile == INVALID_HANDLE_VALUE) { error handling here }
// Figure out how big it is.
LARGE_INTEGER li_size;
if (!GetFileSizeEx(hfile, &li_size)) { error handling here }
// TODO: On a 32-bit build, this won't be able to handle huge files,
// so check that here.
std::size_t size = li_size.QuadPart;
// Create a buffer to store the data, being careful to round up to a
// multiple of sizeof(int). [TODO: Use a std::vector instead.]
int* Data = new int[(size + sizeof(int) - 1) / sizeof(int)];
// Load the data.
const DWORD BytesToRead = static_cast<DWORD>(size);
DWORD BytesRead = 0;
if (!ReadFile(hfile, Data, &BytesRead, nullptr) || BytesRead < BytesToRead) {
error handling here
}
// Close the file
CloseHandle(hfile);
int* Data = new int[size/sizeof(int)];
Why are you doing this? You're dividing the size by 4. You don't want to do this. It should just be int* Data = new int[size]
Also, it should be std::ifstream f("filename.bin", std::ios::binary);
I would like to read the data with fread from file. However, I encounter the problem of setting of NULL terminator.I think the the line (fileMem[fileSize] = 0;) should have been solved. However, I still get rubbish at the with I check the value of "fileMem". Would anyone help me to find the problem?
I followed other posts about setting the NULL terminator but just does not work
File *input = fopen(filePath, "r");
fseek(input, 0, SEEK_END);
auto fileSize = ftell(input);
fseek(input, 0, SEEK_SET);
char* fileMem = new char[fileSize+1];
fileMem[fileSize] = 0;// the NULL terminator problem should have been solved here
clearerr(input);
fread(fileMem, fileSize,1, input);
What is the problem with my code?
fread is reading more bytes than fileSize, because you are specifying a record size of fileSize, and asking it to read only one text record. It then overwrites the 0 at the end with actual data, so you get garbage.
fread returns the number of bytes it actually read, so you can allocate a bigger buffer, then use the return value from fread to determine how much of it is valid (and to set a null-terminator).
Since it is updating your data in this way, I also suggest changing the file type to binary ("rb" rather than "r" in call to fopen).
The reason this is happening is because fread performs translation of text when in text mode ("r" rather than "rb"), such as carriage returns and line feeds.
Assuming you are on Windows, I think the problem is that you are opening the file in text mode and using fread which is meant for binary mode. In text mode, what you read may not be exactly what is in the file. Windows text files have "\r\n" at the end of the file, but in text mode this two character combination is converted to a single character, "\n". So the fileSize you computed will be too large and so your null terminator will be in the wrong place.
To verify this, change your fread to be:
int nr = fread( fileMen, 1, fileSize, input);
Swapping the middle args, will have fread return the number of bytes read. If you look at the value of nr, it will be smaller than fileSize because of the line end translation.
This wouldn't be a problem on a *nix system since there is no translation there in text mode.
When you want to use fread to read the contents of a file, you must open the file in binary mode.
FILE *input = fopen(filePath, "rb");
^^
Otherwise, the size of the file you get by using
fseek(input, 0, SEEK_END);
auto fileSize = ftell(input);
will be greater than the number of characters that what can be read by fread.
If you have a CR and a LF, they will count as two characters by the above method but fread will read only one character. Hence, fread will read less than fileSize characters. You can also change the fread line to:
// Swap the middle arguments.
// The first argument is supposed to be the size of each object.
// The second argument is supposed to be the number of objects to read.
auto n = fread(fileMem, 1, fileSize, input);
if ( n != fileSize )
{
// Surprise
}
fileMem[n] = '\0';
My function gets a FILE* to read from, and it needs to read starting from some non-negative offset.
I could use fseek(file, offset, SEEK_SET), but it fails on stdin, for instance.
How can I determine if fseek works? If it doesn't, I could read and discard offset bytes.
And is there a way to read (and discard) from FILE without allocating read buffer?
You can test if fseek works on that stream, by calling fseek( file, offset, SEEK_SET) and on error, checking that errno == EBADF which is returned to say "The stream specified is not a seekable stream".
I think you need to read and discard, with a buffer, but if it can just be pagesize bytes and you keep a count of bytes read, reading till you did the equivalent of a seek. If it were a memory mappable file, then you can read without reading, but then the seek would have worked.
The return value of fseek() tells you if it worked or not:
Return Value
...Upon successful completion, fgetpos(), fseek(), fsetpos() return 0, ...Otherwise, -1 is returned and errno is set to indicate the error.
So attempt to fseek() from the file and check the return result, and handle your failure case accordingly. Ex:
ret = fseek(stdin, 0, SEEK_SET);
if(ret < 0)
printf("failed because: %s\n", strerror(errno));
will give you something like:
failed because: Illegal seek
So that failed because you can't seek stdin, where as:
FILE * fp = fopen("word.txt", "r");
ret = fseek(fp, 0, SEEK_SET);
if(ret < 0)
printf("failed because: %s\n", strerror(errno));
Wouldn't print anything because you got back 0 indicating success (assuming of course that "word.txt" exists, is readable, was opened successfully, etc).
I don't understand this part of your question:
is there a way to read (and discard) from FILE without allocating read buffer
You can just fseek() to the point you want to read, or you can read into an array to a buffer and then overwrite the results. The answer depends on your goals, but using things like fread() or read() will require a non-null pointer to store data into.
bool CReadWrite::write(unsigned long long offset, void* pvsrc, unsigned long long nbytes)
{ int WriteResult;
pFile = fopen("D:\\myfile.bin","wb");
if (!pFile){
puts("Can't open file");
return false;
}
//offset = fseek(pFile,offset,
WriteResult = fwrite (pvsrc, 1, nbytes, pFile);
if (WriteResult == nbytes){
puts("Wrote to file");
fclose(pFile);
return true;
}
else{
puts("Unable to write to File.");
fclose(pFile);
return false;
}
}
This is my class function so far. I'm basically opening a file, checking to see if it did indeed open if not get out. Writes the file, checks to see if the file see if the file was indeed written to returns true. else return false. As you can tell by my parameters, I'm looking to create an offset where I can give a particular offset i.e. 10, and start from 10 and then from there write. I know for sure I need to use fseek but I can't assume that I'm at the beginning of the file or anywhere in the file. Im pretty sure i need to use SEEK_SET but I may be wrong. Any thoughts on implemented the above desires? Thanks.
If you're using fopen without the append setting (as you are, "wb" creates an empty file), you can assume you're at the beginning.
Regardless, SEEK_SET sets the position to the given offset from the beginning.
If the file doesn't have the offset that you want to seek to (as it is in your case), then the question is what are you required to do? If just pad - then write offset padding bytes, and then your content, otherwise maybe you wanted to use "a" and not "w". "w" truncates the existing content of the file, while "a" opens for append and sets position to the end of the existing content.
More details here.
I am writing a C library that reads a file into memory. It skips the first 54 bytes of the file (header) and then reads the remainder as data. I use fseek to determine the length of the file, and then use fread to read in the file.
The loop runs once and then ends because the EOF is reached (no errors). At the end, bytesRead = 10624, ftell(stream) = 28726, and the buffer contains 28726 values. I expect fread to read 30,000 bytes and the file position to be 30054 when EOF is reached.
C is not my native language so I suspect I've got a dumb beginner mistake somewhere.
Code is as follows:
const size_t headerLen = 54;
FILE * stream;
errno_t ferrno = fopen_s( &stream, filename.c_str(), "r" );
if(ferrno!=0) {
return -1;
}
fseek( stream, 0L, SEEK_END );
size_t bytesTotal = (size_t)(ftell( stream )) - headerLen; //number of data bytes to read
size_t bytesRead = 0;
BYTE* localBuffer = new BYTE[bytesTotal];
fseek(stream,headerLen,SEEK_SET);
while(!feof(stream) && !ferror(stream)) {
size_t result = fread(localBuffer+bytesRead,sizeof(BYTE),bytesTotal-bytesRead,stream);
bytesRead+=result;
}
Depending on the reference you use, it's quite apparent that adding a "b" to the mode flag is the answer. Seeking nominations for the bonehead-badge. :-)
This reference talks about it in the second paragraph, second sentence (though not in their table).
MSDN doesn't discuss the binary flag until halfway down the page.
OpenGroup mentions the existance of the "b" tag, but states that it "shall have no effect".
perhaps it's a binary mode issue. Try opening the file with "r+b" as the mode.
EDIT: as noted in a comment "rb" is likely a better match to your original intent since "r+b" will open it for read/write and "rb" is read-only.
Also worth noting that simply including binmode.obj into your link command will do this for you for all file opens.
A solution, based on the previous answers:
size_t bytesRead = 0;
BYTE* localBuffer = new BYTE[bytesTotal];
fseek(stream,headerLen,SEEK_SET);
while(!feof(stream) && !ferror(stream)) {
size_t result = fread(localBuffer+bytesRead,sizeof(BYTE),bytesTotal-
bytesRead,stream);
bytesRead+=result;
}