I wrote a simple function to load a char * buffer from a file but when compiled through vs2017 it add rubbish at the end of the buffer but mingw compiled exe gives correct output
the function looks like something
#include <stdio.h>
#include <stdlib.h>
using namespace std;
char * loadfromfile(const char * _Filename)
{
char * buffer;
FILE * file = fopen(_Filename, "r");
if (!file)
return nullptr;
fseek(file, 0, SEEK_END);
auto _length = ftell(file);
buffer = new char[_length + 1];
rewind(file);
printf("characters read(loadformfile()) :%i\n",fread(buffer, sizeof(char), _length, file));
buffer[_length] = '\0';
fclose(file);
return buffer;
}
int main() {
char * str = loadfromfile("D:\\shutdown.bat");
printf("%s\n", (str) ? str : "failed to load");
delete[] str;
return 0;
}
VS2017 output:
characters read(loadformfile()) :86
#echo off
Set minutes=30
set /a seconds=%minutes%*60
TIMEOUT /T %seconds%
shutdown /s
\inst
g++ (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 7.1.0 output:
characters read(loadformfile()) :zu
#echo off
Set minutes=30
set /a seconds=%minutes%*60
TIMEOUT /T %seconds%
shutdown /s
file is:
#echo off
Set minutes=30
set /a seconds=%minutes%*60
TIMEOUT /T %seconds%
shutdown /s
EDIT:
A working solution
char * loadfromfile(const char * _Filename)
{
char * buffer;
FILE * file = fopen(_Filename, "r");
if (!file)
return nullptr;
fseek(file, 0, SEEK_END);
auto _length = ftell(file);
buffer = new char[_length + 1];
rewind(file);
buffer[fread(buffer, sizeof(char), _length, file)] = '\0';
fclose(file);
return buffer;
}
You can't portably use fseek()/ftell() to get the size of a file.
Per the C Standard, footnote 268, p 301:
Setting the file position indicator to end-of-file, as with
fseek(file, 0, SEEK_END), has undefined behavior for a binary
stream ...
and
7.21.9.2 The fseek function
... A binary stream need not meaningfully support fseek calls with a
whence value of SEEK_END.
So you can't reliably use fseek() to get to the end of a binary file. In fact, doing so is specifically stated by the C Standard to be undefined behavior.
OK, so you can use fseek() to get to the end of a file opened in text mode, but
7.21.9.4 The ftell function
...
For a text stream, its file position indicator contains unspecified
information, usable by the fseek function for returning the file
position indicator for the stream to its position at the time
of the ftell call; the difference between two such return
values is not necessarily a meaningful measure of the number of
characters written or read.
On a text file, ftell() doesn't return a value useful in getting the file size.
In short, using fseek()/ftell() to get the size of a file is fundamentally broken. The fact that it works sometimes is just an implementation detail.
Related
Scenario: I have a file that is 8,203,685 bytes long in binary, and I am using fread() to read in the file.
Problem: Hexdumping the data after the fread() on both Linux and Windows yields different results. Both hexdump files are the same size, but on Linux it matches the original input file that went in, whereas on Windows starting at byte 8,200,193 the rest of the hexdump contains 0's.
Code:
int main(void)
{
FILE * fp = fopen("input.exe", "rb");
unsigned char * data = NULL;
long size = 0;
if (fp)
{
fseek(fp, 0, SEEK_END);
size = ftell(fp);
fseek(fp, 0, SEEK_SET);
data = (unsigned char *)malloc(size);
size_t read_bytes = fread(data, 1, size, fp);
// print out read_bytes, value is equal to size
// Hex dump using ofstream. Hexdump file is different here on Windows vs
// on Linux. Last ~3000 bytes are all 0's on Windows.
std::ofstream out("hexdump.bin", std::ios::binary | std::ios::trunc);
out.write(reinterpret_cast<char *>(data), size);
out.close();
FILE * out_file = fopen("hexdump_with_FILE.bin", "wb");
fwrite(data, 1, size, out_file);
fflush(out_file);
fclose(out_file);
}
if (fp) fclose(fp);
if (data) free(data);
return 0;
}
Has anyone seen this behavior before, or have an idea of what might be causing the behavior that I am seeing?
P.S. Everything works as expected when using ifstream and its read function
Thanks!
I wanna read and remove the first line from a txt file (without copying, it's a huge file).
I've read the net but everybody just copies the desired content to a new file. I can't do that.
Below a first attempt. This code will be stucked in a loop as no lines are removed. If the code would remove the first line of file at each opening, the code would reach the end.
#include <iostream>
#include <string>
#include <fstream>
#include <boost/interprocess/sync/file_lock.hpp>
int main() {
std::string line;
std::fstream file;
boost::interprocess::file_lock lock("test.lock");
while (true) {
std::cout << "locking\n";
lock.lock();
file.open("test.txt", std::fstream::in|std::fstream::out);
if (!file.is_open()) {
std::cout << "can't open file\n";
file.close();
lock.unlock();
break;
}
else if (!std::getline(file,line)) {
std::cout << "empty file\n"; //
file.close(); // never
lock.unlock(); // reached
break; //
}
else {
// remove first line
file.close();
lock.unlock();
// do something with line
}
}
}
Here's a solution written in C for Windows.
It will execute and finish on a 700,000 line, 245MB file in no time. (0.14 seconds)
Basically, I memory map the file, so that I can access the contents using the functions used for raw memory access. Once the file has been mapped, I just use the strchr function to find the location of one of the pair of symbols used to denote an EOL in windows (\n and \r) - this tells us how long in bytes the first line is.
From here, I just memcpy from the first byte f the second line back to the start of the memory mapped area (basically, the first byte in the file).
Once this is done, the file is unmapped, the handle to the mem-mapped file is closed and we then use the SetEndOfFile function to reduce the length of the file by the length of the first line. When we close the file, it has shrunk by this length and the first line is gone.
Having the file already in memory since I've just created and written it is obviously altering the execution time somewhat, but the windows caching mechanism is the 'culprit' here - the very same mechanism we're leveraging to make the operation complete very quickly.
The test data is the source of the program duplicated 100,000 times and saved as testInput2.txt (paste it 10 times, select all, copy, paste 10 times - replacing the original 10, for a total of 100 times - repeat until output big enough. I stopped here because more seemed to make Notepad++ a 'bit' unhappy)
Error-checking in this program is virtually non-existent and the input is expected not to be UNICODE, i.e - the input is 1 byte per character.
The EOL sequence is 0x0D, 0x0A (\r, \n)
Code:
#include <stdio.h>
#include <windows.h>
void testFunc(const char inputFilename[] )
{
int lineLength;
HANDLE fileHandle = CreateFile(
inputFilename,
GENERIC_READ | GENERIC_WRITE,
0,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL | FILE_FLAG_WRITE_THROUGH,
NULL
);
if (fileHandle != INVALID_HANDLE_VALUE)
{
printf("File opened okay\n");
DWORD fileSizeHi, fileSizeLo = GetFileSize(fileHandle, &fileSizeHi);
HANDLE memMappedHandle = CreateFileMapping(
fileHandle,
NULL,
PAGE_READWRITE | SEC_COMMIT,
0,
0,
NULL
);
if (memMappedHandle)
{
printf("File mapping success\n");
LPVOID memPtr = MapViewOfFile(
memMappedHandle,
FILE_MAP_ALL_ACCESS,
0,
0,
0
);
if (memPtr != NULL)
{
printf("view of file successfully created");
printf("File size is: 0x%04X%04X\n", fileSizeHi, fileSizeLo);
LPVOID eolPos = strchr((char*)memPtr, '\r'); // windows EOL sequence is \r\n
lineLength = (char*)eolPos-(char*)memPtr;
printf("Length of first line is: %ld\n", lineLength);
memcpy(memPtr, eolPos+2, fileSizeLo-lineLength);
UnmapViewOfFile(memPtr);
}
CloseHandle(memMappedHandle);
}
SetFilePointer(fileHandle, -(lineLength+2), 0, FILE_END);
SetEndOfFile(fileHandle);
CloseHandle(fileHandle);
}
}
int main()
{
const char inputFilename[] = "testInput2.txt";
testFunc(inputFilename);
return 0;
}
What you want to do, indeed, is not easy.
If you open the same file for reading and writing in it without being careful, you will end up reading what you just wrote and the result will not be what you want.
Modifying the file in place is doable: just open it, seek in it, modify and close. However, you want to copy all the content of the file except K bytes at the beginning of the file. It means you will have to iteratively read and write the whole file by chunks of N bytes.
Now once done, K bytes will remain at the end that would need to be removed. I don't think there's a way to do it with streams. You can use ftruncate or truncate functions from unistd.h or use Boost.Interprocess truncate for this.
Here is an example (without any error checking, I let you add it):
#include <iostream>
#include <fstream>
#include <unistd.h>
int main()
{
std::fstream file;
file.open("test.txt", std::fstream::in | std::fstream::out);
// First retrieve size of the file
file.seekg(0, file.end);
std::streampos endPos = file.tellg();
file.seekg(0, file.beg);
// Then retrieve size of the first line (a.k.a bufferSize)
std::string firstLine;
std::getline(file, firstLine);
// We need two streampos: the read one and the write one
std::streampos readPos = firstLine.size() + 1;
std::streampos writePos = 0;
// Read the whole file starting at readPos by chunks of size bufferSize
std::size_t bufferSize = 256;
char buffer[bufferSize];
bool finished = false;
while(!finished)
{
file.seekg(readPos);
if(readPos + static_cast<std::streampos>(bufferSize) >= endPos)
{
bufferSize = endPos - readPos;
finished = true;
}
file.read(buffer, bufferSize);
file.seekg(writePos);
file.write(buffer, bufferSize);
readPos += bufferSize;
writePos += bufferSize;
}
file.close();
// No clean way to truncate streams, use function from unistd.h
truncate("test.txt", writePos);
return 0;
}
I'd really like to be able to provide a cleaner solution for in-place modification of the file, but I'm not sure there's one.
I'm trying to make a exe program that can read any file to binary and later use this binary to make the exact same file.
So I figured out that I can use fopen(content,"rb") to read a file as binary,
and using fwrite I can write block of data into stream. But the problem is when I fwrite it doesn't seems copy everything.
For example the text I opened contains 31231232131 in it. When I write it into another file it only copies 3123 (first 4 bytes).
I can see that it's a very simple thing that I'm missing but I don't know what.
#include <stdio.h>
#include <iostream>
using namespace std;
typedef unsigned char BYTE;
long getFileSize(FILE *file)
{
long lCurPos, lEndPos;
lCurPos = ftell(file);
fseek(file, 0, 2);
lEndPos = ftell(file);
fseek(file, lCurPos, 0);
return lEndPos;
}
int main()
{
//const char *filePath = "C:\\Documents and Settings\\Digital10\\MyDocuments\\Downloads\\123123.txt";
const char *filePath = "C:\\Program Files\\NPKI\\yessign\\User\\008104920100809181000405,OU=HNB,OU=personal4IB,O=yessign,C=kr\\SignCert.der";
BYTE *fileBuf;
FILE *file = NULL;
if ((file = fopen(filePath, "rb")) == NULL)
cout << "Could not open specified file" << endl;
else
cout << "File opened successfully" << endl;
long fileSize = getFileSize(file);
fileBuf = new BYTE[fileSize];
fread(fileBuf, fileSize, 1, file);
FILE* fi = fopen("C:\\Documents and Settings\\Digital10\\My Documents\\Downloads\\gcc.txt","wb");
fwrite(fileBuf,sizeof(fileBuf),1,fi);
cin.get();
delete[]fileBuf;
fclose(file);
fclose(fi);
return 0;
}
fwrite(fileBuf,fileSize,1,fi);
You did read fileSize bytes, but are writing sizeof(...) bytes, that is size of pointer, returned by new.
A C++ way to do it:
#include <fstream>
int main()
{
std::ifstream in("Source.txt");
std::ofstream out("Destination.txt");
out << in.rdbuf();
}
You have swapped the arguments of fread and fwrite. Element size precedes the number of elements. Should be like so:
fread(fileBuf, 1, fileSize, file);
And
fwrite(fileBuf, 1, fileSize, fi);
Also address my comment from above:
Enclose the else clause in { and }. Indentation does not determine blocks in c++. Otherwise your code will crash if you fail to open the file.
EDIT: and the another problem - you have been writing sizeof(fileBuf) bytes which is constant. Instead you should write the exact same number of bytes as you've read. Having in mind the rest of your code you could simply replace sizeof(fileBuf) with fileSize as I've done above.
fileBuf = new BYTE[fileSize];
fread(fileBuf, fileSize, 1, file);
FILE* fi = fopen("C:\\Documents and Settings\\[...]\gcc.txt","wb");
fwrite(fileBuf,sizeof(fileBuf),1,fi);
fileBuf is a pointer to BYTE. You declared it yourself, look: BYTE *fileBuf. And so sizeof(filebuf) is sizeof(BYTE *).
Perhaps you wanted:
fwrite(fileBuf, fileSize, 1, fi);
which closely mirrors the earlier fread call.
I strongly recommend that you capture the return values of I/O functions and check them.
I am having problem when i save a single variable of 460 elements in MATLAB using fwrite and when i try to read in MATLAB its fine but trying to access same bin file using fread in Visual C gives fine result for the first 88 values or so but then it experience EOF or so such as it doesn't give the required result for rest of elements. The code used for Visual C is given as under.
Though this question has been asked in the past post at some other forum as well but the answer doesnot solve the issue.
void main()
{
FILE *p;
long lsize;
float *temp;
int i;
size_t nn;
// Name of file
printf("Open File: r0.bin ");
p = fopen("r01.bin", "r");
// Determine the size of file
fseek (p, 0 , SEEK_END);
lsize = ftell (p);
rewind (p);
// Allocate memory
int a=sizeof(float);
lsize /= a;
temp = (float*) malloc (a*lsize);
// Reading the file
nn= fread(temp,a,lsize,p);
// printing the results
for (i=0;i<lsize;i+=4)
printf("\n %g %g %g %g",temp[i],temp[i+1],temp[i+2],temp[i+3] );
getch();
fclose(p);
}
Windows, right? Files are by default open in text mode, and byte 26 is interpreted as EOF marker. Rewrite your fopen as fopen("r01.bin", "rb") to force opening the file in binary mode.
Are you sure that MATLAB is outputting floats and not doubles? and this code is a bit unnecessary:
// get rid of these 2 statements
// int a=sizeof(float);
// lsize /= a;
temp = (float*) malloc( lsize );
// Reading the file
nn = fread( temp, 1, lsize, p );
I am writing a C library that reads a file into memory. It skips the first 54 bytes of the file (header) and then reads the remainder as data. I use fseek to determine the length of the file, and then use fread to read in the file.
The loop runs once and then ends because the EOF is reached (no errors). At the end, bytesRead = 10624, ftell(stream) = 28726, and the buffer contains 28726 values. I expect fread to read 30,000 bytes and the file position to be 30054 when EOF is reached.
C is not my native language so I suspect I've got a dumb beginner mistake somewhere.
Code is as follows:
const size_t headerLen = 54;
FILE * stream;
errno_t ferrno = fopen_s( &stream, filename.c_str(), "r" );
if(ferrno!=0) {
return -1;
}
fseek( stream, 0L, SEEK_END );
size_t bytesTotal = (size_t)(ftell( stream )) - headerLen; //number of data bytes to read
size_t bytesRead = 0;
BYTE* localBuffer = new BYTE[bytesTotal];
fseek(stream,headerLen,SEEK_SET);
while(!feof(stream) && !ferror(stream)) {
size_t result = fread(localBuffer+bytesRead,sizeof(BYTE),bytesTotal-bytesRead,stream);
bytesRead+=result;
}
Depending on the reference you use, it's quite apparent that adding a "b" to the mode flag is the answer. Seeking nominations for the bonehead-badge. :-)
This reference talks about it in the second paragraph, second sentence (though not in their table).
MSDN doesn't discuss the binary flag until halfway down the page.
OpenGroup mentions the existance of the "b" tag, but states that it "shall have no effect".
perhaps it's a binary mode issue. Try opening the file with "r+b" as the mode.
EDIT: as noted in a comment "rb" is likely a better match to your original intent since "r+b" will open it for read/write and "rb" is read-only.
Also worth noting that simply including binmode.obj into your link command will do this for you for all file opens.
A solution, based on the previous answers:
size_t bytesRead = 0;
BYTE* localBuffer = new BYTE[bytesTotal];
fseek(stream,headerLen,SEEK_SET);
while(!feof(stream) && !ferror(stream)) {
size_t result = fread(localBuffer+bytesRead,sizeof(BYTE),bytesTotal-
bytesRead,stream);
bytesRead+=result;
}