I ran across the following code in one of our in-house dlls and I am trying to understand the behavior it was showing:
long GetFD(long* fd, const char* fileName, const char* mode)
{
string fileMode;
if (strlen(mode) == 0 || tolower(mode[0]) == 'w' || tolower(mode[0]) == 'o')
fileMode = string("w");
else if (tolower(mode[0]) == 'a')
fileMode = string("a");
else if (tolower(mode[0]) == 'r')
fileMode = string("r");
else
return -1;
FILE* ofp;
ofp = fopen(fileName, fileMode.c_str());
if (! ofp)
return -1;
*fd = (long)_fileno(ofp);
if (*fd < 0)
return -1;
return 0;
}
long CloseFD(long fd)
{
close((int)fd);
return 0;
}
After repeated calling of GetFD with the appropriate CloseFD, the whole dll would no longer be able to do any file IO. I wrote a tester program and found that I could GetFD 509 times, but the 510th time would error.
Using Process Explorer, the number of Handles did not increase.
So it seems that the dll is reaching the limit for the number of open files; setting _setmaxstdio(2048) does increase the amount of times we can call GetFD. Obviously, the close() is working quite right.
After a bit of searching, I replaced the fopen() call with:
long GetFD(long* fd, const char* fileName, const char* mode)
{
*fd = (long)open(fileName, 2);
if (*fd < 0)
return -1;
return 0;
}
Now, repeatedly calling GetFD/CloseFD works.
What is going on here?
If you open a file with fopen, you have to close it with fclose, symmetrically.
The C++ runtime must be given a chance to clean up/deallocate its inner file-related structures.
You need to use fclose with files opened via fopen, or close with files opened via open.
The standard library you are using has a static array of FILE structures. Because you are not calling fclose(), the standard library doesn't know that the underlying files have been closed, so it doesn't know it can reuse the corresponding FILE structures. You get an error after it has run out of entries in the FILE array.
fopen opens it's own file descriptor, so you'd need to do an fclose(ofp) in your original function to prevent running out of file descriptors. Usually, one either uses the lower level file descriptor functions open, close OR the buffered fopen, fclose functions.
you are open the file fopen() function so u have to close the file useing fclose(), if you are using open() function and try to call fclose() function it will not work
Related
I am mapping a huge file to avoid my app thrashing to main virtual memory, and to be able to run the app with more than the RAM I have. The code is c++ but partly follows old c APIs. When I work with the allocated pointer, the memory does get backed to the file as desired. However, when I run the app next time, I want the memory to be read from this same file which already has the prepared data. For some reason, on the next run, I read back all zeros. What am I doing wrong? Is it the ftruncate call? Is it the fopen call with wrong flag? Is it the mmap flags?
int64_t mmbytes=1<<36;
FILE *file = fopen(filename, "w+");
int fd = fileno(file);
int r = ftruncate(fd, mmbytes );
if (file == NULL || r){
perror("Failed: ");
throw std::runtime_error(std::strerror(errno));
} //
if ((mm = mmap(0, mmbytes,
PROT_READ | PROT_WRITE, MAP_FILE | MAP_SHARED, fd, 0)) == MAP_FAILED)
{
fprintf(stderr,"mmap error for output, errno %d\n", errno);
exit(-1);
}
}
FILE *file = fopen(filename, "w+");
I refer you to fopen's manual page, which describes "w+" as follows:
w+ Open for reading and writing. The file is created if it does
not exist, otherwise it is truncated. The stream is positioned
at the beginning of the file.
I specifically draw your attention to the "it is truncated" part. In other words, if there's anything in an existing file this ends up nuking it from high orbit.
Depending on what else you're doing "a" will work better.
Even better would be to forget fopen entirely, and simply use open:
int fd=open(filename, O_RDWR|O_CREAT, 0666);
There's your file descriptor, without jumping through any hoops. The file gets created, and left untouched if it already exists.
I am working with mmap() to fastly read big files, basing my script on this question answer (Fast textfile reading in c++).
I am using the second version from sehe answer :
#include <algorithm>
#include <iostream>
#include <cstring>
// for mmap:
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
const char* map_file(const char* fname, size_t& length);
int main()
{
size_t length;
auto f = map_file("test.cpp", length);
auto l = f + length;
uintmax_t m_numLines = 0;
while (f && f!=l)
if ((f = static_cast<const char*>(memchr(f, n, l-f))))
m_numLines++, f++;
std::cout << "m_numLines = " << m_numLines << "n";
}
void handle_error(const char* msg) {
perror(msg);
exit(255);
}
const char* map_file(const char* fname, size_t& length)
{
int fd = open(fname, O_RDONLY);
if (fd == -1)
handle_error("open");
// obtain file size
struct stat sb;
if (fstat(fd, &sb) == -1)
handle_error("fstat");
length = sb.st_size;
const char* addr = static_cast<const char*>(mmap(NULL, length, PROT_READ, MAP_PRIVATE, fd, 0u));
if (addr == MAP_FAILED)
handle_error("mmap");
// TODO close fd at some point in time, call munmap(...)
return addr;
}
and it works just great.
But if I implement it over a loop of several files (I just change the main() function name to:
void readFile(std::string &nomeFile) {
and then get the file content in "f" object in main() function with:
size_t length;
auto f = map_file(nomeFile.c_str(), length);
auto l = f + length;
and call it from main() on a loop over a filenames list), after a while I got:
open: Too many open files
I imagine there would be a way to close the open() call after working on a file, but I can not figure out how and where to put it exactly. I tried:
int fc = close(fd);
at the end of the readFile() function but it did change nothing.
Thanks a lot in advance for any help!
EDIT:
after the important suggestions I received I made some performance comparison with different approaches with mmap() and std::cin(), check out: fast file reading in C++, comparison of different strategies with mmap() and std::cin() results interpretation for the results
Limit to the number of concurrently open files
As you can imagine, keeping a file open consumes resources. So there is in any case a practical limit to the number of open file descriptors on your system. This is why it's highly recommended to close files that you no longer need.
The exact limit depends on the OS and the configuration. If you want to know more, there are already a lot of answers available for this kind of question.
Special case of mmap
Obviously, with mmap() you open a file. And doing so repetitively in a loop risk to reach sooner or later the fatal file description limit, as you could experience.
The idea of trying to close the file is not bad. The problem is that it does not work. This is specified in the POSIX documentation:
The mmap() function adds an extra reference to the file associated
with the file descriptor fildes which is not removed by a subsequent
close() on that file descriptor. This reference is removed when there
are no more mappings to the file.
Why ? Because mmap() links the file in a special way to the virtual memory management in your system. And this file will be needed as long as you use the address range to which it was allocated.
So how to remove those mappings ? The answer is to use munmap():
The function munmap() removes any mappings for those entire pages
containing any part of the address space of the process starting at
addr and continuing for len bytes.
And of course, close() the file descriptor that you no longer need. A prudent approach would be to close after munmap(), but in principle, at least on a POSIX compliant system, it should not matter when you're closing. Nevertheless, check your latest OS documentation to be on the safe side :-)
*Note: file mapping is also available on windows; the documentation about closing the handles is ambiguous on potential memory leaks if there are remaining mappings. This is why I recommend prudence on the closing moment. *
RESOLVED
I'm trying to make a simple file loader.
I aim to get the text from a shader file (plain text file) into a char* that I will compile later.
I've tried this function:
char* load_shader(char* pURL)
{
FILE *shaderFile;
char* pShader;
// File opening
fopen_s( &shaderFile, pURL, "r" );
if ( shaderFile == NULL )
return "FILE_ER";
// File size
fseek (shaderFile , 0 , SEEK_END);
int lSize = ftell (shaderFile);
rewind (shaderFile);
// Allocating size to store the content
pShader = (char*) malloc (sizeof(char) * lSize);
if (pShader == NULL)
{
fputs ("Memory error", stderr);
return "MEM_ER";
}
// copy the file into the buffer:
int result = fread (pShader, sizeof(char), lSize, shaderFile);
if (result != lSize)
{
// size of file 106/113
cout << "size of file " << result << "/" << lSize << endl;
fputs ("Reading error", stderr);
return "READ_ER";
}
// Terminate
fclose (shaderFile);
return 0;
}
But as you can see in the code I have a strange size difference at the end of the process which makes my function crash.
I must say I'm quite a beginner in C so I might have missed some subtilities regarding the memory allocation, types, pointers...
How can I solve this size issue?
*EDIT 1:
First, I shouldn't return 0 at the end but pShader; that seemed to be what crashed the program.
Then, I change the type of reult to size_t, and added a end character to pShader, adding pShdaer[result] = '/0'; after its declaration so I can display it correctly.
Finally, as #JamesKanze suggested, I turned fopen_s into fopen as the previous was not usefull in my case.
First, for this sort of raw access, you're probably better off
using the system level functions: CreateFile or open,
ReadFile or read and CloseHandle or close, with
GetFileSize or stat to get the size. Using FILE* or
std::filebuf will only introduce an additional level of
buffering and processing, for no gain in your case.
As to what you are seeing: there is no guarantee that an ftell
will return anything exploitable as a numeric value; it could
very well be just a magic cookie. On most current systems, it
is a byte offset into the physical file, but on any non-Unix
system, the offset into the physical file will not map directly
to the logical file you are reading unless you open the file in
binary mode. If you use "rb" to open the file, you'll
probably see the same values. (Theoretically, you could get
extra 0's at the end of the file, but practically, the OS's
where that happened are either extinct, or only used on legacy
mainframes.)
EDIT:
Since the answer stating this has been deleted: you should loop
on the fread until it returns 0 (setting errno to 0 before
each call, and checking it after the return to see whether the
function returned because of an error or because it reached the
end of file). Having said this: if you're on one of the usual
Windows or Unix systems, and the file is local to the machine,
and not too big, fread will read it all in one go. The
difference in size you are seeing (given the numerical values
you posted) is almost certainly due to the fact that the two
byte Windows line endings are being mapped to a single '\n'
character. To avoid this, you must open in binary mode;
alternatively, if you really are dealing with text (and want
this mapping), you can just ignore the extra bytes in your
buffer, setting the '\0' terminator after the last byte
actually read.
bool CReadWrite::write(unsigned long long offset, void* pvsrc, unsigned long long nbytes)
{ int WriteResult;
pFile = fopen("D:\\myfile.bin","wb");
if (!pFile){
puts("Can't open file");
return false;
}
//offset = fseek(pFile,offset,
WriteResult = fwrite (pvsrc, 1, nbytes, pFile);
if (WriteResult == nbytes){
puts("Wrote to file");
fclose(pFile);
return true;
}
else{
puts("Unable to write to File.");
fclose(pFile);
return false;
}
}
This is my class function so far. I'm basically opening a file, checking to see if it did indeed open if not get out. Writes the file, checks to see if the file see if the file was indeed written to returns true. else return false. As you can tell by my parameters, I'm looking to create an offset where I can give a particular offset i.e. 10, and start from 10 and then from there write. I know for sure I need to use fseek but I can't assume that I'm at the beginning of the file or anywhere in the file. Im pretty sure i need to use SEEK_SET but I may be wrong. Any thoughts on implemented the above desires? Thanks.
If you're using fopen without the append setting (as you are, "wb" creates an empty file), you can assume you're at the beginning.
Regardless, SEEK_SET sets the position to the given offset from the beginning.
If the file doesn't have the offset that you want to seek to (as it is in your case), then the question is what are you required to do? If just pad - then write offset padding bytes, and then your content, otherwise maybe you wanted to use "a" and not "w". "w" truncates the existing content of the file, while "a" opens for append and sets position to the end of the existing content.
More details here.
I'm using code like the following to check whether a file has been created before continuing, thing is the file is showing up in the file browser much before it is being detected by stat... is there a problem with doing this?
//... do something
struct stat buf;
while(stat("myfile.txt", &buf))
sleep(1);
//... do something else
alternatively is there a better way to check whether a file exists?
Using inotify, you can arrange for the kernel to notify you when a change to the file system (such as a file creation) takes place. This may well be what your file browser is using to know about the file so quickly.
The "stat" system call is collecting different information about the file, such as, for example, a number of hard links pointing to it or its "inode" number. You might want to look at the "access" system call which you can use to perform existence check only by specifying "F_OK" flag in "mode".
There is, however, a little problem with your code. It puts the process to sleep for a second every time it checks for file which doesn't exist yet. To avoid that, you have to use inotify API, as suggested by Jerry Coffin, in order to get notified by kernel when file you are waiting for is there. Keep in mind that inotify does not notify you if file is already there, so in fact you need to use both "access" and "inotify" to avoid a race condition when you started watching for a file just after it was created.
There is no better or faster way to check if file exists. If your file browser still shows the file slightly faster than this program detects it, then Greg Hewgill's idea about renaming is probably taking place.
Here is a C++ code example that sets up an inotify watch, checks if file already exists and waits for it otherwise:
#include <cstdio>
#include <cstring>
#include <string>
#include <unistd.h>
#include <sys/inotify.h>
int
main ()
{
const std::string directory = "/tmp";
const std::string filename = "test.txt";
const std::string fullpath = directory + "/" + filename;
int fd = inotify_init ();
int watch = inotify_add_watch (fd, directory.c_str (),
IN_MODIFY | IN_CREATE | IN_MOVED_TO);
if (access (fullpath.c_str (), F_OK) == 0)
{
printf ("File %s exists.\n", fullpath.c_str ());
return 0;
}
char buf [1024 * (sizeof (inotify_event) + 16)];
ssize_t length;
bool isCreated = false;
while (!isCreated)
{
length = read (fd, buf, sizeof (buf));
if (length < 0)
break;
inotify_event *event;
for (size_t i = 0; i < static_cast<size_t> (length);
i += sizeof (inotify_event) + event->len)
{
event = reinterpret_cast<inotify_event *> (&buf[i]);
if (event->len > 0 && filename == event->name)
{
printf ("The file %s was created.\n", event->name);
isCreated = true;
break;
}
}
}
inotify_rm_watch (fd, watch);
close (fd);
}
your code will check if the file is there every second. you can use inotify to get an event instead.