I have a C++ program which opens files in /tmp (on a *nix system) and reads their contents.
To do this, I am using:
ofstream dest;
dest.open(abs_path.c_str(), ios::app);
where abs_path is a string containing the absolute path to the file.
The problem is that some *nix programs create named pipes as files in /tmp. For example,
/tmp/vgdb-pipe-to-vgdb-from-23732-by-myusername-on-???
Is a pipe created by a debugging utility I am using.
In the documentation for ofstream, the open method it says that the method sets an error bit when opening the file fails. However, in my tests it instead hangs trying to open the file (which is actually a pipe) indefinitely. I assume this is because the file is locked by another program (probably the debugger).
So, how can I force ofstream::open to block for a finite amount of time, or not at all? It's easy enough to clean up gracefully if it fails, but it needs to actually fail first..
The simple answer is that you can't. filebuf::open (called by
ofstream) basically delegates to the OS, and supposed that the OS will
do the right thing. And the interface it supports is very, very
limited; many important options to open (O_SYNC, O_NONBLOCK, etc)
aren't mapped, and thus can't be used. The only solutions I've found to
this is either to use std::ostringstream, then write the string to the
file using system level calls, or to write my own streambuf, which
does what I want (much simpler than it sounds, since you typically only
need part of what filebuf offers—you often don't need
bidirectionality, seeking or code translation).
Neither of these solutions are portable, of course.
Finally, I'm not sure why you're writing into /tmp. By convention,
anything you put into /tmp should contain the process id. And for
security reasons, I'd always create a subdirectory, with the process id
in its name, and with very limited access rights, and create any
temporary files in it.
AFAIK, there is no such thing as non-blocking input defined by the C++ language. (There is a method std::streambuf::in_avail(), but still it can't help you)
You can consider using C method
int file_descr = open( "pipe_addr", O_RDONLY |O_NONBLOCK);
instead of std::ofstream
Related
When I open a file in C, I get a file descriptor, if I had not read it's contents, and then someone modifies the file, will I read the old file or the new file?
Let's say a file has lots of lines, what happens that while reading the file, someone edits the beginning, will this somehow corrupt how my file reads the file?
How do programs don't get corrupted while the file is being read? Is the OS that takes care of this problem? If I can still read the old data, where is this data being stored?
The man page of open, has some information about the internals of open, but it is not very clear to me.
The C language standard doesn't acknowledge the existence of other processes nor specify interaction between them and the program (nor does C++). The behaviour depends on the operating system and / or the file system.
Generally, it is safest to assume that file operations are not atomic and therefore accessing a file while another process is editing it would be an example of race condition. Some systems may provide some stricter guarantees.
A general approach to attempt avoiding problems is file locking. The standard C library does not have an API for file locking, but multitasking operating systems generally do.
All this depends heavily on the OS, not at the C++ level. In Windows, for example, opening the file with CreateFile allows you to lock the file for subsequent access. But not at the language level.
You must decide based on the specific OS you work with. There are no assumptions; it all depends on the documentation you are provided with.
Generally, C++ level documentation is not much useful at such problems because there can never be a full standard for something so low level as file access (even the fs was only recently added to C++) and there is no point creating 'portable' code on such. You must make it a habit to immerse in the OS specific documentation and libraries.
We're writing a C++/Objective C app, runnable on OSX from versions 10.7 to present (10.11).
Under windows, there is the concept of a shadow file, which allows you read a file as it exists at a certain point in time, without having to worry about other processes writing to that file in the interim.
However, I can't find any documentation or online articles discussing a similar feature in OS X. I know that OS X will not lock a file when it's being written to, so is it necessary to do something special to make sure I don't pick up a file that is in the middle of being modified?
Or does the Journaled Filesystem make any special handling unnecessary? I'm concerned that if I have one process that is creating or modifying files (within a single context of, say, an fopen call - obviously I can't be guaranteed of "completeness" if the writing process is opening and closing a file repeatedly during what should be an atomic operation), that a reading process will end up getting a "half-baked" file.
And if JFS does guarantee that readers only see "whole" files, does this extend to Fat32 volumes that may be mounted as external drives?
A few things:
On Unix, once you open a file, if it is replaced (as opposed to modified), your file descriptor continues to access the file you opened, not its replacement.
Many apps will replace rather than modify files, using things like -[NSData writeToFile:atomically:] with YES for atomically:.
Cocoa and the other high-level frameworks do, in fact, lock files when they write to them, but that locking is advisory not mandatory, so other programs also have to opt in to the advisory locking system to be affected by that.
The modern approach is File Coordination. Again, this is a voluntary system that apps have to opt in to.
There is no feature quite like what you described on Windows. If the standard approaches aren't sufficient for your needs, you'll have to build something custom. For example, you could make a copy of the file that you're interested in and, after your copy is complete, compare it to the original to see if it was being modified as you were copying it. If the original has changed, you'll have to start over with a fresh copy operation (or give up). You can use File Coordination to at least minimize the possibility of contention from cooperating programs.
I am using MPI together with C++. I want to read information from one file, modify it by some rule, and then write modified content in the same file. I am using temporary file which where I store modified content and at the end I overwrite it by these commands:
temp_file.open("temporary.txt",ios::in);
ofstream output_file(output_name,ios::out);
output_file<<temp_file.rdbuf();
output_file.flush();
temp_file.close();
output_file.close();
remove("temporary.txt");
This function which modify the file is executed by MPI process with rank 0. After exiting from function, MPI_Barrier(MPI_COMM_WORLD); is called to ensure synchronization.
And then, all MPI processes should read modified file and perform some computations. The problem is that, since file is too big, data are not completely written to file when execution of function is finished, and I get wrong results. I also tried to put sleep() command, but sometimes it works, sometimes it doesn't (it depends on the node where I perform computations). Is there general way to solve this problem?
I put MPI as a tag, but I think this problm is inherently connected with c++ standard and manipulating with storage. How to deal with this latency between writing in buffer aand writing in file on storage medium?
Fun topic. You are dealing with two or maybe three consistency semantics here.
POSIX consistency says essentially when a byte is written to a file, it's visible.
NFS consistency says "woah, that's way too hard. you write to this file and I'll make it visible whenever I feel like it. "
MPI-IO consistency semantics (which you aren't using, but are good to know) say that data is visible after specific synchronization events occur. Those two events are "close a file and reopen it" or "sync file, barrier, sync file again".
If you are using NFS, give up now. NFS is horrible. There are a lot of good parallel file systems you can use, several of which you can set up entirely in userspace (such as PVFS).
If you use MPI-IO here, you'll get more well-defined behavior, but the MPI-IO routines are more like C system calls than C++ iostream operators, so think more like open(2) read(2) write(2) and close(2). Text files are usually a headache to deal with but in your case where modifications are appended to file, that shouldn't be too bad.
I'm using an ofstream to write data to a file. I regularly call flush on the file but the backing file doesn't always get updated at that time. I assume this is related to an OS-level cache, or something inside the MSVC libraries.
I need a way to have the data properly flush at that point. Preferably written to disc, but at least enough such that a copy operation from another program would see all data up to the flush point.
What API can I use to do this?
FlushFileBuffers will flush the Windows write file cache and write it to a file. Be aware it can be very slow if called repeatedly.
I also found this KB article which describes the use of _commit(). This might be more useful to you since you are using ofstream.
CXXFileBuf.flush();
_commit(CXXFileBuf.rdbuf()->fd());
I used:
MyOfstreamObject.rdbuf()->pubsync();
I'm using stl_port on Win 7 with ICC 9.1.
I have not tested the solution extensively but it seems to work... Maybe it could solve the problem of the absence of fd() noticed by edA-qa mort-ora-y .
Just add commode.obj to Linker->Input->Additional Dependencies in the project's Property Pages in Visual Studio and call std::ostream::flush(). That way std::ostream's flush will link against another method which has the desired behavior. That's what helped to me.
If this is a windows-only solution, you might want to use FlushFileBuffers(). This means you will have to re-write some of your code to accomodate calls to CreateFile(), WriteFile(), etc. If your application depends on many different operator<< functions, you can write your own std::streambuf.
You also might want to read the remarks section carefully. In particular,
Due to disk caching interactions within the system, the FlushFileBuffers function can be inefficient when used after every write to a disk drive device when many writes are being performed separately. If an application is performing multiple writes to disk and also needs to ensure critical data is written to persistent media, the application should use unbuffered I/O instead of frequently calling FlushFileBuffers.
For C++/linux programs, how does writing to cout (when cout has been redirected to a file during program launch) compare against writing to the target file directly? (via say fstream)
Does the system do the appropriate magic at the start of the program to make these two cases exactly equivalent or is the later gonig to be better than the first?
Thanks!
They are basically equivalent. In both cases, the underlying stream buffer will end up calling the write() system call, for the same effect.
Note however that by default, cout is synchronized to stdio, for backwards compatibility (so you can use C-style standard output as well as cout, and have it work as expected). This additional synchronization can slow down C++ output. If this is important, then you can use std::ios_base::sync_with_stdio(false) to unlink them. Then, a file-redirected cout and an fstream should have essentially identical performance and function.
The former is better for the phylosophy of UNIX tools, that is feeding a program with the output of another.
Let's say your programs prints numbers and you need to sort them. You feed the sort tool with the output of your commands and then write the result to a file, always with output redirection.
On the contrary, if you wrote directly to a file you couldn't to that.
Of course, if you don't plan your application to do this sort of things, you can write directly to a file. But if I were in you, I'll let the user decide. Maybe with a command line argument.
I don't think the latter is necessiarily "better" or "worse". It certainly requires far less code when you simply redirect cout/stdout from the shell. It allows for simple text output (via printf/fprintf/cout).
I prefer using simple cout calls for quick and dirty logging and "printf" style debugging.
In my experience, I use syslog for things that absolutely have to be logged. For example, error cases where a file fails to open or you run out of resources or whatever.
I use printf/fprintf for the other "simple" logging tasks.
Some years ago, I developed a simple debugging system that now I just plug into my new Linux applications. I can then just call the appropriate functions in that code. It's similar to syslog in that it has debug "levels". For example, level 1 always writes to stdout, level 2 writes to stderr, level 4 writes to syslog, level 5 may create a new file and write messages to that, etc.
Yes. If the spawning process has arranged that file descriptor 1 (standard output) is the result of an open() call on a disk file, then I/O from the child process to that descriptor is exactly equivalent to the same I/O done to a file it had opened manually. The same is true for processes spawned with a pipe or socket (i.e. equivalent behavior to having opened their own pipe or socket). In fact, the kernel can't even tell the difference. All that matters is how the file descriptor was created (i.e. using what syscall, and on what filesystem), not which process did it.