How to treat RAM data as if it was a real file? - c++

So I have some temp data in my program (in RAM). I want to somehow make it seem as it is a file (for example for sending it into another program which takes a file link as argument)?
Is it possible?
How to do such thing?

Why not simply write the file to disk? If writing to disk is too slow, you can pass the FILE_ATTRIBUTE_TEMPORARY flag to CreateFile to keep the data in cache (and avoid writing it to the physical device).
Sometimes the obvious solutions are the best...

If supported by your operating system (Unixoid systems and Windows do), you could try to use memory-mapped files.

You can do it in C using the popen() function:
FILE *f = popen("program args", "w");
// write your output to f here using stdio
pclose(f);
This is possible if your external program reads its input from stdin.

You could use pipe()
The pipe() function shall create a pipe and place two file descriptors,
one each into the arguments fildes[0] and fildes[1], that refer to the
open file descriptions for the read and write ends of the pipe. Their
integer values shall be the two lowest available at the time of the
pipe() call. The O_NONBLOCK and FD_CLOEXEC flags shall be clear on both
file descriptors. (The fcntl() function can be used to set both these
flags.)

Yes, it is possible. You can transfer your data to your other application via an interprocess communication mechanism:
Depending on your OS, you have different options here. You could create a pipe, as other posters have mentioned here, as many OSes have pipes.
You could also use shared memory.
You could simply write it out to a file, and then open up that file in your other application.
Many OSes have other techniques you can use.
EDIT: MSDN lists all the IPC mechanisms available for Windows here.

Related

Can you open the same file multiple times for writing?

We are writing a multi threaded application that does a bunch of bit twiddling and writes the binary data to disk. Is it possible to have each thread std::fopen the same file for writing at the same time? The reasoning would be each thread could do its work and have its own access to the writable file.
std::fstream has functionality defined in terms of the C stdio library. I would be surprised if it were actually specified, but the most likely behavior from opening the same file twice is multiple internal buffers bound to the same file descriptor.
The usual way to simultaneously write to multiple points in the same file is POSIX pwrite or writev. This functionality is not wrapped by C stdio, and by extension not by C++ iostreams either. But, having multiple descriptors to the same filesystem file might work too.
Edit: POSIX open called twice on the same file in Mac OS X produces different file descriptors. So, it might work on your platform, but it's probably not portable.
A definitive answer would require connecting these dots:
Where the C++ standard specifies that fstream works like a C (stdio) stream.
Where the C standard defines when a stream is created (fopen is only defined to associate a stream with a newly-opened file).
Where the POSIX standard defines its requirements for C streams.
Where POSIX defines the case of opening the same file twice.
This is a bit more research than I'm up for at the moment, but I'm sure someone out there has done the legwork.
I've written some high speed multi-threaded data capture utilities, but the output went to separate files on separate hard drives, and then were post-processed.
I seem to recall that you can have fopen not lock the file so in theory that would allow different threads to all write to the same file with independent handles. In practice you're going to run into other issues, namely concurrency. Your threads are almost certainly going to step all over each other and scramble the results unless you implement some synchronization. And if you have to do that, why not just use one handle across all the threads?
I/O access is not a parallelizable task (it can't be, you simply can't send two or more data addresses over the device bus at the same time) so you'd better implement a queue in which many threads posts their chunks of data and one single consumer actually writes them to disk.

Why use MPI_File_open instead of fopen?

After reading the MPI documentation, it doesn't sound like this gives you any additional functionality at all. I had assumed that it coordinated network traffic such that all file operations work with the given file on the executing system (the one issuing an mpirun command), as opposed to using the local filesystem on each individual host. This would be useful. Instead, the "user" needs to ensure that they all end up at the same file. Clearly they're not communicating that much about this file... are they?
What does MPI_File_open actually do, and how is it beneficial? Why should I not just use fopen?
Sure, MPI_File_open allows you to seek and read/write at particular blocks, like you would with fopen, in which case each process has a private file pointer. Differences from fopen include the nonblocking IO methods would allow your program to continue execution without waiting for the operation to complete. MPI also supports shared file pointers (e.g. MPI_File_read_shared), although obviously use of shared pointers have a synchronization overhead.

Safe access to file from two processes

Suppose I have two processes. One always resides in memory and periodically reads some settings from a file on a disk. If it detects that settings was changed then it applies them.
The other process runs under command line by demand and modifies the settings. Thus the first process only read the file and never write to it while the second can only write to the file.
Should I synchronize the access to the file to ensure that the first process will always get consistent settings i.e. before or after modifications not some intermediate contents? If yes, what is the simplest way to do this in C++.
I'm interested mainly in cross-platform ways. But also curious about Windows- and/or Linux-specific ones.
Use a named semaphore and require either process to hold the semaphore before editing the file on disk. Named semaphores can be connected to by any running application.
Look at man 7 sem_overview for more information on named semaphores on linux machines.
The closest equivalent for windows I can find is http://msdn.microsoft.com/en-us/library/windows/desktop/ms682438(v=vs.85).aspx
You are using C++ so your first option should be to check through the usual cross-platform libs - POCO, Boost, ACE, and so forth to see if there is anything that already does what you require.
You really have two separate issues: (1) file synchronization and (2) notification.
On Linux to avoid having your daemon constantly polling to see if a file has changed you can use inotify calls and set up events that will tell you when the file has been changed by the command line program. It might be simplest to look for IN_CLOSE_WRITE events since a CL prog will presumably be opening, changing, and closing the file.
For synchronization, since you are in control of both programs, you can just use file or record locking e.g. lockf, flock or fcntl.
The most obvious solution is to open the file in exclusive mode. If the file can not be opened, wait some time and try to open the file again. This will prevent possible access/modification conflicts.
The benefit of this approach is that it's simple and doesn't have significant drawbacks.
Of course you could use some synchronization primitives (Mutex, Semaphore depending on the OS) but this would be an overkill in your scenario, when speedy response is not required (waiting 200 msec between open attempts is fine, and writing of config file won't take more).

Obtaining a FILE * handle without actually creating a file on disk

I need to process some data with a legacy library that I can't modify. My problem is that it requires a plain old FILE handle in order to save its results, and I'm required not to write anything on disk at all.
I understood that there's no standard way to do that, but is it possible using windows API, boost or anything else, to obtain a file handle somewhat pointing to memory ?
I found nowhere a solution for which it's explicitly guarantee that no disc access is (systematically) performed.
I believe you can fopen a pipe, using the pipe syntax:
fopen("\\\\.\\pipe\\WritePipe", "w+");
You need to create the pipe using CreateNamedPipe, beforehand, but once you've done that you should be able to use the pipe for processing the data.
You'll probably have to create a thread to read from the pipe to ensure that your app will not hang, but it should work for your needs (not being able to touch the file system)
Try with
fmemopen
From How to write to a memory buffer with a FILE*?
tbert's answer:
For anyone else who stumbles upon this thread looking for a correct
answer: yes, there is a standards-compliant way to use memory as a
FILE descriptor: fmemopen or open_memstream, depending on the
semantics you want.
http://pubs.opengroup.org/onlinepubs/9699919799/functions/fmemopen.html
http://pubs.opengroup.org/onlinepubs/9699919799/functions/open_memstream.html

Libevent and file I/O

Does the libevent deal with buffered file I/O? I know it handles sockets pretty good, but does it concern also normal files or it's "only" an epoll/... wrapper?
Using libevent (or any of the underlying readiness notification mechanisms such as e.g. epoll or kqueue) with normal file descriptors does not normally make sense. Exceptions are files on NFS or using kernel AIO with an eventfd.
File descriptors on local disks are always ready, there is always sufficient buffer space, and operations always complete "immediately". The write operation merely copies data to the buffer cache, and the actual write to the disk happens ... whenever it happens. (note that this link is Linux-specific, but apart from maybe some little implementation details it works the same on other systems)
libevent is not an epoll wrapper. It selects the highest performance method available on each platform.
Sockets are also file descriptors so you can should be able to use libevent for file io.
You will need to disable epoll usage of libevent though. If I remember right Epoll does not support unix files.
struct event_config *cfg = event_config_new();
event_config_avoid_method(cfg, "epoll");
libevent sits at a lower level than buffered file I/O (what you get with stdio.h), using file descriptors directly. You are correct thinking that it is 'just' an epoll/select/kevent/etc wrapper. Its purpose is to listen for events on descriptors, which is the lowest level of file I/O. However, you can use it in conjunction with the stdio.h file I/O library facilities, as that also eventually uses the file descriptors. You can use fileno(3) to retrieve the file descriptor from the FILE * you want to wait on.