I'm using a third-party library that allows conversion between two file formats A and B. I would like to use this library to load a file of format A and convert it to format B, but I only need the converted representation in memory. So I would like to do the conversion without actually saving a file of the target format to disk and rather obtain an unsigned char* buffer or something similar. Unfortunately the libraries only conversion function is of the form
void saveAsB(A& a, std::FILE *const file);
What can I do? Is there any way to redirect the write operations performed on the handle to some buffer?
If your platform supports it, use open_memstream(3). This will be available on Linux and BSD systems, and it's probably better than fmemopen() for your use case because open_memstream() allocates the output buffer dynamically rather than you having to know the maximum size in advance.
If your platform doesn't have those functions, you can always use a "RAM disk" approach, which again on Linux would be writing a "file" to /dev/shm/ which will never actually reach any disk, but rather be stored in memory.
Edit: OK, so you say you're using Windows. Here's an outline of what you can try:
Open a non-persisted memory-mapped files.
Use _open_osfhandle to convert the HANDLE to an int file descriptor.
Use _fdopen to convert the int file descriptor to FILE*.
Cross your fingers. I haven't tested any of this.
I found this reference useful in putting the pieces together: http://www.codeproject.com/Articles/1044/A-Handy-Guide-To-Handling-Handles
Edit 2: It looks like CreateFileMapping() and _open_osfhandle() may be incompatible with each other--you would be at least the third person to try it:
https://groups.google.com/forum/#!topic/comp.os.ms-windows.programmer.win32/NTGL3h7L1LY
http://www.progtown.com/topic178214-createfilemapping-and-file.html
So, you can try what the last link suggested, which is to use setvbuf() to "trick" the data into flowing to a buffer you control, but even that has potential problems, e.g. it won't work if the library seeks within the FILE*.
So, perhaps you can just write to a file on some temporary/scratch filesystem and be done with it? Or use a platform other than Windows? Or use some "RAM disk" software.
If you can rely on POSIX being available, then use fmemopen().
Related
Can I retrieve information about a file previously opened with fopen() using only the pointer it returned?
The reason I ask is that I am trying to write a RAII-style wrapper class for FILE *s, and I want to make it as general as possible, and one of the functions I imagined for it was a copy-like operation, that would take a FILE * as an argument, and create a new reference to the same file.
Under POSIX, I can create a duplicate of a file descriptor with dup()/dup2(), and even get how the file is being accessed with fnctl()'s F_GETFL operation. However, even if I do that to the underlying descriptor of a FILE *, it isn't enough for guessing properties such as if the stream is text or binary (under POSIX, there no real difference, but I want to be general), or its orientation towards char- or wchar_t-based text.
So, is there is a way of learning about the stream I'm about to create a wrapper for, how far can I go, and how should I do it?
Thank you for you attention.
I working on a constrained embedded system.
Presently we use snprintf to a buffer, then with another statement, print the buffer to the serial port:
char temp_buffer[256];
int bytes_written = snprintf(temp_buffer, sizeof(temp_buffer),
"Solar system has %d planets\n",
10);
if (bytes_written > 0)
{
Serial_Port_Output(temp_buffer, bytes_written);
}
I want to switch to printf to write directly to the serial port. Per our compiler's documentation, I have intercepted the function call for outputting the data to use the serial port. (The interface uses block writing: an address and the number of characters).
The printf function may use a character buffer for formatting, such as integer or floating point to text.
Questions:
Where is the buffer that printf uses for formatting? (Other
inquiring minds want to know, before I make the changes.)
Is this a compiler (platform) dependent issue?
Platform: Arm7tdmi processor, System On a Chip (SOC), IAR EW compiler.
This is completely implementation-specific. printf is under no obligation to use any buffer. Of course it has at its disposal the stdio buffer associated with the FILE (stdout in the case of printf) but that may be zero-length if the program turned off buffering with setbuf/setvbuf. It's also possible that printf has an internal buffer; for a proper C implementation this would need to have automatic storage ("on the stack") but a low-quality embedded one without threads might use a static buffer. In any case, printf is specified to work as if by repeated calls to fputc, and it could certainly be implemented this way without any buffer at all.
It is library rather than compiler dependent and you should consult the library's documentation and possibly where available the source code. This (perhaps out-of-date) IAR C Library documentation says:
Since a complete formatter demands a lot of space there are several
different formatters to choose between. For more information, see the
see the IAR C Compiler Reference Guide.
The current IAR compiler reference discusses formatter selection, though in most cases the linker can automatically select the most appropriate formatter. The documentation even discusses further optimisation available if rebuilding the library (for which you presumably need a source license).
Some implementations (not specifically IAR) use significant stack space. If you want full control, you might consider using an open-source implementation such as Tiny printf. It is not a complete ISO implementation, but suitable for many embedded applications.
Is it possible to delete N bytes from the end of a binary file in C++ using fstream (or something similar)? I don´t want to read the whole file, cut it and write it again, but since it´s from the end of a file it seems like it shouldn't be such a problem.
I'm not aware of a generic C++ (platform independent) way to do this without writing a new file. However, on POSIX systems (Linux, etc.) you can use the ftruncate() function. On Windows, you can use SetEndOfFile().
This also means you'll need to open the file using the native functions instead of fstream since you need the native descriptor/handle for those functions.
EDIT: If you are able to use the Boost library, it has a resize_file() function in its Filesystem library which would do what you want.
Update:
Now in C++17 you can use resize_file from filesystem
Live on Coliru
In case you want to use Qt, QFile also provides two resize() methods that allow to truncate a file.
Is HANDLE similar to file descriptor in Linux? As far as I know, HANDLE is used for handling every resources on Windows, such as font, icons, files, devices..., which in essence is just a void pointer point to a memory block holding data of a specific resource
Yes, Windows handles are very similar to Unix file descriptors (FDs).
Note that a HANDLE is not a pointer to a block of memory. Although HANDLE is typedef'd as void *, that's just to make it more opaque. In practice, a HANDLE is an index that is looked up in a table, just as an FD number is.
This blog post explores some of the similarities and differences:
http://lackingrhoticity.blogspot.com/2015/05/passing-fds-handles-between-processes.html
Yes, they are conceptually similar. File descriptors in unix map integers to a per-process table of pointers to other objects (which can be other things than files, too). File descriptors are not as unified though -- some things exist in a separate "namespace" (e.g., process timers). In that respect, Windows is more orthogonal -- CloseHandle will always free a resource regardless of what it is.
Besides the fact that handles refer to a far broader concept on Windows. Even we restrict the discussion to only file handles, there is significant differences. There is a function called _open_osfhandle() as part of C run-time library on Windows. Its purpose is to, quote "Associates a C run-time file descriptor with an existing operating-system file handle." That is, a glue function between the kernel land and the C Run-time land. The function signature is as below:
int _open_osfhandle (
intptr_t osfhandle,
int flags
);
File handles Windows is actually more feature rich than file descriptors in C, which can be configured when a file handle is created with CreateFileA (ANSI version) or CreateFile (UTF16 version), reflecting the design difference between *Nix and Windows. And the resulted handle carries all these information around with all its implications.
A HANDLE is a void pointer
typedef PVOID HANDLE;
typedef void *PVOID;
Windows Data Types
In C++, on Linux, how can I write a function to return a temporary filename that I can then open for writing?
The filename should be as unique as possible, so that another process using the same function won't get the same name.
Use one of the standard library "mktemp" functions: mktemp/mkstemp/mkstemps/mkdtemp.
Edit: plain mktemp can be insecure - mkstemp is preferred.
tmpnam(), or anything that gives you a name is going to be vulnerable to race conditions. Use something designed for this purpose that returns a handle, such as tmpfile():
#include <stdio.h>
FILE *tmpfile(void);
The GNU libc manual discusses the various options available and their caveats:
http://www.gnu.org/s/libc/manual/html_node/Temporary-Files.html
Long story short, only mkstemp() or tmpfile() should be used, as others have mentioned.
man tmpfile
The tmpfile() function opens a unique temporary file in binary
read/write (w+b) mode. The file will be automatically deleted when it
is closed or the program terminates.ote
mktemp should work or else get one of the plenty of available libraries to generate a UUID.
The tmpnam() function in the C standard library is designed to solve just this problem. There's also tmpfile(), which returns an open file handle (and automatically deletes it when you close it).
You should simply check if the file you're trying to write to already exists.
This is a locking problem.
Files also have owners so if you're doing it right the wrong process will not be able to write to it.