I currently have some code reading files which are not compressed, it uses the following approach to read a file in C++
FILE* id = fopen("myfile.dat", "r");
after obtaining id, different parts of the code access the file using fread, fseek, etc.
I would like to adapt my code so as to open a gzip version of the file, e.g. "myfile.dat.gz" without needing to change too much.
Ideally I would implement a wrapper to fopen, call it fopen2, which can read both myfile.dat and myfile.dat.gz, i.e. it should return a pointer to a FILE object, so that the remaining of the code does not need to be changed.
Any suggestions?
Thank you.
PS: it would be fine to decompress the whole file in memory, if this approach provides a solution
zlib provides analogs of fopen(), fread(), etc. called gzopen(), gzread(), etc. for reading and writing gzip files. If the file is not gzip-compressed, it will be read just as the f functions would. So you would only need to change the function names and link in zlib.
Related
(My previous questions was closed as a duplicate of Are there binary memory streams in C++ which is ridiculous, since i can't change the implementation of the library I'm using)
I'm using a library (Poco) to create zip files. It takes ostream as an input and writes the data of the zip file into it. Something like:
std::ofstream ofs("file.zip", std::ios::binary);
Compress compress(ofs);
// add data to compress ...
compress.close();
// now file.zip contains added file
This works. But I want to be able to create a zip in memory without creating a file. I tried using stringstream instead of ofstream, i get additionl newline characters in the data in the zip file is corrupted. Is there any other stream i can use?
(If someone still thinks it's a duplicate, I'm gonna need an explanation, since I don't see how this other question is helpful for me)
Use a std::stringstream -- that will create an in-memory string that you can write to as an ostream, and WILL NOT add extra newlines. If you later copy the string to an fstream that was opened in text mode (such as std::cout), then that process may add extra CR characters that are not in the string (nor in the original output).
If you are seeing extra characters corrupting your stream, they are coming from somewhere else -- something besides you compress call/lib is writing to the stream, or something with how you are looking at your stream is doing something.
If you're on linux, how about creating an anonymous file using memfd_create? You can then open /proc/self/fd/<fd> and do your stuff. Some implementations of std::ofstream may even provide a constructor that takes a FILE*, you can check if that's the case on your system.
I would like to save a vibe.d stream such as HTTPClientResponse.bodyReader (of type InterfaceProxy!InputStream), but also other potential vibe.d streams to a file, how do I best do that in a memory efficient way without copying all data to RAM?
In general for downloading files using a HTTP client you can use the vibe.inet.urltransfer package which offers a download convenience function which performs a HTTP request, handles redirects and stores the final output to a file.
download(url, file);
However if you want to take a raw input stream (for example when not handling redirects) you can use vibe.core.file : openFile to open/create a file as file stream and then write to that.
To then write to the file stream you've got two options:
Either you directly call file.write(otherStream)
Otherwise you can use vibe.core.stream : pipe
Directly calling write on the FileStream object is what is being used inside the vibe.d urltransfer module and is also recommended for files as it will read directly from the stream into the write buffer instead of using an additional temporary buffer which pipe would use.
Sample:
// createTrunc creates a file if it doesn't exist and clears it if it does exist
// You might want to use readWrite or append instead.
auto fil = openFile(filename, FileMode.createTrunc);
scope(exit) fil.close();
fil.write(inputStream);
I need iostream wrapper for zlib that allow to append data into end of existing stream.
gzopen() function in append mode follow the next rule. "When appending, gzopen does not test whether the file begins with a gzip stream, nor does it look for the end of the gzip streams to begin appending. gzopen will simply append a gzip stream to the existing file." E.g. opening of file with "a" option will create file with multiple compressed streams.
gzofstream wrapper for zlib translate std::ios_base::app in parameter list as "ab" mode of gzopen() function. As result, gzofstream will also create file with multiple streams.
However, in my application I need alternative behavior. Main module should open file in append mode (with std::ios_base::app option), write small data portion in the end of existing stream, and close the file. E.g., file should always contain single compressed stream after several open/close operation.
gzlog example from zlib has functionality close to required. However, it is pure C.
Can you propose some ready solutions for my task?
It seems that ready solution is not exists for this case: iostream wrappers don't support append operation.
I have implemented own code based on gzlog and pure C (not C++).
My programming knowlege is very limited so please take this into account when reading this. I am using Visual C++ MFC and I am looking for a basic function that would overwrite the contents of a file but keep the file the same name. I am sure this is probably fairly simple however I can't seem to find anything online. Thanks in advance for any help.
You can use CFile::Open() there is flags to specify to open an existing file without truncating it. For example if you want to create the file if it not exists, or using the alreading existing without truncating you can use CFile::modeCreate|CFile::modeNoTruncate. You can then seet to the needed position by using CFile::Seek()
It's been a while since I've done any MFC work so I'll just give you the general standard on how to do this in C/C++. This will give you a direction on how to work with MFC.
When you're opening a file, you can choose an "open flag" that tells the file system how to open it. it can be "a" for append, "r" for read, "w" for write over (trunacte), and you can add "b" if it's a binary file.
so to do that just do:
FILE *fp = fopen("my_file.whatever", "wb");
if (fp)
{
//now write to
the file... ....
fclose(fp);
}
I am trying to load a bitmap from an archive. The bitmap class I have takes a character pointer to a filename and then loads it if it is in the same directory. The bitmap loading class is well tested and I don't want to mess with it too much. Problem is it uses a file pointer to load and do all of its file manipulation. Is there any way to emulate a file pointer and actually have it read from a chunk in memory instead?
Sorry if this is a bizarre question.
Refactor it and create functions that takes the exact same parameters as before : If you used fopen, fread and fseek that read from disk, create mopen, mread and mseek that read file from memory. You'll only have to fix the name of the functions.
It should be easy without risk and code won't look like an dirty hack in the end.
You can also use a pipe. A pipe is a piece of memory where you can read and write using file primitives. Which is basically what you want
(Assuming POSIX Operating system)
create a pipe:
int p[2];
pipe(p);
use fdopen() to turn the pipe file descriptor into a FILE*
FILE *emulated_file = fdopen(p[0], "r");
then write whatever you want to the write end of the pipe :
write(p[1], 17 ,"whatevereyouwant");
Now :
buf[32];
fread(&buf,1,32, emulated_file);
cout<<buf<<endl;
willl output "whateveryouwant".
Check out John Ratcliff's File Interface replacement for standard file I/O. It supports the feature you need.
You'll still need to refactor the bitmap loading code to use the new interface. However, this interface supports loading from file on disk, or memory chunk in memory (as well as writing to file on disk, or to expandable memory chunks).