So we have some stringstream with somedata. With help of which cross platform library we can turn its contents into gziped format (you would save it into file with extention .tar.gz)
Lets get into code: so I have:
stringstream body;
body << std::ifstream( path.string().c_str(), ios::binary).rdbuf();
Which includes shall I add, which function should I call to encode stringstream and which finction to call to turn to put that encoded data into stringstream?
BTW: is there any boost library that can encode into gzip
boost.iostreams has optional built-in support for bzip2 and gzip compression/decompression.
It's not too clear what you're asking. If you mean that the
file (specified by path) is in gzipped format, the easiest
solution is probably to define a pipe input stream, and read it
from gunzip < file through that. If you want to unzip it
within your application, some sort of filtering input streambuf
would seem in order (see boost::iostream).
And I wouldn't use a temporary std::ifstream in any case: you
want to verify that the open succeeded before trying to read.
Related
(My previous questions was closed as a duplicate of Are there binary memory streams in C++ which is ridiculous, since i can't change the implementation of the library I'm using)
I'm using a library (Poco) to create zip files. It takes ostream as an input and writes the data of the zip file into it. Something like:
std::ofstream ofs("file.zip", std::ios::binary);
Compress compress(ofs);
// add data to compress ...
compress.close();
// now file.zip contains added file
This works. But I want to be able to create a zip in memory without creating a file. I tried using stringstream instead of ofstream, i get additionl newline characters in the data in the zip file is corrupted. Is there any other stream i can use?
(If someone still thinks it's a duplicate, I'm gonna need an explanation, since I don't see how this other question is helpful for me)
Use a std::stringstream -- that will create an in-memory string that you can write to as an ostream, and WILL NOT add extra newlines. If you later copy the string to an fstream that was opened in text mode (such as std::cout), then that process may add extra CR characters that are not in the string (nor in the original output).
If you are seeing extra characters corrupting your stream, they are coming from somewhere else -- something besides you compress call/lib is writing to the stream, or something with how you are looking at your stream is doing something.
If you're on linux, how about creating an anonymous file using memfd_create? You can then open /proc/self/fd/<fd> and do your stuff. Some implementations of std::ofstream may even provide a constructor that takes a FILE*, you can check if that's the case on your system.
I would like to save a vibe.d stream such as HTTPClientResponse.bodyReader (of type InterfaceProxy!InputStream), but also other potential vibe.d streams to a file, how do I best do that in a memory efficient way without copying all data to RAM?
In general for downloading files using a HTTP client you can use the vibe.inet.urltransfer package which offers a download convenience function which performs a HTTP request, handles redirects and stores the final output to a file.
download(url, file);
However if you want to take a raw input stream (for example when not handling redirects) you can use vibe.core.file : openFile to open/create a file as file stream and then write to that.
To then write to the file stream you've got two options:
Either you directly call file.write(otherStream)
Otherwise you can use vibe.core.stream : pipe
Directly calling write on the FileStream object is what is being used inside the vibe.d urltransfer module and is also recommended for files as it will read directly from the stream into the write buffer instead of using an additional temporary buffer which pipe would use.
Sample:
// createTrunc creates a file if it doesn't exist and clears it if it does exist
// You might want to use readWrite or append instead.
auto fil = openFile(filename, FileMode.createTrunc);
scope(exit) fil.close();
fil.write(inputStream);
I currently have some code reading files which are not compressed, it uses the following approach to read a file in C++
FILE* id = fopen("myfile.dat", "r");
after obtaining id, different parts of the code access the file using fread, fseek, etc.
I would like to adapt my code so as to open a gzip version of the file, e.g. "myfile.dat.gz" without needing to change too much.
Ideally I would implement a wrapper to fopen, call it fopen2, which can read both myfile.dat and myfile.dat.gz, i.e. it should return a pointer to a FILE object, so that the remaining of the code does not need to be changed.
Any suggestions?
Thank you.
PS: it would be fine to decompress the whole file in memory, if this approach provides a solution
zlib provides analogs of fopen(), fread(), etc. called gzopen(), gzread(), etc. for reading and writing gzip files. If the file is not gzip-compressed, it will be read just as the f functions would. So you would only need to change the function names and link in zlib.
I can read file in c++. This is my code:
std::string ReadFile(std::string file)
{
char buff[20480];
std::ifstream fread(file, std::ios::binary | std::ios::app);
fread.read(buff,sizeof(buff));
std::string str = buff;
fread.close();
return str;
}
The variable "file" is the FilePath. And I get a folder .zip, I want to read the file in folder. What should I do? I try to use libzip, but it can't solve my problem, maybe I didn't use it by wrong way.
No. To unzip a file, you must unzip a file.
You don't need to invoke the unzip utility to do it: there are libraries that can expose decompression through a streams API, resulting in code that looks rather similar to what you've written above. But you need to install and learn how to use those libraries.
Unless you have access to the API that allows to unpack your file I don't see how directly in the code.
If you are lazy you could write a small script in whatever language you prefer that does the unpacking and then calls your program on the unpacked file
Assuming, you have unzip available. Did you try something like:
FILE * file = popen("unzip -p filename", "r");
Similarly, popen("gzip -f filename", "r") should work for gzip.
In order to parse the output, I'd refer to this post (with Windows hints). I don't know about a more C++-style way of doing this.
I need iostream wrapper for zlib that allow to append data into end of existing stream.
gzopen() function in append mode follow the next rule. "When appending, gzopen does not test whether the file begins with a gzip stream, nor does it look for the end of the gzip streams to begin appending. gzopen will simply append a gzip stream to the existing file." E.g. opening of file with "a" option will create file with multiple compressed streams.
gzofstream wrapper for zlib translate std::ios_base::app in parameter list as "ab" mode of gzopen() function. As result, gzofstream will also create file with multiple streams.
However, in my application I need alternative behavior. Main module should open file in append mode (with std::ios_base::app option), write small data portion in the end of existing stream, and close the file. E.g., file should always contain single compressed stream after several open/close operation.
gzlog example from zlib has functionality close to required. However, it is pure C.
Can you propose some ready solutions for my task?
It seems that ready solution is not exists for this case: iostream wrappers don't support append operation.
I have implemented own code based on gzlog and pure C (not C++).