I would like to save a vibe.d stream such as HTTPClientResponse.bodyReader (of type InterfaceProxy!InputStream), but also other potential vibe.d streams to a file, how do I best do that in a memory efficient way without copying all data to RAM?
In general for downloading files using a HTTP client you can use the vibe.inet.urltransfer package which offers a download convenience function which performs a HTTP request, handles redirects and stores the final output to a file.
download(url, file);
However if you want to take a raw input stream (for example when not handling redirects) you can use vibe.core.file : openFile to open/create a file as file stream and then write to that.
To then write to the file stream you've got two options:
Either you directly call file.write(otherStream)
Otherwise you can use vibe.core.stream : pipe
Directly calling write on the FileStream object is what is being used inside the vibe.d urltransfer module and is also recommended for files as it will read directly from the stream into the write buffer instead of using an additional temporary buffer which pipe would use.
Sample:
// createTrunc creates a file if it doesn't exist and clears it if it does exist
// You might want to use readWrite or append instead.
auto fil = openFile(filename, FileMode.createTrunc);
scope(exit) fil.close();
fil.write(inputStream);
Related
I am familiar with Ruby and am trying to write a program in Crystal.
I have a file called special_file.txt that I want to read in my Crystal program, how do I do that?
Crystal is inspired by Ruby syntax and so you can often read and perform File operations in a similar manner. For example, Crystal has a File classclass which is an instance of the IO class containing a read method.
To read a file's contents on your filesystem you can instantiate a File object and invoke the gets_to_end method coming from the IO super class:
file = File.new("path/to/file")
content = file.gets_to_end
file.close
The gets_to_end method reads an entire IO objects data to a String variable.
You can also use a block invocation to achieve a similar result:
# Implicit close with `open`
content = File.open("path/to/file") do |file|
file.gets_to_end
end
Finally, the most idiomatic way to read the contents of an entire file would be the one line:
# Shortcut:
content = File.read("path/to/file")
(My previous questions was closed as a duplicate of Are there binary memory streams in C++ which is ridiculous, since i can't change the implementation of the library I'm using)
I'm using a library (Poco) to create zip files. It takes ostream as an input and writes the data of the zip file into it. Something like:
std::ofstream ofs("file.zip", std::ios::binary);
Compress compress(ofs);
// add data to compress ...
compress.close();
// now file.zip contains added file
This works. But I want to be able to create a zip in memory without creating a file. I tried using stringstream instead of ofstream, i get additionl newline characters in the data in the zip file is corrupted. Is there any other stream i can use?
(If someone still thinks it's a duplicate, I'm gonna need an explanation, since I don't see how this other question is helpful for me)
Use a std::stringstream -- that will create an in-memory string that you can write to as an ostream, and WILL NOT add extra newlines. If you later copy the string to an fstream that was opened in text mode (such as std::cout), then that process may add extra CR characters that are not in the string (nor in the original output).
If you are seeing extra characters corrupting your stream, they are coming from somewhere else -- something besides you compress call/lib is writing to the stream, or something with how you are looking at your stream is doing something.
If you're on linux, how about creating an anonymous file using memfd_create? You can then open /proc/self/fd/<fd> and do your stuff. Some implementations of std::ofstream may even provide a constructor that takes a FILE*, you can check if that's the case on your system.
I currently have some code reading files which are not compressed, it uses the following approach to read a file in C++
FILE* id = fopen("myfile.dat", "r");
after obtaining id, different parts of the code access the file using fread, fseek, etc.
I would like to adapt my code so as to open a gzip version of the file, e.g. "myfile.dat.gz" without needing to change too much.
Ideally I would implement a wrapper to fopen, call it fopen2, which can read both myfile.dat and myfile.dat.gz, i.e. it should return a pointer to a FILE object, so that the remaining of the code does not need to be changed.
Any suggestions?
Thank you.
PS: it would be fine to decompress the whole file in memory, if this approach provides a solution
zlib provides analogs of fopen(), fread(), etc. called gzopen(), gzread(), etc. for reading and writing gzip files. If the file is not gzip-compressed, it will be read just as the f functions would. So you would only need to change the function names and link in zlib.
I need iostream wrapper for zlib that allow to append data into end of existing stream.
gzopen() function in append mode follow the next rule. "When appending, gzopen does not test whether the file begins with a gzip stream, nor does it look for the end of the gzip streams to begin appending. gzopen will simply append a gzip stream to the existing file." E.g. opening of file with "a" option will create file with multiple compressed streams.
gzofstream wrapper for zlib translate std::ios_base::app in parameter list as "ab" mode of gzopen() function. As result, gzofstream will also create file with multiple streams.
However, in my application I need alternative behavior. Main module should open file in append mode (with std::ios_base::app option), write small data portion in the end of existing stream, and close the file. E.g., file should always contain single compressed stream after several open/close operation.
gzlog example from zlib has functionality close to required. However, it is pure C.
Can you propose some ready solutions for my task?
It seems that ready solution is not exists for this case: iostream wrappers don't support append operation.
I have implemented own code based on gzlog and pure C (not C++).
So we have some stringstream with somedata. With help of which cross platform library we can turn its contents into gziped format (you would save it into file with extention .tar.gz)
Lets get into code: so I have:
stringstream body;
body << std::ifstream( path.string().c_str(), ios::binary).rdbuf();
Which includes shall I add, which function should I call to encode stringstream and which finction to call to turn to put that encoded data into stringstream?
BTW: is there any boost library that can encode into gzip
boost.iostreams has optional built-in support for bzip2 and gzip compression/decompression.
It's not too clear what you're asking. If you mean that the
file (specified by path) is in gzipped format, the easiest
solution is probably to define a pipe input stream, and read it
from gunzip < file through that. If you want to unzip it
within your application, some sort of filtering input streambuf
would seem in order (see boost::iostream).
And I wouldn't use a temporary std::ifstream in any case: you
want to verify that the open succeeded before trying to read.