I was discussing with a co-worker and I thought this would be a good question to put here on SO.
When designing and API when should your functions accept file paths and when should they accept streams? Are there any guidelines?
void do_something(const std::filesystem::path &file_path);
void do_something(std::istream &stream);
path:
callee is responsible for checking that the file exists and is accessible.
is difficult to unit test. You have to create/have a file on disk to test it.
stream:
caller is responsible for checking that the file exists and is accessible. more repetitive boilerplate code.
unit test is easier you can just pass a stream object
I guess one could add a function to the library to "help" open the file, something of the sorts:
std::ifstream open_input(const std::filesystem::path &file)
{
std::ifstream stream(file);
if (not stream) {
throw std::invalid_argument("failed to open file: " + file.string());
}
return stream;
}
You stated yourself that you could add the "helper" function in order to preserve the istream interface. This is also the better solution in terms of testability and adheres to the single responsibility principle (SRP).
Your helper function has one responsibility (create stream from file) and your actual function another (it "does something" :)).
I'd add that it depends on the context of what does something actually does. For example, if it is a facade for different accesses to an underlying functionality, then it would make sense to have that interface with the actual path. Still you would have a separate helper function and a do_something function which are used from the facade.
You may have your cake and eat it:
#include <fstream>
#include <sstream>
#include <utility>
//
// some booilerplate to allow use of a polymorphic temporary
template<class Stream, std::enable_if_t<std::is_base_of<std::istream, Stream>::value> * = nullptr>
struct stream_holder
{
stream_holder(Stream stream) : stream_(std::move(stream)) {}
operator std::istream&() && { return stream_; }
operator std::istream&() & { return stream_; }
private:
Stream stream_;
};
// helper function
template<class Stream, std::enable_if_t<std::is_base_of<std::istream, Stream>::value> * = nullptr>
auto with_this(Stream&& stream)
{
return stream_holder<std::decay_t<Stream>>(std::forward<Stream>(stream));
}
// express logic in terms of stream
void do_something(std::istream& stream_ref);
// utility functions to create various types of stream
std::ifstream file_stream();
std::stringstream string_stream();
int main()
{
// * composability with succinct syntax
// * lifetime automatically managed
// * no repetitive boilerplate
do_something(with_this(file_stream()));
do_something(with_this(string_stream()));
}
Related
I am trying to create a native nodejs module, using NAN and c ++, I want to transform an existing program that uses std::ifstream stream (filename, std :: ifstream :: in | std :: ifstream :: binary); to load a file into a javascript module that can load a buffer and send it to c ++
The original c ++ code was made to work via command line, I don't want to have to write a file to disk, I would like to send this file using a nodejs buffer.
index.js
const fs = require('fs')
const addon = require('./build/Release/image_edit');
fs.readFile('image.png', function read(err, buffer) {
if (err) {
throw err;
}
var result = addon.edit(buffer, buffer.length);
//console.log(result)
});
main.cpp
#include <node.h>
#include <node_buffer.h>
#include <iostream>
#include <nan.h>
#include <sstream>
#include <string>
#include <fstream>
#include <streambuf>
#include <istream>
using namespace Nan;
using namespace v8;
uint32_t read(std::istream& in)
{
uint32_t v;
in.read(reinterpret_cast<char*>(&v), sizeof(v));
return v;
}
NAN_METHOD(edit) {
unsigned char*buffer = (unsigned char*) node::Buffer::Data(info[0]->ToObject());
unsigned int size = info[1]->Uint32Value();
//the closest I could to manipulating the data was using a vector
std::vector<uint32_t> png_data(buffer, buffer + size);
//The main core of the program uses the in.read function to parse the file, tb uses in.clear () and in.seekg ();
//here an example of how this is done
uint32_t count = readU32(stream);
}
NAN_MODULE_INIT(Init) {
Nan::Set(target, New<String>("edit").ToLocalChecked(),
GetFunction(New<FunctionTemplate>(edit)).ToLocalChecked());
}
NODE_MODULE(image_edit, Init)
I tried using the following code to verify that the data received is valid and if the recorded file is the same as the original, everything looks fine.
std::ofstream FILE("test.png", std::ios::out | std::ofstream::binary);
std::copy(png_data.begin(), png_data.end(), std::ostreambuf_iterator<char>(FILE));
The question is, how do I make this buffer received from nodejs into something read the same way an ifstream does, without having to drastically change the c ++ program?
The main methods called by the program in c ++ are: .seekg (), .push_back, .clear (),
This kind of thing is usually done by implementing a custom subclass of std::streambuf, and then using it to construct a std::istream.
std::istream has a constructor that takes a pointer to a std::streambuf as a parameter, so the basic outline is something like this
class my_streambuf : public std::streambuf {
// ... Your implementation of your subclass
};
my_streambuf msb{ /* Parameters to your class's constructor */ }
std::istream i{&msb};
At this point, i is an ordinary input stream and does everything that any other input stream does. You can seek it. You can read from it.
Of course, the hard part is implementing your custom subclass of std::streambuf. This is not something that can be fully described in one or two paragraphs on stackoverflow.com. You should read std::streambuf's documentation, specifically the descriptions of its virtual methods. Your custom subclass will need to reimplement std::streambuf's virtual methods and make them work with your buffer. It's likely you will not need to reimplement all the virtual methods. For some of them their default implementation will be sufficient. Some of them won't be needed, for what you end up doing with std::istream.
You will have to determine, based on you specific needs to what extent you need to reimplement which std::streambuf's virtual methods, and how.
Of course, another, easy alternative is to use your buffer to construct a std::string, and then using it to construct a std::istringstream, and call it a day. Of course, that'll be somewhat wasteful and require effectively doubling the memory used for the data, with a second copy that's owned by a throw-away std::string, and copying it. If this is a small amount of data that's probably fine, but if your buffer is very big that may not be practical, and a custom std::streambuf subclass that uses the buffer directly is your only option.
Like the other answer mentioned, you can use an std::stringstream if you don't want to go the std::streambuf route:
std::stringstream ss;
std::copy(png_data.begin(), png_data.end(), std::ostreambuf_iterator<uint32_t>(ss));
Then you just use it like an input stream.
I want to make my code more efficient, specifically the reading of data from a text file. Here is a snapshot of what it looks like now:
values V(name);
V.population = read_value(find_line_number(name, find_in_map(pop, mapping)));
V.net_growth = read_value(find_line_number(name, find_in_map(ngr, mapping)));
... // and so on
Basically, the read_value function creates an ifstream object, opens the file, reads one line of data, and closes the file connection. This happens many times. What I want to do is to open the file once, read every line that is needed into the struct, and then close the file connection.
Here is the creating values struct function with parameters:
static values create_struct(std::string name, std::map<std::string, int> mapping) {
values V(name);
V.population = read_value(find_line_number(name, find_in_map(pop, mapping)), file);
V.net_growth = read_value(find_line_number(name, find_in_map(ngr, mapping)), file);
// more values here
return V;
}
The function that calls create_struct is shown below:
void initialize_data(string name) {
// read the appropriate data from file into a struct
value_container = Utility::create_struct(name, this->mapping);
}
I am thinking of instead defining the ifstream object in the function initialize_data. Given what is shown about my program, would that be the best location to create the file object, open the connection, read the values, then close the connection? Also, would I need to pass in the ifstream object into the create_values struct, and if so, by value, reference or pointer?
The short answer is to create your ifstream object first and pass it as reference to your parser. Remember to seek the stream back to the beginning before you leave your function, or when you start to read.
The RAII thing to do would be to create a wrapper object that automatically does this when it goes out of scope.
class ifStreamRef{
ifStreamRef(std::ifstream& _in) : mStream(_in){}
~ifStreamRef(){mStream.seekg(0);}
std::ifstream& mStream;
}
Then you create a wrapper instance when entering a method that will read the fstream.
void read_value(std::ifstream& input, ...){
ifStreamRef autoRewind(input);
}
Or, since the Ctor can do the conversion...
void read_value(ifStreamRef streamRef, ...) {
streamRef.mStream.getLine(...);
}
std::ifstream itself follows RAII, so it will close() the stream for you when your stream goes out of scope.
The long answer is that you should read up on dependency injection. Don't create dependencies inside of objects/functions that can be shared. There are lots of videos and documents on dependency injection and dependency inversion.
Basically, construct the objects that your objects depend on and pass them in as parameters.
The injection now relies on the interface of the objects that you pass in. So if you change your ifStreamRef class to act as an interface:
class ifStreamRef{
ifStreamRef(std::ifstream& _in) : mStream(_in){}
~ifStreamRef(){mStream.seekg(0);}
std::string getLine(){
// todo : mStream.getLine() + return "" on error;
}
bool eof() { return mStream.eof(); }
std::ifstream& mStream;
}
Then later on you can change the internal implementation that would take a reference to vector<string>& instead of ifstream...
class ifStreamRef{
ifStreamRef(std::vector<string>& _in) : mStream(_in), mCursor(0){}
~ifStreamRef(){}
std::string getLine(){
// todo : mStream[mCursor++] + return "" on error;
}
bool eof() { return mCursor >= mStream.size(); }
std::vector<string>& mStream;
size_t mCursor;
}
I have oversimplified a few things.
I've created an fstream object to write info to files.
I write strings to the new file like
fStreamObject << "New message.\n";
because I want each << to print a string to the next line.
I want to be able to set a property and make a call like
fstreamObject << "New message.";
which will write the string to the next line.
Are there flags/settings for fstream objects that allows this to be done?
I've seen the different file modes (i.e. ofstream::in, ofstream::out, etc.), but I couldn't find one that auto writes to a new line. Also, I'm not looking to write my own solution. I want to be able to use a built in feature.
No, there are no readily configurable capabilities of that sort within the standard streams.
You may have to subclass the stream type and fiddle with operator<< to get this to work the way you want, or do it with a helper function of some description:
fstreamObject << nl("New message.");
(but that's hardly easier than just having the \n in there (for a string, anyway).
It depends on what you mean by "setting the stream". If we consider this to be fairly broad then the answer happens to be "yes"!
Here is how:
Create a stream buffer which inserts a newline every time it is flushed, i.e., when sync() is called. Otherwise it just forwards characters.
Change the file stream's stream buffer to use this stream buffer filtering to the file stream's stream buffer.
Set the flag std::ios_base::unitbuf which causes a flush after every [properly written] output operation.
Here are is the example code to do just that:
#include <iostream>
class newlinebuf
: public std::streambuf {
std::ostream* stream;
std::streambuf* sbuf;
int overflow(int c) { return this->sbuf->sputc(c); }
int sync() {
return (this->sbuf->sputc('\n') == std::char_traits::eof()
|| this->sbuf->pubsync() == -1)? -1: 0;
}
public:
newlinebuf(std::ostream& stream)
: stream(&stream)
, sbuf(stream.rdbuf(this)) {
stream << std::unitbuf;
}
~newlinebuf() { this->stream->rdbuf(this->sbuf); }
};
int main() {
newlinebuf sbuf(std::cout);
std::cout << "hello" << "world";
}
Although this approach work, I would recommend against using it! On problem is that all composite output operators, i.e., those using multiple output operators to do their work, will cause multiple newlines. I'm not aware of anything which can be done to prevent this behavior. There isn't anything in the standard library which enables just configuring the stream to do this: you'll need to insert the newline somehow.
No, the C++ streams do not allow that.
There is no way to decide where one insertion stops and the next starts.
For example for custom types, their stream-inserters are often implemented as calls to other stream-inserters and member-functions.
The only things you can do, is write your own class, which delegates to a stream of your choosing, and does that.
That's of strictly limited utiliy though.
struct alwaysenter {
std::ostream& o;
template<class X> alwaysenter& operator<<(X&& x) {
o<<std::forward<X>(x);
return *this;
}
};
I'd like to input a stream parameter to a method which can be either a <stringstream> or <iostream> as in:
void method(? out); // or
void method(? in);
If ? is <istream> or <ostream> it's straightforward. What I don't know is what to do if the parameter is either <istream> or <stringstream> or is either <ostream> or <stringstream>.
Can this be done?
The streams implementing both std::istream and std::ostream, e.g., std::stringstream and std::fstream derive from std::iostream (since they are all class templates, you'd look for basic_... in the standard). That is, if you really need a stream which is used for both input and output, you'd pass an std::iostream&.
The class std::iostream derives from both std::istream and std::ostream. The appropriate types are straight forward:
for reading only use std::istream&
for writing only use std::ostream&
for reading and writing use std::iostream& (I don't think I ever used this in production code)
Note that you need to seek when switching between reading and writing in case the stream may be a file stream: switching between reading and writing without intervening seek, even if the seek is to the current position, results in undefined behavior.
Both the string stream classes and file stream classes publically-derive from the general stream bases classes std::basic_istream<...> and std::basic_ostream<...>. This means that you can pass (for instance) std::ostringstream objects to functions that take std::ostream:
void test(std::ostream& os);
std::ostringstream buf;
test(buf); // Good!
Unless of course, you need additional functionality specific to the derived stream classes. For example, string streams provide an str() method and file streams provide open()/close() member functions. In the case where you need these, you can have test() take string streams or file streams as parameters.
void test(std::istringstream& iss)
{
std::cout << iss.str(); // Works because the type is a stringstream
}
I found this question answered for Python, Java, Linux script, but not C++:
I'd like to write all outputs of my C++ program to both the terminal and an output file. Using something like this:
int main ()
{
freopen ("myfile.txt","w",stdout);
cout<< "Let's try this";
fclose (stdout);
return 0;
}
outputs it to only the output file named "myfile.txt", and prevents it from showing on the terminal. How can I make it output to both simultaneously? I use visual studio 2010 express (if that would make any difference).
Thanks in advance!
Possible solution: use a static stream cout-like object to write both to cout and a file.
Rough example:
struct LogStream
{
template<typename T> LogStream& operator<<(const T& mValue)
{
std::cout << mValue;
someLogStream << mValue;
}
};
inline LogStream& lo() { static LogStream l; return l; }
int main()
{
lo() << "hello!";
return 0;
}
You will probably need to explicitly handle stream manipulators, though.
Here is my library implementation.
There is no built in way to do this in one step. You have to write the data to a file and then write the data out on screen in two steps.
You can write a function that takes in the data and the filename and does this for you, to save you time, some sort of logging function.
I have a method to do this, and it is based on a subscriber model.
In this model all your logging goes to a "logging" manager and you then have "subscribers" that decide what to do with the messages. Messages have topics (for me a number) and loggers subscribe to one or more topic.
For your purpose, you create 2 subscribers, one that outputs to the file and one that outputs to the console.
In the logic of your code you simply output the message, and at this level not need to know what is going to be done with it. In my model though you can check first if there are any "listeners" as this is considered cheaper than constructing and outputting messages that will only end up in /dev/null (well you know what I mean).
One way to do this would be to write a small wrapper to do this, for example:
class DoubleOutput
{
public:
// Open the file in the constructor or any other method
DoubleOutput(const std::string &filename);
// ...
// Write to both the file and the stream here
template <typename T>
friend DoubleOutput & operator<<(const T& file);
// ...
private:
FILE *file;
}
To have a class instead of a function makes you use the RAII idiom (https://en.wikipedia.org/wiki/Resource_acquisition_is_initialization)
To use it:
DoubleOutput mystream("myfile");
mystream << "Hello World";