I've read about custom streams for C++ but it seems that generally people inherit from std::streambuf, std::istream, and std::ostream. By inspecting those type's declarations it becomes clear that these are meant for characters:
typedef basic_streambuf<char> streambuf;
The docs confirm this:
Input stream objects can read and interpret input from sequences of
characters.
Obviously, that makes sense. I'm wondering what would be the correct way of implementing a stream for other types. I do not want to allow text but other forms of binary input/output (I have specific formats). The obvious step seems to be to inherit the basic variants of the above (basic_streambuf, basic_istream, and basic_ostream) and use whatever type I see fit as the template parameter. I failed to find confirmation that this would be the right procedure. So, is it?
Edit for clarification: I have a class called Segment. These streams will send/receive segments and only segments over a WiFi connection as these are used in the communication protocol. Sending anything else would break the protocol. This means that the stream cannot support other types.
This is not an answer to your question in terms of inheriting from std::basic_* with non-char types. But following the comments and given your application, I am questioning the need to reimplement the whole standard stream machinery for your Segment type, when you can simply define a class with a stream operator:
class SegmentStream
{
public:
SegmentStream& operator<< ( const Segment& s );
}
Better yet you could clarify your code by defining methods send and recv instead of >>, <<.
Or perhaps you could explain why this would not be sufficient and why you specifically want to use standard streams?
Related
I need some guidance or pointers understanding how to implement a custom ostream. My requirements are:
A class with a '<<' operator for several data types.
The intention is to send output to database. Each "line" should go to a separate record.
Each record most important field would be the text (or blob), but some other fields such as time, etc. can be mostly deduced automatically
buffering is important, as I don't want to go to database for every record.
First, does it worth deriving from ostream? What do I get by deriving from ostream? What if my class simply implements few operator<< methods (including some custom data types). Which functionality do I get from ostream?
Assuming what I want is a class derived from ostream, I need some guidance understanding the relationship between the ostream and the streambuf classes. Which one do I need to implement? Looking at some samples, it appears that I don't need to derive from ostream at all, and just give the ostream constructor a custom streambuf. Is that true? is that the canonical approach?
Which virtual functions at the custom streambuf do i need to implement? I've seen some samples (including this site: here and here, and few more), some override the sync method, and other override the overflow method. Which one should I override? Also, looking at the stringbuf and filebuf sources (Visual Studio or GCC) both those buffer classes implement many methods of the streambuf.
If a custom class derived from streambuf is required, would there be any benefit deriving from stringbuf (or any other class) instead of directly from streambuf?
As for "lines". I would like at least when my users of the class using the 'endl' manipulator to be a new line (i.e. record in database). Maybe - depends on effort - every '\n' character should be considered as a new record as well. Who do my custom ostream and/or streambuf get notified for each?
A custom destination for ostream means implementing your own ostreambuf. If you want your streambuf to actually buffer (i.e. don't connect to the database after each character), the easiest way to do that is by creating a class inheriting from std::stringbuf. The only function that you'll need to override is the sync() method, which is being called whenever the stream is flushed.
class MyBuf : public std::stringbuf
{
public:
virtual int sync() {
// add this->str() to database here
// (optionally clear buffer afterwards)
}
};
You can then create a std::ostream using your buffer:
MyBuf buff;
std::ostream stream(&buf)
Most people advised against redirecting the stream to a database, but they ignored my description that the database basically has a single blob field where all text is going to.
In rare cases, I might send data to a different field. This can be facilitated with custom attributes understood by my stream. For example:
MyStream << "Some text " << process_id(1234) << "more text" << std::flush
The code above will create a record in the database with:
blob: 'Some text more text'
process_id: 1234
process_id() is a method returning a structure ProcessID. Then, in the implementation of my ostream, I have an operator<<(ProcessID const& pid), which stores the process ID until it gets written. Works great!
The simplest way is to inherit std::streambuf and override just two methods:
std::streamsize xsputn(const char_type* s, std::streamsize n) – to append a given buffer with size provided to your internal buffer, std::string for example;
int_type overflow(int_type c) – to append a single char to your internal buffer.
Your streambuf can be constructed from whatever you want (DB connection for example). After append something into the internal buffer you may try to split it into lines and push something into DB (or just buffer an SQL statements to execute later).
To use it: just attach your streambuf to any std::ostream using constructor.
Simple! I've done something like this to output strings to syslog – everything works fine with any custom operator<< for user defined classes.
my2c - I think you are tackling this the wrong way. A stream may sound like a nice idea, but you'll need a way to indicate the end of the row too (and then what if someone forgets?) I would suggest something along the lines of how the java PreparedStatements and batches work, as in provide a set of methods which accept the types and a column index, then a "batch" method which explicitly makes it clear that you are indeed batching that row and then an execute to push the batch in.
Any stream based operation will rely on type (typically) to indicate which column to fill - but what if you have two ints? IMO, as a user, it doesn't feel like a natural way of inserting records into a database...
To add a new source or destination of character input/output to the iostreams mechanism, you should create a new streambuf class. The task of the stream buffer classes is to communicate with the 'external device' that will store the characters and to provide buffering facilities.
The problem with using iostreams to communicate with your database is that a database table does not match with the concept of a character sequence. A bit like pushing a round peg in a square hole. A streambuf only operates on characters. That is the only thing ever presented to it. This means the streambuf has to parse the character stream presented to it to find the field and record separators.
If you decide to go this route, I predict you will end up writing a CSV-to-SQL converter in your streambuf, just to get it working.
You will probably be better of with just adding a few operator<< overloads to your class(es). You could look at the Qt framework for ideas here. They also have the possibility to use operator<< to add items to a collections and such.
in C++, there is a concept of an ostream. cout is an ostream, as is an ofstream. In this way you can define a function that takes an ostream and writes to it so that the caller can decide where he wants the function to write to.
Is it possible to achieve the same effect in kotlin: defining a function that determines where (possibly the console) it writes to at runtime? Obviously if or when statements don't count.
Kotlin (and Java) have two equivalents: OutputStream for byte streams, and Writer for character streams. These are both abstract classes, with many concrete subclasses writing to different places.
To take the simpler case:
If you want to write byte data to a file, you can create a FileOutputStream instance which writes to a given filename. (That implements OutputStream.)
Or if you want to write to stdout, you can use System.out directly. (That is also an OutputStream.)
Or if you have a network Socket, you can call its getOutputStream() method, which gives you an OutputStream.
Or there are implementations that can write to a byte array, or a pipe, or a CORBA stream, or…
So if you have some code that uses an OutputStream, you could provide it with an instance of any of those classes, and when that calls OutputStream.write() it will write to the appropriate place.
(In practice, you often wrap it in a BufferedOutputStream for efficiency.)
It's very similar for a Writer, too, but in those cases you generally have to tell it which character encoding to use. (Though in many cases you can leave it to the platform default, which is usually UTF-8.)
So if you want to write character data to a file, you can create a FileWriter instance which writes to a given filename.
Or if you want to write to stdout you can create an OutputStreamWriter around System.out.
And so on.
Again, if your code is written to accept any Writer, then it will work regardless of where it writes to, and there's a BufferedWriter wrapper for efficiency.
There are equivalent classes for input, too: InputStream for byte streams, and Reader for character streams, both with lots of implementations for different sources.
There are few steps in java and kotlin :
Get the InputStream or OutputStream, either from file or network or any other resource that can provide you these Streams. (usages and names of Streams vary depending upon abstractions they provide)
You can use these Streams to read or write to them which eventually reads from or writes to the underlying Resource.(eg. a file)
At the end of operations you need to close the respective Streams using close() method.
Then you may or may not close the Resource depending on your use case.
I would recommend that you follow the Kotlin documentation.
Must see the usage of "use". ( It is like Java's try-with-resource)
I am working on an application the will need to sequentially encode/decode a series of bytes, and put them to a stream once they have been processed.
My plan was to subclass ostream and provide a constructor MyEncodeStream(istream) using istream.
I must admit though I am at a loss as to what to do next, which methods should be overridden?
What is the standard facility for doing this.
Please provide example code for a this simple case:
A char is pulled from the input buffer
The char is manipulated by char manipulationFunc(char in)
The char is put to a buffer
MyEncodeStream stops blocking so the char can be read
Does ostream provide a read function that should be overridden which operator << calls, or is operator<< the function that should be overridden?
I figured it out, I was confused. What I wanted to do was to subclass an input stream with an input stream as an argument for the constructor, that way when the custom input stream is read it can read from the standard input mechanism. I was looking up subclassing output streams, which is why google wasn't turning up many results.
Up until now, whenever I wanted to pass some raw data to a function (like a function that loads an image from a buffer), I would do something like this:
void Image::load(const char* buffer, std::size_t size);
Today I took a look at the Boost libraries, more specifically at the property_tree/xml_parser.hpp header, and I noticed this function signature:
template<typename Ptree>
void read_xml(std::basic_istream<typename Ptree::key_type::value_type>&,
Ptree &, int = 0);
This actually made me curious: is this the correct way to pass around raw data in C++, by using streams? Or am I misinterpreting what the function is supposed to be used for?
If it's the former, could you please point me to some resource where I can learn how to use streams for this? I haven't found much myself (mainly API references), and I have't been able to find the Boost source code for the XML parser either.
Edit: Some extra details
Seems there's been some confusion as to what I want. Given a data buffer, how can I convert it to a stream such that it is compatible with the read_xml function I posted above? Here's my specific use case:
I'm using the SevenZip C library to read an XML file from an archive. The library will provide me with a buffer and its size, and I want to put that in stream format such that it is compatible with read_xml. How can I do that?
Well, streams are quite used in C++ because of their conveniences:
- error handling
- they abstract away the data source, so whether you are reading from a file, an audio source, a camera, they are all treated as input streams
- and probably more advantages I don't know of
Here is an overview of the IOstream library, perhaps that might better help you understand what's going on with streams:
http://www.cplusplus.com/reference/iostream/
Understanding what they are exactly will help you understand how and when to use them.
There's no single correct way to pass around data buffers. A combination of pointer and length is the most basic way; it's C-friendly. Passing a stream might allow for sequential/chunked processing - i. e. not storing the whole file in memory at the same time. If you want to pass a mutable buffer (that might potentially grow), a vector<char>& would be a good choice.
Specifically on Windows, a HGLOBAL or a section object handle might be used.
The C++ philosophy explicitly allows for many different styles, depending on context and environment. Get used to it.
Buffers of raw memory in C++ can either be of type unsigned char*, or you can create a std::vector<unsigned char>. You typically don't want to use just a char* for your buffer since char is not guaranteed by the standard to use all the bits in a single byte (i.e., this will end up varying by platform/compiler). That being said, streams have some excellent uses as well, considering that you can use a stream to read bytes from a file or some other input, etc., and from there, store that data in a buffer.
Seems there's been some confusion as to what I want. Given a data buffer, how can I convert it to a stream such that it is compatible with the read_xml function I posted above?
Easily (I hope PTree::Key_type::value_type would be something like char):
istringstream stream(string(data, len));
read_xml(stream, ...);
More on string streams here.
This is essentially using a reference to pass the stream contents. So behind the scene's it's essentially rather similar to what you did so far and it's essentially the same - just using a different notation. Simplified, the reference just hides the pointer aspect, so in your boost example you're essentially working with a pointer to the stream.
References got the advantage avoiding all the referencing/dereferencing and are therefore easier to handle in most situations. However they don't allow you multiple levels of (de-)referencing.
The following two example functions do essentially the same:
void change_a(int &var, myclass &cls)
{
var = cls.convert();
}
void change_b(int *var, myclass *cls)
{
*var = cls->convert();
}
Talking about the passed data itself: It really depends on what you're trying to achieve and what's more effective. If you'd like to modify a string, utilizing an object of class std::string might be more convenient than using a classic pointer to a buffer (char *). Streams got the advantage that they can represent several different things (e.g. data stream on the network, a compressed stream or simply a file or memory stream). This way you can write single functions or methods that accept a stream as input and will instantly work without worrying about the actual stream source. Doing this with classic buffers can be more complicated. On the other side you shouldn't forget that all objects will add some overhead, so depending on the job to be done a simple pointer to a character string might be perfectly fine (and the most effective solution). There's no "the one way to do it".
Some days ago, I decided that it would be fun to write a streambuf subclass that would use mmap and read-ahead.
I looked at how my STL (SGI) implemented filebuf and realized that basic_filebuf contains a FILE*. So inheriting from basic_filebuf is out of the question.
So I inherited from basic_streambuf. Then i wanted to bind my mmapbuf to a fstream.
I thought the only thing that I would have to do would be to copy the implicit interface of filebuf... but that was a clear mistake. In the SGI, basic_fstream owns a basic_filebuf. No matter if I call basic_filestream.std::::ios::rdbuf( streambuf* ), the filestream completely ignores it and uses its own filebuf.
So now I'm a bit confused... sure, I can create my own mmfstream, that would be the exact copy/paste of the fstream but that sounds really not DRY-oriented.
What I can't understand, is: why does fstream is so tightly coupled with filebuf, so that it is not possible to use anything else than a filebuf? The whole point of separating streams and bufs is that one can use a stream with a different buffer.
Solutions:
=> filestream should rely on the implicit interface of filebuf. That is, fstream should be templated by a streambuf class. That would allow everyone to provide its own streambuf subclass to a fstream as long as it implements filebuf's implicit interface. Problem: we cannot add a template parameter to fstream since it would break template selectors while using fstream as template template parameter.
=> filebuf should be a pure virtual class without any additional attributes. So that one can inherit from it without carrying all its FILE* garbage.
Your ideas on the subject ?
In the IO streams' design, most of the actual streams' functionality (as opposed to the stream buffers' functionality) is implemented in std::basic_istream, std::basic_ostream, and their base classes. The string and file stream classes are more or less just convenience wrappers which make sure a stream with the right type of buffer is instantiated.
If you want to extend the streams, you almost always want to provide your own stream buffer class, and you almost never need to provide your own stream class. .
Once you have your own stream buffer type, you can then make it the buffer for any stream object you happen to have around. Or you derive your own classes from std::basic_istream, std::basic_ostream, and std::basic_iostream which instantiates your stream buffer and pass it to their base classes.
The latter is more convenient for users, but requires you to write some boiler-plate code for the buffer's instantiation (namely constructors for the stream class).
To answer your question: File streams and file buffer are coupled so tightly because the former only exists to ease the creation of the latter. Using a file stream makes it easy to set it all up.
Using your own stream class to wrap construction of your own stream buffer shouldn't be a problem, since you shouldn't be passing around file streams anyway, but only (references) to the base classes.
Check out mapped_file in the Boost.Iostreams library. I've never used used it myself, but it seems like it might already do what you need.
EDIT: Oops, reread your questions and I see you're doing this for fun. Perhaps you can draw inspiration from Boost.Iostreams?
fstream in itself is not a big class. It inherits from basic_stream to provide support for all the << and >> operations, contains a specialized steambuf that have to be initialized, and the corresponding constructors to pass the parameters to the streambuf constructor.
In a sense, what you wrote about your templated solution is OK. But basic_stream can also be derived into a tcp_stream for example. In that case, the constructors of fstream are a bit useless. Thus you need to provide a new tcpstream class, inheriting from basic_stream with the correct parameters for the constructors to be able to create the tcp_stream. In the end, you wouldn't use anything from fstream. Creating this new tcpstream is a matter of writing 3 or 4 functions only.
In the end, you would derive from the fstream class without any real reason to. This would add more coupling in the class hierarchy, unneeded coupling.
The whole point of std::fstream is that it is a _F_ile based std::stream. If you want an ordinary std::stream backed by your mmstreambuf, then you should create a mmstreambuf and pass it to std::stream::stream(std::streambuf*)