std::fstream with multiple buffers? - c++

Question

You can specify one buffer for your file stream like that:
char buf[BUFFER_SIZE];
std::ofstream file("file", std::ios_base::binary | std::ios_base::out);
if (file.is_open())
{
file.rdbuf()->pubsetbuf(buf, BUFFER_SIZE);
file << "abcd";
}
What I want to do now, is using more than just one buffer:
char* buf[] = { new char[BUFFER_SIZE], new char[BUFFER_SIZE], new char[BUFFER_SIZE], };
Is it possible without creating a custom derivation of std::streambuf?
EDIT:
I think I need to explain what I want to do in more detail. Please consider the following situation:
- The file(s) I want to read won't fit into memory
- The file while be accessed by some kind of a binary jump search
So, if you split the file into logical pages of a specific size, then I would like to provide multiple buffers which are representing specific pages. This would increase performance when a file location is read and the related page is already in a buffer.

Answer 1

I gather from the comment that you want to do a kind of scatter-gather I/O. I'm pretty sure there's no support for that in the C++ standard I/O streams library, so you'll have to roll your own.
If you want to do this efficiently, you can use OS support for scatter-gather. E.g., POSIX/Unix-like systems have writev for this purpose.

Answer 2

There's nothing like this provided by Standard. However, depending on your platform, you can use Memory Mapped Files, which provide the same functionality. Windows and Linux both provide them.

Answer 3

I will take a look at boost::iostreams::mapped_file, but I think my requirement is much simpler. I've created a custom class derived from basic_filebuf.
template<typename char_type>
class basic_filemultibuf : public std::basic_filebuf<char_type/*, std::char_traits<char_type>*/>
{
private:
char_type** m_buffers;
std::ptrdiff_t m_buffer_count,
m_curent_buffer;
std::streamsize m_buffer_size;
protected:
virtual int_type overflow(int_type meta = traits_type::eof())
{
if (this->m_buffer_count > 0)
{
if (this->m_curent_buffer == this->m_buffer_count)
this->m_curent_buffer = 0;
this->basic_filebuf::setbuf(this->m_buffers[this->m_curent_buffer++], this->m_buffer_size);
}
return this->basic_filebuf::overflow(meta);
}
public:
basic_filemultibuf(basic_filebuf const& other)
: basic_filebuf(other),
m_buffers(NULL),
m_buffer_count(0),
m_curent_buffer(-1),
m_buffer_size(0)
{
}
basic_filemultibuf(basic_filemultibuf const& other)
: basic_filebuf(other),
m_buffers(other.m_buffers),
m_buffer_count(other.m_buffer_count),
m_curent_buffer(other.m_curent_buffer),
m_buffer_size(other.m_buffer_size)
{
}
basic_filemultibuf(FILE* f = NULL)
: basic_filemultibuf(basic_filebuf(f))
{
}
basic_filemultibuf* pubsetbuf(char** buffers, std::ptrdiff_t buffer_count, std::streamsize buffer_size)
{
if ((this->m_buffers = buffers) != NULL)
{
this->m_buffer_count = buffer_count;
this->m_buffer_size = buffer_size;
this->m_curent_buffer = 0;
}
else
{
this->m_buffer_count = 0;
this->m_buffer_size = 0;
this->m_curent_buffer = -1;
}
this->basic_filebuf::setbuf(NULL, 0);
return this;
}
};
Example usage:
typedef basic_filemultibuf<char> filemultibuf;
std::fstream file("file", std::ios_base::binary | std::ios_base::in | std::ios_base::out);
char** buffers = new char*[2];
for (int i = 0; i < n; ++i)
buffers[i] = new char[4096];
filemultibuf multibuf(*file.rdbuf());
multibuf.pubsetbuf(buffers, 2, 4096);
file.set_rdbuf(&multibuf);
//
// do awesome stuff with file ...
//
for (int i = 0; i < n; ++i)
delete[] buffers[i];
That's pretty much it. The only thing I would really like to do is offer this functionally for other streambufs, because the usage of multiple buffers should not be restricted to filebuf. But it seems to me it isn't possible without rewriting the file specific functions.
What do you think about that?