In C++, is there a smart way to mirror output from stdout to both the console and the file?
I'm hoping there is a way to do it like in this question.
Edit: It would be nice to be able to do this with just the standard libraries (ie: no boost)..
Alternatively, just start your program so it's piped to the tee command.
You could try a Tee Device provided by Boost.Iostreams.
A Tee device directs output to multiple streams. As far as I know, you can chain them to reach theoretically infinite output devices from one write call.
This answer shows an example for how to do exactly what you want.
You can do this by creating a class that extends std::streambuf, and has a std::ofstream member. After overriding the std::streambuf::overflow and std::streambuf::sync member functions, you'll be all set.
Most of the code below comes from here. The stuff I've added ("ADDED:") for file-mirroring is pointed out. This might be overly complex as I'm at work and can't pore over it fully to simplify it, but it works - the bonus of doing it this way (instead of just using a std::streambuf* is that any code (say you have an external library) that writes to std::cout will write to your file.
mystreambuf.h
#ifndef MYSTREAMBUF_H
#define MYSTREAMBUF_H
template <typename charT, typename traits = std::char_traits<charT> >
class mystreambuf : public std::basic_streambuf<charT, traits>
{
public:
// The size of the input and output buffers.
static const size_t BUFF_SIZE = 1024;
typedef traits traits_type;
typedef typename traits_type::int_type int_type;
typedef typename traits_type::pos_type pos_type;
typedef typename traits_type::off_type off_type;
// You can use any method that you want, but here, we'll just take in a raw streambuf as a
// slave I/O object. xor_char is what each character is xored with before output.
explicit mystreambuf(std::streambuf* buf)
: out_buf_(new charT[BUFF_SIZE])
{
// ADDED: store the original cout stream and open our output file
this->original_cout = buf;
outfile.open("test.txt");
// Initialize the put pointer. Overflow won't get called until this buffer is filled up,
// so we need to use valid pointers.
this->setp(out_buf_, out_buf_ + BUFF_SIZE - 1);
}
// It's a good idea to release any resources when done using them.
~mystreambuf() {
delete [] out_buf_;
// ADDED: restore cout, close file
std::cout.rdbuf(original_cout);
outfile.flush();
outfile.close();
}
protected:
// This is called when there are too many characters in the buffer (thus, a write needs to be performed).
virtual int_type overflow(int_type c);
// This is called when the buffer needs to be flushed.
virtual int_type sync();
private:
// Output buffer
charT* out_buf_;
// ADDED: tracking the original std::cout stream & the file stream to open
std::streambuf* original_cout;
std::ofstream outfile;
};
#endif
mystreambuf.cpp
// Based on class by perfectly.insane from http://www.dreamincode.net/code/snippet2499.htm
#include <fstream>
#include <iostream>
#include <streambuf>
#include "mystreambuf.h"
// This function is called when the output buffer is filled.
// In this function, the buffer should be written to wherever it should
// be written to (in this case, the streambuf object that this is controlling).
template <typename charT, typename traits>
typename mystreambuf<charT, traits>::int_type
mystreambuf<charT, traits>::overflow(typename mystreambuf<charT, traits>::int_type c)
{
charT* ibegin = this->out_buf_;
charT* iend = this->pptr();
// Reset the put pointers to indicate that the buffer is free
// (at least it will be at the end of this function).
setp(out_buf_, out_buf_ + BUFF_SIZE + 1);
// If this is the end, add an eof character to the buffer.
// This is why the pointers passed to setp are off by 1
// (to reserve room for this).
if(!traits_type::eq_int_type(c, traits_type::eof())) {
*iend++ = traits_type::to_char_type(c);
}
// Compute the write length.
int_type ilen = iend - ibegin;
// ADDED: restore cout to its original stream, output to it, output to the file, then set cout's stream back to this, our streambuf)
std::cout.rdbuf(original_cout);
out_buf_[ilen] = '\0';
std::cout << out_buf_;
outfile << out_buf_;
std::cout.rdbuf(this);
return traits_type::not_eof(c);
}
// This is called to flush the buffer.
// This is called when we're done with the file stream (or when .flush() is called).
template <typename charT, typename traits>
typename mystreambuf<charT, traits>::int_type
mystreambuf<charT, traits>::sync()
{
return traits_type::eq_int_type(this->overflow(traits_type::eof()),
traits_type::eof()) ? -1 : 0;
}
int main(int argc, char* argv[]) {
mystreambuf<char> filter(std::cout.rdbuf());
std::cout.rdbuf( &filter );
std::cout << "Hello World" << std::endl;
return 0;
}
hope this helps; cheers
Related
Is there a (cross-platform) way to get a C FILE* handle from a C++ std::fstream ?
The reason I ask is because my C++ library accepts fstreams and in one particular function I'd like to use a C library that accepts a FILE*.
The short answer is no.
The reason, is because the std::fstream is not required to use a FILE* as part of its implementation. So even if you manage to extract file descriptor from the std::fstream object and manually build a FILE object, then you will have other problems because you will now have two buffered objects writing to the same file descriptor.
The real question is why do you want to convert the std::fstream object into a FILE*?
Though I don't recommend it, you could try looking up funopen().
Unfortunately, this is not a POSIX API (it's a BSD extension) so its portability is in question. Which is also probably why I can't find anybody that has wrapped a std::stream with an object like this.
FILE *funopen(
const void *cookie,
int (*readfn )(void *, char *, int),
int (*writefn)(void *, const char *, int),
fpos_t (*seekfn) (void *, fpos_t, int),
int (*closefn)(void *)
);
This allows you to build a FILE object and specify some functions that will be used to do the actual work. If you write appropriate functions you can get them to read from the std::fstream object that actually has the file open.
There isn't a standardized way. I assume this is because the C++ standardization group didn't want to assume that a file handle can be represented as a fd.
Most platforms do seem to provide some non-standard way to do this.
http://www.ginac.de/~kreckel/fileno/ provides a good writeup of the situation and provides code that hides all the platform specific grossness, at least for GCC. Given how gross this is just on GCC, I think I'd avoid doing this all together if possible.
UPDATE: See #Jettatura what I think it is the best answer https://stackoverflow.com/a/33612982/225186 (Linux only?).
ORIGINAL:
(Probably not cross platform, but simple)
Simplifying the hack in http://www.ginac.de/~kreckel/fileno/ (dvorak answer), and looking at this gcc extension http://gcc.gnu.org/onlinedocs/gcc-4.6.2/libstdc++/api/a00069.html#a59f78806603c619eafcd4537c920f859,
I have this solution that works on GCC (4.8 at least) and clang (3.3 at least) before C++11:
#include<fstream>
#include<ext/stdio_filebuf.h>
typedef std::basic_ofstream<char>::__filebuf_type buffer_t;
typedef __gnu_cxx::stdio_filebuf<char> io_buffer_t;
FILE* cfile_impl(buffer_t* const fb){
return (static_cast<io_buffer_t* const>(fb))->file(); //type std::__c_file
}
FILE* cfile(std::ofstream const& ofs){return cfile_impl(ofs.rdbuf());}
FILE* cfile(std::ifstream const& ifs){return cfile_impl(ifs.rdbuf());}
and can be used this,
int main(){
std::ofstream ofs("file.txt");
fprintf(cfile(ofs), "sample1");
fflush(cfile(ofs)); // ofs << std::flush; doesn't help
ofs << "sample2\n";
}
Note: The stdio_filebuf is not used in newer versions of the library. The static_cast<>() is somewhat dangerous too. Use a dynamic_cast<>() instead of if you get a nullptr you need that's not the right class. You can try with stdio_sync_filebuf instead. Problem with that class is that the file() is not available at all anymore.
Limitations: (comments are welcome)
I find that it is important to fflush after fprintf printing to std::ofstream, otherwise the "sample2" appears before "sample1" in the example above. I don't know if there is a better workaround for that than using fflush. Notably ofs << flush doesn't help.
Cannot extract FILE* from std::stringstream, I don't even know if it is possible. (see below for an update).
I still don't know how to extract C's stderr from std::cerr etc., for example to use in fprintf(stderr, "sample"), in an hypothetical code like this fprintf(cfile(std::cerr), "sample").
Regarding the last limitation, the only workaround I found is to add these overloads:
FILE* cfile(std::ostream const& os){
if(std::ofstream const* ofsP = dynamic_cast<std::ofstream const*>(&os)) return cfile(*ofsP);
if(&os == &std::cerr) return stderr;
if(&os == &std::cout) return stdout;
if(&os == &std::clog) return stderr;
if(dynamic_cast<std::ostringstream const*>(&os) != 0){
throw std::runtime_error("don't know cannot extract FILE pointer from std::ostringstream");
}
return 0; // stream not recognized
}
FILE* cfile(std::istream const& is){
if(std::ifstream const* ifsP = dynamic_cast<std::ifstream const*>(&is)) return cfile(*ifsP);
if(&is == &std::cin) return stdin;
if(dynamic_cast<std::ostringstream const*>(&is) != 0){
throw std::runtime_error("don't know how to extract FILE pointer from std::istringstream");
}
return 0; // stream not recognized
}
Attempt to handle iostringstream
It is possible to read with fscanf from istream using fmemopen, but that requires a lot of book keeping and updating the input position of the stream after each read, if one wants to combine C-reads and C++-reads. I wasn't able to convert this into a cfile function like above. (Maybe a cfile class that keeps updating after each read is the way to go).
// hack to access the protected member of istreambuf that know the current position
char* access_gptr(std::basic_streambuf<char, std::char_traits<char>>& bs){
struct access_class : std::basic_streambuf<char, std::char_traits<char>>{
char* access_gptr() const{return this->gptr();}
};
return ((access_class*)(&bs))->access_gptr();
}
int main(){
std::istringstream iss("11 22 33");
// read the C++ way
int j1; iss >> j1;
std::cout << j1 << std::endl;
// read the C way
float j2;
char* buf = access_gptr(*iss.rdbuf()); // get current position
size_t buf_size = iss.rdbuf()->in_avail(); // get remaining characters
FILE* file = fmemopen(buf, buf_size, "r"); // open buffer memory as FILE*
fscanf(file, "%f", &j2); // finally!
iss.rdbuf()->pubseekoff(ftell(file), iss.cur, iss.in); // update input stream position from current FILE position.
std::cout << "j2 = " << j2 << std::endl;
// read again the C++ way
int j3; iss >> j3;
std::cout << "j3 = " << j3 << std::endl;
}
Well, you can get the file descriptor - I forget whether the method is fd() or getfd(). The implementations I've used provide such methods, but the language standard doesn't require them, I believe - the standard shouldn't care whether your platform uses fd's for files.
From that, you can use fdopen(fd, mode) to get a FILE*.
However, I think that the mechanisms the standard requires for synching STDIN/cin, STDOUT/cout and STDERR/cerr don't have to be visible to you. So if you're using both the fstream and FILE*, buffering may mess you up.
Also, if either the fstream OR the FILE closes, they'll probably close the underlying fd, so you need to make sure you flush BOTH before closing EITHER.
In a single-threaded POSIX application you can easily get the fd number in a portable way:
int fd = dup(0);
close(fd);
// POSIX requires the next opened file descriptor to be fd.
std::fstream file(...);
// now fd has been opened again and is owned by file
This method breaks in a multi-threaded application if this code races with other threads opening file descriptors.
yet another way to do this in Linux:
#include <stdio.h>
#include <cassert>
template<class STREAM>
struct STDIOAdapter
{
static FILE* yield(STREAM* stream)
{
assert(stream != NULL);
static cookie_io_functions_t Cookies =
{
.read = NULL,
.write = cookieWrite,
.seek = NULL,
.close = cookieClose
};
return fopencookie(stream, "w", Cookies);
}
ssize_t static cookieWrite(void* cookie,
const char* buf,
size_t size)
{
if(cookie == NULL)
return -1;
STREAM* writer = static_cast <STREAM*>(cookie);
writer->write(buf, size);
return size;
}
int static cookieClose(void* cookie)
{
return EOF;
}
}; // STDIOAdapter
Usage, for example:
#include <boost/iostreams/filtering_stream.hpp>
#include <boost/iostreams/filter/bzip2.hpp>
#include <boost/iostreams/device/file.hpp>
using namespace boost::iostreams;
int main()
{
filtering_ostream out;
out.push(boost::iostreams::bzip2_compressor());
out.push(file_sink("my_file.txt"));
FILE* fp = STDIOAdapter<filtering_ostream>::yield(&out);
assert(fp > 0);
fputs("Was up, Man", fp);
fflush (fp);
fclose(fp);
return 1;
}
There is a way to get file descriptor from fstream and then convert it to FILE* (via fdopen). Personally I don't see any need in FILE*, but with file descriptor you may do many interesting things such as redirecting (dup2).
Solution:
#define private public
#define protected public
#include <fstream>
#undef private
#undef protected
std::ifstream file("some file");
auto fno = file._M_filebuf._M_file.fd();
The last string works for libstdc++. If you are using some other library you will need to reverse-engineer it a bit.
This trick is dirty and will expose all private and public members of fstream. If you would like to use it in your production code I suggest you to create separate .cpp and .h with single function int getFdFromFstream(std::basic_ios<char>& fstr);. Header file must not include fstream.
I ran in that problem when I was faced with isatty() only working on a file descriptor.
In newer versions of the C++ standard library (at least since C++11), the solution proposed by alfC does not work anymore because that one class was changed to a new class.
The old method will still work if you use very old versions of the compiler. In newer version, you need to use std::basic_filebuf<>(). But that does not work with the standard I/O such as std::cout. For those, you need to use __gnu_cxx::stdio_sync_filebuf<>().
I have a functional example in my implementation of isatty() for C++ streams here. You should be able to lift off that one file and reuse it in your own project. In your case, though, you wanted the FILE* pointer, so just return that instead of the result of ::isatty(fileno(<of FILE*>)).
Here is a copy of the template function:
template<typename _CharT
, typename _Traits = std::char_traits<_CharT>>
bool isatty(std::basic_ios<_CharT, _Traits> const & s)
{
{ // cin, cout, cerr, and clog
typedef __gnu_cxx::stdio_sync_filebuf<_CharT, _Traits> io_sync_buffer_t;
io_sync_buffer_t * buffer(dynamic_cast<io_sync_buffer_t *>(s.rdbuf()));
if(buffer != nullptr)
{
return ::isatty(fileno(buffer->file()));
}
}
{ // modern versions
typedef std::basic_filebuf<_CharT, _Traits> file_buffer_t;
file_buffer_t * file_buffer(dynamic_cast<file_buffer_t *>(s.rdbuf()));
if(file_buffer != nullptr)
{
typedef detail::our_basic_filebuf<_CharT, _Traits> hack_buffer_t;
hack_buffer_t * buffer(static_cast<hack_buffer_t *>(file_buffer));
if(buffer != nullptr)
{
return ::isatty(fileno(buffer->file()));
}
}
}
{ // older versions
typedef __gnu_cxx::stdio_filebuf<_CharT, _Traits> io_buffer_t;
io_buffer_t * buffer(dynamic_cast<io_buffer_t *>(s.rdbuf()));
if(buffer != nullptr)
{
return ::isatty(fileno(buffer->file()));
}
}
return false;
}
Now, you should be asking: But what is that detail class our_basic_filebuf?!?
And that's a good question. The fact is that the _M_file pointer is protected and there is no file() (or fd()) in the std::basic_filebuf. For that reason, I created a shell class which has access to the protected fields and that way I can return the FILE* pointer.
template<typename _CharT
, typename _Traits = std::char_traits<_CharT>>
class our_basic_filebuf
: public std::basic_filebuf<_CharT, _Traits>
{
public:
std::__c_file * file() throw()
{
return this->_M_file.file();
}
};
This is somewhat ugly, but cleanest I could think off to gain access to the _M_file field.
I am writing a program to that regularly stores and reads structs in the form below.
struct Node {
int leftChild = 0;
int rightChild = 0;
std::string value;
int count = 1;
int balanceFactor = 0;
};
How would I read and write nodes to a file? I would like to use the fstream class with seekg and seekp to do the serialization manually but I'm not sure how it works based off of the documentation and am struggling with finding decent examples.
[edit] specified that i do not want to use a serialization library.
This problem is known as serialization. Use a serializing library like e.g. Google's Protocol Buffers or Flatbuffers.
To serialize objects, you will need to stick to the concept that the object is writing its members to the stream and reading members from the stream. Also, member objects should write themselves to the stream (as well as read).
I implemented a scheme using three member functions, and a buffer:
void load_from_buffer(uint8_t * & buffer_pointer);
void store_to_buffer(uint8_t * & buffer_pointer) const;
unsigned int size_on_stream() const;
The size_on_stream would be called first in order to determine the buffer size for the object (or how much space it occupied in the buffer).
The load_from_buffer function loads the object's members from a buffer using the given pointer. The function also increments the pointer appropriately.
The store_to_buffer function stores the objects's members to a buffer using the given pointer. The function also increments the pointer appropriately.
This can be applied to POD types by using templates and template specializations.
These functions also allow you to pack the output into the buffer, and load from a packed format.
The reason for I/O to the buffer is so you can use the more efficient block stream methods, such as write and read.
Edit 1: Writing a node to a stream
The problem with writing or serializing a node (such a linked list or tree node) is that pointers don't translate to a file. There is no guarantee that the OS will place your program in the same memory location or give you the same area of memory each time.
You have two options: 1) Only store the data. 2) Convert the pointers to file offsets. Option 2) is very complicated as it may require repositioning the file pointer because file offsets may not be known ahead of time.
Also, be aware of variable length records like strings. You can't directly write a string object to a file. Unless you use a fixed string width, the string size will change. You will either need to prefix the string with the string length (preferred) or use some kind of terminating character, such as '\0'. The string length first is preferred because you don't have to search for the end of the string; you can use a block read to read in the text.
If you replace the std::string by a char buffer, you can use fwrite and fread to write/read your structure to and from disk as a fixed size block of information. Within a single program that should work ok.
The big bug-a-boo is the fact that compilers will insert padding between fields in order to keep the data aligned. That makes the code less portable as if a module is compiled with different alignment requirements the structure literally can be a different size, throwing your fixed size assumption out the door.
I would lean toward a well worn in serialization library of some sort.
Another approach would be to overload the operator<< and operator>> for the structure so that it knows how to save/load itself. That would reduce the problem to knowing where to read/write the node. In theory, your left and right child fields could be seek addresses to where the nodes actually reside, while a new field could hold the seek location of the current node.
When implementing your own serialization method, the first decision you'll have to make is whether you want the data on disk to be in binary format or textual format.
I find it easier to implement the ability to save to a binary format. The number of functions needed to implement that is small. You need to implement functions that can write the fundamental types, arrays of known size at compile time, dynamic arrays and strings. Everything else can be built on top of those.
Here's something very close to what I recently put into production code.
#include <cstring>
#include <fstream>
#include <cstddef>
#include <stdexcept>
// Class to write to a stream
struct Writer
{
std::ostream& out_;
Writer(std::ostream& out) : out_(out) {}
// Write the fundamental types
template <typename T>
void write(T number)
{
out_.write(reinterpret_cast<char const*>(&number), sizeof(number));
if (!out_ )
{
throw std::runtime_error("Unable to write a number");
}
}
// Write arrays whose size is known at compile time
template <typename T, uint64_t N>
void write(T (&array)[N])
{
for(uint64_t i = 0; i < N; ++i )
{
write(array[i]);
}
}
// Write dynamic arrays
template <typename T>
void write(T array[], uint64_t size)
{
write(size);
for(uint64_t i = 0; i < size; ++i )
{
write(array[i]);
}
}
// Write strings
void write(std::string const& str)
{
write(str.c_str(), str.size());
}
void write(char const* str)
{
write(str, std::strlen(str));
}
};
// Class to read from a stream
struct Reader
{
std::ifstream& in_;
Reader(std::ifstream& in) : in_(in) {}
template <typename T>
void read(T& number)
{
in_.read(reinterpret_cast<char*>(&number), sizeof(number));
if (!in_ )
{
throw std::runtime_error("Unable to read a number.");
}
}
template <typename T, uint64_t N>
void read(T (&array)[N])
{
for(uint64_t i = 0; i < N; ++i )
{
read(array[i]);
}
}
template <typename T>
void read(T*& array)
{
uint64_t size;
read(size);
array = new T[size];
for(uint64_t i = 0; i < size; ++i )
{
read(array[i]);
}
}
void read(std::string& str)
{
char* s;
read(s);
str = s;
delete [] s;
}
};
// Test the code.
#include <iostream>
void writeData(std::string const& file)
{
std::ofstream out(file);
Writer w(out);
w.write(10);
w.write(20.f);
w.write(200.456);
w.write("Test String");
}
void readData(std::string const& file)
{
std::ifstream in(file);
Reader r(in);
int i;
r.read(i);
std::cout << "i: " << i << std::endl;
float f;
r.read(f);
std::cout << "f: " << f << std::endl;
double d;
r.read(d);
std::cout << "d: " << d << std::endl;
std::string s;
r.read(s);
std::cout << "s: " << s << std::endl;
}
void testWriteAndRead(std::string const& file)
{
writeData(file);
readData(file);
}
int main()
{
testWriteAndRead("test.bin");
return 0;
}
Output:
i: 10
f: 20
d: 200.456
s: Test String
The ability to write and read a Node is very easily implemented.
void write(Writer& w, Node const& n)
{
w.write(n.leftChild);
w.write(n.rightChild);
w.write(n.value);
w.write(n.count);
w.write(n.balanceFactor);
}
void read(Reader& r, Node& n)
{
r.read(n.leftChild);
r.read(n.rightChild);
r.read(n.value);
r.read(n.count);
r.read(n.balanceFactor);
}
The process you are referring to are known as serialization. I'd recommend Cereal at http://uscilab.github.io/cereal/
It supports both json, xml and binary serialization and is very easy to use (with good examples).
(Unfortunately it does not support my favourite format yaml)
I am reading a gzipped file using boost iostreams:
The following works fine:
namespace io = boost::iostreams;
io::filtering_istream in;
in.push(boost::iostreams::basic_gzip_decompressor<>());
in.push(io::file_source("test.gz"));
stringstream ss;
copy(in, ss);
However, I don't want to take the memory hit of reading an entire gzipped file
into memory. I want to be able to read the file incrementally.
For example, if I have a data structure X that initializes itself from istream,
X x;
x.read(in);
fails. Presumably this is because we may have to put back characters into the stream
if we are doing partial streams. Any ideas whether boost iostreams supports this?
According to the iostream documentation the type boost::io::filtering_istream derives from std::istream. That is, it should be possible to pass this everywhere an std::istream& is expected. If you have errors at run-time because you need to unget() or putback() characters you should have a look at the pback_size parameter which specifies how many characters are return at most. I haven't seen in the documentation what the default value for this parameter is.
If this doesn't solve your problem can you describe what your problem is exactly? From the looks of it should work.
I think you need to write your own filter. For instance, to read a .tar.gz and output the files contained, I wrote something like
//using namespace std;
namespace io = boost::iostreams;
struct tar_expander
{
tar_expander() : out(0), status(header)
{
}
~tar_expander()
{
delete out;
}
/* qualify filter */
typedef char char_type;
struct category :
io::input_filter_tag,
io::multichar_tag
{ };
template<typename Source>
void fetch_n(Source& src, std::streamsize n = block_size)
{
/* my utility */
....
}
// Read up to n filtered characters into the buffer s,
// returning the number of characters read or -1 for EOF.
// Use src to access the unfiltered character sequence
template<typename Source>
std::streamsize read(Source& src, char* s, std::streamsize n)
{
fetch_n(src);
const tar_header &h = cast_buf<tar_header>();
int r;
if (status == header)
{
...
}
std::ofstream *out;
size_t fsize, stored;
static const size_t block_size = 512;
std::vector<char> buf;
enum { header, store_file, archive_end } status;
}
}
My function read(Source &...) when called receives the unzipped text.
To use the filter:
ifstream file("/home/..../resample-1.8.1.tar.gz", ios_base::in | ios_base::binary);
io::filtering_streambuf<io::input> in;
in.push(tar_expander());
in.push(io::gzip_decompressor());
in.push(file);
io::copy(in, cout);
I am using win32's ReadFile to read from a child process's pipe. That gives me chunks of characters at a time and the size of each chunk, but they may or may not have new lines. I want to process the output line by line. What is the simplest way of doing so?
I thought about appending each chunk to a string, and at the end, using a stringstream to process it line by line, but I'd like to do this as data is coming in. I suppose the trick is, how and where should I detect the new line ending? If only streamreader's getline returned nothing when no delimiter is found...
Append to a string until you encounter newline char or end of data. Then you have a line in the string, process it, empty string and repeat. Reason for using the string: that you don't know how long a line may be, and a string does the reallocation etc. for you as necessary.
Special case: end of data with nothing earlier on that line should probably be not-a-line, ignored.
Cheers & hth.
What I can think of is using a buffer where to store chunks coming to you, you know the size via lpNumberOfBytesRead, so on each chunk you check if it contains new line character/s and if it does, you output all the buffered characters till the new line character, then start buffering until you received another chunk with new line characters and so on.
Some pseudo code might look like:
w_char buffer[BUFFER_SIZE]; // enough for few chunks (use new to allocate dynamic memory)
ReadLine(hFile, lpBuffer, nNumberOfBytesToRead, lpNumberOfBytesRead, lpOverlapped);
if (w_char has new_line_character) {
process_whole_line(); // don't forget to clear up the buffer
}
You can extend std::basic_streambuf and implement xsputn so it stores data in an internal buffer, and detect newlines or do whatever processing you need. If you want to process only complete lines, upon detection you can push the buffer to a std::queue up to the newline, and erase the respective part from the buffer. You'll need to implement overflow too. For example:
template<typename CharT, typename Traits = std::char_traits<CharT> >
class line_streambuf : public std::basic_streambuf<CharT, Traits> {
public:
typedef CharT char_type;
typedef Traits traits_type;
typedef std::basic_string<char_type> string_type;
typedef typename string_type::size_type size_type;
typedef typename traits_type::int_type int_type;
line_streambuf(char_type separator = '\n') : _separator(separator) {}
virtual ~line_streambuf() {}
bool getline(string_type& output) { /* pop from the queue and return */ }
protected:
virtual int_type overflow(int_type v) {
if (v == _separator) {
_processed.push(_buffer);
_buffer.erase(_buffer.begin(), _buffer.end());
} else {
_buffer += v;
}
return v;
}
virtual std::streamsize xsputn(const char_type* p, std::streamsize n) {
_buffer.append(p, p + n);
while (true) {
// The 1st find could be smarter - finding only after old_length+p
size_type pos = _buffer.find(_separator);
if (pos == string_type::npos)
break;
_processed.push(string_type(_buffer.begin(), _buffer.begin()+pos));
_buffer.erase(_buffer.begin(), _buffer.begin() + pos + 1);
}
return n;
}
private:
char_type _separator;
string_type _buffer;
std::queue<string_type> _processed;
};
NOTE: highly untested code. Let me know if you find a problem, or feel free to edit.
Is there a (cross-platform) way to get a C FILE* handle from a C++ std::fstream ?
The reason I ask is because my C++ library accepts fstreams and in one particular function I'd like to use a C library that accepts a FILE*.
The short answer is no.
The reason, is because the std::fstream is not required to use a FILE* as part of its implementation. So even if you manage to extract file descriptor from the std::fstream object and manually build a FILE object, then you will have other problems because you will now have two buffered objects writing to the same file descriptor.
The real question is why do you want to convert the std::fstream object into a FILE*?
Though I don't recommend it, you could try looking up funopen().
Unfortunately, this is not a POSIX API (it's a BSD extension) so its portability is in question. Which is also probably why I can't find anybody that has wrapped a std::stream with an object like this.
FILE *funopen(
const void *cookie,
int (*readfn )(void *, char *, int),
int (*writefn)(void *, const char *, int),
fpos_t (*seekfn) (void *, fpos_t, int),
int (*closefn)(void *)
);
This allows you to build a FILE object and specify some functions that will be used to do the actual work. If you write appropriate functions you can get them to read from the std::fstream object that actually has the file open.
There isn't a standardized way. I assume this is because the C++ standardization group didn't want to assume that a file handle can be represented as a fd.
Most platforms do seem to provide some non-standard way to do this.
http://www.ginac.de/~kreckel/fileno/ provides a good writeup of the situation and provides code that hides all the platform specific grossness, at least for GCC. Given how gross this is just on GCC, I think I'd avoid doing this all together if possible.
UPDATE: See #Jettatura what I think it is the best answer https://stackoverflow.com/a/33612982/225186 (Linux only?).
ORIGINAL:
(Probably not cross platform, but simple)
Simplifying the hack in http://www.ginac.de/~kreckel/fileno/ (dvorak answer), and looking at this gcc extension http://gcc.gnu.org/onlinedocs/gcc-4.6.2/libstdc++/api/a00069.html#a59f78806603c619eafcd4537c920f859,
I have this solution that works on GCC (4.8 at least) and clang (3.3 at least) before C++11:
#include<fstream>
#include<ext/stdio_filebuf.h>
typedef std::basic_ofstream<char>::__filebuf_type buffer_t;
typedef __gnu_cxx::stdio_filebuf<char> io_buffer_t;
FILE* cfile_impl(buffer_t* const fb){
return (static_cast<io_buffer_t* const>(fb))->file(); //type std::__c_file
}
FILE* cfile(std::ofstream const& ofs){return cfile_impl(ofs.rdbuf());}
FILE* cfile(std::ifstream const& ifs){return cfile_impl(ifs.rdbuf());}
and can be used this,
int main(){
std::ofstream ofs("file.txt");
fprintf(cfile(ofs), "sample1");
fflush(cfile(ofs)); // ofs << std::flush; doesn't help
ofs << "sample2\n";
}
Note: The stdio_filebuf is not used in newer versions of the library. The static_cast<>() is somewhat dangerous too. Use a dynamic_cast<>() instead of if you get a nullptr you need that's not the right class. You can try with stdio_sync_filebuf instead. Problem with that class is that the file() is not available at all anymore.
Limitations: (comments are welcome)
I find that it is important to fflush after fprintf printing to std::ofstream, otherwise the "sample2" appears before "sample1" in the example above. I don't know if there is a better workaround for that than using fflush. Notably ofs << flush doesn't help.
Cannot extract FILE* from std::stringstream, I don't even know if it is possible. (see below for an update).
I still don't know how to extract C's stderr from std::cerr etc., for example to use in fprintf(stderr, "sample"), in an hypothetical code like this fprintf(cfile(std::cerr), "sample").
Regarding the last limitation, the only workaround I found is to add these overloads:
FILE* cfile(std::ostream const& os){
if(std::ofstream const* ofsP = dynamic_cast<std::ofstream const*>(&os)) return cfile(*ofsP);
if(&os == &std::cerr) return stderr;
if(&os == &std::cout) return stdout;
if(&os == &std::clog) return stderr;
if(dynamic_cast<std::ostringstream const*>(&os) != 0){
throw std::runtime_error("don't know cannot extract FILE pointer from std::ostringstream");
}
return 0; // stream not recognized
}
FILE* cfile(std::istream const& is){
if(std::ifstream const* ifsP = dynamic_cast<std::ifstream const*>(&is)) return cfile(*ifsP);
if(&is == &std::cin) return stdin;
if(dynamic_cast<std::ostringstream const*>(&is) != 0){
throw std::runtime_error("don't know how to extract FILE pointer from std::istringstream");
}
return 0; // stream not recognized
}
Attempt to handle iostringstream
It is possible to read with fscanf from istream using fmemopen, but that requires a lot of book keeping and updating the input position of the stream after each read, if one wants to combine C-reads and C++-reads. I wasn't able to convert this into a cfile function like above. (Maybe a cfile class that keeps updating after each read is the way to go).
// hack to access the protected member of istreambuf that know the current position
char* access_gptr(std::basic_streambuf<char, std::char_traits<char>>& bs){
struct access_class : std::basic_streambuf<char, std::char_traits<char>>{
char* access_gptr() const{return this->gptr();}
};
return ((access_class*)(&bs))->access_gptr();
}
int main(){
std::istringstream iss("11 22 33");
// read the C++ way
int j1; iss >> j1;
std::cout << j1 << std::endl;
// read the C way
float j2;
char* buf = access_gptr(*iss.rdbuf()); // get current position
size_t buf_size = iss.rdbuf()->in_avail(); // get remaining characters
FILE* file = fmemopen(buf, buf_size, "r"); // open buffer memory as FILE*
fscanf(file, "%f", &j2); // finally!
iss.rdbuf()->pubseekoff(ftell(file), iss.cur, iss.in); // update input stream position from current FILE position.
std::cout << "j2 = " << j2 << std::endl;
// read again the C++ way
int j3; iss >> j3;
std::cout << "j3 = " << j3 << std::endl;
}
Well, you can get the file descriptor - I forget whether the method is fd() or getfd(). The implementations I've used provide such methods, but the language standard doesn't require them, I believe - the standard shouldn't care whether your platform uses fd's for files.
From that, you can use fdopen(fd, mode) to get a FILE*.
However, I think that the mechanisms the standard requires for synching STDIN/cin, STDOUT/cout and STDERR/cerr don't have to be visible to you. So if you're using both the fstream and FILE*, buffering may mess you up.
Also, if either the fstream OR the FILE closes, they'll probably close the underlying fd, so you need to make sure you flush BOTH before closing EITHER.
In a single-threaded POSIX application you can easily get the fd number in a portable way:
int fd = dup(0);
close(fd);
// POSIX requires the next opened file descriptor to be fd.
std::fstream file(...);
// now fd has been opened again and is owned by file
This method breaks in a multi-threaded application if this code races with other threads opening file descriptors.
yet another way to do this in Linux:
#include <stdio.h>
#include <cassert>
template<class STREAM>
struct STDIOAdapter
{
static FILE* yield(STREAM* stream)
{
assert(stream != NULL);
static cookie_io_functions_t Cookies =
{
.read = NULL,
.write = cookieWrite,
.seek = NULL,
.close = cookieClose
};
return fopencookie(stream, "w", Cookies);
}
ssize_t static cookieWrite(void* cookie,
const char* buf,
size_t size)
{
if(cookie == NULL)
return -1;
STREAM* writer = static_cast <STREAM*>(cookie);
writer->write(buf, size);
return size;
}
int static cookieClose(void* cookie)
{
return EOF;
}
}; // STDIOAdapter
Usage, for example:
#include <boost/iostreams/filtering_stream.hpp>
#include <boost/iostreams/filter/bzip2.hpp>
#include <boost/iostreams/device/file.hpp>
using namespace boost::iostreams;
int main()
{
filtering_ostream out;
out.push(boost::iostreams::bzip2_compressor());
out.push(file_sink("my_file.txt"));
FILE* fp = STDIOAdapter<filtering_ostream>::yield(&out);
assert(fp > 0);
fputs("Was up, Man", fp);
fflush (fp);
fclose(fp);
return 1;
}
There is a way to get file descriptor from fstream and then convert it to FILE* (via fdopen). Personally I don't see any need in FILE*, but with file descriptor you may do many interesting things such as redirecting (dup2).
Solution:
#define private public
#define protected public
#include <fstream>
#undef private
#undef protected
std::ifstream file("some file");
auto fno = file._M_filebuf._M_file.fd();
The last string works for libstdc++. If you are using some other library you will need to reverse-engineer it a bit.
This trick is dirty and will expose all private and public members of fstream. If you would like to use it in your production code I suggest you to create separate .cpp and .h with single function int getFdFromFstream(std::basic_ios<char>& fstr);. Header file must not include fstream.
I ran in that problem when I was faced with isatty() only working on a file descriptor.
In newer versions of the C++ standard library (at least since C++11), the solution proposed by alfC does not work anymore because that one class was changed to a new class.
The old method will still work if you use very old versions of the compiler. In newer version, you need to use std::basic_filebuf<>(). But that does not work with the standard I/O such as std::cout. For those, you need to use __gnu_cxx::stdio_sync_filebuf<>().
I have a functional example in my implementation of isatty() for C++ streams here. You should be able to lift off that one file and reuse it in your own project. In your case, though, you wanted the FILE* pointer, so just return that instead of the result of ::isatty(fileno(<of FILE*>)).
Here is a copy of the template function:
template<typename _CharT
, typename _Traits = std::char_traits<_CharT>>
bool isatty(std::basic_ios<_CharT, _Traits> const & s)
{
{ // cin, cout, cerr, and clog
typedef __gnu_cxx::stdio_sync_filebuf<_CharT, _Traits> io_sync_buffer_t;
io_sync_buffer_t * buffer(dynamic_cast<io_sync_buffer_t *>(s.rdbuf()));
if(buffer != nullptr)
{
return ::isatty(fileno(buffer->file()));
}
}
{ // modern versions
typedef std::basic_filebuf<_CharT, _Traits> file_buffer_t;
file_buffer_t * file_buffer(dynamic_cast<file_buffer_t *>(s.rdbuf()));
if(file_buffer != nullptr)
{
typedef detail::our_basic_filebuf<_CharT, _Traits> hack_buffer_t;
hack_buffer_t * buffer(static_cast<hack_buffer_t *>(file_buffer));
if(buffer != nullptr)
{
return ::isatty(fileno(buffer->file()));
}
}
}
{ // older versions
typedef __gnu_cxx::stdio_filebuf<_CharT, _Traits> io_buffer_t;
io_buffer_t * buffer(dynamic_cast<io_buffer_t *>(s.rdbuf()));
if(buffer != nullptr)
{
return ::isatty(fileno(buffer->file()));
}
}
return false;
}
Now, you should be asking: But what is that detail class our_basic_filebuf?!?
And that's a good question. The fact is that the _M_file pointer is protected and there is no file() (or fd()) in the std::basic_filebuf. For that reason, I created a shell class which has access to the protected fields and that way I can return the FILE* pointer.
template<typename _CharT
, typename _Traits = std::char_traits<_CharT>>
class our_basic_filebuf
: public std::basic_filebuf<_CharT, _Traits>
{
public:
std::__c_file * file() throw()
{
return this->_M_file.file();
}
};
This is somewhat ugly, but cleanest I could think off to gain access to the _M_file field.