I am reading from a pipe (Linux) or a pipe-like device object (Windows) using std::ifstream::read. However, when there is no more data, read reads 0 bytes and sets EOF. Is there a way to make a blocking read from an ifstream, such that it only returns when there is some more data?
I'd rather not busy wait for the EOF flag to clear.
If it is not possible with the C++ standard library, what is the closest other option? Can I do it in plain C, or do I have to resort to operating system specific APIs?
Unfortunately, std is very poor on any non-algorithmic functionality, like IO. You always have to rely on 3rd-party solutions. Fortunately, there is Boost and, if you do not mind, I will suggest to use it to reduce OS specific code.
namespace bs = boost::iostreams;
int fd; // Create, for example, Posix file descriptor and specify necessary flags for it.
bs::file_descriptor_source fds(fd);
bs::stream<bs::file_descriptor_source> stream(fds);
// Work with the stream as it is std stream
In this small example I use Boost IO Streams and specifically file_descriptor_source that works as an underlying stream device and hides Windows or Posix specific pipe inside. The pipe you open yourself, so you can configure the pipe as you want.
well there seems no way to do a blocking read. clearing the error bit will not help. Only a re-open of the fifo like in this example:
int main(int argc, char **argv)
{
int rc=0;
enum FATAL {ERR_ARGV,ERR_OPEN_FILE};
try
{
if( argv[1] == NULL) throw ERR_ARGV;
std::ifstream fifo;
while(1)
{
fifo.open(argv[1],std::ifstream::in);
if( !fifo.is_open() ) throw ERR_OPEN_FILE;
std::string line;
while(std::getline(fifo,line))
{
std::cout << line << "\n"; fflush(stdout);
}
fifo.close();
}
// never should come here
}
catch(FATAL e)
{
rc=e;
switch(e)
{
case ERR_ARGV:
std::cerr << "ERROR: argument 1 should be a fifo file name\n";
break;
case ERR_OPEN_FILE:
std::cerr << "ERROR: unabel to open file " << argv[1] << "\n";
break;
}
}
return(rc);
}
I have tested this code and it works to do an endless read from a fifo.
Related
I want to do a simple thing: read the first line from a file, and do a proper error reporting in case there is no such file, no permission to read the file and so on.
I considered the following options:
std::ifstream. Unfortunately, there is no portable way to report system errors. Some other answers suggest checking errno after reading failed, but the standard does not guarantee that errno is set by any functions in iostreams library.
C style fopen/fread/fclose. This works, but is not as convenient as iostreams with std::getline. I'm looking for C++ solution.
Is there any way to accomplish this using C++14 and boost?
Disclaimer: I am the author of AFIO. But exactly what you are looking for is https://ned14.github.io/afio/ which is the v2 library incorporating the feedback from its Boost peer review in August 2015. See the list of features here.
I will of course caveat that this is an alpha quality library, and you should not use it in production code. However, quite a few people already are doing so.
How to use AFIO to solve the OP's problem:
Note that AFIO is a very low level library, hence you have to type a lot more code to achieve the same as iostreams, on the other hand you get no memory allocation, no exception throwing, no unpredictable latency spikes:
// Try to read first line from file at path, returning no string if file does not exist,
// throwing exception for any other error
optional<std::string> read_first_line(filesystem::path path)
{
using namespace AFIO_V2_NAMESPACE;
// The result<T> is from WG21 P0762, it looks quite like an `expected<T, std::error_code>` object
// See Outcome v2 at https://ned14.github.io/outcome/ and https://lists.boost.org/boost-announce/2017/06/0510.php
// Open for reading the file at path using a null handle as the base
result<file_handle> _fh = file({}, path);
// If fh represents failure ...
if(!_fh)
{
// Fetch the error code
std::error_code ec = _fh.error();
// Did we fail due to file not found?
// It is *very* important to note that ec contains the *original* error code which could
// be POSIX, or Win32 or NT kernel error code domains. However we can always compare,
// via 100% C++ 11 STL, any error code to a generic error *condition* for equivalence
// So this comparison will work as expected irrespective of original error code.
if(ec == std::errc::no_such_file_or_directory)
{
// Return empty optional
return {};
}
std::cerr << "Opening file " << path << " failed with " << ec.message() << std::endl;
}
// If errored, result<T>.value() throws an error code failure as if `throw std::system_error(fh.error());`
// Otherwise unpack the value containing the valid file_handle
file_handle fh(std::move(_fh.value()));
// Configure the scatter buffers for the read, ideally aligned to a page boundary for DMA
alignas(4096) char buffer[4096];
// There is actually a faster to type shortcut for this, but I thought best to spell it out
file_handle::buffer_type reqs[] = {{buffer, sizeof(buffer)}};
// Do a blocking read from offset 0 possibly filling the scatter buffers passed in
file_handle::io_result<file_handle::buffers_type> _buffers_read = read(fh, {reqs, 0});
if(!_buffers_read)
{
std::error_code ec = _fh.error();
std::cerr << "Reading the file " << path << " failed with " << ec.message() << std::endl;
}
// Same as before, either throw any error or unpack the value returned
file_handle::buffers_type buffers_read(_buffers_read.value());
// Note that buffers returned by AFIO read() may be completely different to buffers submitted
// This lets us skip unnecessary memory copying
// Make a string view of the first buffer returned
string_view v(buffers_read[0].data, buffers_read[0].len);
// Sub view that view with the first line
string_view line(v.substr(0, v.find_first_of('\n')));
// Return a string copying the first line from the file, or all 4096 bytes read if no newline found.
return std::string(line);
}
People on boost-users mailing list pointed out that the boost.beast library has OS-independent API for basic file IO including proper error handling. There are three implementations of the file concept out-of-the-box: POSIX, stdio and win32. The implementations support RAII (automatic closing on destruction) and move semantics. The POSIX file model automatically handles EINTR error. Basically this is sufficient and convenient to portably read a file chunk by chunk and, for example, explicitly handle the situation of absence of a file:
using namespace boost::beast;
using namespace boost::system;
file f;
error_code ec;
f.open("/path/to/file", file_mode::read, ec);
if(ec == errc::no_such_file_or_directory) {
// ...
} else {
// ...
}
The best thing to do could be to wrap Boost WinAPI and or POSIX APIs.
The "naive" C++ standard library thing (with bells and wistles) doesn't get you too far:
Live On Coliru
#include <iostream>
#include <fstream>
#include <vector>
template <typename Out>
Out read_file(std::string const& path, Out out) {
std::ifstream s;
s.exceptions(std::ios::badbit | std::ios::eofbit | std::ios::failbit);
s.open(path, std::ios::binary);
return out = std::copy(std::istreambuf_iterator<char>{s}, {}, out);
}
void test(std::string const& spec) try {
std::vector<char> data;
read_file(spec, back_inserter(data));
std::cout << spec << ": " << data.size() << " bytes read\n";
} catch(std::ios_base::failure const& f) {
std::cout << spec << ": " << f.what() << " code " << f.code() << " (" << f.code().message() << ")\n";
} catch(std::exception const& e) {
std::cout << spec << ": " << e.what() << "\n";
};
int main() {
test("main.cpp");
test("nonexistent.cpp");
}
Prints...:
main.cpp: 823 bytes read
nonexistent.cpp: basic_ios::clear: iostream error code iostream:1 (iostream error)
Of course you can add more diagnostics perusing <filesystem> but
that's susceptible to races, as mentioned (depending on your application, these can even open up security vulnerabilities, so just say "No").
Using boost::filesystem::ifstream doesn't change the exceptions raised
Worse still, using Boost IOstream fails to raise any errors:
template <typename Out>
Out read_file(std::string const& path, Out out) {
namespace io = boost::iostreams;
io::stream<io::file_source> s;
s.exceptions(std::ios::badbit | std::ios::eofbit | std::ios::failbit);
s.open(path, std::ios::binary);
return out = std::copy(std::istreambuf_iterator<char>{s}, {}, out);
}
Happily prints:
main.cpp: 956 bytes read
nonexistent.cpp: 0 bytes read
Live On Coliru
#include <iostream>
#include <fstream>
#include <string>
#include <system_error>
using namespace std;
int
main()
{
ifstream f("testfile.txt");
if (!f.good()) {
error_code e(errno, system_category());
cerr << e.message();
//...
}
// ...
}
ISO C++ Standard:
The contents of the header
"cerrno"
are the same as the POSIX header
"errno.h"
, except that
errno
shall
be defined as a macro. [
Note:
The intent is to remain in close alignment with the POSIX standard.
— end
note
] A separate
errno
value shall be provided for each thread.
check this code:
uSTL is a partial implementation of the C++ standard library that focuses on
decreasing the memory footprint of user executables.
https://github.com/msharov/ustl/blob/master/fstream.cc
I am writing a C++11 program that will run in a Unix environment (portability isn't a concern). Currently, I have a Makefile that invokes two processes, one which writes to a file and the second which reads from that file.
target:
mkfifo myfile
other_program > myfile &
my_program myfile
For various reasons, I want invoke all of this from within my_program. It looks like popen is promising, as it invokes an external process and provides a FILE* I can read. However, the existing file processing code I've already written uses ifstream:
std::ifstream stream1(argv[1]);
Is there a decent way to connect popen's FILE* to an ifstream? Is there something else I should use instead of popen?
You can create a stream buffer which reads from a FILE*. Clearly, you may need to change your code to use std::istream in case you use std::ifstream in other places than creating the stream but this should be a straight forward change. Here is a simple demo showing how to create a corresponding stream buffer and how to use it:
#include <cstdio>
#include <iostream>
struct FILEbuf
: std::streambuf {
FILEbuf(FILE* fp): fp_(fp) {}
int underflow() {
if (this->gptr() == this->egptr()) {
int size = fread(this->buffer_, 1, int(s_size), this->fp_);
if (0 < size) {
this->setg(this->buffer_, this->buffer_, this->buffer_ + size);
}
}
return this->gptr() == this->egptr()
? traits_type::eof()
: traits_type::to_int_type(*gptr());
}
FILE* fp_;
enum { s_size = 1024 };
char buffer_[s_size];
};
int main()
{
FILEbuf sbuf(popen("ls -l", "r"));
std::istream in(&sbuf);
for (std::string line; std::getline(in, line); ) {
std::cout << line << '\n';
}
}
In the past I have been told off for using popen() or system() because these calls are considered to be unsafe: both of these calls spawn a shell which can be used to hijack their behavior. The alternative is to create a stream buffer using file descriptors and using pipe(), dup() (or one of its siblings), close(), fork(), and execl() (or one of its siblings) to build the pipe directly.
I am trying out try, catch, throw statements in C++ for file handling, and I have written a dummy code to catch all errors. My question is in order to check if I have got these right, I need an error to occur. Now I can easily check infile.fail() by simply not creating a file of the required name in the directory. But how will I be able to check the same for outfile.fail() (outfile is ofstream where as infile is ifstream). In which case, will the value for outfile.fail() be true?
sample code [from comments on unapersson's answer, simplified to make issue clearer -zack]:
#include <fstream>
using std::ofstream;
int main()
{
ofstream outfile;
outfile.open("test.txt");
if (outfile.fail())
// do something......
else
// do something else.....
return 0;
}
The open(2) man page on Linux has about 30 conditions. Some intresting ones are:
If the file exists and you don't have permission to write it.
If the file doesn't exist, and you don't have permission (on the diretory) to create it.
If you don't have search permission on some parent directory.
If you pass in a bogus char* for the filename.
If, while opening a device file, you press CTRL-C.
If the kernel encountered too many symbolic links while resolving the name.
If you try to open a directory for writing.
If the pathname is too long.
If your process has too many files open already.
If the system has too many files open already.
If the pathname refers to a device file, and there is no such device in the system.
If the kernel has run out of memory.
If the filesystem is full.
If a component of the pathname is not a directory.
If the file is on a read-only filesystem.
If the file is an executable file which is currently being executed.
By default, and by design, C++ streams never throw exceptions on error. You should not try to write code that assumes they do, even though it is possible to get them to. Instead, in your application logic check every I/O operation for an error and deal with it, possibly throwing your own exception if that error cannot be dealt with at the specific place it occurs in your code.
The canonical way of testing streams and stream operations is not to test specific stream flags, unless you have to. Instead:
ifstream ifs( "foo.txt" );
if ( ifs ) {
// ifs is good
}
else {
// ifs is bad - deal with it
}
similarly for read operations:
int x;
while( cin >> x ) {
// do something with x
}
// at this point test the stream (if you must)
if ( cin.eof() ) {
// cool - what we expected
}
else {
// bad
}
To get ofstream::open to fail, you need to arrange for it to be impossible to create the named file. The easiest way to do this is to create a directory of the exact same name before running the program. Here's a nearly-complete demo program; arranging to reliably remove the test directory if and only if you created it, I leave as an exercise.
#include <iostream>
#include <fstream>
#include <sys/stat.h>
#include <cstring>
#include <cerrno>
using std::ofstream;
using std::strerror;
using std::cerr;
int main()
{
ofstream outfile;
// set up conditions so outfile.open will fail:
if (mkdir("test.txt", 0700)) {
cerr << "mkdir failed: " << strerror(errno) << '\n';
return 2;
}
outfile.open("test.txt");
if (outfile.fail()) {
cerr << "open failure as expected: " << strerror(errno) << '\n';
return 0;
} else {
cerr << "open success, not as expected\n";
return 1;
}
}
There is no good way to ensure that writing to an fstream fails. I would probably create a mock ostream that failed writes, if I needed to test that.
I've been trying to call another program from c++, and save the stout of that program to a text file. popen() seems to be the appropriate function, but saving it to a text file isn't working.
ofstream delaunayfile;
delaunayfile.open ("triangulation/delaunayedpoints.txt");
FILE *fp;
fp = popen("qdelaunay < triangulation/rawpoints.txt", "r");
delaunayfile << fp;
delaunayfile.close();
Any help? Thanks in advance!
You cannot write a FILE* directly into a stream. It will write a memory address instead of the actual file contents, therefore it will not give you the desired result.
The ideal solution would be to read from an ifstream and write to your ofstream, but there's no way to construct an ifstream from a FILE*.
However, we can extend the streambuf class, make it work over a FILE*, and then pass it to an istream instead. A quick search revealed someone already implemented that, and properly named popen_streambuf. See this specific answer.
Your code then would look like this:
std::ofstream output("triangulation/delaunayedpoints.txt");
popen_streambuf popen_buf;
if (popen_buf.open("qdelaunay < triangulation/rawpoints.txt", "r") == NULL) {
std::cerr << "Failed to popen." << std::endl;
return;
}
char buffer[256];
std::istream input(&popen_buf);
while (input.read(buffer, 256)) {
output << buffer;
}
output.close();
As pointed by Simon Richter in comments, there's an operator<< that accepts streambuf and writes data to ostream until EOF is reached. This way, the code would be simplified to:
std::ofstream output("triangulation/delaunayedpoints.txt");
popen_streambuf popen_buf;
if (popen_buf.open("qdelaunay < triangulation/rawpoints.txt", "r") == NULL) {
std::cerr << "Failed to popen." << std::endl;
return;
}
output << &popen_buf;
output.close();
There are two ways to do this: The simple way
int rc = system("qdelaunay < triangulation/rawpoints.txt >triangulation/delaunayedpoints.txt");
and the slightly more elaborate way, using fork(), dup2() and execve(), the latter working without a shell interpreter installed on the system. Given that this looks like you are doing computation work, I suspect this is not an embedded system, so you can assume a working shell.
popen opens a pipe but I am not aware you can just stream it into delaunayfile that way.
Of course it would be very nice if you could just do that and it would read from the pipe until it was complete.
The normal way to check for data on the pipe is to use select(). I found a useful link http://codenewbie.com/forums/threads/2908-Using-std-fstream-in-a-pipe that integrates pipes with fstream though and it may help you achieve what you want.
In this instance though as all you want to do is write the output to a file, why not redirect the output of the process to it rather than to a pipe? The purpose of a pipe is Inter-Process communication but your process does not appear to be using the data it receives from the other process for any practical purpose.
I'm working on a file sharing application in C++. I want to write console output into a separate file and at the same time I want to see the output in console also. Can anybody help me...Thanks in advance.
Here we go...
#include <fstream>
using std::ofstream;
#include <iostream>
using std::cout;
using std::endl;
int main( int argc, char* argv[] )
{
ofstream file( "output.txt" ); // create output file stream to file output.txt
if( !file ) // check stream for error (check if it opened the file correctly)
cout << "error opening file for writing." << endl;
for( int i=0; i<argc; ++i ) // argc contains the number of arguments
{
file << argv[i] << endl; // argv contains the char arrays of commandline arguments
cout << argv[i] << endl;
}
file.close(); // always close a file stream when you're done with it.
return 0;
}
PS: OK, read your question wrong (console output/input mixup), but you still get the idea I think.
The idea is to create a derivate of std::streambuf which will output data to both the file and cout. Then create an instance of it and use cout.rdbuf(...);
Here is the code (tested with MSVC++ 2010, should work on any compiler):
class StreambufDoubler : public std::streambuf {
public:
StreambufDoubler(std::streambuf* buf1, std::streambuf* buf2) :
_buf1(buf1), _buf2(buf2), _buffer(128)
{
assert(_buf1 && _buf2);
setg(0, 0, 0);
setp(_buffer.data(), _buffer.data(), _buffer.data() + _buffer.size());
}
~StreambufDoubler() {
sync();
}
void imbue(const std::locale& loc) {
_buf1->pubimbue(loc);
_buf2->pubimbue(loc);
}
std::streampos seekpos(std::streampos sp, std::ios_base::openmode which) {
return seekoff(sp, std::ios_base::cur, which);
}
std::streampos seekoff(std::streamoff off, std::ios_base::seekdir way, std::ios_base::openmode which) {
if (which | std::ios_base::in)
throw(std::runtime_error("Can't use this class to read data"));
// which one to return? good question
// anyway seekpos and seekoff should never be called
_buf1->pubseekoff(off, way, which);
return _buf2->pubseekoff(off, way, which);
}
int overflow(int c) {
int retValue = sync() ? EOF : 0;
sputc(c);
return retValue;
}
int sync() {
_buf1->sputn(pbase(), pptr() - pbase());
_buf2->sputn(pbase(), pptr() - pbase());
setp(_buffer.data(), _buffer.data(), _buffer.data() + _buffer.size());
return _buf1->pubsync() | _buf2->pubsync();
}
private:
std::streambuf* _buf1;
std::streambuf* _buf2;
std::vector<char> _buffer;
};
int main() {
std::ofstream myFile("file.txt");
StreambufDoubler doubler(std::cout.rdbuf(), myFile.rdbuf());
std::cout.rdbuf(&doubler);
// your code here
return 0;
}
However note that a better implementation would use templates, a list of streambufs instead of just two, etc. but I wanted to keep it as simple as possible.
What you want actually is to follow in real time the lines added to the log your application writes.
In the Unix world, there's a simple tool that has that very function, it's called tail.
Call tail -f your_file and you will see the file contents appearing in almost real time in the console.
Unfortunately, tail is not a standard tool in Windows (which I suppose you're using, according to your question's tags).
It can however be found in the GnuWin32 package, as well as MSYS.
There are also several native tools for Windows with the same functionality, I'm personally using Tail For Win32, which is licensed under the GPL.
So, to conclude, I think your program should not output the same data to different streams, as it might slow it down without real benefits, while there are established tools that have been designed specifically to solve that problem, without the need to develop anything.
i don't program in c++ but here is my advice: create new class, that takes InputStream (istream in c++ or smth), and than every incoming byte it will transfer in std.out and in file.
I am sure there is a way to change standard output stream with forementioned class. As i remember, std.out is some kind of property of cout.
And again, i spent 1 week on c++ more than half a year ago, so there is a chance that all i've said is garbage.