My requirements are simple: start a process, wait for it to finish, then capture and process it's output.
For the longest time I've been using the following:
struct line : public std∷string {
friend std∷istream& operator>> (std∷istream &is, line &l) {
return std∷getline(is, l);
}
};
void capture(std::vector<std::string> &output, const char *command)
{
output.clear();
FILE *f = popen(command, "r");
if(f) {
__gnu_cxx::stdio_filebuf<char> fb(f, ios∷in) ;
std::istream fs(&fb);
std::istream_iterator<line> start(fs), end;
output.insert(output.end(), start, end);
pclose(f);
}
}
And it works really well on single threaded programs.
However, if I call this function from inside a thread, sometimes the popen() call hangs and never return.
So, as a proof-of-concept I replaced the function for this ugly hack:
void capture(std::vector<std::string> &output, const char *command)
{
output.clear();
std::string c = std::string(command) + " > /tmp/out.txt";
::system(c.c_str());
ifstream fs("/tmp/out.txt", std::ios::in);
output.insert(output.end(), istream_iterator<line>(fs), istream_iterator<line>());
unlink("/tmp/out.txt");
}
It's ugly but works, however it kept me wondering what would be the proper way to capture a process output on a multi-threaded program.
The program runs on linux in a embedded powerquiccII processor.
See this: popen - locks or not thread safe? and other references do not seem conclusive that popen() needs to be thread-safe, so perhaps since you are using a less-popular platform, your implementation is not. Any chance you can view the source code of the implementation for your platform?
Otherwise, consider creating a new process and waiting upon it. Or hey, stick with the silly system() hack, but do handle its return code!
Related
I'm using Visual Studio 2019 on Windows 10, using the boost.process library. I'm trying to make chess, and I'm using the stockfish engine as a separate executable. I need the engine to run throughout the entirety of the game, as that's how it's designed to be used.
Currently I have in ChessGame.h
class ChessGame
{
public:
void startStockFish();
void beginGame();
void parseCommand(std::string cmd);
private:
boost::process::child c;
boost::process::ipstream input;
boost::process::opstream output;
}
And in ChessGame.cpp
#include ChessGame.h
void ChessGame::startStockFish()
{
std::string exec = "stockfish_10_x32.exe";
std::vector<std::string> args = { };
boost::process::child c(exec, args, boost::process::std_out > input,
boost::process::std_in < output);
//c.wait()
}
void ChessGame::beginGame()
{
parseCommand("uci");
parseCommand("ucinewgame");
parseCommand("position startpos");
parseCommand("go");
}
void ChessGame::parseCommand(std::string cmd)
{
output << cmd << std::endl;
std::string line;
while (std::getline(input, line) && !line.empty())
{
std::cout << line << std::endl;
}
}
And in main.cpp
ChessGame chessGame = ChessGame(isWhite); //isWhite is a boolean that control who the player is, irrelevent to the question
//std::thread t(&ChessGame::startStockFish, chessGame);
chessGame.startStockFish();
chessGame.beginGame();
The problem is that I believe as soon as the function startStockFish finishes it terminates c, as nothing is outputted to the terminal as described above, but if I use beginGame() within startStockFish() it outputs as expected. Also, if I uncomment the line c.wait() and the funtion waits for stockfish to exit, it gets stuck as stockfish never gets the exit command. If I instead try running startStockFish on a separate thread in main (as seen above) I
get the following two errors:
the argument to a feature-test macro must be a simple identifier.
In file 'boost\system\detail\config.hpp' line 51
and
'std::tuple::tuple': no overloaded function takes 2 arguments.
In file 'memory' line 2042
Also, I don't want to use threads as I can imagine that will have its own issues with the input and output streams.
So is there a way for me to keep the process alive out of this function, or do I need to reorganise my code some other way? I believe having the process being called in main would work, but I really don't want to do that as I want to keep all the chess-related code in ChessGame.cpp.
Ok I believe that adding c.detach(); after initialising the boost.process child in startStockFish() has done what I want, as the program no longer terminates c when the function ends. Input appears to work fine with a detached process, simply writing output << cmd << std::endl; where cmd is the desired command as a std::string has no issues. However, output does have some issues, the usual method of
std::string line;
while (std::getline(input, line) && !line.empty())
{
// Do something with line
}
somewhat works, but std::getline(input, line) will get stuck in an infinite loop when there are no more lines to output. I couldn't find a direct solution to this, but I did find a work around.
Firstly I changed the initialisation of the boost.process child to
boost::process::child c(exec, args, boost::process::std_out > "console.txt", boost::process::std_in < output);
And then changed input to a std::ifstream, a file reader stream. Then to get the output I used
input.open("console.txt");
std::string line;
while (std::getline(input, line))
{
// Do something with line
}
input.close();
I also added remove("console.txt"); to the beginning of startStockFish() to attain a fresh text file.
I'm not confident that this is the best solution, as I am worried about what would happen if stockfish tried to write to console.txt as input was reading from it, but that hasn't seemed to occur or doesn't seemed to be an issue if it has occurred, so right now it is an adequate solution.
I am developing a Qt interface for a 3D printer. When I cilck the Print button (the printer starts printing) the interface crashes. I am using this code:
*future= QtConcurrent::run(Imprimir,filename.toUtf8().data());
What can I use to solve it?? What types of threads can I use????
I need to use the interface while the printer is printing (it may take several minutes).
Thank you for advance.
Edit:
Imprimir function:
int Imprimir(char *fich)
{
char *aux = new char;
FILE *f;
f=fopen(fich, "r");
while(!feof(f)){
fgets(aux, 200, f);
Enviar(aux);
while(!seguir_imprimiendo);
}
Sleep(7000);
return 0;
}
It's making life harder than necessary by not using QFile. When you use QFile, you don't have to deal with silly things like passing C-string filenames around. You're likely to do it wrong, since who's to guarantee that the platform expects them to be encoded in UTF-8. The whole point of Qt is that it helps you avoid such issues. They are taken care of, the code is tested on multiple platforms to ensure that the behavior is correct in each case.
By not using QByteArray and QFile, you're liable to commit silly mistakes like your C-classic bug of allocating a single character buffer and then pretending that it's 200 characters long.
I see no reason to sleep in that method. It also makes no sense to wait for the continue flag seguir_imprimiendo to change, since Enviar runs in the same thread. It should block until the data is sent.
I presume that you've made Enviar run its code through QtConcurrent::run, too. This is unnecessary and leads to a deadlock. Think of what happens if a free thread can never be available while Imprimir is running. It's valid for the pool Imprimir runs on to be limited to just one thread. You can't simply pretend that it can't happen.
bool Imprimir(const QString & fileName)
{
QFile src(fileName);
if (! src.open(QIODevice::ReadOnly)) return false;
QByteArray chunk;
do {
chunk.resize(4096);
qint64 read = src.read(chunk.data(), chunk.size());
if (read < 0) return false;
if (read == 0) break; //we're done
chunk.resize(read);
if (!Enviar(chunk)) return false;
} while (! src.atEnd());
return true;
}
bool Enviar(const QByteArray & data)
{
...
return true; // if successful
}
Assuming there's no problem with Imprimir, the issue is probably with filename.toUtf8().data(). The data pointer you get from this function is only valid while filename is in-scope. When filename goes out of scope, the data may be deleted and any code accessing the data will crash.
You should change the Imprimir function to accept a QString parameter instead of char* to be safe.
If you can't change the Imprimir function (because it's in another library, for example), then you will have to wrap it in your own function which accepts a QString. If you're using C++11, you can use a lambda expression to do the job:
QtConcurrent::run([](QString filename) {
Imprimir(filename.toUtf8().data());
}, filename);
If not, you will have to write a separate ImprimirWrapper(QString filename) function and invoke it using QtConcurrent::run.
I have some long loop that I need to write some data to a file on every iteration. The problem is that writing to a file can be slow, so I would like to reduce the time this takes by doing the writing asynchronously.
Does anyone know a good way to do this? Should I be creating a thread that consumes whatever is put into it's buffer by writing it out ( in this case, a single producer, single consumer )?
I am interested mostly in solutions that don't involve anything but the standard library (C++11).
Before going into asynchronous writing, if you are using IOStreams you might want to try to avoid flushing the stream accidentally, e.g., by not using std::endl but rather using '\n' instead. Since writing to IOStreams is buffered this can improve performance quite a bit.
If that's not sufficient, the next question is how the data is written. If there is a lot of formatting going on, there is a chance that the actual formatting takes most of the time. You might be able to push the formatting off into a separate thread but that's quite different from merely passing off writing a couple of bytes to another thread: you'd need to pass on a suitable data structure holding the data to be formatted. What is suitable depends on what you are actually writing, though.
Finally, if writing the buffers to a file is really the bottleneck and you want to stick with the standard C++ library, it may be reasonable to have a writer thread which listens on a queue filled with buffers from a suitable stream buffer and writes the buffers to an std::ofstream: the producer interface would be an std::ostream which would send off probably fixed sized buffers either when the buffer is full or when the stream is flushed (for which I'd use std::flush explicitly) to a queue on which the other read listens. Below is a quick implementation of that idea using only standard library facilities:
#include <condition_variable>
#include <fstream>
#include <mutex>
#include <queue>
#include <streambuf>
#include <string>
#include <thread>
#include <vector>
struct async_buf
: std::streambuf
{
std::ofstream out;
std::mutex mutex;
std::condition_variable condition;
std::queue<std::vector<char>> queue;
std::vector<char> buffer;
bool done;
std::thread thread;
void worker() {
bool local_done(false);
std::vector<char> buf;
while (!local_done) {
{
std::unique_lock<std::mutex> guard(this->mutex);
this->condition.wait(guard,
[this](){ return !this->queue.empty()
|| this->done; });
if (!this->queue.empty()) {
buf.swap(queue.front());
queue.pop();
}
local_done = this->queue.empty() && this->done;
}
if (!buf.empty()) {
out.write(buf.data(), std::streamsize(buf.size()));
buf.clear();
}
}
out.flush();
}
public:
async_buf(std::string const& name)
: out(name)
, buffer(128)
, done(false)
, thread(&async_buf::worker, this) {
this->setp(this->buffer.data(),
this->buffer.data() + this->buffer.size() - 1);
}
~async_buf() {
std::unique_lock<std::mutex>(this->mutex), (this->done = true);
this->condition.notify_one();
this->thread.join();
}
int overflow(int c) {
if (c != std::char_traits<char>::eof()) {
*this->pptr() = std::char_traits<char>::to_char_type(c);
this->pbump(1);
}
return this->sync() != -1
? std::char_traits<char>::not_eof(c): std::char_traits<char>::eof();
}
int sync() {
if (this->pbase() != this->pptr()) {
this->buffer.resize(std::size_t(this->pptr() - this->pbase()));
{
std::unique_lock<std::mutex> guard(this->mutex);
this->queue.push(std::move(this->buffer));
}
this->condition.notify_one();
this->buffer = std::vector<char>(128);
this->setp(this->buffer.data(),
this->buffer.data() + this->buffer.size() - 1);
}
return 0;
}
};
int main()
{
async_buf sbuf("async.out");
std::ostream astream(&sbuf);
std::ifstream in("async_stream.cpp");
for (std::string line; std::getline(in, line); ) {
astream << line << '\n' << std::flush;
}
}
Search the web for "double buffering."
In general, one thread will write to one or more buffers. Another thread reads from the buffers, "chasing" the writing thread.
This may not make your program more efficient. Efficiency with files is achieved by writing in huge blocks so that the drive doesn't get a chance to spin down. One write of many bytes is more efficient than many writes of a few bytes.
This could be achieved by having the writing thread only write when the buffer content has exceeded some threshold like 1k.
Also research the topic of "spooling" or "print spooling".
You'll need to use C++11 since previous versions don't have threading support in the standard library. I don't know why you limit yourself, since Boost has some good stuff in it.
I am implementing some file system in C++. Up to now I was using fstream but I realized that it is impossible to open it in exclusive mode. Since there are many threads I want to allow multiple reads, and when opening file in writing mode I want to open the file in exclusive mode?
What is the best way to do it? I think Boost offers some features. And is there any other possibility? I would also like to see simple example. If it is not easy / good to do in C++ I could write in C as well.
I am using Windows.
On many operating systems, it's simply impossible, so C++
doesn't support it. You'll have to write your own streambuf.
If the only platform you're worried about is Windows, you can
possibly use the exclusive mode for opening that it offers.
More likely, however, you would want to use some sort of file
locking, which is more precise, and is available on most, if not
all platforms (but not portably—you'll need LockFileEx
under Windows, fcntl under Unix).
Under Posix, you could also use pthread_rwlock. Butenhof
gives an implementation of this using classical mutex and
condition variables, which are present in C++11, so you could
actually implement a portable version (provided all of the
readers and writers are in the same process—the Posix
requests will work across process boundaries, but this is not
true for the C++ threading primitives).
if your app only works on Windows, the Win32 API function CreateFile() is your choice.
For example:
HANDLE hFile = ::CreateFileW(lpszFileFullPathName, GENERIC_WRITE, FILE_SHARE_READ, NULL, OPEN_EXISTING, NULL, NULL);
If you are open to using boost, then I would suggest you use the file_lock class. This means you want to keep the filename of the files you open/close because fstream does not do so for you.
They have two modes lock() that you can use for writing (i.e. only one such lock at a time, the sharable lock prevents this lock too) and lock_sharable() that you can use for reading (i.e. any number of threads can obtain such a lock).
Note that you will find it eventually complicated to manage both, read and write, in this way. That is, if there is always someone to read, the sharable lock may never get released. In that case, the exclusive lock will never be given a chance to take....
// add the lock in your class
#include <boost/interprocess/sync/file_lock.hpp>
class my_files
{
...
private:
...
boost::file_lock m_lock;
};
Now when you want to access a file, you can lock it one way or the other. If the thread is in charge of when they do that, you could add functions for the user to have access to the lock. If your implementation of the read and write functions in my_files are in charge, you want to get a stack based object that locks and unlocks for you (RAII):
class safe_exclusive_lock
{
public:
safe_exclusive_lock(file_lock & lock)
: m_lock_ref(lock)
{
m_lock_ref.lock();
}
~safe_exclusive_lock()
{
m_lock_ref.unlock();
}
private:
file_lock & m_lock_ref;
};
Now you can safely lock the file (i.e. you lock, do things that may throw, you always unlock before exiting your current {}-block):
ssize_t my_files::read(char *buf, size_t len)
{
safe_exclusive_lock guard(m_lock);
...your read code here...
return len;
} // <- here we get the unlock()
ssize_t my_files::write(char const *buf, size_t len)
{
safe_exclusive_lock guard(m_lock);
...your write code here...
return len;
} // <- here we get the unlock()
The file_lock uses a file, so you will want to have the fstream file already created whenever the file_lock is created. If the fstream file may not be created in your constructor, you probably will want to transform the m_lock variable in a unique pointer:
private:
std::unique_ptr<file_lock> m_lock;
And when you reference it, you now need an asterisk:
safe_exclusive_lock guard(*m_lock);
Note that for safety, you should check whether the pointer is indeed allocated, if not defined, it means the file is not open yet so I would suggest you throw:
if(m_lock)
{
safe_exclusive_lock guard(*m_lock);
...do work here...
}
else
{
throw file_not_open();
}
// here the lock was released so you cannot touch the file anymore
In the open, you create the lock:
bool open(std::string const & filename)
{
m_stream.open(...);
...make sure it worked...
m_lock.reset(new file_lock(filename));
// TODO: you may want a try/catch around the m_lock and
// close the m_stream if it fails or use a local
// variable and swap() on success...
return true;
}
And do not forget to release the lock object in your close:
void close()
{
m_lock.reset();
}
Well you can manually prevent yourself from opening a file if it has been opened in write mode already. Just keep track internally of which files you've opened in write mode.
Perhaps you could hash the filename and store it in a table upon open with write access. This would allow fast lookup to see if a file has been opened or not.
You could rename the file, update it under the new name, and rename it back. I've done it, but it's a little heavy.
Since C++17 there are two options:
In C++23 by using the openmode std::ios::noreplace.
In C++17 by using the std::fopen mode x (exclusive).
Note: The x mode was added to c in C11.
C++23 and later:
#include <cerrno>
#include <cstring>
#include <fstream>
#include <iostream>
int main() {
std::ofstream ofs("the_file", std::ios::noreplace);
if (ofs) {
std::cout << "success\n";
} else {
std::cerr << "Error: " << std::strerror(errno) << '\n';
}
}
Demo
C++17 and later:
#include <cerrno>
#include <cstdio>
#include <cstring>
#include <fstream>
#include <iostream>
#include <memory>
struct FILE_closer {
void operator()(std::FILE* fp) const { std::fclose(fp); }
};
// you may want overloads for `std::filesystem::path`, `std::string` etc too:
std::ofstream open_exclusively(const char* filename) {
bool excl = [filename] {
std::unique_ptr<std::FILE, FILE_closer> fp(std::fopen(filename, "wx"));
return !!fp;
}();
auto saveerr = errno;
std::ofstream stream;
if (excl) {
stream.open(filename);
} else {
stream.setstate(std::ios::failbit);
errno = saveerr;
}
return stream;
}
int main() {
std::ofstream ofs = open_exclusively("the_file");
if (ofs) {
std::cout << "success\n";
} else {
std::cout << "Error: " << std::strerror(errno) << '\n';
}
}
Demo
Having multiple processes, all writing on the same output stream (e.g. with std::cout), is there a way to lock the stream so that, when a process starts writing his own message, it can do it till the end (e.g. with std::endl)?
I need a portable way of doing it.
It's not clear if it would fit the parameters of your situation, but you could potentially funnel all data to a separate worker process that aggregates the data (with its own internal locking) before dumping them to stdout.
You are out of luck. You will have to use whatever your taget OS provides. This means using global/system-wide mutexes or lockf() like functions. You could use some 3rd party library to satisfy the portability requirement, like Boost.Interprocess.
If you are on a UNIX like OS, then you may be able to mimic the behavior you want with a stringstream adapter. This may not be the best way to accomplish it, but the idea is to trigger a single write call whenever std::endl is encountered.
// Assume fd is in blocking mode
class fdostream : public std::ostringstream {
typedef std::ostream & (*manip_t) (std::ostream &);
struct fdbuf : public std::stringbuf {
int fd_;
fdbuf (int fd) : fd_(fd) {}
int sync () {
int r = ::write(fd_, str().data(), str().size());
str(std::string());
return (r > 0) ? 0 : -1;
}
} buf_;
std::ostream & os () { return *this; }
public:
fdostream (int fd) : buf_(fd) { os().rdbuf(&buf_); }
};
fdostream my_cout(1);
my_cout << "Hello," << " world!" << std::endl;
This should achieve the effect of synchronized writes, at the cost of buffering input into a stringstream and then clearing the internal string after each flush.
For greater portability, you could modify the code to use fwrite, and specify unbuffered writes with setvbuf. But, the atomicity of fwrite would depend on the C implementation of the library function.