I am using boost::process to read asynchronously the output of a console application in Windows. I noticed that the reading events is triggered after about 4k of data every-time.
If I set my buffer 'buf' to a small value nothing changes: the event is triggered multiple times ONLY after 4k of data has been transferred.
As per my understanding this could be a safe mechanism used in Windows to avoid dead-lock while reading from the pipe.
Is there any way in boost::process to change the size of the buffer used by the PIPE to transfer the data?
#include <boost/process.hpp>
#include <boost/asio.hpp>
using namespace boost::process;
boost::asio::io_service ios;
std::vector<char> buf(200);
async_pipe ap(ios);
void read_from_buffer(const boost::system::error_code &ec, std::size_t size)
{
if (ec)
{
std::cout << "error" << std::endl;
return;
}
std::cout << "--read-- " << size << std::endl;
for (size_t i = 0; i < size; i++) std::cout << buf[i];
std::cout << std::endl;
ap.async_read_some(boost::asio::buffer(buf), read_from_buffer);
}
int main()
{
child c("MyApp.exe --args", std_out > ap);
ap.async_read_some(boost::asio::buffer(buf), read_from_buffer);
ios.run();
int result = c.exit_code();
}
You might have to control the "sending" side (so, MyApp.exe).
On UNIX there's stdbuf (using setvbuff), unbuffer and similar. Tools might have some support built-in (e.g. grep --line-buffered).
On Windows, I'm not sure. Here's some pointers: Disable buffering on redirected stdout Pipe (Win32 API, C++)
Related
I have this requirement where my app have to connect to another app via sockets and will have to maintain persistent connection for quiet long time. My app will be a TCP client and the other is a TCP server. My app will send commands and the server will respond accordingly.
The problem am facing right now is how to read the whole data from server a string and return for app which will issue the next command. Reading synchronously (with asio::read) looked like a good option up until I observed socket hanging up until I terminate the server. Looking at the documentation I found that the library is correctly working.
his function is used to read a certain number of bytes of data from a stream. The call will block until one of the following conditions is true:
1. The supplied buffers are full. That is, the bytes transferred is equal to the sum of the buffer sizes.
2. An error occurred.
The problem is I don't know correct buffer size as the response from the server varies. So If I put a too small buffer it returns fine but missing some data. If I put too big it will hang forever until server quits.
So I thought I would do the async reading. It works only once and I don't know how to make it fetch data until whole data it read.
here is the relevant async code
#define ASIO_STANDALONE 1
#include <iostream>
#include <asio.hpp>
int main()
{
asio::io_context context;
size_t reply_length;
size_t length = 1024;
std::vector<char> buffer;
//create socket
asio::ip::tcp::socket socket(context);
socket.connect(asio::ip::tcp::endpoint(asio::ip::address::from_string("127.0.0.1"), 8088));
std::string dataOut = "list --files"; //some command to write
std::error_code error;
asio::write(socket, asio::buffer(dataOut), error);
if (!error)
{
std::cout << "Receiving...!" << std::endl;
buffer.resize(length);
asio::async_read(socket, asio::buffer(buffer), [&buffer, &context](const asio::error_code &ec, std::size_t bytes_transferred) {
std::copy(buffer.begin(), buffer.end(), std::ostream_iterator<char>(std::cout, ""));
std::cout << "\nRead total of:" << bytes_transferred << "\n";
context.run();
});
}
else
{
std::cout << "send failed: " << error.message() << std::endl;
}
context.run();
}
Searching didn't help much solving my issue.
So my question is, how can I read all the data in a persistent socket with asio? Am not using boost.
You need to loop async_read calls. If you don't want your client to hang on read operation you can define the smallest possible buffer i.e. 1 byte.
Define function which takes socket, buffer and two additional parameters according to async_read's handler signature, and this function calls itself with async_read to make the loop of async_read calls - it reads until some error occures:
void onRead (
asio::ip::tcp::socket& socket,
std::array<char,1>& buf,
const system::error_code& ec,
std::size_t bytes)
{
if (ec)
{
if (ec == asio::error::eof && bytes == 1)
std::cout << buf[0];
return;
}
std::cout << buf[0];
asio::async_read(socket,asio::buffer(buf),
std::bind(onRead, std::ref(socket), std::ref(buf),
std::placeholders::_1, // error code
std::placeholders::_2)); // transferred bytes
}
and the changes in main:
std::array<char,1> buf;
asio::write(socket, asio::buffer(dataOut), error);
if (!error)
{
std::cout << "Receiving...!" << std::endl;
asio::async_read(socket, asio::buffer(buf),
std::bind(onRead, std::ref(socket), std::ref(buf),
std::placeholders::_1,
std::placeholders::_2));
context.run();
}
else
{
std::cout << "send failed: " << error.message() << std::endl;
}
(I am using Boost, so you should replace system::error_code on asio::error_code).
Here is my code.
boost::asio::async_write(*serialPort, boost::asio::buffer(*something),handler);
boost::asio::async_write(*serialPort, boost::asio::buffer(*something2),handler);
The above code will get an error "the requested resource is in use" on the second line (Note that the async stream is a serial port). But when I changed the stream to a tcp socket, everthing works fine. Why?
Now I know I can’t use these composed asynchronous operation these way, but the first line code may be a heartbeat package, the second line may be a package which don’t sent regually. And these send operations buffers can’t gather together in the same time. Is there a way to synchronize these asynchronous operations in a single thread ( or multi-thread)?
The commenters are right. In this case you can easily use a buffer sequence ("scatter/gather IO"):
std::vector<boost::asio::const_buffer> buffers {
boost::asio::buffer(*something),
boost::asio::buffer(*something2)
};
boost::asio::async_write(*serialPort, buffers, handler);
See it Compiling On Coliru
#include <iostream>
#include <string>
#include <boost/asio.hpp>
#include <boost/asio/serial_port.hpp>
void handler(boost::system::error_code ec, size_t) {
std::cout << __PRETTY_FUNCTION__ << ": " << ec.message() << "\n";
}
int main() {
boost::asio::io_service svc;
auto serialPort = std::make_shared<boost::asio::serial_port>(svc);
auto something = std::make_shared<std::string>("hello world\n");
auto something2 = std::make_shared<std::string>("bye world\n");
std::vector<boost::asio::const_buffer> buffers {
boost::asio::buffer(*something),
boost::asio::buffer(*something2)
};
boost::asio::async_write(*serialPort, buffers, handler);
}
I am trying to listen for input on a named pipe. I'm using Boost.Asio's stream_descriptor and async_read under Linux. The problem is, the call to io_service::run() only blocks like I want it to until the first read. After that, it just keeps calling the handler immediately with the "End of file" error, even though I try to attach more async_reads to it. The code I have is equivalent to the following:
boost::asio::io_service io_service;
int fifo_d = open("/tmp/fifo", O_RDONLY);
boost::asio::posix::stream_descriptor fifo(io_service, fifo_d);
while (true)
{
// buffer and handler probably aren't important for the purposes of this question
boost::asio::async_read(fifo, buffer, handler);
io_service.run();
}
Only the first async_read works as I expect it to. Subsequent async_reads just return immediately. The only way I found to make it work like I want is to close and reopen the named pipe, but it seems like a hack:
boost::asio::io_service io_service;
while (true)
{
int fifo_d = open("/tmp/fifo", O_RDONLY);
boost::asio::posix::stream_descriptor fifo(io_service, fifo_d);
boost::asio::async_read(fifo, buffer, handler);
io_service.run();
close(fifo_d);
}
Can anyone tell me what am I doing wrong?
UPDATE: Here's a simple "read" version, which allowed for some code simplification, the problem remains the same:
int fifo_d = open("/tmp/fifo", O_RDONLY);
boost::asio::posix::stream_descriptor fifo(io_service, fifo_d);
while (true) {
try {
boost::asio::read(fifo, boost::asio::buffer(buffer));
}
catch (boost::system::system_error& err) {
// It loops here with "read: End of file" error
std::cout << err.what() << std::endl;
}
}
This is not how works. run() is not intended to be called in a loop. If you insist, you need to call reset() in between (as per the documentation).
Also, if you /want/ blocking behaviour, why are you using the async_* interface?
Demos
Consider using a simple iostream to read the fd:
Live On Coliru
#include <iostream>
#include <fstream>
int main() {
std::ifstream fifo("/tmp/fifo");
std::string word;
size_t lineno = 0;
while (fifo >> word) {
std::cout << "word: " << ++lineno << "\t" << word << "\n";
}
}
Or if you must attach to some fd you get from somewhere else, use file_descriptor from Boost IOstreams:
Live On Coliru
#include <boost/iostreams/device/file_descriptor.hpp>
#include <boost/iostreams/stream.hpp>
#include <iostream>
#include <fcntl.h>
int main() {
namespace io = boost::iostreams;
using src = io::file_descriptor_source;
io::stream<src> fifo(src(open("./fifo", O_RDONLY), io::file_descriptor_flags::close_handle));
std::string word;
size_t number = 0;
while (fifo >> word) {
std::cout << "word: " << ++number << "\t" << word << "\n";
}
}
Both examples print the expected:
word: 1 hello
word: 2 world
As also sehe reported, that's not the way boost::asio works.
The ioservice::run() method runs in blocking mode while it has some work. When the ioservice goes out of work you have to call the reset() method before putting other work, so that's why in your first code the async_read is done only once.
A common pattern in this case would look something like:
void handler(...) {
if (!error) {
// do your work
boost::asio::async_read(fifo, buffer, handler); // <-- at the end of the handler a subsequent async_read is put to the ioservice, so it never goes out-of-work
}
}
boost::asio::io_service io_service;
int fifo_d = open("/tmp/fifo", O_RDONLY);
boost::asio::posix::stream_descriptor fifo(io_service, fifo_d);
boost::asio::async_read(fifo, buffer, handler); // <-- you call async_read only once here.
io_service.run(); //<-- this blocks till an error occurs
I am writing some binary data to a device fie like /dev/itun.
void ahaConnector::asyncWriteData(vector<uint8_t> packedMessage) {
cout<<"\n async write data packed message";
deviceStreamDescriptor.assign(device);
boost::asio::write (
deviceStreamDescriptor,
boost::asio::buffer(packedMessage)
);
readbuffer.resize(1024);
deviceStreamDescriptor.async_read_some(boost::asio::buffer(readbuffer),
boost::bind(&ahaConnector::readHeader, this,
boost::asio::placeholders::error(),
boost::asio::placeholders::bytes_transferred()
));
io_service.run();
}
void ahaConnector::readHeader(const boost::system::error_code &ec, std::size_t bytes_transferred) {
if(!ec) {
std::cout<<"\n Bytes transfereed :"<<bytes_transferred<<" "<<readbuffer.size();
deviceStreamDescriptor.async_read_some(boost::asio::buffer(readbuffer),
boost::bind(&ahaConnector::readHeader, this,
boost::asio::placeholders::error(),
boost::asio::placeholders::bytes_transferred()
));
Callbacks callbacks;
callbacks.OnReceivingPackedMessage();
io_service.run();
}
else {
cout<<"\n System Error Code "<<ec;
}
}
The callback function readhandler is getting executed successfully, however I am not able to transfer the control from my Callback function to another class.
Is something wrong from the design perspective. I need to handle the message received from the callback function for further logic. Should I use another thread here ?
Looking at this code you might just want to replace the read(device,...) by boost Asio's support for Posix streams:
#include <boost/asio.hpp>
#include <boost/asio/posix/stream_descriptor.hpp>
#include <boost/function.hpp>
#include <iostream>
static int device = 0;
using namespace boost;
int main() {
boost::asio::io_service io_svc;
boost::asio::posix::stream_descriptor iodevice(io_svc, device);
char buffer[1024];
function<void(system::error_code const&, size_t)> callback;
callback = [&](boost::system::error_code const& ec, size_t bytes_transferred) {
if (ec)
{
std::cout << "Error '" << ec.message() << "' during asynchronous operation\n";
}
else
{
std::cout << "Read exactly " << bytes_transferred << " bytes\n";
std::cout << "Data: '";
std::cout.write(buffer, bytes_transferred);
std::cout << "'\n";
iodevice.async_read_some(asio::buffer(buffer), callback);
}
};
iodevice.async_read_some(asio::buffer(buffer), callback);
io_svc.run();
}
See it Live On Coliru.
Sadly on Coliru it can't work because input is redirected from a non-stream. But if you run it interactively it will work and print the first 10 characters entered.
The answer depends on exactly what are the properties of the device. Check the documentation for the device driver you're trying to use. If the device supports non-blocking I/O, open the device with O_NONBLOCK, and use poll() to wait for device to be available for reading or writing.
If the device does not support non-blocking I/O, the only viable option would be to use a separate thread to read and/or write to the device, and use the background thread to construct facade that pretends and behaves like a non/blocking data source and sink.
I created a console application that sends data on a network link. I used the boost library, both the thread and the asio ones; currently i'm running it under Windows. If I run a single application it works perfectly, but if I open two instances on two different consoles, the CPU load goes to 100%, if I close one of the application it goes back to normal. I just used a simple socket with async reads and writes, and threads with condition variables and mutexes. Is there any special thingh to do when dealing with such a situation? I can show you some code, but I think it's nothing special:
socket->connect(tcp::endpoint(address::from_string(getAddress()),getPort()));
for connecting
and
socket->async_read_some(buffer(receiveData),bind(&NetworkLink::handle_response, this,placeholders::error,placeholders::bytes_transferred));
inside the handle_response function for async reading.
For the thread I use
boost::unique_lock<boost::mutex> messages_lock(message_received_mutex);
Before deleting everything and starting a simple test project from scratch I would like to know if there are any special care to be taken in this situation.
Ok it seems I've tracked down the problem. First of all, the 100% CPU usage was due to the fact that each instance was using 50% of the CPU (I'm on a dual core PC). So I run all over the code and found out this. I had this in my code, inside the NetworkLink::handle_response function:
socket->async_read_some(
boost::asio::buffer(receiveData),
boost::bind(&NetworkLink::handle_response, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
io_service.run();
I used the io_service.run(); because before the software was not receiving data. Now I removed the line, I don't get the 50% CPU usage, but the handler response is not called so I cannot receive any data. Any thought about this?
Thanks
PS: I created a small app that shows this problem:
#include <iostream>
#include <boost/asio.hpp>
#include "boost/thread.hpp"
#include "boost/thread/mutex.hpp"
#ifdef _WIN32
#include "Windows.h"
#endif
using namespace boost::asio::ip;
using namespace std;
std::vector<uint8_t> buf;
boost::asio::io_service io_service;
boost::asio::ip::tcp::socket mysocket(io_service);
int handle_response(const boost::system::error_code &err,
size_t bytes_transferred)
{
// cout << bytes_transferred << ' ';
if (bytes_transferred > 0)
cout << buf.data() << ' ';
boost::asio::async_read(mysocket, boost::asio::buffer(buf),
boost::asio::transfer_at_least(1), &handle_response);
}
int mythread()
{
boost::asio::async_read(mysocket, boost::asio::buffer(buf),
boost::asio::transfer_at_least(1), &handle_response);
io_service.run();
}
int main()
{
int m_nPort = 12345;
buf.resize(100, '0');
boost::condition_variable message_received_condition;
boost::thread message_receiver_thread(&mythread);
boost::mutex messages_mutex;
tcp::endpoint endpoint(boost::asio::ip::address::from_string("127.0.0.1"),
m_nPort);
boost::unique_lock<boost::mutex> messages_lock(messages_mutex);
message_received_condition.notify_one();
cout << "Waiting for connection..." << endl;
Sleep(10000);
mysocket.connect(endpoint);
cout << "connection accepted" << endl;
try
{
while (true)
{
boost::system::error_code ec;
boost::asio::socket_base::bytes_readable command(true);
mysocket.io_control(command);
std::size_t bytes_readable = command.get();
mysocket.async_read_some(boost::asio::buffer(buf),
&handle_response);
io_service.run();
}
} catch (exception &e)
{
cerr << e.what() << endl; //"The parameter is incorrect" exception
}
}
If you remove the comment from th line
// cout << bytes_transferred << ' ';
in the handle response functionyou get a lower CPU usage, I guess because of the delay for writing to the screen.
You are not checking the error. If there is a failure, you continue to read anyway, which will probably immediately post back a completion with a failure, ad infinitum.