I'm reading stdin using Boost.ASIO, but when I pipe into it I would expect that the pipe would close when the input has been fully consumed. I.e. I'm doing this at the commmand line:
cat somefile.txt | myprog
And I'd expect that myprog will see the file close. Instead it waits forever.
The code looks like this:
boost::asio::posix::stream_descriptor as_stdin(ios);
{
boost::system::error_code error;
as_stdin.assign(dup(STDIN_FILENO), error);
if ( error ) {
exit(2);
}
}
auto proc = [&as_stdinr](auto yield) {
boost::asio::streambuf buffer;
while ( as_stdin.is_open() ) {
auto bytes = boost::asio::async_read_until(as_stdin, buffer, '\n', yield);
if ( bytes ) {
buffer.commit(bytes);
std::istream in(&buffer);
std::string line;
std::getline(in, line);
std::cerr << line << std::endl;
} else {
std::cerr << "No bytes read" << std::endl;
}
}
std::cerr << "Done" << std::endl;
};
boost::asio::spawn(ios, proc);
All of the file content is properly echoed, so reading from the pipe works fine, but neither of the "No bytes read" or "Done" messages are ever printed. I've tried both with and without the dup system call.
Am I misunderstanding how the pipe works, or am I doing something wrong or missing something else?
I think this comes down to "How do I detect EOF when using coroutines?"
You could catch the exception from async_read_until
size_t bytes = 0;
bool eof = false;
try {
bytes = boost::asio::async_read_until(as_stdin, buffer, '\n', yield);
} catch(std::exception const& e) {
std::cerr << "Exception: " << e.what() << "\n";
bytes = 0;
eof = true;
}
// ...
if (eof) break;
Or use the error_code:
boost::system::error_code ec;
auto bytes = boost::asio::async_read_until(as_stdin, buffer, '\n', yield[ec]);
// ...
if (ec) {
std::cerr << "Error: " << ec.message() << "\n";
break;
}
Output is very similar in both cases
Exception: End of file
No bytes read
Done
Or
No bytes read
Error: End of file
Done
Limitations
Regular files cannot be used with POSIX stream_descriptor, see https://stackoverflow.com/a/23631715/85371
Related
I'm new to the boost::asio, and boost::process libraries and I've come across a problem which I'm struggling to find a solution for...
Consider that I have a small toy program that does the following:
Firstly, fork()s itself into a parent-branch and a child-branch.
The child-branch then uses the boost::process::child class to invoke the unix command ls in an asynchronous context.
The child-branch supplies the boost::process::child class with a boost::process::async_pipe to direct std_out to.
The parent-branch wishes to read what has been written to the pipe, line by line, and process it further.
Currently, my implementation of this works up to a point. However, the read_loop() call in the parent-branch does not terminate. It is almost as if it never reaches EOF, or is blocked. Why is this?
Here is my MWE:
#include <boost/process.hpp>
#include <boost/asio.hpp>
#include <iostream>
#include <string>
#include <unistd.h>
void read_loop(boost::process::async_pipe& pipe)
{
static boost::asio::streambuf buffer;
boost::asio::async_read_until(
pipe,
buffer,
'\n',
[&](boost::system::error_code error_code, std::size_t bytes) {
if (!error_code) {
std::istream is(&buffer);
if (std::string line; std::getline(is, line)) {
std::cout << "Read Line: " << line << "\n";
}
read_loop(pipe);
}
else {
std::cout << "Error in read_loop()!\n";
pipe.close();
}
}
);
}
int main(int argc, char* argv[])
{
boost::asio::io_context io_context{};
boost::process::async_pipe pipe{ io_context };
io_context.notify_fork(boost::asio::io_context::fork_prepare);
pid_t pid{ fork() };
if (pid == 0) {
io_context.notify_fork(boost::asio::io_context::fork_child);
boost::process::child child(
boost::process::args({ "/usr/bin/ls", "/etc/" }),
boost::process::std_out > pipe,
boost::process::on_exit([&](int exit, std::error_code error_code) { std::cout << "[Exited with code " << exit << " (" << error_code.message() << ")]\n"; }),
io_context
);
io_context.run();
}
else {
io_context.notify_fork(boost::asio::io_context::fork_parent);
read_loop(pipe);
io_context.run();
}
return 0;
}
Which will successfully give the (abridged) output, as expected:
Read Line: adduser.conf
...
[Exited with code 0 (Success)]
...
Read Line: zsh_command_not_found
but will then just hang until it is forcibly killed.
Which leaves the main question, why does my read_loop() function end up blocking/not exiting correctly?
Thanks in advance!
Chasing The Symptom
The process not "seeing" EOF makes me think you have to close either end of the pipe. This is somewhat hacky, but works:
Live On Coliru
#include <boost/asio.hpp>
#include <boost/process.hpp>
#include <iostream>
namespace bp = boost::process;
void read_loop(bp::async_pipe& pipe) {
static boost::asio::streambuf buffer;
using boost::system::error_code;
async_read_until( //
pipe, buffer, '\n', [&](error_code ec, [[maybe_unused]] size_t bytes) {
// std::cout << "Handler " << ec.message() << " bytes:" << bytes << " (" <<
// buffer.size() << ")" << std::endl;
if (!ec) {
std::istream is(&buffer);
if (std::string line; std::getline(is, line)) {
std::cout << "Read Line: " << line << "\n";
}
read_loop(pipe);
} else {
std::cout << "Loop exit (" << ec.message() << ")" << std::endl;
pipe.close();
}
});
}
int main() {
boost::asio::io_context ioc{};
bp::async_pipe pipe{ioc};
ioc.notify_fork(boost::asio::io_context::fork_prepare);
pid_t pid{fork()};
if (pid == 0) {
ioc.notify_fork(boost::asio::io_context::fork_child);
bp::child child( //
bp::args({"/usr/bin/ls", "/etc/"}), bp::std_out > pipe, bp::std_in.close(),
bp::on_exit([&](int exit, std::error_code ec) {
std::cout << "[Exited with code " << exit << " (" << ec.message() << ")]\n";
pipe.close();
}),
ioc);
ioc.run();
} else {
ioc.notify_fork(boost::asio::io_context::fork_parent);
std::move(pipe).sink().close();
read_loop(pipe);
ioc.run();
}
}
Side note: I guess it would be nice to have a more unhacky way to specify this, like (bp::std_in < pipe).close() or so.
Fixing The Root Cause
When using Boost Process, the fork is completely redundant. Boost Process literally does the fork for you, complete with correct service notification and file descriptor handling.
You'll find the code becomes a lot simpler and also handles the closing correctly (likely because some assumptions within Boost Process implementation details):
Live On Coliru
#include <boost/asio.hpp>
#include <boost/process.hpp>
#include <iostream>
namespace bp = boost::process;
void read_loop(bp::async_pipe& pipe) {
static boost::asio::streambuf buffer;
static std::string line; // re-used because we can
async_read_until( //
pipe, buffer, '\n',
[&](boost::system::error_code ec, size_t /*bytes*/) {
if (ec) {
std::cout << "Loop exit (" << ec.message() << ")" << std::endl;
return;
}
if (getline(std::istream(&buffer), line))
std::cout << "Read Line: " << line << "\n";
read_loop(pipe);
});
}
int main() {
boost::asio::io_context ioc{};
bp::async_pipe pipe{ioc};
bp::child child( //
bp::args({"/bin/ls", "/etc/"}), bp::std_out > pipe,
bp::on_exit([&](int exit, std::error_code ec) {
std::cout << "[Exited with " << exit << " (" << ec.message()
<< ")]\n";
}));
read_loop(pipe);
ioc.run();
}
I am building an networking application, and being a newbie to Boost asio and networking as a whole had this doubt which might be trivial. I have this application which reads from a file and calls apis accordingly. I am reading json (example):
test.json
{
"commands":
[
{
"type":"login",
"Username": 0,
"Password": "kk"
}
]
}
My main program looks like this :
int main() {
ba::io_service ios;
tcp::socket s(ios);
s.connect({{},8080});
IO io;
io.start_read(s);
io.interact(s);
ios.run();
}
void start_read(tcp::socket& socket) {
char buffer_[MAX_LEN];
socket.async_receive(boost::asio::null_buffers(),
[&](const boost::system::error_code& ec, std::size_t bytes_read) {
(void)bytes_read;
if (likely(!ec)) {
boost::system::error_code errc;
int br = 0;
do {
br = socket.receive(boost::asio::buffer(buffer_, MAX_LEN), 0, errc);
if (unlikely(errc)) {
if (unlikely(errc != boost::asio::error::would_block)) {
if (errc != boost::asio::error::eof)
std::cerr << "asio async_receive: error " << errc.value() << " ("
<< errc.message() << ")" << std::endl;
interpret_read(socket,nullptr, -1);
//close(as);
return;
}
break; // EAGAIN
}
if (unlikely(br <= 0)) {
std::cerr << "asio async_receive: error, read " << br << " bytes" << std::endl;
interpret_read(socket,nullptr, br);
//close(as);
return;
}
interpret_read(socket,buffer_, br);
} while (br == (int)MAX_LEN);
} else {
if (socket.is_open())
std::cerr << "asio async_receive: error " << ec.value() << " (" << ec.message() << ")"
<< std::endl;
interpret_read(socket,nullptr, -1);
//close(as);
return;
}
start_read(socket);
});
}
void interpret_read(tcp::socket& s,const char* buf, int len) {
if(len<0)
{
std::cout<<"some error occured in reading"<<"\n";
}
const MessageHeaderOutComp *obj = reinterpret_cast<const MessageHeaderOutComp *>(buf);
int tempId = obj->TemplateID;
//std::cout<<tempId<<"\n";
switch(tempId)
{
case 10019: //login
{
//const UserLoginResponse *obj = reinterpret_cast<const UserLoginResponse *>(buf);
std::cout<<"*********[SERVER]: LOGIN ACKNOWLEDGEMENT RECEIVED************* "<<"\n";
break;
}
}
std::cout << "RX: " << len << " bytes\n";
if(this->input_type==2)
interact(s);
}
void interact(tcp::socket& s)
{
if(this->input_type == -1){
std::cout<<"what type of input you want ? option 1 : test.json / option 2 : manually through command line :";
int temp;
std::cin>>temp;
this->input_type = temp;
}
if(this->input_type==1)
{
//std::cout<<"reading from file\n";
std::ifstream input_file("test.json");
Json::Reader reader;
Json::Value input;
reader.parse(input_file, input);
for(auto i: input["commands"])
{
std::string str = i["type"].asString();
if(str=="login")
this->login_request(s,i);
}
std::cout<<"File read completely!! \n Do you want to continue or exit?: ";
}
}
The sending works fine, the message is sent and the server responds in a correct manner, but what I need to understand is why is the control not going to on_send_completed (which prints sent x bytes). Neither it prints the message [SERVER]: LOGIN ACKNOWLEDGEMENT RECEIVED, I know I am missing something basic or am doing something wrong, please correct me.
login_request function:
void login_request(tcp::socket& socket,Json::Value o) {
/*Some buffer being filled*/
async_write(socket, boost::asio::buffer(&info, sizeof(info)), on_send_completed);
}
Thanks in advance!!
From a cursory scan it looks like you redefined buffer_ that was already a class member (of IO, presumably).
It's hidden by the local in start_read, which is both UB (because the lifetime ends before the async read operation completes) and also makes it so the member _buffer isn't used.
I see a LOT of confusing code though. Why are you doing synchronous reads from within completion handlers?
I think you might be looking for the composed-ooperation reads (boost::asio::async_read and boost::asio::async_until)
I copy websocket example from boost::beast website and run it Websocket session work fine but I don't know how to convert received multi_buffer to string.
below code is websocket session handler.
void
do_session(tcp::socket &socket) {
try {
// Construct the stream by moving in the socket
websocket::stream <tcp::socket> ws{std::move(socket)};
// Accept the websocket handshake
ws.accept();
while (true) {
// This buffer will hold the incoming message
boost::beast::multi_buffer buffer;
// Read a message
boost::beast::error_code ec;
ws.read(buffer, ec);
if (ec == websocket::error::closed) {
break;
}
// Echo the message back
ws.text(ws.got_text());
ws.write(buffer);
}
cout << "Close" << endl;
}
catch (boost::system::system_error const &se) {
// This indicates that the session was closed
if (se.code() != websocket::error::closed)
std::cerr << "Error: " << se.code().message() << std::endl;
}
catch (std::exception const &e) {
std::cerr << "Error: " << e.what() << std::endl;
}
}
Is there way to convert buffer to string ?
Since original question was about converting to string directly, without using streams, I decided to add my answer.
You can use beast::buffers_to_string(buffer.data()).
You can use buffers on buffer.data()
std::cout << "Data read: " << boost::beast::buffers(buffer.data()) <<
std::endl;
Looking for a boost::asio (and with himself boost) decided to write asynchronous server. To store incoming data I use boost::asio::streambuf.
Here I have a problem. When I receive a second message from the client and subsequent I see that in the buffer contains a data from previous messages.
Although I call Consume method at the input buffer. What's wrong?
class tcp_connection
// Using shared_ptr and enable_shared_from_this
// because we want to keep the tcp_connection object alive
// as long as there is an operation that refers to it.
: public boost::enable_shared_from_this<tcp_connection>
{
...
boost::asio::streambuf receive_buffer;
boost::asio::io_service::strand strand;
}
...
void tcp_connection::receive()
{
// Read the response status line. The response_ streambuf will
// automatically grow to accommodate the entire line. The growth may be
// limited by passing a maximum size to the streambuf constructor.
boost::asio::async_read_until(m_socket, receive_buffer, "\r\n",
strand.wrap(boost::bind(&tcp_connection::handle_receive, shared_from_this()/*this*/,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred)));
}
void tcp_connection::handle_receive(const boost::system::error_code& error,
std::size_t bytes_transferred)
{
if (!error)
{
// process the data
/* boost::asio::async_read_until remarks
After a successful async_read_until operation,
the streambuf may contain additional data beyond the delimiter.
An application will typically leave that data in the streambuf for a
subsequent async_read_until operation to examine.
*/
/* didn't work
std::istream is(&receive_buffer);
std::string line;
std::getline(is, line);
*/
// clean up incomming buffer but it didn't work
receive_buffer.consume(bytes_transferred);
receive();
}
else if (error != boost::asio::error::operation_aborted)
{
std::cout << "Client Disconnected\n";
m_connection_manager.remove(shared_from_this());
}
}
Either using a std::istream and reading from it, such as by std::getline(), or explicitly invoking boost::asio::streambuf::consume(n), will remove data from the input sequence.
If the application is performing either of these and subsequent read_until() operations results in duplicated data in receive_buffer's input sequence, then the duplicated data is likely originating from the remote peer. If the remote peer is writing to the socket and directly using a streambuf's input sequence, then the remote peer needs to explicitly invoke consume() after each successful write operation.
As noted in the documentation, successful read_until() operations may contain additional data beyond the delimiter, including additional delimiters. For instance, if "a#b#" is written to a socket, a read_until() operation using '#' as a delimiter may read and commit "a#b#" to the streambuf's input sequence. However, the operation will indicate that the amount of bytes transferred is that up to and including the first delimiter. Thus, bytes_transferred would be 2 and streambuf.size() would be 4. After 2 bytes have been consumed, the streambuf's input sequence would contain "b#", and a subsequent call to read_until() will return immediately, as the streambuf already contains the delimiter.
Here is a complete example demonstrating streambuf usage for reading and writing, and how the input sequence is consumed:
#include <iostream>
#include <boost/asio.hpp>
#include <boost/bind.hpp>
// This example is not interested in the handlers, so provide a noop function
// that will be passed to bind to meet the handler concept requirements.
void noop() {}
std::string make_string(boost::asio::streambuf& streambuf)
{
return {buffers_begin(streambuf.data()),
buffers_end(streambuf.data())};
}
int main()
{
using boost::asio::ip::tcp;
boost::asio::io_service io_service;
// Create all I/O objects.
tcp::acceptor acceptor(io_service, tcp::endpoint(tcp::v4(), 0));
tcp::socket server_socket(io_service);
tcp::socket client_socket(io_service);
// Connect client and server sockets.
acceptor.async_accept(server_socket, boost::bind(&noop));
client_socket.async_connect(acceptor.local_endpoint(), boost::bind(&noop));
io_service.run();
// Write to server.
boost::asio::streambuf write_buffer;
std::ostream output(&write_buffer);
output << "a#"
"b#";
write(server_socket, write_buffer.data());
std::cout << "Wrote: " << make_string(write_buffer) << std::endl;
assert(write_buffer.size() == 4); // Data not consumed.
// Read from the client.
boost::asio::streambuf read_buffer;
// Demonstrate consuming via istream.
{
std::cout << "Read" << std::endl;
auto bytes_transferred = read_until(client_socket, read_buffer, '#');
// Verify that the entire write_buffer (data pass the first delimiter) was
// read into read_buffer.
auto initial_size = read_buffer.size();
assert(initial_size == write_buffer.size());
// Read from the streambuf.
std::cout << "Read buffer contains: " << make_string(read_buffer)
<< std::endl;
std::istream input(&read_buffer);
std::string line;
getline(input, line, '#'); // Consumes from the streambuf.
assert("a" == line); // Note getline discards delimiter.
std::cout << "Read consumed: " << line << "#" << std::endl;
assert(read_buffer.size() == initial_size - bytes_transferred);
}
// Write an additional message to the server, but only consume 'a#'
// from write buffer. The buffer will contain 'b#c#'.
write_buffer.consume(2);
std::cout << "Consumed write buffer, it now contains: " <<
make_string(write_buffer) << std::endl;
assert(write_buffer.size() == 2);
output << "c#";
assert(write_buffer.size() == 4);
write(server_socket, write_buffer.data());
std::cout << "Wrote: " << make_string(write_buffer) << std::endl;
// Demonstrate explicitly consuming via the streambuf.
{
std::cout << "Read" << std::endl;
auto initial_size = read_buffer.size();
auto bytes_transferred = read_until(client_socket, read_buffer, '#');
// Verify that the read operation did not attempt to read data from
// the socket, as the streambuf already contained the delimiter.
assert(initial_size == read_buffer.size());
// Read from the streambuf.
std::cout << "Read buffer contains: " << make_string(read_buffer)
<< std::endl;
std::string line(
boost::asio::buffers_begin(read_buffer.data()),
boost::asio::buffers_begin(read_buffer.data()) + bytes_transferred);
assert("b#" == line);
assert(read_buffer.size() == initial_size); // Nothing consumed.
read_buffer.consume(bytes_transferred); // Explicitly consume.
std::cout << "Read consumed: " << line << std::endl;
assert(read_buffer.size() == 0);
}
// Read again.
{
std::cout << "Read" << std::endl;
read_until(client_socket, read_buffer, '#');
// Read from the streambuf.
std::cout << "Read buffer contains: " << make_string(read_buffer)
<< std::endl;
std::istream input(&read_buffer);
std::string line;
getline(input, line, '#'); // Consumes from the streambuf.
assert("b" == line); // Note "b" is expected and not "c".
std::cout << "Read consumed: " << line << "#" << std::endl;
std::cout << "Read buffer contains: " << make_string(read_buffer)
<< std::endl;
}
}
Output:
Wrote: a#b#
Read
Read buffer contains: a#b#
Read consumed: a#
Consumed write buffer, it now contains: b#
Wrote: b#c#
Read
Read buffer contains: b#
Read consumed: b#
Read
Read buffer contains: b#c#
Read consumed: b#
Read buffer contains: c#
I'm trying to build a thread that checks for user input and if the input equals "exit", it turns off all other threads.
The way that I use cin seems to stop the thread. The thread should run, check for user input, and if there is any and it equals "exit", turn runProcesses off.
This is my code that doesn't work as expected as "newline stopped" never is printed, "running" is printed only once:
void check_for_cin() {
while ( runProcesses ) {
cout << "running";
string input;
std::getline( std::cin, input );
//while ( std::getline( std::cin, input ) ) {
if ( !input.empty() ) {
if ( input == "exit" ) {
runProcesses = false;
cout << "exiting" << ", run processes: " << runProcesses;
}
}
cout << "newline stopped";
boost::this_thread::sleep( boost::posix_time::seconds( 1 ) );
}
cout << "no longer checking for input";
}
How can my intent be done?
Look at Asio's file descriptor service objects.
Posix has a 'reactor' style asynchrony, so you don't actually need threads to achieve asynchronicity.
My example shows a reading loop that exits when 'exit' is typed /or/ when a timeout expires (10s).
#include <boost/asio.hpp>
#include <boost/asio/posix/stream_descriptor.hpp>
boost::asio::io_service my_io_service;
boost::asio::posix::stream_descriptor in(my_io_service, ::dup(STDIN_FILENO));
boost::asio::deadline_timer timer(my_io_service);
// handle timeout
void timeout_expired(boost::system::error_code ec) {
if (!ec)
std::cerr << "Timeout expired\n";
else if (ec == boost::asio::error::operation_aborted) // this error is reported on timer.cancel()
std::cerr << "Leaving early :)\n";
else
std::cerr << "Exiting for another reason: " << ec.message() << "\n";
// stop the reading loop
in.cancel();
in.close();
}
// set timeout timer
void arm_timeout()
{
timer.expires_from_now(boost::posix_time::seconds(10));
timer.async_wait(timeout_expired);
}
// perform reading loop
void reading_loop()
{
std::cerr << "(continueing input...)\n";
static boost::asio::streambuf buffer; // todo some encapsulation :)
async_read_until(in, buffer, '\n', [&](boost::system::error_code ec, size_t bytes_transferred) {
if (!ec)
{
std::string line;
std::istream is(&buffer);
if (std::getline(is, line) && line == "exit")
ec = boost::asio::error::operation_aborted;
else
reading_loop(); // continue
}
if (ec)
{
std::cerr << "Exiting due to: " << ec.message() << "\n";
// in this case, we don't want to wait until the timeout expires
timer.cancel();
}
});
}
int main() {
arm_timeout();
reading_loop();
my_io_service.run();
}
On windows, you can use the equivalent Windows Stream Handle
You could trivially add threads by running my_io_service.run() on more than one thread.