What is the meaning of boost::asio::placeholders::bytes_transferred in async_read_until()? In the callback function it returns smaller value, than streambuf.size(). streambuf was clear before the callback. To sum up,...bytes_transferred is not the actual number of bytes went through the socket, but less. Do I have misunderstood all of this, or what?
EDIT: I read the following protocol from a socket:
Y43,72,0,,91009802000000603=0000000000000000000
"Y43," - is the header.
"Y" - is message type.
"43" - additional bytes to read
"," - delimiter. The header is the until the first "," encountered.
My code is for reading is like:
void handle_write(const boost::system::error_code& error,
size_t bytes_transferred)
{
if (!error)
{
boost::asio::async_read_until(
socket_,
inputStreamBuffer_,
',',
boost::bind(
&client::handle_read1, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred
)
);
}
else
{
std::cout << "Write failed: " << error << "\n";
}
}
void handle_read1(const boost::system::error_code& error,
size_t bytes_transferred)
{
cout << "bytes_transferred=" << bytes_transferred << endl;
if (!error)
{
cout << "0 size=" << inputStreamBuffer_.size() << endl;
istream is(&inputStreamBuffer_);
char c[1000];
is.read(c,bytes_transferred);
c[bytes_transferred]=0;
for (int i=0;i<bytes_transferred;++i)
{
cout << dec << "c[" << i << "]=" << c[i] << " hex=" << hex << static_cast<int>(c[i]) << "#" << endl;
}
}
else
{
std::cout << "Read failed: " << error << "\n";
}
}
For stream sent from the other side:
Y43,71,0,,91009802000000595=0000000000000000000
Some times, I read this:
bytes_transferred=4
0 size=47
c[0]=Y hex=59#
c[1]=4 hex=34#
c[2]=3 hex=33#
c[3]=, hex=2c#
For stream sent from the other side:
Y43,72,0,,91009802000000603=0000000000000000000
But other times, I read this:
bytes_transferred=7
0 size=47
c[0]= hex=0#
c[1]= hex=0#
c[2]= hex=0#
c[3]= hex=0#
c[4]=7 hex=37#
c[5]=2 hex=32#
c[6]=, hex=2c#
The socket is secured with SSL, and the client and server apps are slightly modified examples from boost_asio/example/ssl/* .
In the second example I loose the entire header :(
There's four overloads of the function but let's just assume the first one is used. If you look at the documentation, then you'll see that bytes_transferred is the amount of bytes to and including the delimiter specified.
And furthermore:
After a successful async_read_until operation, the streambuf may contain additional data beyond the delimiter. An application will typically leave that data in the streambuf for a subsequent async_read_until operation to examine.
Resolved. I was passing std::string object to boost::asio::buffer(), instead of std::string.c_str() when sending the reply from the server.
As the docs suggest, you should be able to ignore anything beyond bytes_transferred and just call async_read_until again.
However if you happen to be using the all-new SSL implementation in ASIO 1.5.3 (which is not officially part of boost yet), you might run into the same issue I did (for which I submitted a patch):
http://comments.gmane.org/gmane.comp.lib.boost.asio.user/4803
It doesn't look like you're using the new version or running into the same problem, but it's something to be aware of if you hit some limitations and are tempted by the advantages of the new implementation:
The new implementation compiles faster, shows substantially improved performance, and supports custom memory allocation and handler invocation. It includes new API features such as certificate verification callbacks and has improved error reporting. The new implementation is source-compatible with the old for most uses.
Related
I have this requirement where my app have to connect to another app via sockets and will have to maintain persistent connection for quiet long time. My app will be a TCP client and the other is a TCP server. My app will send commands and the server will respond accordingly.
The problem am facing right now is how to read the whole data from server a string and return for app which will issue the next command. Reading synchronously (with asio::read) looked like a good option up until I observed socket hanging up until I terminate the server. Looking at the documentation I found that the library is correctly working.
his function is used to read a certain number of bytes of data from a stream. The call will block until one of the following conditions is true:
1. The supplied buffers are full. That is, the bytes transferred is equal to the sum of the buffer sizes.
2. An error occurred.
The problem is I don't know correct buffer size as the response from the server varies. So If I put a too small buffer it returns fine but missing some data. If I put too big it will hang forever until server quits.
So I thought I would do the async reading. It works only once and I don't know how to make it fetch data until whole data it read.
here is the relevant async code
#define ASIO_STANDALONE 1
#include <iostream>
#include <asio.hpp>
int main()
{
asio::io_context context;
size_t reply_length;
size_t length = 1024;
std::vector<char> buffer;
//create socket
asio::ip::tcp::socket socket(context);
socket.connect(asio::ip::tcp::endpoint(asio::ip::address::from_string("127.0.0.1"), 8088));
std::string dataOut = "list --files"; //some command to write
std::error_code error;
asio::write(socket, asio::buffer(dataOut), error);
if (!error)
{
std::cout << "Receiving...!" << std::endl;
buffer.resize(length);
asio::async_read(socket, asio::buffer(buffer), [&buffer, &context](const asio::error_code &ec, std::size_t bytes_transferred) {
std::copy(buffer.begin(), buffer.end(), std::ostream_iterator<char>(std::cout, ""));
std::cout << "\nRead total of:" << bytes_transferred << "\n";
context.run();
});
}
else
{
std::cout << "send failed: " << error.message() << std::endl;
}
context.run();
}
Searching didn't help much solving my issue.
So my question is, how can I read all the data in a persistent socket with asio? Am not using boost.
You need to loop async_read calls. If you don't want your client to hang on read operation you can define the smallest possible buffer i.e. 1 byte.
Define function which takes socket, buffer and two additional parameters according to async_read's handler signature, and this function calls itself with async_read to make the loop of async_read calls - it reads until some error occures:
void onRead (
asio::ip::tcp::socket& socket,
std::array<char,1>& buf,
const system::error_code& ec,
std::size_t bytes)
{
if (ec)
{
if (ec == asio::error::eof && bytes == 1)
std::cout << buf[0];
return;
}
std::cout << buf[0];
asio::async_read(socket,asio::buffer(buf),
std::bind(onRead, std::ref(socket), std::ref(buf),
std::placeholders::_1, // error code
std::placeholders::_2)); // transferred bytes
}
and the changes in main:
std::array<char,1> buf;
asio::write(socket, asio::buffer(dataOut), error);
if (!error)
{
std::cout << "Receiving...!" << std::endl;
asio::async_read(socket, asio::buffer(buf),
std::bind(onRead, std::ref(socket), std::ref(buf),
std::placeholders::_1,
std::placeholders::_2));
context.run();
}
else
{
std::cout << "send failed: " << error.message() << std::endl;
}
(I am using Boost, so you should replace system::error_code on asio::error_code).
I have following implementation of a function reading serialized data.
The only problem is that buffer to which the data is written doesn't seem to overwrite the data.
Instead after each function call, buffer appends new data.
I read about consume() which I belive would make it work, but calling it doesn't empty the buffer.
Code below:
void Client::read_msg() {
boost::asio::async_read_until(socket, stream_buf, "\n", [this](boost::system::error_code ec, std::size_t) {
if (!ec) {
std::istream is(&stream_buf);
std::getline(is, read_msg_string);
ss << read_msg_string;
cereal::BinaryInputArchive iarchive(ss);
iarchive(txt);
std::cerr << txt.header << " " << txt.body;
stream_buf.consume(stream_buf.size());
this->read_msg();
} else {
socket.close();
}
});
}
A few ideas:
Reading a binary archive with a single getline is definitely a bad idea (you would never read past a byte having 0x0a (linefeed)).
close streams/archives before modifying the underlying buffers/streams
use the transferred size or keep buffer_size from before the read operations (I don't think this should matter, but it's one variable to eliminate)
So, for a start there would be
std::stringstream ss;
ss << &stream_buf;
{
cereal::BinaryInputArchive iarchive(ss);
iarchive(txt);
std::cerr << txt.header << " " << txt.body;
}
But I think it could just be
{
std::istream is(&stream_buf);
cereal::BinaryInputArchive iarchive(is);
iarchive(txt);
std::cerr << txt.header << " " << txt.body;
}
As far as I know, the streambuf operations shown will already call consume() as required.
If you expect trailing data to be present, you probably will not want to consume it, but rather close the socket (if you blindly ignore trailing data, the conversation partner will end up confused about what you have received)
I would like to create client/server communication programs pair using Boost ASIO + SSL. So I started off with the examples provided by boost, and I learned how that works, and I'm almost ready to develop my communication protocol, except that there's one problem.
So starting from this example, I'm modifying the handle_read() callback function after the handshake. The following is my code. My only modification is: Add another callback function called startComm(), which will start the communication.
void handle_read(const boost::system::error_code& error,
size_t bytes_transferred)
{
if (!error)
{
std::cout << "Reply: ";
std::cout.write(reply_, bytes_transferred);
std::cout << "\n";
boost::asio::async_write(socket_,
boost::asio::buffer(std::string("Now?")),
boost::bind(&SSLClient::startComm, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
else
{
std::cout << "Read failed: " << error.message() << "\n";
}
}
void startComm(const boost::system::error_code& error,
size_t bytes_transferred)
{
if (!error)
{
std::cout << "Reply: ";
std::cout.write(reply_, bytes_transferred); //problem here, bytes transferred should contain the number of received chars not number of written chars
std::cout << "\n";
}
else
{
std::cout << "Read failed: " << error.message() << "\n";
}
}
In the async_write() above, there's an argument boost::asio::placeholders::bytes_transferred which parametrizes my callback function to provide the number of bytes that were sent to the server. Now I would like to know the number of bytes the server responded with. How can I do that in my simple example?
Thanks. If you require any additional details, please ask.
The write call sends data.
Since it doesn't, at all, receive data the number of bytes received is by definition 0.
If you want to receive data, use (async_)read and it will tell you the number of bytes received.
These call backs use the same placeholder (bytes_transferred) but it carries different meaning depending on the direction of the transfer that has been completed.
Here's a solution that technically does what you want: define an extra parameter of startComm and bind it (not with a placeholder).
void handle_read(const boost::system::error_code &error, size_t bytes_received) {
if (!error) {
std::cout << "Reply: ";
std::cout.write(reply_, bytes_received);
std::cout << "\n";
boost::asio::async_write(socket_, boost::asio::buffer(std::string("Now?")),
boost::bind(&SSLClient::startComm,
this,
boost::asio::placeholders::error,
bytes_received,
boost::asio::placeholders::bytes_transferred));
} else {
std::cout << "Read failed: " << error.message() << "\n";
}
}
void startComm(const boost::system::error_code &error, size_t had_received, size_t bytes_sent) {
if (!error) {
std::cout << "Reply: ";
std::cout.write(reply_, had_received);
std::cout << "\n";
} else {
std::cout << "Write failed: " << error.message() << "\n";
}
}
Note that I still think you might mistakenly expect async_write to receive a reply, which (obviously?) isn't the case
I have studied the existing examples:
Sending Protobuf Messages with boost::asio
Reading Protobuf objects using boost::asio::read_async
Google Protocol Buffers: parseDelimitedFrom and writeDelimitedTo for C++
Are there C++ equivalents for the Protocol Buffers delimited I/O functions in Java?
Sending Protobuf Messages with boost::asio
but I still can not figure out how to pass Google Protobuf messages using the Boost::asio API. In particular I have no clear understanding of the following problems:
Interaction between boost::asio::streambuf and google::protobuf::io objects (and the necessity of the applying of the last ones)
Correct implementation of the message streaming (due to the lack of writeDelimitedTo and parseDelimitedFrom methods in C++ API)
Here is my implementation based on boost::asio v. 1.39 ssl_client from examples.
class client
{
public:
client(boost::asio::io_service& io_service, boost::asio::ssl::context& context,
boost::asio::ip::tcp::resolver::iterator endpoint_iterator)
: socket_(io_service, context),
request_stream(&b),
raw_output(&request_stream),
coded_output(&raw_output)
{
...
}
void handle_connect(const boost::system::error_code& error,
boost::asio::ip::tcp::resolver::iterator endpoint_iterator)
{
...
}
//Debugging function
void print_buffers_condition(const char *step)
{
std::cout << "\nBuffer conditions after " << step << std::endl;
std::cout << "boost::asio::streambuf\t\tb: " << b.size() << std::endl;
std::cout << "google::protobuf::io::OstreamOutputStream raw_output: " << raw_output.ByteCount() << std::endl;
std::cout << "google::protobuf::io::CodedOutputStream coded_output: " << coded_output.ByteCount() << std::endl;
std::cout << std::endl;
}
//Sending test message after SSL Handshake
void handle_handshake(const boost::system::error_code& error)
{
std::cout << "-----------------------------SENDING-----------------------------" << std::endl;
print_buffers_condition("handle handshake");
if (!error)
{
SearchRequest msg;
msg.set_query("qwerty");
msg.set_code(12345);
std::cout << "Debugged" << std::endl;
msg.PrintDebugString();
//Writing the length of the message before and serializing
print_buffers_condition("before serialising");
coded_output.WriteVarint32(msg.ByteSize());
if (!msg.SerializeToCodedStream(&coded_output))
{
std::cout << "serailizing error" << std::endl;
}
else
{
std::cout << "serializing success" << std::endl;
}
//Sending
buffers_condition("before async write");
boost::asio::async_write(socket_,
b,
boost::bind(&client::handle_write, this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
buffers_condition("after async write");
}
else
{
std::cout << "Handshake failed: " << error << "\n";
}
}
void handle_write(const boost::system::error_code& error,
size_t bytes_transferred)
{
std::cout << " bytes_trransferred: " << bytes_transferred << std::endl;
if (!error)
{
std::cout << "No error" << std::endl;
...
}
else
{
std::cout << "Write failed: " << error << "\n";
}
}
void handle_read(const boost::system::error_code& error,
size_t bytes_transferred)
{
...
}
private:
boost::asio::ssl::stream<boost::asio::ip::tcp::socket> socket_;
boost::asio::streambuf b;
std::ostream request_stream;
google::protobuf::io::OstreamOutputStream raw_output;
google::protobuf::io::CodedOutputStream coded_output;
};
This code is operational, so after creating the message we fall into the void handle_write(const boost::system::error_code& error, size_t bytes_transferred) function. Printing the bytes_transferred_ value returns 0: server (implemented on the base of examples too) recieves nothing.
The usage of the debugging function void print_buffers_condition(const char *step) hints at loss of message during its transmission through a stack of different buffering objects:
$ ./client 127.0.0.1 5000
-----------------------------SENDING-----------------------------
Buffer conditions after handle handshake
boost::asio::streambuf b: 0
google::protobuf::io::OstreamOutputStream raw_output: 8192
google::protobuf::io::CodedOutputStream coded_output: 0
Debugged:
query: "qwerty"
code: 12345
Buffer conditions after before serialization
boost::asio::streambuf b: 0
google::protobuf::io::OstreamOutputStream raw_output: 8192
google::protobuf::io::CodedOutputStream coded_output: 0
serializing success
Buffer conditions after before async write
boost::asio::streambuf b: 0
google::protobuf::io::OstreamOutputStream raw_output: 8192
google::protobuf::io::CodedOutputStream coded_output: 13
Buffer conditions after after async write
boost::asio::streambuf b: 0
google::protobuf::io::OstreamOutputStream raw_output: 8192
google::protobuf::io::CodedOutputStream coded_output: 13
bytes_trransferred: 0
I have no idea how to do it in a proper way.
OS is RHEL 6.4.
Thank you.
I'm not familiar with asio, but it looks to me like the problem is that you aren't flushing your buffers. The data is stuck in CodedOutputStream and never finds its way into asio.
CodedOutputStream should be allocated on the stack, such that it is destroyed as soon as you're done writing the message. The destructor will flush the buffer. Note that CodedOutputStream is cheap to allocate so there's no performance problem with putting it on the stack (in fact, it's probably better that way).
OstreamOutputStream can similarly be allocated on the stack, but it heap-allocates a buffer which you might want to reuse. If you choose to reuse the same object, make sure to call Flush() to flush the buffer after the CodedOutputStream is destroyed.
Incidentally, OstreamOutputStream is not particularly efficient, because it has to do its own layer of buffering on top of what ostream is already doing. You may want to serialize to a string (str = message.SerializeAsString() or message.SerializeToString(&str)) and then write that directly to the socket (if asio allows this), as it will probably avoid a redundant copy.
I have searched other posts, but didn't found anything relevant.
Now, I have a protocol consisting of header and body.
Protocol is like:
Z24,91009802,123456789ABCDEF
Where Z24, is the header. Z is message type, 24 is remaining bytes to read. Remaining bytes is variable, so I read until first ',' is found.
void handle_handshake(const boost::system::error_code& error)
{
if (!error)
{
boost::asio::async_read_until(
socket_,
inputStreamBuffer_,
',',
boost::bind(
&session::doReadHeader, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred)
);
}
else
{
delete this;
}
}
void doReadHeader(
const boost::system::error_code& error,
size_t bytes_transferred)
{
if (!error)
{
istream is(&inputStreamBuffer_);
vector<char> v(bytes_transferred);
is.read(&(v[0]),bytes_transferred);
request_.append(v.begin(),v.end());
cout << "request_=#" << request_ << "#" << endl;
int nBytes=string_to_llint(request_.substr(1,request_.size()-2));
cout << "nBytes=" << nBytes << endl;
cout << "size=" << inputStreamBuffer_.size() << endl;
boost::asio::async_read(
socket_,
inputStreamBuffer_,
boost::asio::transfer_at_least(nBytes),
boost::bind(
&session::doReadBody, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred)
);
}
else
{
delete this;
}
}
void doReadBody(
const boost::system::error_code& error,
size_t bytes_transferred)
{
if (!error)
{
istream is(&inputStreamBuffer_);
vector<char> v(bytes_transferred);
is.read(&(v[0]),bytes_transferred);
request_.append(v.begin(),v.end());
string response=cardIssueProcessor_.process(request_);
cout << "request=#" << request_ << "#" << endl;
cout << "response=#" << response << "#" << endl;
request_.clear();
boost::asio::async_write(
socket_,
boost::asio::buffer(response, response.size()),
boost::bind(
&session::doWriteResponse, this,
boost::asio::placeholders::error)
);
}
else
{
delete this;
}
}
Now, the header is read. But reading the footer blocks. Apparently the entire message is read in the header call. When I do the second async_read() with boost::asio::transfer_at_least(nBytes), nBytes are already in inputStreamBuffer_, but I think the call doesn't check this?
This is dump from the output:
request_=#Z24,#
nBytes=24
size=24
What is the problem, or how can I workaround it. I am a boost newbie, so all help appreciated. Thank you.
EDIT:
I tried to check the buffer fullness, and don't make async_read() call for the body if it happens to be already read by previous call.
It kind of works, but is it the right solution?
void doReadHeader(
const boost::system::error_code& error,
size_t bytes_transferred)
{
if (!error)
{
istream is(&inputStreamBuffer_);
vector<char> v(bytes_transferred);
is.read(&(v[0]),bytes_transferred);
request_.assign(v.begin(),v.end());
cout << "request_=#" << request_ << "#" << endl;
int nBytes=string_to_llint(request_.substr(1,request_.size()-2));
cout << "nBytes=" << nBytes << endl;
cout << "size=" << inputStreamBuffer_.size() << endl;
size_t toReadBytes=nBytes-inputStreamBuffer_.size();
if (toReadBytes>0)
{
boost::asio::async_read(
socket_,
inputStreamBuffer_,
boost::asio::transfer_at_least(toReadBytes),
boost::bind(
&session::doReadBody, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred)
);
}
else
{
doReadBody(error,nBytes);
}
}
else
{
delete this;
}
}
The Boost ASIO documentation indicates that the async_read_until call may read data into the buffer that is beyond the delimiter (see the Remarks section). That being said, your solution for checking whether the buffer has more data is a good solution given your input.
As I mentioned in my comment above, if your requirements will allow you to do so, using an integral value for the remaining bytes instead of a string is probably going to make your life easier and the code a bit cleaner and less error prone.
async_read_until can read bytes past the delimiter
Remarks
After a successful async_read_until
operation, the streambuf may contain
additional data beyond the delimiter.
An application will typically leave
that data in the streambuf for a
subsequent async_read_until operation
to examine.