boost read_until stops at white space - c++

I am trying to setup a tcp socketing server with boost. For some reason the function read_until seems to be stopping at white space rather then going all the way until the delimiter.
for example:
sent
┼N▀]»ü rx}q╠Cä≥è┘Y\║ï2æ╨╬ΣfV╠παÇ/S┬0è3à ╫VR∞{εoÆ?LeN≡╬.lÖnÖ1⌡&âm&ù╫ä╛'°≈L▀_ °çF¿P»2ß|╪+96#3kα≥
╟¬─╣╩í▄¢hú╤fûº╢5~AcbF┌Zd╒∞?╓)a.ƒ¿B■αZº=■uΣ╔nÜ┌╬▌╝>┌iE┌y≈ÿ≤┴Kå ²å£∩¢R>╒S(y╙cPjA▀▀Z2O╓? ÆÉ#τß╢ªy╗▒*Γ▓σ&K₧#╦╩∙⌠%ßΩ-x*Ü╞7ε_█zâ╡C
╧╩║╗Q■═TM╠<æ┤päi^▓'àiUóα<«3Çÿ ─╗E Σ]ππa╒εk»╣╕(╔╡╙ä╝y≡╥¡╠▌╪┼¡Ö
a|MC├₧\y╚üßσ√⌡ÿ±2<æq}ÿ┌Mzçα∩òΣÆ{end}
recived:
┼N▀]»ü
My code that reads from the socket is
string read_(tcp::socket& socket, CryptoPP::RSA::PrivateKey *privateKey) {
boost::asio::streambuf buf;
boost::asio::read_until(socket, buf, "{end}");
string data = boost::asio::buffer_cast<const char*>(buf.data());
std::cout << "recived:\n" << data << std::endl;
return data
}
solution:
string read_(tcp::socket& socket, CryptoPP::RSA::PrivateKey *privateKey) {
boost::asio::streambuf buf;
boost::asio::read_until(socket, buf, "{end}");
streambuf::const_buffers_type buf2 = buf.data();
string data(buffers_begin(buf2), buffers_begin(buf2) + buf.size());
std::cout << "recived:\n" << data << std::endl;
return data
}

You should output the characters in a safe way, e.g. by printing each byte as 2 hex digits:
auto read_(tcp::socket& socket, CryptoPP::RSA::PrivateKey * /*privateKey*/) {
std::vector<uint8_t> data;
boost::asio::read_until(
socket,
boost::asio::dynamic_buffer(data),
"{end}");
std::cout << "received:\n";
for (int ch : data) {
std::cout << " " << std::hex << std::setw(2) << std::setfill('0') << ch;
}
return data;
}
Note that I elected to:
omit the intermediate streambuf
used the opportunity to demonstrate that you can easily use more apt data structures (std::vector<uint8_t>), although you can, of course, replace that with std::string

Related

Boost::asio::streambuf consume() doesn't empty the buffer

I have following implementation of a function reading serialized data.
The only problem is that buffer to which the data is written doesn't seem to overwrite the data.
Instead after each function call, buffer appends new data.
I read about consume() which I belive would make it work, but calling it doesn't empty the buffer.
Code below:
void Client::read_msg() {
boost::asio::async_read_until(socket, stream_buf, "\n", [this](boost::system::error_code ec, std::size_t) {
if (!ec) {
std::istream is(&stream_buf);
std::getline(is, read_msg_string);
ss << read_msg_string;
cereal::BinaryInputArchive iarchive(ss);
iarchive(txt);
std::cerr << txt.header << " " << txt.body;
stream_buf.consume(stream_buf.size());
this->read_msg();
} else {
socket.close();
}
});
}
A few ideas:
Reading a binary archive with a single getline is definitely a bad idea (you would never read past a byte having 0x0a (linefeed)).
close streams/archives before modifying the underlying buffers/streams
use the transferred size or keep buffer_size from before the read operations (I don't think this should matter, but it's one variable to eliminate)
So, for a start there would be
std::stringstream ss;
ss << &stream_buf;
{
cereal::BinaryInputArchive iarchive(ss);
iarchive(txt);
std::cerr << txt.header << " " << txt.body;
}
But I think it could just be
{
std::istream is(&stream_buf);
cereal::BinaryInputArchive iarchive(is);
iarchive(txt);
std::cerr << txt.header << " " << txt.body;
}
As far as I know, the streambuf operations shown will already call consume() as required.
If you expect trailing data to be present, you probably will not want to consume it, but rather close the socket (if you blindly ignore trailing data, the conversation partner will end up confused about what you have received)

C++ no output, boost.asio

I'm about to write an IRCBot using Boost.Asio and I have the function getMsg:
std::string getMsg()
{
buffer.clear(); //make sure buffer is empty
buffer.resize(512); //make sure it's big enough for 512char
socket.read_some(boost::asio::buffer(&buffer[0],buffer.size()));
std::size_t pos = buffer.find("PING :");
if(pos != std::string::npos)
{
sendMsg("PONG :" + buffer.substr(pos + 6));
}
return buffer;
}
In my main function when using std::cout << Text; I get an output, but when trying std::cout << "Hello", nothing seems to happen:
while(true)
{
std::string Text = Test.getMsg();
std::cout << Text; //OUTPUT
}
while(true)
{
std::string Text = Test.getMsg();
std::cout << "TEST"; //NO OUTPUT ---- WHY?
}
The error you are asking about most likely occurs because you don't flush the stdout: std::cout << "TEST" << std::flush; This has nothing to do with boost::asio.
However your asio code also has a possible error: You are looking for PING : there in a single read call which might never be received within a single read call, due to the fact of how TCP works (it's a stream, not packets). If it's UDP socket it would work.

Sending data through ZeroMQ (zmqpp) using MsgPack gives 'msgpack::v1::insufficient_bytes' error

I made a PUB/SUB connection using zmqpp and now I want to send data from the publisher to the subscribers using the header-only, C++11 version of msgpack-c.
The publisher has to send 2 int64_t numbers -- header_1 and header_2 -- followed by a std::vector<T> -- data --, where T is determined by the (header_1, header_2) combination.
Sinse there aren't that many examples on how to combine msgpack and zmqpp, the idea I came up with is to send a 3-part message by using zmqpp::message::add/add_raw. Each part would be packed/unpacked using msgpack.
The publisher packs a single data part as follows:
zmqpp::message msg;
int64_t header_1 = 1234567;
msgpack::sbuffer buffer;
msgpack::pack(buffer, header_1);
msg.add(buffer.data(), buffer.size());
And the receiver unpacks it like this:
zmqpp::message msg;
subscriberSock.receive(msg);
int64_t header_1;
msgpack::unpacked unpackedData;
// crash !
msgpack::unpack(unpackedData,
static_cast<const char*>(msg.raw_data(0)),
msg.size(0));
unpackedData.get().convert(&header_1);
When I run the code, I get the following error on the subscriber side:
terminate called after throwing an instance of 'msgpack::v1::insufficient_bytes'
what(): insufficient bytes
Aborted
Also, it seems that zmqpp has generated a 5-part message, even though I called add() only 3 times.
Q1: Am I packing/unpacking the data correctly ?
Q2: Is this the proper method for sending msgpack buffers using zmqpp ?
Here are the important parts of the code:
Publisher
zmqpp::socket publisherSock;
/* connection setup stuff ...*/
// forever send data to the subscribers
while(true)
{
zmqpp::message msg;
// meta info about the data
int64_t header_1 = 1234567;
int64_t header_2 = 89;
// sample data
std::vector<double> data;
data.push_back(1.2);
data.push_back(3.4);
data.push_back(5.6);
{
msgpack::sbuffer buffer;
msgpack::pack(buffer, header_1);
msg.add(buffer.data(), buffer.size());
cout << "header_1:" << header_1 << endl; // header_1:1234567
}
{
msgpack::sbuffer buffer;
msgpack::pack(buffer, header_2);
msg.add(buffer.data(), buffer.size());
cout << "header_2:" << header_2 << endl; // header_2:89
}
{
msgpack::sbuffer buffer;
msgpack::pack(buffer, data);
msg.add_raw(buffer.data(), buffer.size());
std::cout << "data: " << data << std::endl; // data:[1.2 3.4 5.6]
}
std::cout << msg.parts() << " parts" << std::endl; // prints "5 parts"... why ?
publisherSock.send(msg);
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
Subscriber
zmqpp::socket subscriberSock;
/* connection setup stuff ...*/
zmqpp::message msg;
subscriberSock.receive(msg);
int64_t header_1;
int64_t header_2;
std::vector<double> data;
std::cout << msg.parts() << " parts" << std::endl; // prints "5 parts"
{
// header 1
{
msgpack::unpacked unpackedData;
// crash !
msgpack::unpack(unpackedData,
static_cast<const char*>(msg.raw_data(0)),
msg.size(0));
unpackedData.get().convert(&header_1);
cout << "header_1:" << header_1 << endl;
}
// header 2
{
msgpack::unpacked unpackedData;
msgpack::unpack(unpackedData,
static_cast<const char*>(msg.raw_data(1)),
msg.size(1));
unpackedData.get().convert(&header_2);
cout << "header_2:" << header_2 << endl;
}
// data
{
msgpack::unpacked unpacked_data;
msgpack::unpack(unpacked_data,
static_cast<const char*>(msg.raw_data(2)),
msg.size(2));
unpacked_data.get().convert(&data);
std::cout << "data:" << data << std::endl;
}
}
EDIT: Problem solved: As pointed out by #Jens, the correct way of packing/sending data is by using zmqpp::message::add_raw()
zmqpp::message msg;
int64_t header_1 = 1234567;
msgpack::sbuffer buffer;
msgpack::pack(buffer, header_1);
msg.add_raw(buffer.data(), buffer.size());
I think the calls to msg.add(buffer.data(), buffer.size()do not add a array of buffer.size() bytes, but call message::add(Type const& part, Args &&...args), which
msg << buffer.data(), which probably calls message::operator<<(bool) since a pointer converts to bool
add(buffer.size()) which then calls msg << buffer.size(), which adds a size_t value as the next part.
Looking at the zmqpp::message class, using message::add_raw should do the trick.
PS: This is all without any guarantee because I have never used zmqpp or msgpack.

Working with boost::asio::streambuf

Looking for a boost::asio (and with himself boost) decided to write asynchronous server. To store incoming data I use boost::asio::streambuf.
Here I have a problem. When I receive a second message from the client and subsequent I see that in the buffer contains a data from previous messages.
Although I call Consume method at the input buffer. What's wrong?
class tcp_connection
// Using shared_ptr and enable_shared_from_this
// because we want to keep the tcp_connection object alive
// as long as there is an operation that refers to it.
: public boost::enable_shared_from_this<tcp_connection>
{
...
boost::asio::streambuf receive_buffer;
boost::asio::io_service::strand strand;
}
...
void tcp_connection::receive()
{
// Read the response status line. The response_ streambuf will
// automatically grow to accommodate the entire line. The growth may be
// limited by passing a maximum size to the streambuf constructor.
boost::asio::async_read_until(m_socket, receive_buffer, "\r\n",
strand.wrap(boost::bind(&tcp_connection::handle_receive, shared_from_this()/*this*/,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred)));
}
void tcp_connection::handle_receive(const boost::system::error_code& error,
std::size_t bytes_transferred)
{
if (!error)
{
// process the data
/* boost::asio::async_read_until remarks
After a successful async_read_until operation,
the streambuf may contain additional data beyond the delimiter.
An application will typically leave that data in the streambuf for a
subsequent async_read_until operation to examine.
*/
/* didn't work
std::istream is(&receive_buffer);
std::string line;
std::getline(is, line);
*/
// clean up incomming buffer but it didn't work
receive_buffer.consume(bytes_transferred);
receive();
}
else if (error != boost::asio::error::operation_aborted)
{
std::cout << "Client Disconnected\n";
m_connection_manager.remove(shared_from_this());
}
}
Either using a std::istream and reading from it, such as by std::getline(), or explicitly invoking boost::asio::streambuf::consume(n), will remove data from the input sequence.
If the application is performing either of these and subsequent read_until() operations results in duplicated data in receive_buffer's input sequence, then the duplicated data is likely originating from the remote peer. If the remote peer is writing to the socket and directly using a streambuf's input sequence, then the remote peer needs to explicitly invoke consume() after each successful write operation.
As noted in the documentation, successful read_until() operations may contain additional data beyond the delimiter, including additional delimiters. For instance, if "a#b#" is written to a socket, a read_until() operation using '#' as a delimiter may read and commit "a#b#" to the streambuf's input sequence. However, the operation will indicate that the amount of bytes transferred is that up to and including the first delimiter. Thus, bytes_transferred would be 2 and streambuf.size() would be 4. After 2 bytes have been consumed, the streambuf's input sequence would contain "b#", and a subsequent call to read_until() will return immediately, as the streambuf already contains the delimiter.
Here is a complete example demonstrating streambuf usage for reading and writing, and how the input sequence is consumed:
#include <iostream>
#include <boost/asio.hpp>
#include <boost/bind.hpp>
// This example is not interested in the handlers, so provide a noop function
// that will be passed to bind to meet the handler concept requirements.
void noop() {}
std::string make_string(boost::asio::streambuf& streambuf)
{
return {buffers_begin(streambuf.data()),
buffers_end(streambuf.data())};
}
int main()
{
using boost::asio::ip::tcp;
boost::asio::io_service io_service;
// Create all I/O objects.
tcp::acceptor acceptor(io_service, tcp::endpoint(tcp::v4(), 0));
tcp::socket server_socket(io_service);
tcp::socket client_socket(io_service);
// Connect client and server sockets.
acceptor.async_accept(server_socket, boost::bind(&noop));
client_socket.async_connect(acceptor.local_endpoint(), boost::bind(&noop));
io_service.run();
// Write to server.
boost::asio::streambuf write_buffer;
std::ostream output(&write_buffer);
output << "a#"
"b#";
write(server_socket, write_buffer.data());
std::cout << "Wrote: " << make_string(write_buffer) << std::endl;
assert(write_buffer.size() == 4); // Data not consumed.
// Read from the client.
boost::asio::streambuf read_buffer;
// Demonstrate consuming via istream.
{
std::cout << "Read" << std::endl;
auto bytes_transferred = read_until(client_socket, read_buffer, '#');
// Verify that the entire write_buffer (data pass the first delimiter) was
// read into read_buffer.
auto initial_size = read_buffer.size();
assert(initial_size == write_buffer.size());
// Read from the streambuf.
std::cout << "Read buffer contains: " << make_string(read_buffer)
<< std::endl;
std::istream input(&read_buffer);
std::string line;
getline(input, line, '#'); // Consumes from the streambuf.
assert("a" == line); // Note getline discards delimiter.
std::cout << "Read consumed: " << line << "#" << std::endl;
assert(read_buffer.size() == initial_size - bytes_transferred);
}
// Write an additional message to the server, but only consume 'a#'
// from write buffer. The buffer will contain 'b#c#'.
write_buffer.consume(2);
std::cout << "Consumed write buffer, it now contains: " <<
make_string(write_buffer) << std::endl;
assert(write_buffer.size() == 2);
output << "c#";
assert(write_buffer.size() == 4);
write(server_socket, write_buffer.data());
std::cout << "Wrote: " << make_string(write_buffer) << std::endl;
// Demonstrate explicitly consuming via the streambuf.
{
std::cout << "Read" << std::endl;
auto initial_size = read_buffer.size();
auto bytes_transferred = read_until(client_socket, read_buffer, '#');
// Verify that the read operation did not attempt to read data from
// the socket, as the streambuf already contained the delimiter.
assert(initial_size == read_buffer.size());
// Read from the streambuf.
std::cout << "Read buffer contains: " << make_string(read_buffer)
<< std::endl;
std::string line(
boost::asio::buffers_begin(read_buffer.data()),
boost::asio::buffers_begin(read_buffer.data()) + bytes_transferred);
assert("b#" == line);
assert(read_buffer.size() == initial_size); // Nothing consumed.
read_buffer.consume(bytes_transferred); // Explicitly consume.
std::cout << "Read consumed: " << line << std::endl;
assert(read_buffer.size() == 0);
}
// Read again.
{
std::cout << "Read" << std::endl;
read_until(client_socket, read_buffer, '#');
// Read from the streambuf.
std::cout << "Read buffer contains: " << make_string(read_buffer)
<< std::endl;
std::istream input(&read_buffer);
std::string line;
getline(input, line, '#'); // Consumes from the streambuf.
assert("b" == line); // Note "b" is expected and not "c".
std::cout << "Read consumed: " << line << "#" << std::endl;
std::cout << "Read buffer contains: " << make_string(read_buffer)
<< std::endl;
}
}
Output:
Wrote: a#b#
Read
Read buffer contains: a#b#
Read consumed: a#
Consumed write buffer, it now contains: b#
Wrote: b#c#
Read
Read buffer contains: b#
Read consumed: b#
Read
Read buffer contains: b#c#
Read consumed: b#
Read buffer contains: c#

C++ , return string from function; boost::asio read / write

I get a compile error, additionally I cannot boost::asio::read buf without giving it array elements.
std::string eport::read_data (void)
{
io_service io; // create the I/O service that talks to the serial device
serial_port port (io, PORT); // create the serial device, note it takes the io service and the port name
error_code ec; // address used for error checking
std::string buf [100]; // data with crc on end
try
{
read (port, buffer (buf), ec);
std::cout << "eport::read: result: " << buf << std::endl;
}
catch (error_code &ec)
{
std::cout << "eport::read: ERROR: " << ec << std::endl;
return "error";
}
std::cout << "eport::read: SUCCESS" << std::endl;
return buf;
The error:
eport.cc:83:9: error: could not convert ‘(std::string*)(& buf)’ from ‘std::string* {aka std::basic_string<char>*}’ to ‘std::string {aka std::basic_string<char>}’
Does the function need to be cast as const char* ? I am not sure what is wrong. Any help is appreciated, thank you.
UPDATED CODE
This is my code. I hope it can help someone because asio lacks good examples on the web. I know my write function could be written better, and this code has not been tested so I'm not sure if I'm doing this right or not. Thanks.
#include "../include/main.H"
#include <boost/asio.hpp> // asynchronous input/output
#include <boost/crc.hpp> // cyclic redundancy code (for data checking)
using namespace::boost::system;
using namespace::boost::asio;
const char *PORT = "/dev/ttyS0";
// serial port communication setup
serial_port_base::baud_rate BAUD (9600); // what baud rate do we communicate at (default is 9600)
serial_port_base::character_size C_SIZE (8); // how big is each "packet" of data (default is 8 bits)
serial_port_base::flow_control FLOW (serial_port_base::flow_control::none); // what flow control is used (default is none)
serial_port_base::parity PARITY (serial_port_base::parity::none); // what parity is used (default is none)
serial_port_base::stop_bits STOP (serial_port_base::stop_bits::one); // how many stop bits are used (default is one)
int eport::initialize (void)
{
io_service io; // create the I/O service that talks to the serial device
serial_port port (io, PORT); // create the serial device, note it takes the io service and the port name
// set serial port options
port.set_option (BAUD);
port.set_option (C_SIZE);
port.set_option (FLOW);
port.set_option (PARITY);
port.set_option (STOP);
return 0;
}
int eport::write_data (std::string data)
{
io_service io; // create the I/O service that talks to the serial device
serial_port port (io, PORT); // create the serial device, note it takes the io service and the port name
error_code ec; // address used for error checking
boost::crc_32_type crcresult; // used for communication checking
char buf [1024]; // buffer to hold data
int crc; // holds crc value
std::ostringstream convert; // used to convert int to string
std::string data_crc; // data with crc on end
std::stringstream ss; // used to add strings
strncpy (buf, data.c_str(), sizeof(buf)); // put data into buffer
buf [sizeof(buf) - 1] = 0; // make sure the last element has a null
crcresult.process_bytes (buf, sizeof(buf)); // get crc value from buffer contents
crc = crcresult.checksum(); // put crc value into integer
convert << crc; // convert integer to string
ss << data << convert.str (); // add crc string to data string
data_crc = ss.str (); // data string with crc appended to be used in reading / writing
std::cout << "eport::write: data with crc: " << data_crc << std::endl;
std::cout << "eport::write: writing: " << data_crc << std::endl;
write (port, buffer (data_crc, sizeof(data_crc)), ec); // write data with crc to serial device
if (ec) // if error code is true, print and return
{
std::cout << "eport::write: ERROR: " << ec << std::endl;
return -1;
}
std::cout << "eport::write: SUCCESS" << std::endl;
return crc;
}
std::string eport::read_data (void)
{
io_service io; // create the I/O service that talks to the serial device
serial_port port (io, PORT); // create the serial device, note it takes the io service and the port name
error_code ec; // address used for error checking
streambuf sb; // asio stream buffer to hold read data
std::string buf; // read buffer will be put into this string
size_t transferred = read (port, sb, ec); // read data from serial device
buf.resize (transferred); // resize the string to the read data size
sb.sgetn (&buf[0], buf.size ()); // stores characters from the stream to the array
std::cout << "eport::read: result: " << buf << std::endl;
if (ec)
{
std::cout << "eport::read: ERROR: " << ec << std::endl;
return "error";
}
std::cout << "eport::read: SUCCESS" << std::endl;
return buf;
}
The most generic way would be use a asio::streambuf
streambuf sb;
size_t transferred = read (port, sb, ec);
According to the docs:
This function is used to read a certain number of bytes of data from a stream. The call will block until one of the following conditions is true:
The supplied buffer is full (that is, it has reached maximum size).
An error occurred.
This operation is implemented in terms of zero or more calls to the stream's read_some function.
Then, copy it to a string:
std::string buf;
buf.resize(transferred);
sb.sgetn(&buf[0], buf.size());
Alternatively, preallocate a buffer of the expected size:
std::string buf(100u, '\0');
size_t transferred = read (port, buffer(buf), ec);
buf.resize(transferred);
For more complicated scenarios, use read_until:
streambuf sb;
size_t transferred = read_until(port, sb, "\r\n", ec);
This will read until "\r\n" was encountered (note: may read more than that, but won't invoke read_some again after seeing the delimiter).
Even more complicated stop conditions could use the overload that takes a MatchCondition functor.
Note on exception handling
If you pass ec to receive the error_code there will be no exceptions thrown
buf is an array of std::string. You should change your prototype or return just one string. buf[0] for example.
Most possibly what you want is:
std::string buf; // No [100]
There are issues with your code that you will need to answer, more specifically, how do you know the number of characters that will be sent to your read function?
However, the general answer to your question is to use a character array, and then return this as the std::string:
std::string eport::read_data (void)
{
io_service io; // create the I/O service that talks to the serial device
serial_port port (io, PORT); // create the serial device, note it takes the io service and the port name
error_code ec; // address used for error checking
char buf [100]; // data with crc on end
try
{
read (port, buf, ec);
std::cout << "eport::read: result: " << buf << std::endl;
}
catch (error_code &ec)
{
std::cout << "eport::read: ERROR: " << ec << std::endl;
return "error";
}
std::cout << "eport::read: SUCCESS" << std::endl;
return buf;
}
The std::string constructor will take care of copying the buf at the end to a std::string.
Now, if there is a way to determine the number of characters read, then the function has to be written differently. Most read functions have a parameter specifying the maximum number of characters to read, and somewhere it is returned the number of characters that are read.
Assuming you could rewrite (or call) a different read function that has both of these properties, the code would look like this:
std::string eport::read_data (void)
{
io_service io; // create the I/O service that talks to the serial device
serial_port port (io, PORT); // create the serial device, note it takes the io service and the port name
error_code ec; // address used for error checking
char buf [100]; // data with crc on end
int numCharsRead = 0;
try
{
numCharsRead = read2 (port, buf, 100, ec);
std::cout << "eport::read: result: " << buf << std::endl;
}
catch (error_code &ec)
{
std::cout << "eport::read: ERROR: " << ec << std::endl;
return "error";
}
std::cout << "eport::read: SUCCESS" << std::endl;
return std::string(buf, numCharsRead);
}
Note the difference in the return. A std::string is constructed from the character array, but only up to numCharsRead characters.