This question already has answers here:
boost asio async_write : how to not interleaving async_write calls?
(2 answers)
Closed 6 years ago.
I use boost::asio for asynchronous client and server.
In working process a client send to server data of different type: small service messages (5-50 B) and largest messages (40-200 KB) with raw image data.
When I call Client::send in order (in one thread, successively):
send "small service message";
send "large image message";
I get mixed data (wrong) on the server lake as:
|begin of large message||small message||end of large message|
void Client::send(MessageType type, const void* data, int size, bool read_header_after) {
assert(cstatus.is_connected());
header.type = type;
size_t buf_size = sizeof(ProtocolHeader) + size;
Bytes *p = new Bytes();
p->resize(buf_size);
std::memcpy(&p->front(), &header, sizeof(ProtocolHeader));
if (size) {
std::memcpy(&p->at(sizeof(ProtocolHeader)), data, size);
}
std::cout << "***** SEND start: " << p->size() << " bytes *****" << std::endl;
ba::async_write(*socket, ba::buffer(&p->front(), buf_size),
ba::transfer_exactly(buf_size),
[this, p, read_header_after](const boost::system::error_code& ec, std::size_t length) {
std::cout << "***** SEND complete: "
<< p->size() << " bytes; ec="
<< ec.value() << " (" << ec.message() << ") bufsize="
<< p->size()
<< " *****"
<< std::endl;
size_t buf_size = p->size();
delete p; // remove sent data
if (ec) {
cstatus.set_last_network_error("Client::send " + ec.message());
connection_failed("send - ec");
} else if (length < buf_size) {
connection_failed("send - len");
} else {
if (read_header_after) {
read_header();
}
start_check_timer(NORMAL_INTERVAL_DATA_SEND_MILLISEC);
}
});
}
Output show that small message send to async_write as second but executed (finished) as the first before large message.
***** SEND start: 53147 bytes *****
***** SEND start: 5 bytes *****
***** SEND complete: 5 bytes; ec=0 (Success) bufsize=5 *****
***** SEND complete: 53147 bytes; ec=0 (Success) bufsize=53147 *****
How it possible and how sync this? Thanks!
Update
I don't need queue of sync tasks. I need sync two async_write operations with different buffers sizes.
It is not a bug. The documentation (http://www.boost.org/doc/libs/1_62_0/doc/html/boost_asio/reference/async_write/overload1.html) explicitly sais that no other write operations should be executed on the same socket until there's another one in progress.
This is because an async_write is translated in several calls to async_write_some.
If you are on the same thread, my suggestion is to use a write queue in which you add the data to be send, to then extract it on the write callback to perform another operation.
Related
I'm trying to use Boost.Asio to read from and write to a serial port. Here is my code:
void async_read(boost::asio::serial_port& serial_port)
{
auto buffer = std::make_shared<std::vector<uint8_t>>(64);
serial_port.async_read_some(boost::asio::buffer(*buffer),
[buffer, &serial_port](const boost::system::error_code& error, size_t bytes_read)
{
if (error)
{
std::cout << "Error reading serial port: " << error.message() << std::endl;
return;
}
std::string message(buffer->begin(), buffer->end());
std::cout << "Read " << bytes_read << " bytes:\t" << message << std::endl;
async_read(serial_port);
});
}
void async_write(boost::asio::serial_port& serial_port)
{
auto timer = std::make_shared<boost::asio::deadline_timer>(serial_port.get_io_service(), boost::posix_time::seconds(1));
timer->async_wait(
[&serial_port, timer](const boost::system::error_code& error)
{
if (error)
{
std::cout << "Timer error: " << error.message() << std::endl;
return;
}
auto message = std::make_shared<std::string>("Hello\n");
boost::asio::async_write(serial_port, boost::asio::buffer(*message),
[message, &serial_port](const boost::system::error_code& error, size_t bytes_sent)
{
if (error)
{
std::cout << "Error writing to serial port: " << error.message() << std::endl;
return;
}
std::cout << "Wrote " << bytes_sent << " bytes" << std::endl;
async_write(serial_port);
});
});
}
int main()
{
boost::asio::io_service service;
auto serial_port = boost::asio::serial_port(service, "/dev/ttyUSB0");
serial_port.set_option(boost::asio::serial_port::baud_rate(9600));
async_read(serial_port);
async_write(serial_port);
service.run();
}
At the opposite end of the serial cable, I have a separate machine running cat /dev/ttyTHS0.
My issue is that whenever the program performs an async_write, those same bytes are immediately handled by the async_read completion handler - even though the remote machine isn't sending anything.
I'm not sure if the root cause is the code, or because I'm using cat at the other end. When cat is not running, I'm not having the issue.
Running the above code, (with cat /dev/ttyTHS0 at the other end) gives output like this:
Wrote 6 bytes
Read 7 bytes: Hello
Wrote 6 bytes
Read 3 bytes: Hel
Read 4 bytes: lo
Wrote 6 bytes
Read 7 bytes: Hello
Wrote 6 bytes
Read 1 bytes: H
Read 6 bytes: ello
Wrote 6 bytes
Read 7 bytes: Hello
Maybe I'm missing something obvious, but any help is appreciated! Thank you!
This is the expected behaviour for a serial port. These were typically used to connect a terminal like a vt100 to a computer. When you type on the keyboard, the character is sent to the computer, and the serial port echoes it back to the vt100 screen where it is displayed.
If you run stty -a -F /dev/ttyTHS0 on the remote you will see the setting echo is on (before you ran your program on the remote). You can switch it off with stty -echo -F /dev/ttyTHS0, when it will show as -echo.
Typically, when serial ports are used for comms between computers, the application will set the port to raw, noecho. This is to avoid certain translations being done by the serial port driver, so the data arrives unchanged at the application.
I have a server which is written with by boost.asio. This server gets the file from the client and write it to disk. I have just a problem with that. When server get the file, it write it to disk when it recieved the file completely. I wanted server write the buffer to disk in real-time fashion. For example, server write to disk every 100kb size of the file it get from the client. I have written the following code but I don't know how can I edited to get to this goal.
void Session::DoReadFileContent(size_t arg_bytes_transferred)
{
if (arg_bytes_transferred > 0)
{
m_outputFile.write(m_buffer.data(), static_cast<std::streamsize>(arg_bytes_transferred));
if (m_outputFile.tellp() >= static_cast<std::streamsize>(m_fileSize))
{
std::cout << "Received file: " << m_fileName << std::endl;
return;
}
}
auto self = shared_from_this();
m_socket.async_read_some(boost::asio::buffer(m_buffer.data(), m_buffer.size()),
[this, self](boost::system::error_code arg_error_code, size_t arg_bytes)
{
DoReadFileContent(arg_bytes);
});
}
First off, in that case it seems better to read explicit sizes of data instead of read_some which reads whatever is available.
In this pattern, it becomes easier to track "remaining bytes receivable" than m_fileSize.
Here's some minor re-shufflings that made your code into a self-contained example. It expects a server to send a line of text giving the payload size and output filename, followed by the contents of that file. An example server can be run with netcat e.g.:
(stat -c '%soutput.dat' main.cpp; cat main.cpp) | netcat -l -p 6969
Live On Coliru
#include <boost/asio.hpp>
#include <fstream>
#include <iostream>
using boost::system::error_code;
using boost::asio::ip::tcp;
struct Session : std::enable_shared_from_this<Session> {
Session(boost::asio::io_context& io, uint16_t port)
: m_socket(io)
{
m_socket.connect({{}, port});
}
void Start();
void DoReadFileContent(size_t transferred = 0);
private:
std::array<char, 1024> m_buffer;
std::streamsize m_remainingSize = 0;
std::string m_fileName = "noname.dat";
std::ofstream m_outputFile;
tcp::socket m_socket;
};
void Session::Start() {
// Reading a size (in text for simplicity) and subsequently receive as many bytes
//
// I'm keeping this sync for simplicity, because you probably already have
// this coded somehwere
boost::asio::streambuf buf;
error_code ec;
auto n = read_until(m_socket, buf, "\n", ec);
std::istream is(&buf);
if (is >> m_remainingSize && getline(is, m_fileName)) {
std::cerr << "Protocol trace: n:" << n << ", fileName:" << m_fileName << " payload_size:" << m_remainingSize << "\n";
m_outputFile.exceptions(std::ios::failbit | std::ios::badbit);
m_outputFile.open(m_fileName, std::ios::binary);
// write excess buffer contents as part of payload
if (buf.size()) {
std::cerr << "Writing " << buf.size() << " bytes\n";
m_remainingSize -= buf.size();
m_outputFile << &buf;
}
DoReadFileContent();
} else {
std::cerr << "Protocol error, payload_size expected\n";
}
}
void Session::DoReadFileContent(size_t transferred) {
if (transferred > 0) {
std::cerr << "Writing " << transferred << " bytes\n";
m_remainingSize -= transferred;
m_outputFile.write(m_buffer.data(), transferred);
}
if (m_remainingSize <= 0) {
std::cout << "Completed file: " << m_fileName << std::endl;
return;
}
auto self = shared_from_this();
auto expect = std::min(size_t(m_remainingSize), m_buffer.size());
std::cout << "Trying to receive next " << expect << " bytes" << std::endl;
async_read(m_socket,
boost::asio::buffer(m_buffer.data(), expect),
[this, self](error_code ec, size_t arg_bytes) {
std::cerr << "async_read: " << ec.message() << " - " << arg_bytes << " bytes\n";
if (!ec) {
DoReadFileContent(arg_bytes);
}
});
}
int main() {
boost::asio::io_context io;
std::make_shared<Session>(io, 6868) // download from port 6868
->Start();
io.run(); // complete
}
Testing with
(stat -c '%soutput.dat' main.cpp; cat main.cpp) | netcat -l -p 6868&
./a.out
md5sum main.cpp output.dat
Prints, e.g.:
Protocol trace: n:15, fileName:output.dat payload_size:2654
Trying to receive next 1024 bytes
async_read: Success - 1024 bytes
Writing 1024 bytes
Trying to receive next 1024 bytes
async_read: Success - 1024 bytes
Writing 1024 bytes
Trying to receive next 606 bytes
async_read: Success - 606 bytes
Writing 606 bytes
Completed file: output.dat
The last two lines
b4eec7203f6a1dcbfbf3d298c7ec0832 main.cpp
b4eec7203f6a1dcbfbf3d298c7ec0832 output.dat
indicate that the received file is identical to the original.
Notes:
packets are delivered in unspecified sizes, on my system e.g. the same file is received as:
Protocol trace: n:15, fileName:output.dat payload_size:2654
Writing 497 bytes
Trying to receive next 1024 bytes
async_read: Success - 1024 bytes
Writing 1024 bytes
Trying to receive next 1024 bytes
async_read: Success - 1024 bytes
Writing 1024 bytes
Trying to receive next 109 bytes
async_read: Success - 109 bytes
Writing 109 bytes
Completed file: output.dat
b4eec7203f6a1dcbfbf3d298c7ec0832 main.cpp
b4eec7203f6a1dcbfbf3d298c7ec0832 output.dat
Note that it starts out with 497 bytes already in the input buffer from the read_until.
The protocol is not secure:
the file names should be validated. Just imagine what happens if the file would be '/home/sehe/myimportant_file.txt' or worse, say /dev/sde1 and we have permissions to do raw block device access...
you might want to specify a amximum size for streambuf, so that if you get a fuzzer that doesn't ever send a '\n' you wouldn't just gobble up all RAM
the error handling on file IO is very rough. I used io exceptions, but you probably want to check for m_outputFile.good() instead at various places
I have this requirement where my app have to connect to another app via sockets and will have to maintain persistent connection for quiet long time. My app will be a TCP client and the other is a TCP server. My app will send commands and the server will respond accordingly.
The problem am facing right now is how to read the whole data from server a string and return for app which will issue the next command. Reading synchronously (with asio::read) looked like a good option up until I observed socket hanging up until I terminate the server. Looking at the documentation I found that the library is correctly working.
his function is used to read a certain number of bytes of data from a stream. The call will block until one of the following conditions is true:
1. The supplied buffers are full. That is, the bytes transferred is equal to the sum of the buffer sizes.
2. An error occurred.
The problem is I don't know correct buffer size as the response from the server varies. So If I put a too small buffer it returns fine but missing some data. If I put too big it will hang forever until server quits.
So I thought I would do the async reading. It works only once and I don't know how to make it fetch data until whole data it read.
here is the relevant async code
#define ASIO_STANDALONE 1
#include <iostream>
#include <asio.hpp>
int main()
{
asio::io_context context;
size_t reply_length;
size_t length = 1024;
std::vector<char> buffer;
//create socket
asio::ip::tcp::socket socket(context);
socket.connect(asio::ip::tcp::endpoint(asio::ip::address::from_string("127.0.0.1"), 8088));
std::string dataOut = "list --files"; //some command to write
std::error_code error;
asio::write(socket, asio::buffer(dataOut), error);
if (!error)
{
std::cout << "Receiving...!" << std::endl;
buffer.resize(length);
asio::async_read(socket, asio::buffer(buffer), [&buffer, &context](const asio::error_code &ec, std::size_t bytes_transferred) {
std::copy(buffer.begin(), buffer.end(), std::ostream_iterator<char>(std::cout, ""));
std::cout << "\nRead total of:" << bytes_transferred << "\n";
context.run();
});
}
else
{
std::cout << "send failed: " << error.message() << std::endl;
}
context.run();
}
Searching didn't help much solving my issue.
So my question is, how can I read all the data in a persistent socket with asio? Am not using boost.
You need to loop async_read calls. If you don't want your client to hang on read operation you can define the smallest possible buffer i.e. 1 byte.
Define function which takes socket, buffer and two additional parameters according to async_read's handler signature, and this function calls itself with async_read to make the loop of async_read calls - it reads until some error occures:
void onRead (
asio::ip::tcp::socket& socket,
std::array<char,1>& buf,
const system::error_code& ec,
std::size_t bytes)
{
if (ec)
{
if (ec == asio::error::eof && bytes == 1)
std::cout << buf[0];
return;
}
std::cout << buf[0];
asio::async_read(socket,asio::buffer(buf),
std::bind(onRead, std::ref(socket), std::ref(buf),
std::placeholders::_1, // error code
std::placeholders::_2)); // transferred bytes
}
and the changes in main:
std::array<char,1> buf;
asio::write(socket, asio::buffer(dataOut), error);
if (!error)
{
std::cout << "Receiving...!" << std::endl;
asio::async_read(socket, asio::buffer(buf),
std::bind(onRead, std::ref(socket), std::ref(buf),
std::placeholders::_1,
std::placeholders::_2));
context.run();
}
else
{
std::cout << "send failed: " << error.message() << std::endl;
}
(I am using Boost, so you should replace system::error_code on asio::error_code).
I'm developing a client-server app, both sides of which use boost::asio.
I'm trying to send a large package of data over TCP (356 kb)
On server side, I write like:
boost::asio::async_write(Msocket,
boost::asio::buffer(sendBuffer,dataLen),
boost::bind(&ServerSession::onDataWrite,
this, boost::asio::placeholders::error,
boost::asio::placeholders::bytes_trasferred));
The onDataWrite is simple:
void ServerSession::onDataWrite(const boost::system::error_code& error, const std::size_t bytesSent) {
if (error) {
std::cout << "Error " << error << " while sending data" << std::endl;
}
}
On client side:
int readSize = ...; // defined from msg header, in this case equals 300 kbytes.
boost::asio::async_read(*Msocket,
boost::asio::buffer(recvBuffer, 50*1024*1024),
boost::asio::transfer_exactly(readSize),
boost::bind(&ClientSession::onDataRead,
this, boost::asio::placeholders::error,
boost::asio::placeholders::bytes_trasferred,
readSize));
And onDataRead is:
void ClientSession::onDataRead(const boost::system::error_code& error, const std::size_t bytesRecvd, const int readSize) {
if (error || bytesRecvd != readSize) {
std::cout << "Error " << error << " while getting data, expect " << readSize <<", got " << bytesRecvd << std::endl;
}
}
During write, server side prints
Error system:10014 while sending data
And client prints
Error system:0 while getting data, expect 393216, got 131064
While 131064 = 128kb - 8 bytes of header.
It looks like this 128-kb issue is caused by send/receive buffer overflow. But I though Boost will take care about those buffers itself, transparently for me.
What do I misunderstand?
I am developing an application with C++ and having some difficulty with boost sockets. The server sends an image but not all the bytes are received by the client; the client always receives about 500 bytes less than the server sent. Provided below is the pertinent code and screenshots of the program running.
Server code:
int sent = boost::asio::write(*socket, response, boost::asio::transfer_all(), error);
std::cout << "Sent: " << sent << std ::endl;
Client code (I know that read_some will block if the total bytes sent by the server is divisible by 10000; this code is just for testing):
int len = 0;
int count = 0;
do {
len = socket->read_some( boost::asio::buffer( imageData, 10000 ) );
count += len;
std::cout << "len: " << len << std::endl;
std::cout << "count: " << count << std::endl;
} while(len == 10000);
std::cout << "Image Received of size: " << count << std::endl;
Screenshot of server:
Screenshot of client:
Thanks for your time; any help or suggestions would be greatly appreciated.
There're no guarantee you'll receive complete buffers of 10000 bytes.
I'd recommend following approach:
To send some binary data w/o any explicit terminator, first send its size and only then data itself. In this case client will know how many data in this chunk it should receive.