Converting asio read_some to async version - c++

I have the following code that reads from a TCP socket using boost asio read_some function. Currently the code is synchronous and I need to convert it to the async version. The issue is initially that some bytes are read to identify the packettype and to get the length of the packet. Then we have a loop that reads the data. Would I need to use two callbacks to do this asynchronously or can it be done with one ( which would be preferable).
void Transport::OnReadFromTcp()
{
int read = 0;
// read 7 bytes from TCP into mTcpBuffer
m_sslsock->read_some(asio::buffer(mTcpBuffer, 7));
bool tag = true;
for (unsigned char i = 0; i < 5; i++)
{
tag = tag && (mTcpBuffer[i] == g_TcpPacketTag[i]);
}
// get the length from the last two bytes
unsigned short dataLen = (mTcpBuffer[5] ) | (mTcpBuffer[6] << 8);
mBuff = new char[dataLen];
int readTotal = 0;
while (readTotal < dataLen)
{
// read lengths worth of data from tcp pipe into buffer
int readlen = dataLen;
size_t read = m_sslsock->read_some(asio::buffer(&mBuff[readTotal], readlen));
readlen = dataLen - read;
readTotal += read;
}
// Process data .....
}

The first step is to realize that you can remove the read_some loop entireyl using the free function read:
void Transport::OnReadFromTcp() {
int read = 0;
// read 7 bytes from TCP into mTcpBuffer
size_t bytes = asio::read(*m_sslsock, asio::buffer(mTcpBuffer, 7), asio::transfer_all());
assert(bytes == 7);
bool tag = g_TcpPacketTag.end() == std::mismatch(
g_TcpPacketTag.begin(), g_TcpPacketTag.end(),
mTcpBuffer.begin(), mTcpBuffer.end())
.first;
// get the length from the last two bytes
uint16_t const dataLen = mTcpBuffer[5] | (mTcpBuffer[6] << 8);
mBuff.resize(dataLen);
size_t readTotal = asio::read(*m_sslsock, asio::buffer(mBuff), asio::transfer_exactly(dataLen));
assert(mBuff.size() == readTotal);
assert(dataLen == readTotal);
}
That's even regardless of whether execution is asynchronous.
Making it asynchronous is slightly involved, because it requires assumptions about lifetime of the buffers/Transport instance as well as potential multi-threading. I'll provide a sample of that after my morning coffee :)
Demo without threading/lifetime complications:
Live On Coliru
#include <boost/asio.hpp>
#include <boost/asio/ssl.hpp>
#include <boost/bind.hpp>
#include <iostream>
#include <array>
#include <cassert>
namespace asio = boost::asio;
namespace ssl = asio::ssl;
namespace {
static std::array<char, 5> g_TcpPacketTag {{'A','B','C','D','E'}};
}
struct Transport {
using tcp = asio::ip::tcp;
using SslSocket = std::shared_ptr<asio::ssl::stream<tcp::socket> >;
Transport(SslSocket s) : m_sslsock(s) { }
void OnReadFromTcp();
void OnHeaderReceived(boost::system::error_code ec, size_t transferred);
void OnContentReceived(boost::system::error_code ec, size_t transferred);
private:
uint16_t datalen() const {
return mTcpBuffer[5] | (mTcpBuffer[6] << 8);
}
SslSocket m_sslsock;
std::array<char, 7> mTcpBuffer;
std::vector<char> mBuff;
};
void Transport::OnReadFromTcp() {
// read 7 bytes from TCP into mTcpBuffer
asio::async_read(*m_sslsock, asio::buffer(mTcpBuffer, 7), asio::transfer_all(),
boost::bind(&Transport::OnHeaderReceived, this, asio::placeholders::error, asio::placeholders::bytes_transferred)
);
}
#include <boost/range/algorithm/mismatch.hpp> // I love sugar
void Transport::OnHeaderReceived(boost::system::error_code ec, size_t bytes) {
if (ec) {
std::cout << "Error: " << ec.message() << "\n";
}
assert(bytes == 7);
bool tag = (g_TcpPacketTag.end() == boost::mismatch(g_TcpPacketTag, mTcpBuffer).first);
if (tag) {
// get the length from the last two bytes
mBuff.resize(datalen());
asio::async_read(*m_sslsock, asio::buffer(mBuff), asio::transfer_exactly(datalen()),
boost::bind(&Transport::OnContentReceived, this, asio::placeholders::error, asio::placeholders::bytes_transferred)
);
} else {
std::cout << "TAG MISMATCH\n"; // TODO handle error
}
}
void Transport::OnContentReceived(boost::system::error_code ec, size_t readTotal) {
assert(mBuff.size() == readTotal);
assert(datalen() == readTotal);
std::cout << "Successfully completed receive of " << datalen() << " bytes\n";
}
int main() {
asio::io_service svc;
using Socket = Transport::SslSocket::element_type;
// connect to localhost:6767 with SSL
ssl::context ctx(ssl::context::sslv23);
auto s = std::make_shared<Socket>(svc, ctx);
s->lowest_layer().connect({ {}, 6767 });
s->handshake(Socket::handshake_type::client);
// do transport
Transport tx(s);
tx.OnReadFromTcp();
svc.run();
// all done
std::cout << "All done\n";
}
When using against a sample server that accepts SSL connections on port 6767:
(printf "ABCDE\x01\x01F"; cat main.cpp) |
openssl s_server -accept 6767 -cert so.crt -pass pass:test
Prints:
Successfully completed receive of 257 bytes
All done

Related

asynchronous io is working but never ends in linux

In a previous question, I asked how to implement asynchronous I/O. This code now works, except that at the end it never stops. It seems that aio_read reads starting at offset, for length, and if it is past the end of the file, the operation succeeds? This code builds and runs on Ubuntu 20.04LTS and successfully reads blocks 1-5, each 512 bytes, then when it runs out of file it keeps oscillating between block 4 and 5. It never terminates.
Here is the code:
#include <aio.h>
#include <fcntl.h>
#include <signal.h>
#include <unistd.h>
#include <condition_variable>
#include <cstring>
#include <iostream>
#include <thread>
using namespace std;
using namespace std::chrono_literals;
constexpr uint32_t blockSize = 512;
mutex readMutex;
bool readReady = false;
condition_variable cv;
bool operation_completed = false;
int fh;
int bytesRead;
void process(char* buf, uint32_t bytesRead) {
cout << "processing..." << endl;
usleep(100000);
}
void aio_completion_handler(sigval_t sigval) {
struct aiocb* req = (struct aiocb*)sigval.sival_ptr;
// check whether asynch operation is complete
int status;
if ((status = aio_error(req)) != 0) {
cout << "Error: " << status << '\n';
return;
}
int ret = aio_return(req);
bytesRead = req->aio_nbytes;
cout << "ret == " << ret << endl;
cout << (char*)req->aio_buf << endl;
unique_lock<mutex> readLock(readMutex);
operation_completed = true;
cv.notify_one();
}
void thready() {
char* buf1 = new char[blockSize];
char* buf2 = new char[blockSize];
aiocb cb;
char* processbuf = buf1;
char* readbuf = buf2;
fh = open("smallfile.dat", O_RDONLY);
if (fh < 0) {
throw std::runtime_error("cannot open file!");
}
memset(&cb, 0, sizeof(aiocb));
cb.aio_fildes = fh;
cb.aio_nbytes = blockSize;
cb.aio_offset = 0;
// Fill in callback information
/*
Using SIGEV_THREAD to request a thread callback function as a notification
method
*/
cb.aio_sigevent.sigev_notify_attributes = nullptr;
cb.aio_sigevent.sigev_notify = SIGEV_THREAD;
cb.aio_sigevent.sigev_notify_function = aio_completion_handler;
/*
The context to be transmitted is loaded into the handler (in this case, a
reference to the aiocb request itself). In this handler, we simply refer to
the arrived sigval pointer and use the AIO function to verify that the request
has been completed.
*/
cb.aio_sigevent.sigev_value.sival_ptr = &cb;
int cursor = 0;
int currentBytesRead = read(fh, buf1, blockSize); // read the 1st block
while (true) {
cb.aio_buf = readbuf;
operation_completed = false; // set predicate to true and wait until asynch changes it
cb.aio_offset = cursor;
aio_read(&cb); // each next block is read asynchronously
process(processbuf, currentBytesRead); // process while waiting
{
unique_lock<mutex> readLock(readMutex);
cv.wait( readLock, []{ return operation_completed; } );
}
if (!operation_completed)
break;
currentBytesRead = bytesRead; // make local copy of global modified by the asynch code
cursor += bytesRead;
if (currentBytesRead < blockSize) {
break; // last time, get out
}
cout << "back from wait" << endl;
swap(processbuf, readbuf); // switch to other buffer for next time
currentBytesRead = bytesRead; // create local copy
}
delete[] buf1;
delete[] buf2;
}
int main() {
try {
thready();
} catch (std::exception& e) {
cerr << e.what() << '\n';
}
return 0;
}
First, is the above code an appropriate way to do this to get the length of the file and figure out exactly how many reads to do?
Second, if this is so, fine, but how can aio_read just return success if I try to read past the end of file? Error status is always zero. I am confused about what it is supposed to do.
with 512 bytes of each of 1,2,3,4,5

How to read Inexhaustible data from LAN port in C++

This is my code for read a data from file:
void sendfile(string pathname)
{
ifstream ifs(pathname, std::ifstream::in);
if(ifs.fail())
{
throw "error opening";
}
const size_t BUFFER_SIZE = 1024;
char* buffer;
buffer = new char[BUFFER_SIZE];
// get file size
ifs.seekg(0, ios::end);
int file_size = ifs.tellg();
ifs.seekg(0, ios::beg);
cout <<"File size : "<<file_size<< "bytes"<<endl;
unsigned char *data = new unsigned char [file_size];
ifs.read((char*)data, file_size);
for(int i=0; i < file_size; i += 2)
{
if (data[i] == 0xAF
&& data[i+1] == 0xDB
&& data[i+2] == 0xAA
&& data[i+3] == 0x00)
{
if (data[i + BUFFER_SIZE - 4] == 0xFF
&& data[i + BUFFER_SIZE - 3] == 0xAA
&& data[i + BUFFER_SIZE - 2] == 0xDF
&& data[i + BUFFER_SIZE - 1] == 0x00)
{
cout << "Packet" << endl;
}
}
}
delete[] data;
ifs.close();
But now I try to read Inexhaustible data from file or serial port byte by byte and check them to HEADER and FOOTER then send data with TCP/IP port to server.
I have written the client and server part codes and now I am trying to get this part of the inexhaustible data byte by byte. I mean, I am still unaware of the next byte and I want to check the HEADER and FOOTER and then send the packet[1024 byte].
using boost::asio for this example.
State machine:
What should I do?
UPDATE
Client header file:
#include <iostream>
#include <fstream>
#include <bitset>
#include <boost/asio.hpp>
using namespace boost::asio;
using ip::tcp;
using std::string;
using std::cout;
using std::endl;
using byte = unsigned char;
class Client
{
public:
Client(io_context& io_context,
char host[10],
unsigned int port)
: socket_(io_context)
{
start(host, port);
}
~Client()
{
stop();
}
private:
enum { CLIENT_DATA_BUFFER_SIZE = 1024 };
char buffer[CLIENT_DATA_BUFFER_SIZE];
boost::asio::streambuf receive_buffer;
tcp::socket socket_;
void start(char[10], unsigned int);
void stop();
};
Server Header file:
TCPServer::TCPServer(io_context& io_service, char host[10], unsigned int port)
: io_context_(io_service),
acceptor_(io_service, tcp::endpoint(tcp::v4(), port))
{
cout << "Server is running ..." << endl;
cout << "Server: got connection from " << host << " port " << port << endl;
//Timer await
timer = new deadline_timer(io_service, boost::posix_time::milliseconds(3100));
timer->wait();
std::cout << "Blocking wait(): " << 3 << " second-wait\n";
start_accept();
}
//Creates a socket and initiates an asynchronous accept operation to wait for a new connection.
void TCPServer::start_accept()
{
// socket definition
connection = connectionHandler::create(io_context_);
// asynchronous accept operation and wait for a new connection.
acceptor_.async_accept
(
connection->socket(),
boost::bind(&TCPServer::handle_accept,
this,
connection,
boost::asio::placeholders::error)
);
}
//It services the client request, and then calls start_accept() to initiate the next accept operation.
void TCPServer::handle_accept(connectionHandler::pointer connection, const boost::system::error_code& err)
{
if (!err)
{
connection->start();
}
start_accept();
}
void TCPServer::stop()
{
connection->stop();
}

How to use boost::beast, download a file no blocking and with responses

I have started with this example so won't post all the code. My objective is to download a large file without blocking my main thread. The second objective is to get notifications so I can update a progress bar. I do have the code working a couple of ways. First is to just ioc.run(); and let it go to work, I get the file downloaded. But I can not find anyway to start the session without blocking.
The second way I can make the calls down to http::async_read_some and the call works but I can not get a response that I can use. I don't know if there is a way to pass a lambda that captures.
The #if 0..#else..#endif switches the methods. I'm sure there is a simple way but I just can not see it. I'll clean up the code when I get it working, like setting the local file name. Thanks.
std::size_t on_read_some(boost::system::error_code ec, std::size_t bytes_transferred)
{
if (ec);//deal with it...
if (!bValidConnection) {
std::string_view view((const char*)buffer_.data().data(), bytes_transferred);
auto pos = view.find("Content-Length:");
if (pos == std::string_view::npos)
;//error
file_size = std::stoi(view.substr(pos+sizeof("Content-Length:")).data());
if (!file_size)
;//error
bValidConnection = true;
}
else {
file_pos += bytes_transferred;
response_call(ec, file_pos);
}
#if 0
std::cout << "in on_read_some caller\n";
http::async_read_some(stream_, buffer_, file_parser_, std::bind(
response_call,
std::placeholders::_1,
std::placeholders::_2));
#else
std::cout << "in on_read_some inner\n";
http::async_read_some(stream_, buffer_, file_parser_, std::bind(
&session::on_read_some,
shared_from_this(),
std::placeholders::_1,
std::placeholders::_2));
#endif
return buffer_.size();
}
The main, messy but.....
struct lambda_type {
bool bDone = false;
void operator ()(const boost::system::error_code ec, std::size_t bytes_transferred) {
;
}
};
int main(int argc, char** argv)
{
auto const host = "reserveanalyst.com";
auto const port = "443";
auto const target = "/downloads/demo.msi";
int version = argc == 5 && !std::strcmp("1.0", argv[4]) ? 10 : 11;
boost::asio::io_context ioc;
ssl::context ctx{ ssl::context::sslv23_client };
load_root_certificates(ctx);
//ctx.load_verify_file("ca.pem");
auto so = std::make_shared<session>(ioc, ctx);
so->run(host, port, target, version);
bool bDone = false;
auto const lambda = [](const boost::system::error_code ec, std::size_t bytes_transferred) {
std::cout << "data lambda bytes: " << bytes_transferred << " er: " << ec.message() << std::endl;
};
lambda_type lambda2;
so->set_response_call(lambda);
ioc.run();
std::cout << "not in ioc.run()!!!!!!!!" << std::endl;
so->async_read_some(lambda);
//pseudo message pump when working.........
for (;;) {
std::this_thread::sleep_for(250ms);
std::cout << "time" << std::endl;
}
return EXIT_SUCCESS;
}
And stuff I've added to the class session
class session : public std::enable_shared_from_this<session>
{
using response_call_type = void(*)(boost::system::error_code ec, std::size_t bytes_transferred);
http::response_parser<http::file_body> file_parser_;
response_call_type response_call;
//
bool bValidConnection = false;
std::size_t file_pos = 0;
std::size_t file_size = 0;
public:
auto& get_result() { return res_; }
auto& get_buffer() { return buffer_; }
void set_response_call(response_call_type the_call) { response_call = the_call; }
I've updated this as I finally put it to use and I wanted the old method where I could download to a file or a string. Link to how asio works, great talk.
CppCon 2016 Michael Caisse Asynchronous IO with BoostAsio
As for my misunderstanding of how to pass a lambda, here is Adam Nevraumont's answer
There are two ways to compile this using a type to select the method. Both are shown at the beginning of main. You can construct either a file downloader or string downloader by selecting the type of beast parser. The parsers don't have the same constructs so an if constexpr compile time conditions are used. And I checked, a release build of the downloader is about 1K so pretty light weight for what it does. In the case of a small string you don't have to handle the call backs. either pass an empty lambda or add the likes of:
if(response_call)
response_call(resp_ok, test);
This looks to be a pretty clean way to get the job done so I've updated this post as of 11/27/2202.
The code:
//
// Copyright (c) 2016-2019 Vinnie Falco (vinnie dot falco at gmail dot com)
//
// Distributed under the Boost Software License, Version 1.0. (See accompanying
// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
//
// Official repository: https://github.com/boostorg/beast
//------------------------------------------------------------------------------
//
// Example: HTTP SSL client, synchronous, usable in a thread with a message pump
// Added code to use from a message pump
// Also useable as body to a file download, or body to string
//
//------------------------------------------------------------------------------
#include <boost/beast/core.hpp>
#include <boost/beast/http.hpp>
#include <boost/beast/ssl.hpp>
#include <boost/beast/version.hpp>
#include <boost/asio/connect.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/asio/ssl/error.hpp>
#include <boost/asio/ssl/stream.hpp>
#include <cstdlib>
#include <iostream>
#include <string>
#include <fstream>
//the boost shipped certificates
#include <boost/../libs/beast/example/common/root_certificates.hpp>
//TODO add your ssl libs as you would like
#ifdef _M_IX86
#pragma comment(lib, "libcrypto.lib")
#pragma comment(lib, "libssl.lib")
#elif _M_X64
#pragma comment(lib, "libcrypto-3-x64.lib")
#pragma comment(lib, "libssl-3-x64.lib")
#endif
namespace downloader {
namespace beast = boost::beast; // from <boost/beast.hpp>
namespace http = beast::http; // from <boost/beast/http.hpp>
namespace net = boost::asio; // from <boost/asio.hpp>
namespace ssl = net::ssl; // from <boost/asio/ssl.hpp>
using tcp = net::ip::tcp; // from <boost/asio/ip/tcp.hpp>
//specialization if using < c++17; see both 'if constexpr' below.
//this is not needed otherwise
//namespace detail {
// template<typename Type>
// void open_file(http::parser < false, Type>& p, const char* name, boost::system::error_code& file_open_ec) { }
// template<>
// void open_file(http::parser<false, http::file_body>& p, const char* name, boost::system::error_code& file_open_ec) {
// p.get().body().open(name, boost::beast::file_mode::write, file_open_ec);
// }
// template<typename Type>
// std::string get_string(http::parser < false, Type>& p) { return std::string{}; }
// template<>
// std::string get_string(http::parser<false, http::string_body>& p) {
// return p.get().body();
// }
//} //namespace detail
enum responses {
resp_null,
resp_ok,
resp_done,
resp_error,
};
using response_call_type = std::function< void(responses, std::size_t)>;
template<typename ParserType>
struct download {
//as these can be set with array initialization
const char* target_ = "/";
const char* filename_ = "test.txt";
const char* host_ = "lakeweb.net";
std::string body_;
using response_call_type = std::function< void(responses, std::size_t)>;
response_call_type response_call;
boost::asio::io_context ioc_;
ssl::context ctx_{ ssl::context::sslv23_client };
ssl::stream<tcp::socket> stream_{ ioc_, ctx_ };
tcp::resolver resolver_{ ioc_ };
boost::beast::flat_buffer buffer_;
uint64_t file_size_{};
int version{ 11 };
void set_response_call(response_call_type the_call) { response_call = the_call; }
uint64_t get_file_size() { return file_size_; }
void stop() { ioc_.stop(); }
bool stopped() { return ioc_.stopped(); }
std::string get_body() { return std::move(body_); }
void run() {
try {
// TODO should have a timer in case of a hang
load_root_certificates(ctx_);
// Set SNI Hostname (many hosts need this to handshake successfully)
if (!SSL_set_tlsext_host_name(stream_.native_handle(), host_)) {
boost::system::error_code ec{ static_cast<int>(::ERR_get_error()), boost::asio::error::get_ssl_category() };
throw boost::system::system_error{ ec };
}
//TODO resolve is depreciated, use endpoint
auto const results = resolver_.resolve(host_, "443");
boost::asio::connect(stream_.next_layer(), results.begin(), results.end());
stream_.handshake(ssl::stream_base::client);
// Set up an HTTP GET request message
http::request<http::string_body> req{ http::verb::get, target_, version };
req.set(http::field::host, host_);
req.set(http::field::user_agent, "mY aGENT");
// Send the HTTP request to the remote host
http::write(stream_, req);
// Read the header
boost::system::error_code file_open_ec;
http::parser<false, ParserType> p;
p.body_limit((std::numeric_limits<std::uint32_t>::max)());
//detail::open_file(p, filename_, file_open_ec);
//or => c++17
if constexpr (std::is_same_v<ParserType, http::file_body>)
p.get().body().open(filename_, boost::beast::file_mode::write, file_open_ec);
http::read_header(stream_, buffer_, p);
file_size_ = p.content_length().has_value() ? p.content_length().value() : 0;
//Read the body
uint64_t test{};
boost::system::error_code rec;
for (;;) {
test += http::read_some(stream_, buffer_, p, rec);
if (test >= file_size_) {
response_call(resp_done, 0);
break;
}
response_call(resp_ok, test);
}
// Gracefully close the stream
boost::system::error_code ec;
stream_.shutdown(ec);
if (ec == boost::asio::error::eof)
{
// Rationale:
// http://stackoverflow.com/questions/25587403/boost-asio-ssl-async-shutdown-always-finishes-with-an-error
ec.assign(0, ec.category());
}
if (ec)
throw boost::system::system_error{ ec };
//value = detail::get_string(p);
//or => c++17
if constexpr (std::is_same_v<ParserType, http::string_body>)
body_ = p.get().body();
}
catch (std::exception const& e)
{
std::cerr << "Error: " << e.what() << std::endl;
response_call(resp_error, -1);
}
ioc_.stop();
}
};
}//namespace downloadns
//comment to test with string body
#define THE_FILE_BODY_TEST
int main(int argc, char** argv)
{
using namespace downloader;
#ifdef THE_FILE_BODY_TEST
download<http::file_body> dl{"/Nasiri%20Abarbekouh_Mahdi.pdf", "test.pdf"};
#else //string body test
download<http::string_body> dl{ "/robots.txt" };
#endif
responses dl_response{ resp_null };
size_t cur_size{};
auto static const lambda = [&dl_response, &dl, &cur_size](responses response, std::size_t bytes_transferred) {
if ((dl_response = response) == resp_ok) {
cur_size += bytes_transferred;
size_t sizes = dl.get_file_size() - cur_size;//because size is what is left
//drive your progress bar from here in a GUI app
}
};
dl.set_response_call(lambda);
std::thread thread{ [&dl]() { dl.run(); } };
//thread has started, now the pseudo message pump
bool quit = false; //true: as if a cancel button was pushed; won't finish download
for (int i = 0; ; ++i) {
switch (dl_response) { //ad hoc as if messaged
case resp_ok:
std::cout << "from sendmessage: " << cur_size << std::endl;
dl_response = resp_null;
break;
case resp_done:
std::cout << "from sendmessage: done" << std::endl;
dl_response = resp_null;
break;
case resp_error:
std::cout << "from sendmessage: error" << std::endl;
dl_response = resp_null;
}//switch
if (!(i % 5))
std::cout << "in message pump, stopped: " << std::boolalpha << dl.stopped() << std::endl;
std::this_thread::sleep_for(std::chrono::milliseconds(100));
if (quit && i == 10) //the cancel message
dl.stop();
if (!(i % 20) && dl.stopped()) {//dl job was quit or error or finished
std::cout << "dl is stopped" << std::endl;
break;
}
}
#ifdef THE_FILE_BODY_TEST
std::cout << "file written named: 'test.txt'" << std::endl;
#else
std::string res = dl.get_body();
std::cout << "body retrieved:\n" << res << std::endl;
#endif
if (thread.joinable())//in the case a thread was never started
thread.join();
std::cout << "exiting, program all done" << std::endl;
return EXIT_SUCCESS;
}
I strongly recommend against using the low-level [async_]read_some function instead of using http::[async_]read as intended with http::response_parser<http::buffer_body>
I do have an example of that - which is a little bit complicated by the fact that it also uses Boost Process to concurrently decompress the body data, but regardless it should show you how to use it:
How to read data from Internet using muli-threading with connecting only once?
I guess I could tailor it to your specific example given more complete code, but perhaps the above is good enough? Also see "Relay an HTTP message" in libs/beast/example/doc/http_examples.hpp which I used as "inspiration".
Caution: the buffer arithmetic is not intuitive. I think this is unfortunate and should not have been necessary, so pay (very) close attention to these samples for exactly how that's done.

Boost Tcp Client Receive Crash

I have been working recently with the Boost library for windows visual studio 2015. I have been recently working on a server and client that can easily transfer data via tcp connection. But recently I was trying to send a Base64 encoded file string to the client which is roughly 366,660 Bytes of data. To do this I split up the data into packets that are about 1000 in size. (not sure what the max size is) But anyway the server sends the data completely fine but when the client receives more than 160,000 bytes it crashes with no exceptions.
Client:
try
{
for (static int i = 1000; i <= sizeofpackets /*(366,660) (currently 170,000 for testing)*/; i += 1000)
{
char Buffer[1000];
memset(Buffer, 0, 1000);
boost::asio::read(s, boost::asio::buffer(Buffer, 1000));
std::cout << i << std::endl;
}
}
catch (std::exception& e)
{
std::cerr << "Exception: " << e.what() << "\n";
}
Server:
for (int i = 1000; i <= 170000; i += 1000)
{
std::string NewData = encodedBuffer.substr(i - 1000, i);
NewData.erase(0, i - 1000);
boost::asio::write(*sock, boost::asio::buffer(NewData, 1000));
std::cout << NewData.size() + i - 1000 << std::endl;
NewData.clear();
}
Any comments or suggestion would help greatly!
You haven't posted your complete code, so it is hard to tell, but your problem might be the way you are handling strings, not the way you are handling the socket transfers (the substr() member function takes a start offset and a size, and it looks like you are trying to use an ever-increasing size for NewData).
The following (complete) code, which uses your client and server code as a reference, but with the string handling changed, does transfer 170000 bytes just fine on my machine, so it may help:
#include <boost/asio/io_service.hpp>
#include <boost/asio/write.hpp>
#include <boost/asio/read.hpp>
#include <boost/asio/buffer.hpp>
#include <boost/asio/streambuf.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <array>
#include <string>
#include <iostream>
namespace
{
const size_t BIG_MESSAGE_SIZE = 170000;
const size_t BLOCK_SIZE = 1000;
void doTX(boost::asio::ip::tcp::socket& socket, size_t& TXsize)
{
char rawBuffer[BIG_MESSAGE_SIZE] = { 0 }; // raw data to send
std::string encodedBuffer(rawBuffer, BIG_MESSAGE_SIZE);
for (size_t i = 0; i < BIG_MESSAGE_SIZE; i += BLOCK_SIZE)
{
std::string NewData = encodedBuffer.substr(i, BLOCK_SIZE);
TXsize += boost::asio::write(socket, boost::asio::buffer(NewData, BLOCK_SIZE));
}
}
void doRX(boost::asio::ip::tcp::socket& socket, size_t& RXsize)
{
for (size_t i = 0; i < BIG_MESSAGE_SIZE; i += BLOCK_SIZE)
{
char Buffer[BLOCK_SIZE];
memset(Buffer, 0, BLOCK_SIZE);
RXsize += boost::asio::read(socket, boost::asio::buffer(Buffer, BLOCK_SIZE));
}
}
}
int main(int argc, char * argv[])
{
std::cout << "Running:" << std::endl;
int port = 9876;
boost::asio::io_service ios;
boost::asio::ip::tcp::endpoint endpoint(boost::asio::ip::address::from_string("127.0.0.1"), port);
boost::asio::ip::tcp::acceptor acceptor(ios, endpoint);
boost::asio::ip::tcp::socket TXsocket(ios);
boost::asio::ip::tcp::socket RXsocket(ios);
TXsocket.connect(endpoint);
acceptor.accept(RXsocket);
size_t TXsize = 0;
size_t RXsize = 0;
doTX(TXsocket, TXsize);
doRX(RXsocket, RXsize);
std::cout << TXsize << " " << RXsize << std::endl;
return 0;
}

How to synchronize the data being processed in a multithread program?

I am using boost library to develop a asynchronous udp communication. A data received at the receiver side is being precessed by another thread. Then my problem is when I read the received data in another thread rather than the receiver thread it self it gives a modified data or updated data which is not the data that is supposed to be.
My code is working on unsigned character buffer array at sender side and receiver side. The reason is I need consider unsigned character buffer as a packet of data
e.g buffer[2] = Engine_start_ID
/* global buffer to store the incomming data
unsigned char received_buffer[200];
/*
global buffer accessed by another thread
which contains copy the received_buffer
*/
unsigned char read_hmi_buffer[200];
boost::mutex hmi_buffer_copy_mutex;
void udpComm::start_async_receive() {
udp_socket.async_receive_from(
boost::asio::buffer(received_buffer, max_length), remote_endpoint,
boost::bind(&udpComm::handle_receive_from, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
/* the data received is stored in the unsigned char received_buffer data buffer*/
void udpComm::handle_receive_from(const boost::system::error_code& error,
size_t bytes_recvd) {
if (!error && bytes_recvd > 0) {
received_bytes = bytes_recvd;
hmi_buffer_copy_mutex.lock();
memcpy(&read_hmi_buffer[0], &received_buffer[0], received_bytes);
hmi_buffer_copy_mutex.unlock();
/*data received here is correct 'cus i printed in the console
checked it
*/
cout<<(int)read_hmi_buffer[2]<<endl;
}
start_async_receive();
}
/* io_service is running in a thread
*/
void udpComm::run_io_service() {
udp_io_service.run();
usleep(1000000);
}
The above code is the asynchronous udp communication running a thread
/* My second thread function is */
void thread_write_to_datalink()
{ hmi_buffer_copy_mutex.lock();
/* here is my problem begins*/
cout<<(int)read_hmi_buffer[2]<<endl;
hmi_buffer_copy_mutex.unlock();
/* all data are already changed */
serial.write_to_serial(read_hmi_buffer, 6);
}
/* threads from my main function
are as below */
int main() {
receive_from_hmi.start_async_receive();
boost::thread thread_receive_from_hmi(&udpComm::run_io_service,
&receive_from_hmi);
boost::thread thread_serial(&thread_write_to_datalink);
thread_serial.join();
thread_receive_from_hmi.join();
return 0;
}
/* The Serial_manager class contains functions for writting and reading from serial port*/
#include <iostream>
#include <boost/thread.hpp>
#include <boost/asio.hpp>
#include <boost/date_time/posix_time/posix_time.hpp>
using namespace boost::asio;
class Serial_manager {
public:
Serial_manager(boost::asio::io_service &serial_io_service,char *dev_name);
void open_serial_port();
void write_to_serial(void *data, int size);
size_t read_from_serial(void *data, int size);
void handle_serial_exception(std::exception &ex);
virtual ~Serial_manager();
void setDeviceName(char* deviceName);
protected:
io_service &port_io_service;
serial_port datalink_serial_port;
bool serial_port_open;
char *device_name;
};
void Serial_manager::setDeviceName(char* deviceName) {
device_name = deviceName;
}
Serial_manager::Serial_manager(boost::asio::io_service &serial_io_service,char *dev_name):
port_io_service(serial_io_service),
datalink_serial_port(serial_io_service) {
device_name = dev_name;
serial_port_open = false;
open_serial_port();
}
void Serial_manager::open_serial_port() {
bool temp_port_status = false;
bool serial_port_msg_printed = false;
do {
try {
datalink_serial_port.open(device_name);
temp_port_status = true;
} catch (std::exception &ex) {
if (!serial_port_msg_printed) {
std::cout << "Exception-check the serial port device "
<< ex.what() << std::endl;
serial_port_msg_printed = true;
}
datalink_serial_port.close();
temp_port_status = false;
}
} while (!temp_port_status);
serial_port_open = temp_port_status;
std::cout <<std::endl <<"serial port device opened successfully"<<std::endl;
datalink_serial_port.set_option(serial_port_base::baud_rate(115200));
datalink_serial_port.set_option(
serial_port_base::flow_control(
serial_port_base::flow_control::none));
datalink_serial_port.set_option(
serial_port_base::parity(serial_port_base::parity::none));
datalink_serial_port.set_option(
serial_port_base::stop_bits(serial_port_base::stop_bits::one));
datalink_serial_port.set_option(serial_port_base::character_size(8));
}
void Serial_manager::write_to_serial(void *data, int size) {
boost::asio::write(datalink_serial_port, boost::asio::buffer(data, size));
}
size_t Serial_manager::read_from_serial(void *data, int size) {
return boost::asio::read(datalink_serial_port, boost::asio::buffer(data, size));
}
void Serial_manager::handle_serial_exception(std::exception& ex) {
std::cout << "Exception-- " << ex.what() << std::endl;
std::cout << "Cannot access data-link, check the serial connection"
<< std::endl;
datalink_serial_port.close();
open_serial_port();
}
Serial_manager::~Serial_manager() {
// TODO Auto-generated destructor stub
}
I think my area of problem is about thread synchronization and notification and I will be happy if you help me. You should not worry about the sender it is works perfectly as I already checked it the data is received at the receiver thread. I hope you understand my question.
Edit: Here is the modification.My whole idea here is to develop a simulation for the Manual flight control so according my design i have client application that sends commands through
udp communication. At the receiver side intended to use 3 threads. one thread receives input from sticks i.e void start_hotas() the second thread is a thread that receives commands from sender(client): void udpComm::run_io_service() and 3rd is the void thread_write_to_datalink().
/* a thread that listens for input from sticks*/
void start_hotas() {
Hotas_manager hotasobj;
__s16 event_value; /* value */
__u8 event_number; /* axis/button number */
while (1) {
hotasobj.readData_from_hotas();
event_number = hotasobj.getJoystickEvent().number;
event_value = hotasobj.getJoystickEvent().value;
if (hotasobj.isAxisPressed()) {
if (event_number == 0) {
aileron = (float) event_value / 32767;
} else if (event_number == 1) {
elevator = -(float) event_value / 32767;
} else if (event_number == 2) {
rudder = (float) event_value / 32767;
} else if (event_number == 3) {
brake_left = (float) (32767 - event_value) / 65534;
} else if (event_number == 4) {
} else if (event_number == 6) {
} else if (event_number == 10) {
} else if (event_number == 11) {
} else if (event_number == 12) {
}
} else if (hotasobj.isButtonPressed()) {
}
usleep(1000);
}
}
/*
* Hotas.h
*
* Created on: Jan 31, 2013
* Author: metec
*/
#define JOY_DEV "/dev/input/js0"
#include <iostream>
#include <boost/thread.hpp>
#include <boost/asio.hpp>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <linux/joystick.h>
bool message_printed = false;
bool message2_printed = false;
class Hotas_manager {
public:
Hotas_manager();
virtual ~Hotas_manager();
void open_hotas_device();
/*
*
* read from hotas input
* used to the updated event data and status of the joystick from the
* the file.
*
*/
void readData_from_hotas();
js_event getJoystickEvent() {
return joystick_event;
}
int getNumOfAxis() {
return num_of_axis;
}
int getNumOfButtons() {
return num_of_buttons;
}
bool isAxisPressed() {
return axis_pressed;
}
bool isButtonPressed() {
return button_pressed;
}
int* getAxis() {
return axis;
}
char* getButton() {
return button;
}
private:
int fd;
js_event joystick_event;
bool hotas_connected;
int num_of_axis;
int num_of_buttons;
int version;
char devName[80];
/*
* the the variables below indicates
* the state of the joystick.
*/
int axis[30];
char button[30];
bool button_pressed;
bool axis_pressed;
};
Hotas_manager::Hotas_manager() {
// TODO Auto-generated constructor stub
hotas_connected = false;
open_hotas_device();
std::cout << "joystick device detected" << std::endl;
}
Hotas_manager::~Hotas_manager() {
// TODO Auto-generated destructor stub
}
void Hotas_manager::open_hotas_device() {
bool file_open_error_printed = false;
while (!hotas_connected) {
if ((fd = open(JOY_DEV, O_RDONLY)) > 0) {
ioctl(fd, JSIOCGAXES, num_of_axis);
ioctl(fd, JSIOCGBUTTONS, num_of_buttons);
ioctl(fd, JSIOCGVERSION, version);
ioctl(fd, JSIOCGNAME(80), devName);
/*
* NON BLOCKING MODE
*/
ioctl(fd, F_SETFL, O_NONBLOCK);
hotas_connected = true;
} else {
if (!file_open_error_printed) {
std::cout << "hotas device not detected. check "
"whether it is "
"plugged" << std::endl;
file_open_error_printed = true;
}
close(fd);
hotas_connected = false;
}
}
}
void Hotas_manager::readData_from_hotas() {
int result;
result = read(fd, &joystick_event, sizeof(struct js_event));
if (result > 0) {
switch (joystick_event.type & ~JS_EVENT_INIT) {
case JS_EVENT_AXIS:
axis[joystick_event.number] = joystick_event.value;
axis_pressed = true;
button_pressed = false;
break;
case JS_EVENT_BUTTON:
button[joystick_event.number] = joystick_event.value;
button_pressed = true;
axis_pressed = false;
break;
}
message2_printed = false;
message_printed = false;
} else {
if (!message_printed) {
std::cout << "problem in reading the stick file" << std::endl;
message_printed = true;
}
hotas_connected = false;
open_hotas_device();
if (!message2_printed) {
std::cout << "stick re-connected" << std::endl;
message2_printed = true;
}
}
}
I updated the main function to run 3 threads .
int main() {
boost::asio::io_service receive_from_hmi_io;
udpComm receive_from_hmi(receive_from_hmi_io, 6012);
receive_from_hmi.setRemoteEndpoint("127.0.0.1", 6011);
receive_from_hmi.start_async_receive();
boost::thread thread_receive_from_hmi(&udpComm::run_io_service,
&receive_from_hmi);
boost::thread thread_serial(&thread_write_to_datalink);
boost::thread thread_hotas(&start_hotas);
thread_hotas.join();
thread_serial.join();
thread_receive_from_hmi.join();
return 0;
}
The void thread_write_to_datalink() also writes the data come from the hotas_manager(joysticks).
void thread_write_to_datalink() {
/*
* boost serial communication
*/
boost::asio::io_service serial_port_io;
Serial_manager serial(serial_port_io, (char*) "/dev/ttyUSB0");
cout << "aileron " << "throttle " << "elevator " << endl;
while (1) {
// commands from udp communication
serial.write_to_serial(read_hmi_buffer, 6);
// data come from joystick inputs
//cout << aileron<<" "<<throttle<<" "<<elevator<< endl;
memcpy(&buffer_manual_flightcontrol[4], &aileron, 4);
memcpy(&buffer_manual_flightcontrol[8], &throttle, 4);
memcpy(&buffer_manual_flightcontrol[12], &elevator, 4);
unsigned char temp;
try {
serial.write_to_serial(buffer_manual_flightcontrol, 32);
//serial.write_to_serial(buffer_manual_flightcontrol, 32);
} catch (std::exception& exp) {
serial.handle_serial_exception(exp);
}
try {
serial.write_to_serial(buffer_payloadcontrol, 20);
} catch (std::exception& exp) {
serial.handle_serial_exception(exp);
}
usleep(100000);
}
}
My question is how better can I design to synchronize these 3 threads. If your answer says you do not need to use 3 threads I need you to tell me how.
Let's back up a little bit from multi-threading, your program mixes synchronous and asynchronous operations. You don't need to do this, as it will only cause confusion. You can asynchronously write the buffer read from the UDP socket to the serial port. This can all be achieved with a single thread running the io_service, eliminating any concurrency concerns.
You will need to add buffer management to keep the data read from the socket in scope for the lifetime of the async_write for the serial port, study the async UDP server as an example. Also study the documentation, specifically the requirements for buffer lifetime in async_write
buffers
One or more buffers containing the data to be written.
Although the buffers object may be copied as necessary, ownership of
the underlying memory blocks is retained by the caller, which must
guarantee that they remain valid until the handler is called.
Once you have completed that design, then you can move to more advanced techniques such as a thread pool or multiple io_services.
You need to make your access to read_hmi_buffer synchronized.
Therefore you need a mutex (std::mutex, pthread_mutex_t, or the windows equivalent), to lock onto whenever a piece of code read or write in that buffer.
See this question for a few explanations on the concept and links to other tutorials.