I'm using a C library which rips PDF data and provides me with that data via callbacks. Two callbacks are used, one which provides me with the job header and another which provides me with the the ripped data ranging from 1 - 50MB chunks.
I'm then taking that data and sending it across the wire via TCP to someone who cares.
I'm using the boost async_write to send that data across the wire. I want to synchronize access to the async_write until it's done sending the previous chunk of data.
The C callback functions:
void __stdcall HeaderCallback( void* data, int count )
{
// The Send function is a member of my AsyncTcpClient class.
// This is how I'm currently providing my API with the PDF data.
client.Send( data, count );
}
void __stdcall DataCallback( void* data, int count )
{
client.Send( data, count );
}
I receive the provided data in my AsyncTcpClient class's Send method.
void AsyncTcpClient::Send( void* buffer, size_t length )
{
// Write to the remote server.
boost::asio::async_write( _session->socket,
boost::asio::buffer( ( const char* )buffer, length ),
[ this ]( boost::system::error_code const& error, std::size_t bytesTransfered )
{
if ( error )
{
_session->errorCode = error;
OnRequestComplete( _session );
return;
}
std::unique_lock<std::mutex> cancelLock( _session->cancelGuard );
if ( _session->cancelled )
{
OnRequestComplete( _session );
return;
}
} );
}
How can I synchronize access to the async_write function?
Using a mutex at the start of the Send function would be pointless as the async_write returns immediately.
It's also pointless to store the mutex in a unique_lock member variable and attempt to unlock it in the async_write callback lambda as that'll blow up.
How can I synchronize access to the async_write function without using strand?
The first iteration of the program wont use strand for synchronization, I will be implementing that later.
You should use an io_context::strand.
One example from many others, but that answer will help you.
Related
When calling asio::async_read() using a future, is there a way to get the number of bytes transferred when a boost:asio::error::eof exception occurs? It would seem that there are many cases when one would want to get the data transferred even if the peer disconnects.
For example:
namespace ba = boost::asio;
int32_t Session::read (unsigned char* pBuffer, uint32_t bufferSizeToRead)
{
// Create a mutable buffer
ba::mutable_buffer buffer (pBuffer, bufferSizeToRead);
int32_t result = 0;
// We do an async call using a future. A thread from the io_context pool does the
// actual read while the the thread calling this method will blocks on the
// std::future::get()
std::future<std::size_t> future =
ba::async_read(m_socket, buffer, ba::bind_executor(m_sessionStrand, ba::use_future));
try
{
// We block the calling thread here until we get the results of the async_read_some()...
result = future.get();
}
catch (boost::system::system_error &ex) // boost::system::system_error
{
auto exitCode = ex.code().value();
if ( exitCode == ba::error::eof )
{
log ("Connection closed by the peer");
}
}
return results; // This is zero if eof occurs
}
The code sample above represents our issue. It was designed to support a 3rd-party library. The library expects a blocking call. The new code under development is using ASIO with a minimal number of network threads. The expectation is that this 3rd party library calls session::read using its dedicated thread and we adapt the call to an asynchronous call. The network call must be async since we are supporting many such calls from different libraries with minimal threads.
What was unexpected and discovered late is that ASIO treats a connection closed as an error. Without the future, using a handler we could get the bytes transferred up to the point where the disconnect occurred. However, using a future, the exception is thrown and the bytes transferred becomes unknown.
void handler (const boost::system::error_code& ec,
std::size_t bytesTransferred );
Is there a way to do the above with a future and also get the bytes transferred?
Or ss there an alternative approach where we can provide the library a blocking call by still use an asio::async_read or similar.
Our expectation is that we could get the bytes transferred even if the client closed the connection. We're puzzled that when using a future this does not seem possible.
It's an implementation limitation of futures.
Modern async_result<> specializations (that use the initiate member approach) can be used together with as_tuple, e.g.:
ba::awaitable<std::tuple<boost::system::error_code, size_t>> a =
ba::async_read(m_socket, buffer, ba::as_tuple(ba::use_awaitable));
Or, more typical:
auto [ec, n] = co_await async_read(m_socket, buffer, ba::as_tuple(ba::use_awaitable));
However, the corresponding:
auto future = ba::async_read(m_socket, buffer, ba::as_tuple(ba::use_future));
isn't currently supported. It arguably could, but you'd have to create your own completion token, or ask Asio devs to add support to use_future: https://github.com/chriskohlhoff/asio/issues
Side-note: if you construct the m_socket from the m_sessioStrand executor, you do not need to bind_executor to the strand:
using Executor = net::io_context::executor_type;
struct Session {
int32_t read(unsigned char* pBuffer, uint32_t bufferSizeToRead);
net::io_context m_ioc;
net::strand<Executor> m_sessionStrand{m_ioc.get_executor()};
tcp::socket m_socket{m_sessionStrand};
};
Background
I'm new to using Boost::Asio library and am having trouble getting the behaviour I want. I am trying to implement some network communication for custom hardware solution. The communication protocol stack we are using relies heavily on Boost::Asio async methods and I don't believe it is entirely thread safe.
I have successfully implemented sending but encountered a problem when trying to setup the await for receiving. Most boost::asio examples I have found rely on socket behaviour to implement async await with socket_.async_read_some() or other similar functions. However this doesn't work for us as our hardware solution requires calling driver function directly rather than utilising sockets.
The application uses an io_service that is passed into boost::asio::generic::raw_protocol::socket as well as other classes.
Example code from protocol stack using sockets
This is the example code from the protocol stack. do_receive() is called in the constructor of RawSocketLink.
void RawSocketLink::do_receive()
{
namespace sph = std::placeholders;
socket_.async_receive_from(
boost::asio::buffer(receive_buffer_), receive_endpoint_,
std::bind(&RawSocketLink::on_read, this, sph::_1, sph::_2));
}
void RawSocketLink::on_read(const boost::system::error_code& ec, std::size_t read_bytes)
{
if (!ec) {
// Do something with received data...
do_receive();
}
}
Our previous receive code without the protocol stack
Prior to implementing the stack we had been using the threading library to create separate threads for send and recieve. The receive method is shown below. Mostly it relies on calling the receive_data() function from the hardware drivers and waiting for it to return. This is a blocking call but is required to return data.
void NetworkAdapter::Receive() {
uint8_t temp_rx_buffer[2048];
rc_t rc;
socket_t *socket_ptr;
receive_params_t rx_params;
size_t rx_buffer_size;
char str[100];
socket_ptr = network_if[0];
while (1) {
rx_buffer_size = sizeof(temp_rx_buffer);
// Wait until receive_data returns then process
rc = receive_data(socket_ptr,
temp_rx_buffer,
&rx_buffer_size,
&rx_params,
WAIT_FOREVER);
if (rc_error(rc)) {
(void)fprintf(stderr, "Receive failed");
continue;
}
// Do something with received packet ....
}
return;
}
Note that the socket_t pointer in this code is not the same thing as a TCP/UDP socket for Boost::Asio.
Current implement of async receive
This is my current code and where I need help. I'm not sure how to use boost::asio method to wait for receive_data to return. We are trying to replicate the behaviour of socket.async_read_from(). The NetworkAdapter has access to the io_service.
void NetworkAdapter::do_receive() {
rc_t rc;
socket_t *socket_ptr;
receive_params_t rx_params;
size_t rx_buffer_size;
socket_ptr = network_if[0];
rx_buffer_size = receive_buffer_.size();
// What do I put here to await for this to return asynchronously?
rc = receive_data(socket_ptr, receive_buffer_.data(), &rx_buffer_size, &rx_params, ATLK_WAIT_FOREVER);
on_read(rc, rx_buffer_size, rx_params);
}
void NetworkAdapter::on_read(const rc_t &rc, std::size_t read_bytes, const receive_params_t &rx_params) {
if (!rc) {
// Do something with received data...
} else {
LOG(ERROR) << "Packet receieve failure";
}
do_receive();
}
Summary
How do I use boost::asio async/await functions to await a function return? In particular I want to replicate the behaviour of socket.async_receive_from() but with a function rather than a socket.
*Some function names and types have been changed due to data protection requirements.
N4045 Library Foundations for Asynchronous Operations, Revision 2
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4045.pdf
On page 24 there is an example on how to implement an asio async API in terms of callback-based os API.
// the async version of your operation, implementing all kinds of async paradigm in terms of callback async paradigm
template <class CompletionToken>
auto async_my_operation(/* any parameters needed by the sync version of your operation */, CompletionToken&& token)
{
// if CompletionToken is a callback function object, async_my_operation returns void, the callback's signature should be void(/* return type of the sync version of your operation, */error_code)
// if CompletionToken is boost::asio::use_future, async_my_operation returns future</* return type of the sync version of your operation */>
// if CompletionToken is ..., ...
// you are not inventing new async paradigms so you don't have to specialize async_completion or handler_type, you should focus on implement the os_api below
async_completion<CompletionToken, void(/* return type of the sync version of your operation, */error_code)/* signature of callback in the callback case */> completion(token);
typedef handler_type_t<CompletionToken, void(error_code)> Handler;
unique_ptr<wait_op<Handler>> op(new wait_op<Handler>(move(completion.handler))); // async_my_operation initates your async operation and exits, so you have to store completion.handler on the heap, the completion.handler will be invoked later on a thread pool (e.g. threads blocked in IOCP if you are using os api, threads in io_context::run() if you are using asio (sockets accept an io_context during construction, so they know to use which io_context to run completion.handler))
// most os api accepts a void* and a void(*)(result_t, void*) as its C callback function, this is type erasure: the void* points to (some struct that at least contains) the C++ callback function object (can be any type you want), the void(*)(result_t, void*) points to a C callback function to cast the void* to a pointer to C++ callback function object and call it
os_api(/* arguments, at least including:*/ op.get(), &wait_callback<Handler>);
return completion.result.get();
}
// store the handler on the heap
template <class Handler>
struct wait_op {
Handler handler_;
explicit wait_op(Handler handler) : handler_(move(handler)) {}
};
// os post a message into your process's message queue, you have several threads blocking in a os api (such as IOCP) or asio api (such as io_context::run()) that continuously takes message out from the queue and then call the C callback function, the C callback function calls your C++ callback function
template <class Handler>
void wait_callback(result_t result, void* param)
{
unique_ptr<wait_op<Handler>> op(static_cast<wait_op<Handler>*>(param));
op‐>handler_(/* turn raw result into C++ classes before passing it to C++ code */, error_code{});
}
//trivial implementation, you should consult the socket object to get the io_context it uses
void os_api(/* arguments needed by your operation */, void* p_callback_data, void(*p_callback_function)(result_t, void*))
{
std::thread([](){
get the result, blocks
the_io_context_of_the_socket_object.post([](){ (*p_callback_function)(result, p_callback_data); });
}).detach();
}
boost.asio has changed from async_completion and handler_type to async_result, so the above code is outdated.
Requirements on asynchronous operations - 1.75.0
https://www.boost.org/doc/libs/1_75_0/doc/html/boost_asio/reference/asynchronous_operations.html
I am implementing a connection multiplexer - class, which wraps a single connection in order to provide an ability to create so-called Stream-s over it. There can be dozens of such streams over one physical connection.
Messages sent over that connection are defined by a protocol and can be service ones (congestion control, etc), which are never seen by the clients, and data ones - they contain some data for the streams, for which one - defined in the header of the corresponding message.
I have encountered a problem when implementing a method read for a Stream. It must be blocking, but asynchronous, so that it returns some value - data read or error happened - but the request itself must be is some kind of async queue.
To implement asynchronous network IO we have used Boost's async_read-s, async_write-s, etc with a completion token, taken from another library. So, a call to MyConnection::underlying_connection::read(size_t) is asynchronous already in the terms I described before.
One solution I have implemented is function MyConnection::processFrame(), which is reading from the connection, processing message and, if it is a data message, puts the data into the corresponding stream's buffer. The function is to be called in a while loop by the stream's read. But, in that case there can be more than one simulteneous calls to async_read, which is UB. Also, this would mean that even service messages are to wait until some stream wants to read the data, which is not appropriate as well.
Another solution I came up is using future-s, but as I checked, their methods wait/get would block the whole thread (even with defered policy or paired promise), which must be avoided too.
Below is a simplified example with only methods, which are needed to understand the question. This is current implementation, which contains bugs.
struct LowLevelConnection {
/// completion token of 3-rd part library - ufibers
yield_t yield;
/// boost::asio socket
TcpSocket socket_;
/// completely async (in one thread) method
std::vector<uint8_t> read(size_t bytes) {
std::vector<uint8_t> res;
res.reserve(bytes);
boost::asio::async_read(socket_, res, yield);
return res;
}
}
struct MyConnection {
/// header is always of that length
constexpr uint32_t kHeaderSize = 12;
/// underlying connection
LowLevelConnection connection_;
/// is running all the time the connection is up
void readLoop() {
while (connection_.isActive()) {
auto msg = connection_.read(kHeaderSize);
if (msg.type == SERVICE) { handleService(msg); return; }
// this is data message; read another part of it
auto data = connection_.read(msg.data_size);
// put the data into the stream's buffer
streams_.find(data.stream_id).buffer.put(data);
}
}
}
struct Stream {
Buffer buffer;
// also async blocking method
std::vector<uint8_t> read(uint32_t bytes) {
// in perfect scenario, this should look like this
async_wait([]() { return buffer.size() >= bytes; });
// return the subbuffer of 'bytes' size and remove them
return subbufer...
}
}
Thanks for future answers!
I am learning Boost::asio socket; I saw some examples, where they use the member function of socket class to read and receive messages, or use the boost::asio common function which passes the socket as the first param.
So I am wondering what the difference is between the two approaches? thanks!
//one kind
tcp::socket sock;
sock.async_write_some(***);
//another kind
boost::asio::async_write(socket,***);
async_write as static function guarantees that all data in buffer
is written before this function returns.
async_write_some as function member guarantees that at least one byte
is written from buffer before this function ends.
So if you want to use async_write_some you need to provide more code
to handle the situation when not all data from buffer was written.
Suppose you have string with 10 bytes, it is your buffer and you want to ensure
all buffer is send:
// Pseudocode:
string str;
// add 10 bytes to str
SIZE = 10;
int writtenBytes = 0;
socket.async_write_some (buffer(str.c_str(),SIZE), makeCallback());
void callback (
error_code ec,
size_t transferredBytes)
{
// check errors
// [1]
writtenBytes += transferredBytes;
if (writtenBytes == SIZE)
return;
// call async_write_some
socket.async_write_some (buffer(str.c_str()+writtenBytes,SIZE-writtenBytes),makeCallback());
}
in callback [1] you need to check how many bytes were written,
if result is different from SIZE you need to call async_write_some again
to send the remainder of data and so on, your callback may be invoked many times.
The use of async_write is simpler:
string str; // add 10 bytes
async_write (buffer(str.c_str(),SIZE),makeCallback());
void callback() {
// check errors
[1]
}
if no errors occured in [1] you know that all data was sent.
I have a multi-threaded C++ app using lib event. The receive all happens in the libevent process and then a flag is set so that the data received is processed later. On the send tho, things go wrong. I am in a "main thread" and the data to send is assembled and then the following function is invoked.
int SocketWrapper :: SendData( const U8* buffer, int length )
{
if( m_useLibeventToSend )
{
bufferevent* bev = GetBufferEvent();
struct evbuffer* outputBuffer = bufferevent_get_output( bev );
evbuffer_lock( outputBuffer );
int result = evbuffer_add( outputBuffer, buffer, length );
evbuffer_unlock( outputBuffer );
return result;
}
return send( m_socketId, (const char* )buffer, length, 0 );
}
This function crashes on occasion at the point of the evbuffer_add invocation but 99.9% of the time it works fine. This smells like a concurrency bug and it may be related to clients crashing or coming-and-going. I made sure that during the initial creation of the socket by libevent, I did the following:
struct evbuffer* outputBuffer = bufferevent_get_output( GetBufferEvent() );
evbuffer_enable_locking( outputBuffer, NULL );
Do you have any notion of some other special initialization I should be doing? Should I not invoke "SendData" from my main thread and instead send an event to the bufferevent so that the send should happen in the same process as libevent?
All design ideas are open. So far, my workaround is to not use libevent for the send, but to write directly to the socket.
This crash happens in both release and debug, VS 2008, libevent 2.0. It's deep in the library so I will be resorting to including the c files in my project to try and track down my problem, but maybe someone here knows instantly what's wrong. :-)