My program has a buffer when sending data. Instead of directly calling async_write every time for send small packets, try to make the send method return quickly, use streambuf as the sending buffer, and try to send large packets.
The problem encountered now is that when multiple threads call send at the same time, there is a small probability that the opposite end may receive duplicate data packets or messy data. Here is my code:
void ClientConnection::send(const string* buffer, function<void (bool status)> callback) {
{
unique_lock<mutex> lck(*_ioLockPtr);
ostream os(_sendBufferPtr.get());
os << *buffer;
}
delete buffer;
callback(true);
_sendBuffer();
}
void ClientConnection::_sendBuffer() {
unique_lock<mutex> lck(*_ioLockPtr);
size_t bufferSize = _sendBufferPtr->size();
if (!bufferSize || _sendingBufferCount > 0) {
return;
}
++_sendingBufferCount;
async_write(*_socketPtr, _sendBufferPtr->data(), boost::asio::transfer_exactly(bufferSize), boost::bind(&ClientConnection::_handleWrite,
shared_from_this(), boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
_sendBufferPtr->consume(bufferSize);
}
void ClientConnection::_handleWrite(const boost::system::error_code& error, size_t bytes_transferred) {
if (!error) {
unique_lock<mutex> lck(*_ioLockPtr);
size_t bufferSize = _sendBufferPtr->size();
if (bufferSize) {
async_write(*_socketPtr, _sendBufferPtr->data(), boost::asio::transfer_exactly(bufferSize), boost::bind(&ClientConnection::_handleWrite,
shared_from_this(), boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
_sendBufferPtr->consume(bufferSize);
} else {
--_sendingBufferCount;
}
} else {
{
unique_lock<mutex> lck(*_ioLockPtr);
--_sendingBufferCount;
}
_close();
}
}
The relevant variables are defined as follows：
shared_ptr<boost::asio::streambuf> _sendBufferPtr;
uint8_t _sendingBufferCount;
Please help me to understand how to solve this problem, thanks!
The problem encountered now is that when multiple threads call send at the same time
This is strictly prohibited as per the documentation:
This operation is implemented in terms of zero or more calls to the stream's async_write_some function, and is known as a composed operation. The program must ensure that the stream performs no other write operations (such as async_write, the stream's async_write_some function, or any other composed operations that perform writes) until this operation completes.
To serialize the async operations, you may want to use a strand.
Related
I have started N same async operations(e.g. N requests to database), so i need to do something after all this operations end. How i can do this? (After one async operation end, my callback will be called).
I use C++14
Example
i use boost.asio to write some data to socket.
for (int i = 0; i < N; ++i)
{
boost::asio::async_write(
m_socket,
boost::asio::buffer(ptr[i], len[i]),
[this, callback](const boost::system::error_code& ec, std::size_t )
{
callback(ec);
});
}
So i need to know when all my writes ends;
first of all, never call async_write in a loop. Each socket may have only one async_write and one async_read outstanding at any one time.
boost already has provision for scatter/gather io.
This snippet should give you enough information to go on.
Notice that async_write can take a vector of vectors as a 'buffer' and it will fire the handler exactly once, once all the buffers have been written.
struct myclass {
boost::asio::ip::tcp::socket m_socket;
std::vector<std::vector<char>> pending_buffers;
std::vector<std::vector<char>> writing_buffers;
void write_all()
{
assert(writing_buffers.size() == 0);
writing_buffers = std::move(pending_buffers);
boost::asio::async_write(
m_socket,
boost::asio::buffer(writing_buffers),
std::bind(&myclass::write_all_handler,
this,
std::placeholders::_1,
std::placeholders::_2));
}
void write_all_handler(const boost::system::error_code& ec, size_t bytes_written)
{
writing_buffers.clear();
// send next load of data
if (pending_buffers.size())
write_all();
// call your callback here
}
};
Now I have a Connection class as shown below (irrelevant things are omitted):
class Connection : public std::enable_shared_from_this<Connection> {
public:
virtual void write() {
socket_->async_write_some(boost::asio::buffer(buffer_.data(),
buffer_.size()),
std::bind(&Connection::on_written,
shared_from_this(),
std::placeholders::_1,
std::placeholders::_2));
}
void on_written(const boost::system::error_code& e, std::size_t length) {
if(e) {
// handle error here
return;
}
buffer_.consume(length);
}
void add_to_buf(const std::string& data) {
// add the string data to buffer_ here
}
private:
boost::asio::io_service& service_;
std::unique_ptr<socket> socket_;
boost::asio::streambuf buffer_;
};
As you see, the write() operation will write data in buffer_ and the buffer_ is only cleaned in the write operation completion handler. However, the problem comes, now I have the following invocation code (Note: it is multi-threaded):
Connection conn;
// initialization code here
conn.add_to_buf("first ");
conn.write();
conn.add_to_buf("second");
conn.write();
The output I want is first second, but, sometimes the output could be first first second. It happens when the second operation starts but the first completion handler has not been called. I have read about strand to serialize things, however, it can only serialize tasks, it cannot serialize a completion handler and a task.
Someone may suggest to call the second write operation in the first's completion handler, but, per the design, this cannot be achieved.
So, any suggestions? maybe a lock on buffer_?
Locking the buffer per se wont change anything. If you call write before the first write has completed it will send the same data again. In my opinion the best way is to drop the add_to_buf method and just stick to a write function that does both, add data to the buffer and if neccessary triggers a send.
class Connection : public std::enable_shared_from_this<Connection> {
public:
virtual void write(const std::string& data) {
std::lock_guard<std::mutex> l(lock_);
bool triggerSend = buffer_.size() == 0;
// add data to buffer
if (triggerSend) {
do_send_chunk();
}
}
void on_written(const boost::system::error_code& e, std::size_t length) {
if (e) {
// handle error here
return;
}
std::lock_guard<std::mutex> l(lock_);
buffer_.consume(length);
if (buffer_.size() > 0) {
do_send_chunk();
}
}
private:
void do_send_chunk() {
socket_->async_write_some(boost::asio::buffer(buffer_.data(),
buffer_.size()),
std::bind(&Connection::on_written,
shared_from_this(),
std::placeholders::_1,
std::placeholders::_2));
}
boost::asio::io_service& service_;
std::unique_ptr<socket> socket_;
boost::asio::streambuf buffer_;
std::mutex lock_;
};
The idea is, that the write function checks if there is still data left in the buffer. In this case it does not have to trigger a do_send_chunk call as sooner or later on_written will be called which then will cause another do_send_chunk as the new data will stay in the buffer and the if(buffer_.size() > 0) will be true inside on_written. If however there is no data left it has to trigger a send operation.
I'm implementing a tcp server with boost asio library.
In the server, I use asio::async_read_some to get data, and use asio::write to write data. The server code is something like that.
std::array<char, kBufferSize> buffer_;
std::string ProcessMessage(const std::string& s) {
if (s == "msg1") return "resp1";
if (s == "msg2") return "resp2";
return "";
}
void HandleRead(const boost::system::error_code& ec, size_t size) {
std::string message(buffer_.data(), size);
std::string resp = ProcessMessage(message);
if (!resp.empty()) {
asio::write(socket, boost::asio::buffer(message), WriteCallback);
}
socket.async_read_some(boost::asio::buffer(buffer_));
}
Then I write a client to test the server, the code is something like
void MessageCallback(const boost::system::error_code& ec, size_t size) {
std::cout << string(buffer_.data(), size) << std::endl;
}
//Init socket
asio::write(socket, boost::asio::buffer("msg1"));
socket.read_some(boost::asio::buffer(buffer_), MessageCallback);
// Or async_read
//socket.async_read_some(boost::asio::buffer(buffer_), MessageCallback);
asio::write(socket, boost::asio::buffer("msg1"));
socket.read_some(boost::asio::buffer(buffer_), MessageCallback);
// Or async_read
//socket.async_read_some(boost::asio::buffer(buffer_), MessageCallback);
If I run the client, the code will be waiting at second read_some, and output is:resp1.
If I remove the first read_some, the ouput is resp1resp2, that means the server done the right thing.
It seems the first read_some EAT the second response but don't give the response to MessageCallback function.
I've read the quesion at What is a message boundary?, I think if this problem is a "Message Boundary" problem, the second read_some should print something as the first read_some only get part of stream from the tcp socket.
How can I solve this problem?
UPDATE:
I've try to change the size of client buffer to 4, that output will be:
resp
resp
It seems the read_some function will do a little more than read from the socket, I'll read the boost code to find out is that true.
The async_read_some() member function is very likely not doing what you intend, pay special attention to the Remarks section of the documentation
The read operation may not read all of the requested number of bytes.
Consider using the async_read function if you need to ensure that the
requested amount of data is read before the asynchronous operation
completes.
Note that async_read() free function does offer the guarantee that you are looking for
This operation is implemented in terms of zero or more calls to the
stream's async_read_some function, and is known as a composed
operation. The program must ensure that the stream performs no other
read operations (such as async_read, the stream's async_read_some
function, or any other composed operations that perform reads) until
this operation completes.
I'm having a hard time understand the correct way I should structure a tcp client when using async_read and async_write. The examples seem to do a async_read after connecting and then have async_write in the handler.
In the case of my client and sever, when the client connects it needs to check a queue of messages to write and check to see if anything needs to be read. One of the things I'm having a hard time with is understanding how this would work asynchronously.
What I envision is in the async_connect handler, the thread would call async_write if anything is in the sendQueue and call async_read over and over. Or should it check if anything is available to be read before it does an async_read?
Below is an example of what I'm talking about.
void BoostTCPConnection::connectHandler()
{
setRunning(true);
while (isRunning())
{
//If send Queue has messages
if ( sendSize > 0)
{
//Calls to async_write
send();
}
boost::shared_ptr<std::vector<char> > sizeBuffer(new std::vector<char>(4));
boost::asio::async_read(socket_, boost::asio::buffer(data, size), boost::bind(&BoostTCPConnection::handleReceive, shared_from_this(), boost::asio::placeholders::error, sizeBuffer));
}
}
void BoostTCPConnection::handleReceive(const boost::system::error_code& error, boost::shared_ptr<std::vector<char> > sizeBuffer)
{
if (error)
{
//Handle Error
return;
}
size_t messageSize(0);
memcpy((void*)(&messageSize),(void*)sizeBuffer.data(),4);
boost::shared_ptr<std::vector<char> > message(new std::vector<char>(messageSize) );
//Will this create a race condition with other reads?
//Should a regular read happen here
boost::asio::async_read(socket_, boost::asio::buffer(data, size),
boost::bind(&BoostTCPConnection::handleReceiveMessage, shared_from_this(),
boost::asio::placeholders::error, message));
}
void BoostTCPConnection::handleReceiveMessage(const boost::system::error_code& error, boost::shared_ptr<std::vector<char> > rcvBuffer)
{
if (error)
{
//Handle Error
return;
}
boost::shared_ptr<std::string> message(new std::string(rcvBuffer.begin(),rcvBuffer.end()));
receivedMsgs_.push_back(message);
}
void BoostTCPConnection::handleWrite(const boost::system::error_code& error,size_t bytes_transferred)
{
//Success
if (error.value() == 0)
return;
//else handleError
}
Conceptually, async_read waits for data to be received. You should call it any time you want something to happen after data is received and a read isn't already pending. Similarly, async_write waits for data to be written. You should call it any time you need to write data and a write isn't already pending.
You should call async_read when you complete the connection. Before your async_read handler returns, it should probably call async_read again.
When you need to write to the connection, you should call async_write (if a write isn't already pending). In your async_write handler, if you still need to write more, you should call async_write again.
If no read is already pending, you can call async_read in your write handler, if you wish to resume reading after you finish writing. You can also just keep a read always pending. That's up to you.
You should not check if there's anything to read before calling async_read. The point of async_read is for it to complete when there's something to read. It's a smart way of waiting and doing other things in the meantime.
bool Connection::Receive(){
std::vector<uint8_t> buf(1000);
boost::asio::async_read(socket_,boost::asio::buffer(buf,1000),
boost::bind(&Connection::handler, this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
int rcvlen=buf.size();
ByteBuffer b((std::shared_ptr<uint8_t>)buf.data(),rcvlen);
if(rcvlen <= 0){
buf.clear();
return false;
}
OnReceived(b);
buf.clear();
return true;
}
The method works fine but only when I make a breakpoint inside it. Is there an issue with timing as it waits to receive? Without the breakpoint, nothing is received.
You are trying to read from the receive buffer immediately after starting the asynchronous operation, without waiting for it to complete, that is why it works when you set a breakpoint.
The code after your async_read belongs into Connection::handler, since that is the callback you told async_read to invoke after receiving some data.
What you usually want is a start_read and a handle_read_some function:
void connection::start_read()
{
socket_->async_read_some(boost::asio::buffer(read_buffer_),
boost::bind(&connection::handle_read_some, shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
void connection::handle_read_some(const boost::system::error_code& error, size_t bytes_transferred)
{
if (!error)
{
// Use the data here!
start_read();
}
}
Note the shared_from_this, it's important if you want the lifetime of your connection to be automatically taken care of by the number of outstanding I/O requests. Make sure to derive your class from boost::enable_shared_from_this<connection> and to only create it with make_shared<connection>.
To enforce this, your constructor should be private and you can add a friend declaration (C++0x version; if your compiler does not support this, you will have to insert the correct number of arguments yourself):
template<typename T, typename... Arg> friend boost::shared_ptr<T> boost::make_shared(const Arg&...);
Also make sure your receive buffer is still alive by the time the callback is invoked, preferably by using a statically sized buffer member variable of your connection class.