Boost asio - multiple client connections to different servers - c++
I am trying to evaluate using async boost udp/tcp socket operations vs synchronous for my application. I have been trying to find an example that is similar to my design but did not find anything which led me to believe I might be trying to fit async ops into my design even though it is not the right path.
I want to connect to multiple (read: between 1-10) servers and communicate with them using different protocols; I have 4-5 threads which are producing data that needs to be communicated to any one of these server connections.
My current design is synchronous and uses an io_service object per server-connection thread and then using a thread safe queue between the producing threads and each connection thread.
This design does not seem scalable in terms of throughput performance, this is something I would like to maximize.
Are there any examples which provide this multiple connections to different servers pattern?
I have written a client to connect to 6 different servers using TCP/IP SSL/TLS which is implemented with ASIO. All 6 use the same protocol. So, if it helps, here is my code:
SSLSocket.H
#pragma once
#include <cstdlib>
#include <iostream>
#include <queue>
#include <boost/bind.hpp>
#include <boost/asio.hpp>
#include <boost/asio/ssl.hpp>
#include <boost/thread.hpp>
#include <boost/thread/mutex.hpp>
#include <boost/shared_ptr.hpp>
using namespace std;
//
#include "BufferManagement.h"
#include "Logger.h"
#include "Common Classes\Locking.h"
#include "Message.h"
class SSLSocket;
class ConcurrentMsgQueue;
#define BOOST_ASIO_ENABLE_HANDLER_TRACKING
typedef void (__stdcall *Callback)(const SSLSocket* pSSLS, const int bytesInMsg, const void* pBuf);
// typedef std::vector<boost::asio::ssl::stream<boost::asio::ip::tcp::socket> SocketVectorType;
enum {MsgLenBytes = 4};
class SSLSocket
{
// This class handles all communications between the client and the server
// using TCP/IP SSL v1. The Boost ASIO (Asynchronous I/O) library is used to accomplish this.
// Initally written by Bob Bryan on 1/21/2013.
//
public:
SSLSocket(const bool logToFile, const bool logToConsole, const bool displayInHex, const LogLevel levelOfLog, const string& logFileName, const int bufMangLen);
~SSLSocket();
void Connect(SSLSocket* psSLS, const string& serverPath, string& port);
void SendToServer(const int bytesInMsg, Byte* pBuf);
void Stop();
static void SetCallback(Callback callbackFunction)
{
// This method is required in order to be able to do a reverse pinvoke from C#.
// This callback function pointer is what is used to communicate back to the C# code.
CallbackFunction = callbackFunction;
}
static Byte* AllocateMem(int length)
{
// Allocate some memory. This method winds up getting called when the C# client needs to allocate some memory for a message.
Byte* pBuf = BufMang.GetPtr(length);
return pBuf;
}
//
static Logger Log; // Object used to log info to a file and/or to the console.
static Callback CallbackFunction; // Callback function object used to communicate with the worker thread in C#.
private:
void InitAsynchIO();
void HandleConnect(const boost::system::error_code& error);
void HandleHandshake(const boost::system::error_code& error);
void HandleFirstWrite(const boost::system::error_code& error, size_t bytes_transferred);
void HandleRead(const boost::system::error_code& error, size_t bytesTransferred);
// void HandleRead(const boost::system::error_code& error, size_t bytes_transferred);
void Terminate();
void static RcvWorkerThread(SSLSocket* sSLS);
void static SendWorkerThread(SSLSocket* psSLS);
void ProcessSendRequests();
void HandleWrite(const boost::system::error_code& error, size_t bytesTransferred);
static void WorkerThread(boost::shared_ptr< boost::asio::io_service > io_service);
//
struct Bytes
{
// Used to convert 4 bytes to an int.
unsigned char B1;
unsigned char B2;
unsigned char B3;
unsigned char B4;
};
union Bytes4ToInt
{
// Converts 4 bytes to an int.
int IntVal;
Bytes B;
};
inline int BytesToInt(const Byte * pBuf)
{
// This method converts 4 bytes from an array of bytes to a 4-byte int.
B2I.B.B1 = *pBuf++;
B2I.B.B2 = *pBuf++;
B2I.B.B3 = *pBuf++;
B2I.B.B4 = *pBuf;
int Value = B2I.IntVal;
return Value;
}
//
boost::thread_group WorkerThreads; // Used to handle creating threads.
CRITICAL_SECTION SocketLock; // Used in conjuction with the Locking object to handle single threading the code.
boost::asio::ssl::stream<boost::asio::ip::tcp::socket>* pSocket; // Pointer to the socket object.
Bytes4ToInt B2I; // Used to translate 4 bytes in the buffer to an int representing the number of bytes in the msg.
std::string sClientIp; // Client IP address. Used for logging.
unsigned short uiClientPort; // Port number. Used for logging.
// static MessageList* pRepMsgs; // Link list of the msgs to send to the server.
Byte* pDataBuf; // Pointer to the data for the current message to be read.
static boost::shared_ptr< boost::asio::io_service > IOService; // Object required for use by ASIO to perform certain functions.
static bool RcvThreadCreated; // Set when the rcv thread is created so that it won't try to create it again.
static int StaticInit; // Indicates whether or not the static members have been initialized or not.
static bool DisplayInHex; // Specifies whether to display a buffer in hex or not.
static BufferManagement BufMang; // Smart pointer to the buffer used to handle requests coming to and from the server for all sockets.
volatile static bool ReqAlive; // Used to indicate whether the request thread should die or not.
// static bool RepAlive; // Used to indicate whether the response thread should die or not.
static ConcurrentMsgQueue SendMsgQ; // Holds the messages waiting to be sent to the server.
static HANDLE hEvent; // Used for signalling between threads.
};
SSLSocket.cpp
#include "StdAfx.h"
#include "SSLSocket.h"
boost::shared_ptr< boost::asio::io_service > SSLSocket::IOService;
int SSLSocket::StaticInit = 0;
Callback SSLSocket::CallbackFunction;
BufferManagement SSLSocket::BufMang;
volatile bool SSLSocket::ReqAlive = true;
Logger SSLSocket::Log;
HANDLE SSLSocket::hEvent;
bool SSLSocket::DisplayInHex;
ConcurrentMsgQueue SSLSocket::SendMsgQ;
bool SSLSocket::RcvThreadCreated = 0;
BufferManagement* Message::pBufMang;
SSLSocket::SSLSocket(const bool logToFile, const bool logToConsole, const bool displayInHex,
const LogLevel levelOfLog, const string& logFileName, const int bufMangLen) : pSocket(0)
{
// SSLSocket Constructor.
// If the static members have not been intialized yet, then initialize them.
if (!StaticInit)
{
DisplayInHex = displayInHex;
BufMang.Init(bufMangLen);
Message::SetBufMang(&BufMang);
// This constructor enables logging according to the vars passed in.
Log.Init(logToFile, logToConsole, levelOfLog, logFileName);
// Create the crit section object
// Locking::InitLocking(ReadLock);
// Locking::InitLocking(WriteLock);
StaticInit++;
hEvent = CreateEvent(NULL, false, false, NULL);
// Define the ASIO IO service object.
// IOService = new boost::shared_ptr<boost::asio::io_service>(new boost::asio::io_service);
boost::shared_ptr<boost::asio::io_service> IOServ(new boost::asio::io_service);
IOService = IOServ;
}
}
SSLSocket::~SSLSocket(void)
{
delete pSocket;
if (--StaticInit == 0)
CloseHandle(hEvent);
}
void SSLSocket::Connect(SSLSocket* psSLS, const string& serverPath, string& port)
{
// Connects to the server.
// serverPath - specifies the path to the server. Can be either an ip address or url.
// port - port server is listening on.
//
try
{
Locking CodeLock(SocketLock); // Single thread the code.
// If the user has tried to connect before, then make sure everything is clean before trying to do so again.
if (pSocket)
{
delete pSocket;
pSocket = 0;
}
// If serverPath is a URL, then resolve the address.
// Note that this code expects the first server to always have a url.
if ((serverPath[0] < '0') || (serverPath[0] > '9')) // Assumes that the first char of the server path is not a number when resolving to an ip addr.
{
// Create the resolver and query objects to resolve the host name in serverPath to an ip address.
boost::asio::ip::tcp::resolver resolver(*IOService);
boost::asio::ip::tcp::resolver::query query(serverPath, port);
boost::asio::ip::tcp::resolver::iterator EndpointIterator = resolver.resolve(query);
// Set up an SSL context.
boost::asio::ssl::context ctx(*IOService, boost::asio::ssl::context::tlsv1_client);
// Specify to not verify the server certificiate right now.
ctx.set_verify_mode(boost::asio::ssl::context::verify_none);
// Init the socket object used to initially communicate with the server.
pSocket = new boost::asio::ssl::stream<boost::asio::ip::tcp::socket>(*IOService, ctx);
//
// The thread we are on now, is most likely the user interface thread. Create a thread to handle all incoming socket work messages.
if (!RcvThreadCreated)
{
WorkerThreads.create_thread(boost::bind(&SSLSocket::RcvWorkerThread, this));
RcvThreadCreated = true;
WorkerThreads.create_thread(boost::bind(&SSLSocket::SendWorkerThread, this));
}
// Try to connect to the server. Note - add timeout logic at some point.
boost::asio::async_connect(pSocket->lowest_layer(), EndpointIterator,
boost::bind(&SSLSocket::HandleConnect, this, boost::asio::placeholders::error));
}
else
{
// serverPath is an ip address, so try to connect using that.
//
// Create an endpoint with the specified ip address.
const boost::asio::ip::address IP(boost::asio::ip::address::from_string(serverPath));
int iport = atoi(port.c_str());
const boost::asio::ip::tcp::endpoint EP(IP, iport);
// Set up an SSL context.
boost::asio::ssl::context ctx(*IOService, boost::asio::ssl::context::tlsv1_client);
// Specify to not verify the server certificiate right now.
ctx.set_verify_mode(boost::asio::ssl::context::verify_none);
// Init the socket object used to initially communicate with the server.
pSocket = new boost::asio::ssl::stream<boost::asio::ip::tcp::socket>(*IOService, ctx);
//
// Try to connect to the server. Note - add timeout logic at some point.
//pSocket->core_.engine_.do_connect(void*, int);
// pSocket->next_layer_.async_connect(EP, &SSLSocket::HandleConnect)
// pSocket->next_layer().async_connect(EP, &SSLSocket::HandleConnect);
boost::system::error_code EC;
pSocket->next_layer().connect(EP, EC);
if (EC)
{
// Log an error. This worker thread should exit gracefully after this.
stringstream ss;
ss << "SSLSocket::Connect: connect failed to " << sClientIp << " : " << uiClientPort << ". Error: " << EC.message() + ".\n";
Log.LogString(ss.str(), LogError);
}
HandleConnect(EC);
// boost::asio::async_connect(pSocket->lowest_layer(), EP,
// boost::bind(&SSLSocket::HandleConnect, this, boost::asio::placeholders::error));
}
}
catch (std::exception& e)
{
stringstream ss;
ss << "SSLSocket::Connect: threw an error - " << e.what() << ".\n";
Log.LogString(ss.str(), LogError);
Stop();
}
}
void SSLSocket::SendToServer(const int bytesInMsg, Byte* pBuf)
{
// This method creates a msg object and saves it in the SendMsgQ object.
// sends the number of bytes specified by bytesInMsg in pBuf to the server.
//
Message* pMsg = Message::GetMsg(this, bytesInMsg, pBuf);
SendMsgQ.Push(pMsg);
// Signal the send worker thread to wake up and send the msg to the server.
SetEvent(hEvent);
}
void SSLSocket::SendWorkerThread(SSLSocket* psSLS)
{
// This thread method that gets called to process the messages to be sent to the server.
//
// Since this has to be a static method, call a method on the class to handle server requests.
psSLS->ProcessSendRequests();
}
void SSLSocket::ProcessSendRequests()
{
// This method handles sending msgs to the server.
//
std::stringstream ss;
DWORD WaitResult;
Log.LogString("SSLSocket::ProcessSendRequests: Worker thread " + Logger::NumberToString(boost::this_thread::get_id()) + " started.\n", LogInfo);
// Loop until the user quits, or an error of some sort is thrown.
try
{
do
{
// If there are one or more msgs that need to be sent to a server, then send them out.
if (SendMsgQ.Count() > 0)
{
Message* pMsg = SendMsgQ.Front();
SSLSocket* pSSL = pMsg->pSSL;
SendMsgQ.Pop();
const Byte* pBuf = pMsg->pBuf;
const int BytesInMsg = pMsg->BytesInMsg;
boost::system::error_code Error;
{
Locking CodeLock(SocketLock); // Single thread the code.
boost::asio::async_write(*pSSL->pSocket, boost::asio::buffer(pBuf, BytesInMsg), boost::bind(&SSLSocket::HandleWrite, this,
boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
}
ss << "SSLSocket::ProcessSendRequests: # bytes sent = " << BytesInMsg << "\n";
Log.LogString(ss.str(), LogDebug2);
Log.LogBuf(pBuf, BytesInMsg, DisplayInHex, LogDebug3);
}
else
{
// Nothing to send, so go into a wait state.
WaitResult = WaitForSingleObject(hEvent, INFINITE);
if (WaitResult != 0L)
{
Log.LogString("SSLSocket::ProcessSendRequests: WaitForSingleObject event error. Code = " + Logger::NumberToString(GetLastError()) + ". \n", LogError);
}
}
} while (ReqAlive);
Log.LogString("SSLSocket::ProcessSendRequests: Worker thread " + Logger::NumberToString(boost::this_thread::get_id()) + " done.\n", LogInfo);
}
catch (std::exception& e)
{
stringstream ss;
ss << "SSLSocket::ProcessSendRequests: threw an error - " << e.what() << ".\n";
Log.LogString(ss.str(), LogError);
Stop();
}
}
void SSLSocket::HandleWrite(const boost::system::error_code& error, size_t bytesTransferred)
{
// This method is called after a msg has been written out to the socket. Nothing to do really since reading is handled by the HandleRead method.
std::stringstream ss;
try
{
if (error)
{
ss << "SSLSocket::HandleWrite: failed - " << error.message() << ".\n";
Log.LogString(ss.str(), LogError);
Stop();
}
}
catch (std::exception& e)
{
stringstream ss;
ss << "SSLSocket::HandleHandshake: threw an error - " << e.what() << ".\n";
Log.LogString(ss.str(), LogError);
Stop();
}
}
void SSLSocket::RcvWorkerThread(SSLSocket* psSLS)
{
// This is the method that gets called when the receive thread is created by this class.
// This thread method focuses on processing messages received from the server.
//
// Since this has to be a static method, call a method on the class to handle server requests.
psSLS->InitAsynchIO();
}
void SSLSocket::InitAsynchIO()
{
// This method is responsible for initiating asynch i/o.
boost::system::error_code Err;
string s;
stringstream ss;
//
try
{
ss << "SSLSocket::InitAsynchIO: Worker thread - " << Logger::NumberToString(boost::this_thread::get_id()) << " started.\n";
Log.LogString(ss.str(), LogInfo);
// Enable the handlers for asynch i/o. The thread will hang here until the stop method has been called or an error occurs.
// Add a work object so the thread will be dedicated to handling asynch i/o.
boost::asio::io_service::work work(*IOService);
IOService->run();
Log.LogString("SSLSocket::InitAsynchIO: receive worker thread done.\n", LogInfo);
}
catch (std::exception& e)
{
stringstream ss;
ss << "SSLSocket::InitAsynchIO: threw an error - " << e.what() << ".\n";
Log.LogString(ss.str(), LogError);
Stop();
}
}
void SSLSocket::HandleConnect(const boost::system::error_code& error)
{
// This method is called asynchronously when the server has responded to the connect request.
std::stringstream ss;
try
{
if (!error)
{
pSocket->async_handshake(boost::asio::ssl::stream_base::client,
boost::bind(&SSLSocket::HandleHandshake, this, boost::asio::placeholders::error));
ss << "SSLSocket::HandleConnect: From worker thread " << Logger::NumberToString(boost::this_thread::get_id()) << ".\n";
Log.LogString(ss.str(), LogInfo);
}
else
{
// Log an error. This worker thread should exit gracefully after this.
ss << "SSLSocket::HandleConnect: connect failed to " << sClientIp << " : " << uiClientPort << ". Error: " << error.message() + ".\n";
Log.LogString(ss.str(), LogError);
Stop();
}
}
catch (std::exception& e)
{
stringstream ss;
ss << "SSLSocket::InitAsynchIO: threw an error - " << e.what() << ".\n";
Log.LogString(ss.str(), LogError);
Stop();
}
}
void SSLSocket::HandleHandshake(const boost::system::error_code& error)
{
// This method is called asynchronously when the server has responded to the handshake request.
std::stringstream ss;
try
{
if (!error)
{
// Try to send the first message that the server is expecting. This msg tells the server we want to start communicating.
// This is the only msg specified in the C++ code. All other msg processing is done in the C# code.
//
unsigned char Msg[27] = {0x17, 0x00, 0x00, 0x00, 0x06, 0x00, 0x01, 0x00, 0x00, 0x00, 0x0b, 0x00, 0x41,
0x74, 0x74, 0x61, 0x63, 0x6b, 0x50, 0x6f, 0x6b, 0x65, 0x72, 0x02, 0x00, 0x65, 0x6e};
boost::system::error_code Err;
sClientIp = pSocket->lowest_layer().remote_endpoint().address().to_string();
uiClientPort = pSocket->lowest_layer().remote_endpoint().port();
ReqAlive = true;
// boost::asio::async_write(*pSocket, boost::asio::buffer(Msg), boost::bind(&SSLSocket::HandleFirstWrite, this,
// boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
int Count = boost::asio::write(*pSocket, boost::asio::buffer(Msg), boost::asio::transfer_exactly(27), Err);
if (Err)
{
ss << "SSLSocket::HandleHandshake: write failed - " << error.message() << ".\n";
Log.LogString(ss.str(), LogInfo);
}
HandleFirstWrite(Err, Count);
// boost::asio::async_write(pSocket, boost::asio::buffer(Msg, 27), boost::bind(&SSLSocket::HandleWrite, this,
// boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
ss.str("");
ss << "SSLSocket::HandleHandshake: From worker thread " << boost::this_thread::get_id() << ".\n";
}
else
{
ss << "SSLSocket::HandleHandshake: failed - " << error.message() << ".\n";
IOService->stop();
}
Log.LogString(ss.str(), LogInfo);
}
catch (std::exception& e)
{
stringstream ss;
ss << "SSLSocket::HandleHandshake: threw an error - " << e.what() << ".\n";
Log.LogString(ss.str(), LogError);
Stop();
}
}
void SSLSocket::HandleFirstWrite(const boost::system::error_code& error, size_t bytesTransferred)
{
// This method is called after a msg has been written out to the socket.
std::stringstream ss;
try
{
if (!error)
{
// boost::asio::async_read(pSocket, boost::asio::buffer(reply_, bytesTransferred), boost::bind(&SSLSocket::handle_read,
// this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
// boost::asio::async_read(pSocket, boost::asio::buffer(reply_, 84), boost::bind(&SSLSocket::handle_read,
// this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
// Locking CodeLock(ReadLock); // Single thread the code.
// Signal the other threads that msgs are now ready to be sent and received.
// boost::asio::async_read(pSocket, boost::asio::buffer(pRepBuf), boost::asio::transfer_exactly(4), boost::bind(&SSLSocket::HandleRead,
// this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
//
// Notify the UI that we are now connected. Create a 6 byte msg for this.
pDataBuf = BufMang.GetPtr(6);
BYTE* p = pDataBuf;
// Create msg type 500
*p = 244;
*++p = 1;
CallbackFunction(this, 2, (void*)pDataBuf);
// Get the 1st 4 bytes of the next msg, which is always the length of the that msg.
pDataBuf = BufMang.GetPtr(MsgLenBytes);
// int i1=1,i2=2,i3=3,i4=4,i5=5,i6=6,i7=7,i8=8,i9=9;
// (boost::bind(&nine_arguments,_9,_2,_1,_6,_3,_8,_4,_5,_7))
// (i1,i2,i3,i4,i5,i6,i7,i8,i9);
// boost::asio::read(*pSocket, boost::asio::buffer(pReqBuf, MsgLenBytes), boost::asio::transfer_exactly(MsgLenBytes), Err);
// boost::asio::async_read(pSocket, boost::asio::buffer(pReqBuf, MsgLenBytes), boost::bind(&SSLSocket::HandleRead, _1,_2,_3))
// (this, pReqBuf, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred);
// boost::asio::async_read(*pSocket, boost::asio::buffer(reply_), boost::asio::transfer_exactly(ByteCount), boost::bind(&Client::handle_read,
// this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
// boost::asio::async_write(*pSocket, boost::asio::buffer(pDataBuf, MsgLenBytes), boost::bind(&SSLSocket::HandleWrite, this,
// boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
Locking CodeLock(SocketLock); // Single thread the code.
boost::asio::async_read(*pSocket, boost::asio::buffer(pDataBuf, MsgLenBytes), boost::bind(&SSLSocket::HandleRead, this,
boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
}
else
{
ss << "SSLSocket::HandleFirstWrite: failed - " << error.message() << ".\n";
Log.LogString(ss.str(), LogError);
Stop();
}
}
catch (std::exception& e)
{
stringstream ss;
ss << "SSLSocket::HandleFirstWrite: threw an error - " << e.what() << ".\n";
Log.LogString(ss.str(), LogError);
Stop();
}
}
void SSLSocket::HandleRead(const boost::system::error_code& error, size_t bytesTransferred)
{
// This method is called to process an incomming message.
//
std::stringstream ss;
int ByteCount;
try
{
ss << "SSLSocket::HandleRead: From worker thread " << boost::this_thread::get_id() << ".\n";
Log.LogString(ss.str(), LogInfo);
// Set to exit this thread if the user is done.
if (!ReqAlive)
{
// IOService->stop();
return;
}
if (!error)
{
// Get the number of bytes in the message.
if (bytesTransferred == 4)
{
ByteCount = BytesToInt(pDataBuf);
}
else
{
// Call the C# callback method that will handle the message.
ss << "SSLSocket::HandleRead: From worker thread " << boost::this_thread::get_id() << "; # bytes transferred = " << bytesTransferred << ".\n";
Log.LogString(ss.str(), LogDebug2);
Log.LogBuf(pDataBuf, (int)bytesTransferred, true, LogDebug3);
Log.LogString("SSLSocket::HandleRead: sending msg to the C# client.\n\n", LogDebug2);
CallbackFunction(this, bytesTransferred, (void*)pDataBuf);
// Prepare to read in the next message length.
ByteCount = MsgLenBytes;
}
pDataBuf = BufMang.GetPtr(ByteCount);
boost::system::error_code Err;
// boost::asio::async_read(pSocket, boost::asio::buffer(pDataBuf, ByteCount), boost::bind(&SSLSocket::HandleRead,
// this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
Locking CodeLock(SocketLock); // Single thread the code.
boost::asio::async_read(*pSocket, boost::asio::buffer(pDataBuf, ByteCount), boost::bind(&SSLSocket::HandleRead,
this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
// boost::asio::read(pSocket, boost::asio::buffer(reply_), boost::asio::transfer_exactly(ByteCount), Err);
}
else
{
Log.LogString("SSLSocket::HandleRead failed: " + error.message() + "\n", LogError);
Stop();
}
}
catch (std::exception& e)
{
stringstream ss;
ss << "SSLSocket::HandleRead: threw an error - " << e.what() << ".\n";
Log.LogString(ss.str(), LogError);
Stop();
}
}
void SSLSocket::Stop()
{
// This method calls the shutdown method on the socket in order to stop reads or writes that might be going on. If this is not done, then an exception will be thrown
// when it comes time to delete this object.
ReqAlive = false;
SetEvent(hEvent);
IOService->stop();
}
So, here are the key points:
When connecting to a server for the first time, a new instance of the SSLSocket class is created. The io_service object is static and created just once. It is used by all 6 instances of the SSLSocket class.
There are 2 threads that are used for everything having to do with socket communication across all 6 servers. One thread is for processing messages received from a server. The other thread is used for sending messages to a server.
This code uses SSL/TSL. If you are using straight TCP, then you can simply remove the 3 lines in SSLSocket::Connect method as well as the ssl #include line.
The technique used in HandleRead uses a double read method. The first read gets the number of bytes (since the protocol uses the first 4 bytes as the message length) and the second one obtains the total number of bytes in that message. This may not be the most efficient or even most desirable way to handle reading data off of the socket. But, it is the easiest and simplest to understand. You might consider using a different approach if your protocol is different and/or the message size is much larger and you have the ability to begin processing messages before the entire message has been received.
This code uses Boost 1.52.0 with Visual Studio 2008 for Windows.
There are no direct examples of the one-to-many client-server design included with the Asio examples. If your design is fixed at a maximum of 10 connections, using synchronous communication with a thread for each should be fine. However if you intend this to scale to much more than that, it is obvious to see the diminishing returns from creating a few hundred or thousand threads.
That said, using async_connect combined with async_read and async_write is not difficult to understand or implement. I've used this same concept to manage several thousand connections on the world's fastest supercomputer using only a handful of threads. The async TCP client example is probably the best one to study if you choose this route.
If you are looking for more than just examples, there are several open source projects using Asio that you might find useful.
Related
boost asio async reading and writing to socket using queue
I am working on a simple TCP server that reads and writes it's messages to thread safe queue. The application can then use those queue to safely read and write to the socket even from different threads. The problem I am facing is that I cannot async_read. My queue has the pop operation which returns the next element to be processed but it blocks if nothing is available. So once I call pop the async_read callback of course isn't fired anymore. Is there a way I can integrate such a queue into boost asio or do I have to completely rewrite? Below is a short example I made to show the problem I am having. Once a TCP connection is estabilished I create a new thread that will run the application under that tcp_connection. Afterwards I want to start async_read and async_write. I have been breaking my head on this for a couple of hours and I really don't know how to solve this. class tcp_connection : public std::enable_shared_from_this<tcp_connection> { public: static std::shared_ptr<tcp_connection> create(boost::asio::io_service &io_service) { return std::shared_ptr<tcp_connection>(new tcp_connection(io_service)); } boost::asio::ip::tcp::socket& get_socket() { return this->socket; } void app_start() { while(1) { // Pop is a blocking call. auto inbound_message = this->inbound_messages.pop(); std::cout << "Got message in app thread: " << inbound_message << ". Sending it back to client." << std::endl; this->outbound_messages.push(inbound_message); } } void start() { this->app_thread = std::thread(&tcp_connection::app_start, shared_from_this()); boost::asio::async_read_until(this->socket, this->input_stream, "\r\n", strand.wrap(boost::bind(&tcp_connection::handle_read, shared_from_this(), boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred))); // Start async writing here. The message to send are in the outbound_message queue. But a Pop operation blocks // empty() is also available to check whether the queue is empty. // So how can I async write without blocking the read. // block... auto message = this->outbound_messages.pop(); boost::asio::async_write(this->socket, boost::asio::buffer(message), strand.wrap(boost::bind(&tcp_connection::handle_write, shared_from_this(), boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred))); } void handle_read(const boost::system::error_code& e, size_t bytes_read) { std::cout << "handle_read called" << std::endl; if (e) { std::cout << "Error handle_read: " << e.message() << std::endl; return; } if (bytes_read != 0) { std::istream istream(&this->input_stream); std::string message; message.resize(bytes_read); istream.read(&message[0], bytes_read); std::cout << "Got message: " << message << std::endl; this->inbound_messages.push(message); } boost::asio::async_read_until(this->socket, this->input_stream, "\r\n", strand.wrap(boost::bind(&tcp_connection::handle_read, shared_from_this(), boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred))); } void handle_write(const boost::system::error_code& e, size_t /*bytes_transferred*/) { if (e) { std::cout << "Error handle_write: " << e.message() << std::endl; return; } // block... auto message = this->outbound_messages.pop(); boost::asio::async_write(this->socket, boost::asio::buffer(message), strand.wrap(boost::bind(&tcp_connection::handle_write, shared_from_this(), boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred))); } private: tcp_connection(boost::asio::io_service& io_service) : socket(io_service), strand(io_service) { } boost::asio::ip::tcp::socket socket; boost::asio::strand strand; boost::asio::streambuf input_stream; std::thread app_thread; concurrent_queue<std::string> inbound_messages; concurrent_queue<std::string> outbound_messages; }; class tcp_server { public: tcp_server(boost::asio::io_service& io_service) : acceptor(io_service, boost::asio::ip::tcp::endpoint(boost::asio::ip::tcp::v4(), 9001)) { start_accept(); } private: void start_accept() { std::shared_ptr<tcp_connection> new_connection = tcp_connection::create(acceptor.get_io_service()); acceptor.async_accept(new_connection->get_socket(), boost::bind(&tlcp_tcp_server::handle_accept, this, new_connection, boost::asio::placeholders::error)); } void handle_accept(std::shared_ptr<tcp_connection> new_connection, const boost::system::error_code& error) { if (!error) { new_connection->start(); } start_accept(); } boost::asio::ip::tcp::acceptor acceptor; };
It looks to me as if you want an async_pop method which takes an error message placeholder and callback handler. When you receive a message, check whether there is an outstanding handler and if so, pop the message, deregister the handler and call it. Similarly when registering the async_pop, if there is already a message waiting, pop the message and post a call to the handler without registering it. You might want to derive the async_pop class from a polymorphic base base of type pop_operation or similar.
Boost Asio pattern with GUI and worker thread
I would like to implement a Boost Asio pattern using a thread for GUI and a worker thread for some socket IO. The worker thread will use boost::asio::io_service to manage a socket client. All operations on sockets will be performed by the worker thread only. The GUI thread needs to send and receive messages from the worker thread. I can't exactly figure how to implement this pattern using Boost Asio. I've already implemented the socket communication in the standard Asio way (I call io_service.run() from the worker thread and I use async_read_some/async_send). I don't need strands because io_service.run() is called from the worker thread only. Now I'm trying to add the cross thread message queue. How I can I implement it? Should I run the io_service from the GUI thread too? Or should I just use strands with post to post messages from the GUI thread to the worker thread (without calling io_service.run() or io_service.poll_one() from the GUI thread), and use the operating system's GUI message loop to post messages from the worker thread to the GUI thread? If I need to call io_service.run() or io_service.poll_one() from the GUI thread too, do I need to use strands on the socket operations, since the io_service is shared between two threads? EDIT: to clarify my question, I would like to do whatever I can, to implement the message queue, using Boost Asio, relying on other libraries only if Boost Asio can't do the job.
Message passing is fairly generic. There are various ways to approach the problem, and the solution will likely be dependent on the desired behavioral details. For example, blocking or non-blocking, controlling memory allocation, context, etc. Boost.Lockfree provides thread-safe lock-free non-blocking queues for singe/multi consumer/producers. It tends to lend itself fairly nicely to event loops, where it is not ideal for the consumer to be blocked, waiting for the producer to signal a synchronization construct. boost::lockfree::queue<message_type> worker_message_queue; void send_worker_message(const message_type& message) { // Add message to worker message queue. worker_message_queue.push(message); // Add work to worker_io_service that will process the queue. worker_io_service.post(&process_message); } void process_message() { message_type message; // If the message was not retrieved, then return early. if (!worker_message_queue.pop(message)) return; ... } Alternatively, Boost.Asio's io_service can function as a queue. The message just needs to be bound to the specified handler. void send_worker_message(const message_type& message) { // Add work to worker_io_service that will process the message. worker_io_service.post(boost::bind(&process_message, message)); } void process_message(message_type& message) { ... } This comment suggest that the desire is more than message passing. It sounds as though the end goal is to allow one thread to cause another thread to invoke arbitrary functions. If this is the case, then consider: Using Boost.Signals2 for a managed signals and slots implementation. This allows arbitrary functions to register with a signal. Using Boost.Asio's io_service to setup signal emissions. If the GUI thread and worker thread each have their own io_service, then the worker thread can post a handler into the GUI thread's io_service that will emit a signal. In the GUI thread's main loop, it will poll the io_service, emit the signal, and cause slots to be invoked from within the GUI thread's context. Here is complete example where two threads pass a message (as an unsigned int) to one another, as well as causing arbitrary functions to be invoked within another thread. #include <iostream> #include <boost/asio.hpp> #include <boost/bind.hpp> #include <boost/signals2.hpp> #include <boost/thread.hpp> /// #brief io_service dedicated to gui. boost::asio::io_service gui_service; /// #brief io_service dedicated to worker. boost::asio::io_service worker_service; /// #brief work to keep gui_service from stopping prematurely. boost::optional<boost::asio::io_service::work> gui_work; /// #brief hello slot. void hello(int x) { std::cout << "hello with " << x << " from thread " << boost::this_thread::get_id() << std::endl; } /// #brief world slot. void world(int x) { std::cout << "world with " << x << " from thread " << boost::this_thread::get_id() << std::endl; } /// #brief Type for signals. typedef boost::signals2::signal<void (int)> signal_type; void emit_then_notify_gui(signal_type& signal, unsigned int x); /// #brief Emit signals then message worker. void emit_then_notify_worker(signal_type& signal, unsigned int x) { // Emit signal, causing registered slots to run within this thread. signal(x); // If x has been exhausted, then cause gui service to run out of work. if (!x) { gui_work = boost::none; } // Otherwise, post work into worker service. else { std::cout << "GUI thread: " << boost::this_thread::get_id() << " scheduling other thread to emit signals" << std::endl; worker_service.post(boost::bind( &emit_then_notify_gui, boost::ref(signal), --x)); } } /// #brief Emit signals then message worker. void emit_then_notify_gui(signal_type& signal, unsigned int x) { // Emit signal, causing registered slots to run within this thread. signal(x); // If x has been exhausted, then cause gui service to run out of work. if (!x) { gui_work = boost::none; } // Otherwise, post more work into gui. else { std::cout << "Worker thread: " << boost::this_thread::get_id() << " scheduling other thread to emit signals" << std::endl; gui_service.post(boost::bind( &emit_then_notify_worker, boost::ref(signal), --x)); } } void worker_main() { std::cout << "Worker thread: " << boost::this_thread::get_id() << std::endl; worker_service.run(); } int main() { signal_type signal; // Connect slots to signal. signal.connect(&hello); signal.connect(&world); boost::optional<boost::asio::io_service::work> worker_work( boost::ref(worker_service)); gui_work = boost::in_place(boost::ref(gui_service)); std::cout << "GUI thread: " << boost::this_thread::get_id() << std::endl; // Spawn off worker thread. boost::thread worker_thread(&worker_main); // Add work to worker. worker_service.post(boost::bind( &emit_then_notify_gui, boost::ref(signal), 3)); // Mocked up GUI main loop. while (!gui_service.stopped()) { // Do other GUI actions. // Perform message processing. gui_service.poll_one(); } // Cleanup. worker_work = boost::none; worker_thread.join(); } And its output: GUI thread: b7f2f6d0 Worker thread: b7f2eb90 hello with 3 from thread b7f2eb90 world with 3 from thread b7f2eb90 Worker thread: b7f2eb90 scheduling other thread to emit signals hello with 2 from thread b7f2f6d0 world with 2 from thread b7f2f6d0 GUI thread: b7f2f6d0 scheduling other thread to emit signals hello with 1 from thread b7f2eb90 world with 1 from thread b7f2eb90 Worker thread: b7f2eb90 scheduling other thread to emit signals hello with 0 from thread b7f2f6d0 world with 0 from thread b7f2f6d0
If you have only one worker, then it's rather easy. ASIO's handlers are executed by the thread(s) that are calling io_service.run(). In your case, that means that only one thread, the worker one, can execute callback handler. So you need not to worry about thread safety here. Your GUI thread, assuming that it has access to one's socket, can call boost::asio::async_write() without problem. The callback handler, however, will be executed in the worker thread. From my experience (admitedly limited), I used this pattern: The business logic thread (could be your GUI thread) can schedule a write to one of its client easily, by calling boost::asio::async_write(): the worker thread will take care of it. The worker thread start some boost::asio::async_read(), and could be building "business logic packet". What I mean here, is that it construct meaningfull message (could be a subclass of a custom class Packet or Event or w/e you what) from raw data. When the worker thread has enough data to build such a message, it does, and then enqueue it to a thread-safe queue that the GUI thread will be pulling. The GUI (or business logic) thread then process the message. Let me know if its not clear / if I can be of more help.
The way that I exchange messages between 2+ threads is to use a container like a queue and store them in there and then use an event to notify the worker thread to wake up and process them. Here is an example: void SSLSocket::SendToServer(const int bytesInMsg, Byte* pBuf) { // This method creates a msg object and saves it in the SendMsgQ object. // Message* pMsg = Message::GetMsg(this, bytesInMsg, pBuf); SendMsgQ.Push(pMsg); // Signal the send worker thread to wake up and send the msg to the server. SetEvent(hEvent); } In the header file: std::queue<Message*> SendMsgQueue; // Queue of msgs to send to the server. The above code is for Microsoft VC++. You might have to use a different class or methods if your development environment is different. But, the idea should be the same. Edit - More Complete Code Example #include "StdAfx.h" #include "SSLSocket.h" boost::shared_ptr< boost::asio::io_service > SSLSocket::IOService; bool SSLSocket::LobbySocketOpen = false; SSLSocket* SSLSocket::pSSLLobby = 0; int SSLSocket::StaticInit = 0; Callback SSLSocket::CallbackFunction; BufferManagement SSLSocket::BufMang; volatile bool SSLSocket::ReqAlive = true; Logger SSLSocket::Log; HANDLE SSLSocket::hEvent; bool SSLSocket::DisplayInHex; ConcurrentMsgQueue SSLSocket::SendMsgQ; bool SSLSocket::RcvThreadCreated = 0; BufferManagement* Message::pBufMang; bool SSLSocket::ShuttingDown = false; std::vector<SSLSocket *> SocketList; SSLSocket::SSLSocket(const bool logToFile, const bool logToConsole, const bool displayInHex, const LogLevel levelOfLog, const string& logFileName, const int bufMangLen) : pSocket(0) { // SSLSocket Constructor. // If the static members have not been intialized yet, then initialize them. LockCode = new Lock(); if (!StaticInit) { SocketList.push_back(this); DisplayInHex = displayInHex; BufMang.Init(bufMangLen); Message::SetBufMang(&BufMang); // This constructor enables logging according to the vars passed in. Log.Init(logToFile, logToConsole, levelOfLog, logFileName); StaticInit = 1; hEvent = CreateEvent(NULL, false, false, NULL); // Define the ASIO IO service object. // IOService = new boost::shared_ptr<boost::asio::io_service>(new boost::asio::io_service); boost::shared_ptr<boost::asio::io_service> IOServ(new boost::asio::io_service); IOService = IOServ; pSSLLobby = this; } } SSLSocket::~SSLSocket(void) { if (pSocket) delete pSocket; if (--StaticInit == 0) CloseHandle(hEvent); } void SSLSocket::Connect(SSLSocket* psSLS, const string& serverPath, string& port) { // Connects to the server. // serverPath - specifies the path to the server. Can be either an ip address or url. // port - port server is listening on. // try { LockCode->Acquire(); // Single thread the code. // If the user has tried to connect before, then make sure everything is clean before trying to do so again. if (pSocket) { delete pSocket; pSocket = 0; } // If serverPath is a URL, then resolve the address. if ((serverPath[0] < '0') || (serverPath[0] > '9')) // Assumes that the first char of the server path is not a number when resolving to an ip addr. { // Create the resolver and query objects to resolve the host name in serverPath to an ip address. boost::asio::ip::tcp::resolver resolver(*IOService); boost::asio::ip::tcp::resolver::query query(serverPath, port); boost::asio::ip::tcp::resolver::iterator EndpointIterator = resolver.resolve(query); // Set up an SSL context. boost::asio::ssl::context ctx(*IOService, boost::asio::ssl::context::tlsv1_client); // Specify to not verify the server certificiate right now. ctx.set_verify_mode(boost::asio::ssl::context::verify_none); // Init the socket object used to initially communicate with the server. pSocket = new boost::asio::ssl::stream<boost::asio::ip::tcp::socket>(*IOService, ctx); // // The thread we are on now, is most likely the user interface thread. Create a thread to handle all incoming socket work messages. // Only one thread is created to handle the socket I/O reading and another thread is created to handle writing. if (!RcvThreadCreated) { WorkerThreads.create_thread(boost::bind(&SSLSocket::RcvWorkerThread, this)); RcvThreadCreated = true; WorkerThreads.create_thread(boost::bind(&SSLSocket::SendWorkerThread, this)); } // Try to connect to the server. Note - add timeout logic at some point. boost::asio::async_connect(pSocket->lowest_layer(), EndpointIterator, boost::bind(&SSLSocket::HandleConnect, this, boost::asio::placeholders::error)); } else { // serverPath is an ip address, so try to connect using that. // stringstream ss1; boost::system::error_code EC; ss1 << "SSLSocket::Connect: Preparing to connect to game server " << serverPath << " : " << port << ".\n"; Log.LogString(ss1.str(), LogInfo); // Create an endpoint with the specified ip address. const boost::asio::ip::address IP(boost::asio::ip::address::from_string(serverPath)); int iport = atoi(port.c_str()); const boost::asio::ip::tcp::endpoint EP(IP, iport); // Set up an SSL context. boost::asio::ssl::context ctx(*IOService, boost::asio::ssl::context::tlsv1_client); // Specify to not verify the server certificiate right now. ctx.set_verify_mode(boost::asio::ssl::context::verify_none); // Init the socket object used to initially communicate with the server. pSocket = new boost::asio::ssl::stream<boost::asio::ip::tcp::socket>(*IOService, ctx); // // Try to connect to the server. Note - add timeout logic at some point. pSocket->next_layer().connect(EP, EC); if (EC) { // Log an error. This worker thread should exit gracefully after this. stringstream ss; ss << "SSLSocket::Connect: connect failed to " << sClientIp << " : " << uiClientPort << ". Error: " << EC.message() + ".\n"; Log.LogString(ss.str(), LogError); } stringstream ss; ss << "SSLSocket::Connect: Calling HandleConnect for game server " << serverPath << " : " << port << ".\n"; Log.LogString(ss.str(), LogInfo); HandleConnect(EC); } } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::Connect: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } LockCode->Release(); } void SSLSocket::SendToServer(const int bytesInMsg, Byte* pBuf) { // This method creates a msg object and saves it in the SendMsgQ object. // sends the number of bytes specified by bytesInMsg in pBuf to the server. // Message* pMsg = Message::GetMsg(this, bytesInMsg, pBuf); SendMsgQ.Push(pMsg); // Signal the send worker thread to wake up and send the msg to the server. SetEvent(hEvent); } void SSLSocket::SendWorkerThread(SSLSocket* psSLS) { // This thread method gets called to process the messages to be sent to the server. // // Since this has to be a static method, call a method on the class to handle server requests. psSLS->ProcessSendRequests(); } void SSLSocket::ProcessSendRequests() { // This method handles sending msgs to the server. // std::stringstream ss; DWORD WaitResult; Log.LogString("SSLSocket::ProcessSendRequests: Worker thread " + Logger::NumberToString(boost::this_thread::get_id()) + " started.\n", LogInfo); // Loop until the user quits, or an error of some sort is thrown. try { do { // If there are one or more msgs that need to be sent to a server, then send them out. if (SendMsgQ.Count() > 0) { Message* pMsg = SendMsgQ.Front(); SSLSocket* pSSL = pMsg->pSSL; SendMsgQ.Pop(); const Byte* pBuf = pMsg->pBuf; const int BytesInMsg = pMsg->BytesInMsg; boost::system::error_code Error; LockCode->Acquire(); // Single thread the code. try { boost::asio::async_write(*pSSL->pSocket, boost::asio::buffer(pBuf, BytesInMsg), boost::bind(&SSLSocket::HandleWrite, this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred)); } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::ProcessSendRequests: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); } ss.str(std::string()); ss << "SSLSocket::ProcessSendRequests: # bytes sent = " << BytesInMsg << "\n"; Log.LogString(ss.str(), LogDebug2); Log.LogBuf(pBuf, BytesInMsg, DisplayInHex, LogDebug3); LockCode->Release(); } else { // Nothing to send, so go into a wait state. WaitResult = WaitForSingleObject(hEvent, INFINITE); if (WaitResult != 0L) { Log.LogString("SSLSocket::ProcessSendRequests: WaitForSingleObject event error. Code = " + Logger::NumberToString(GetLastError()) + ". \n", LogError); } } } while (ReqAlive); Log.LogString("SSLSocket::ProcessSendRequests: Worker thread " + Logger::NumberToString(boost::this_thread::get_id()) + " done.\n", LogInfo); } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::ProcessSendRequests: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } void SSLSocket::HandleWrite(const boost::system::error_code& error, size_t bytesTransferred) { // This method is called after a msg has been written out to the socket. Nothing to do really since reading is handled by the HandleRead method. // std::stringstream ss; try { if (error) { ss << "SSLSocket::HandleWrite: failed - " << error.message() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::HandleHandshake: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } void SSLSocket::RcvWorkerThread(SSLSocket* psSLS) { // This is the method that gets called when the receive thread is created by this class. // This thread method focuses on processing messages received from the server. // // Since this has to be a static method, call an instance method on the class to handle server requests. psSLS->InitAsynchIO(); } void SSLSocket::InitAsynchIO() { // This method is responsible for initiating asynch i/o. boost::system::error_code Err; string s; stringstream ss; // try { ss << "SSLSocket::InitAsynchIO: Worker thread - " << Logger::NumberToString(boost::this_thread::get_id()) << " started.\n"; Log.LogString(ss.str(), LogInfo); // Enable the handlers for asynch i/o. The thread will hang here until the stop method has been called or an error occurs. // Add a work object so the thread will be dedicated to handling asynch i/o. boost::asio::io_service::work work(*IOService); IOService->run(); Log.LogString("SSLSocket::InitAsynchIO: receive worker thread done.\n", LogInfo); } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::InitAsynchIO: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } void SSLSocket::HandleConnect(const boost::system::error_code& error) { // This method is called asynchronously when the server has responded to the connect request. std::stringstream ss; try { if (!error) { LockCode->Acquire(); // Single thread the code. pSocket->async_handshake(boost::asio::ssl::stream_base::client, boost::bind(&SSLSocket::HandleHandshake, this, boost::asio::placeholders::error)); LockCode->Release(); ss << "SSLSocket::HandleConnect: From worker thread " << Logger::NumberToString(boost::this_thread::get_id()) << ".\n"; Log.LogString(ss.str(), LogInfo); } else { // Log an error. This worker thread should exit gracefully after this. ss << "SSLSocket::HandleConnect: connect failed. Error: " << error.message() + ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::InitAsynchIO: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } void SSLSocket::HandleHandshake(const boost::system::error_code& error) { // This method is called asynchronously when the server has responded to the handshake request. std::stringstream ss; try { if (!error) { // Try to send the first message that the server is expecting. This msg tells the server we want to connect. // unsigned char Msg[5] = {0x17, 0x00, 0x00, 0x00, 0x06}; boost::system::error_code Err; // if (pSSLLobby == this) LobbySocketOpen = true; sClientIp = pSocket->lowest_layer().remote_endpoint().address().to_string(); uiClientPort = pSocket->lowest_layer().remote_endpoint().port(); ReqAlive = true; LockCode->Acquire(); // Single thread the code. int Count = boost::asio::write(*pSocket, boost::asio::buffer(Msg), boost::asio::transfer_exactly(5), Err); if (Err) { ss << "SSLSocket::HandleHandshake: write failed - " << error.message() << ".\n"; Log.LogString(ss.str(), LogInfo); } HandleFirstWrite(Err, Count); LockCode->Release(); ss.str(""); ss << "SSLSocket::HandleHandshake: From worker thread " << boost::this_thread::get_id() << ".\n"; } else { ss << "SSLSocket::HandleHandshake: failed - " << error.message() << ".\n"; IOService->stop(); } Log.LogString(ss.str(), LogInfo); } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::HandleHandshake: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } void SSLSocket::HandleFirstWrite(const boost::system::error_code& error, size_t bytesTransferred) { // This method is called after a msg has been written out to the socket. This method is only called from HandleHandShake. std::stringstream ss; try { if (!error) { // Notify the UI that we are now connected. Create a 6 byte msg for this. pDataBuf = BufMang.GetPtr(6); BYTE* p = pDataBuf; // Create msg type 500 *p = 244; *++p = 1; CallbackFunction(this, 2, (void*)pDataBuf); // Get the 1st 4 bytes of the next msg, which is always the length of the msg. pDataBuf = BufMang.GetPtr(MsgLenBytes); try { boost::asio::async_read(*pSocket, boost::asio::buffer(pDataBuf, MsgLenBytes), boost::bind(&SSLSocket::HandleRead, this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred)); } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::HandleFirstWrite: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } else { ss << "SSLSocket::HandleFirstWrite: failed - " << error.message() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::HandleFirstWrite: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } void SSLSocket::HandleRead(const boost::system::error_code& error, size_t bytesTransferred) { // This method is called to process an incoming message. // std::stringstream ss; int ByteCount; try { // ss << "SSLSocket::HandleRead: From worker thread " << boost::this_thread::get_id() << ".\n"; // Log.LogString(ss.str(), LogInfo); // Set to exit this thread if the user is done. if (!ReqAlive) { // IOService->stop(); return; } if (!error) { // Get the number of bytes in the message. if (bytesTransferred == 4) { ByteCount = BytesToInt(pDataBuf); } else { // Call the C# callback method that will handle the message. ss << "SSLSocket::HandleRead: From worker thread " << boost::this_thread::get_id() << "; # bytes transferred = " << bytesTransferred << ".\n"; Log.LogString(ss.str(), LogDebug2); if (bytesTransferred > 0) { Log.LogBuf(pDataBuf, (int)bytesTransferred, true, LogDebug3); Log.LogString("SSLSocket::HandleRead: sending msg to the C# client.\n\n", LogDebug2); CallbackFunction(this, bytesTransferred, (void*)pDataBuf); } else { // # of bytes transferred = 0. Don't do anything. bytesTransferred = 0; // For debugging. } // Prepare to read in the next message length. ByteCount = MsgLenBytes; } pDataBuf = BufMang.GetPtr(ByteCount); boost::system::error_code Err; try { boost::asio::async_read(*pSocket, boost::asio::buffer(pDataBuf, ByteCount), boost::bind(&SSLSocket::HandleRead, this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred)); } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::HandleRead: threw this error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); } } else { Log.LogString("SSLSocket::HandleRead failed: " + error.message() + "\n", LogError); Stop(); } } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::HandleRead: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } void SSLSocket::Stop() { // This method calls the shutdown method on the socket in order to stop reads or writes that might be going on. If this is not done, then an exception will be thrown // when it comes time to delete this object. // boost::system::error_code EC; try { // This method can be called from the handler as well. So once the ShuttingDown flag is set, don't go throught the same code again. if (ShuttingDown) return; LockCode->Acquire(); // Single thread the code. if (!ShuttingDown) { ShuttingDown = true; pSocket->next_layer().cancel(); pSocket->shutdown(EC); if (EC) { stringstream ss; ss << "SSLSocket::Stop: socket shutdown error - " << EC.message() << ".\n"; } else { pSocket->next_layer().close(); } delete pSocket; pSocket = 0; ReqAlive = false; SetEvent(hEvent); IOService->stop(); LobbySocketOpen = false; WorkerThreads.join_all(); } LockCode->Release(); delete LockCode; LockCode = 0; } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::Stop: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } So, in answer to your question about whether you have to use a queue or not. In your comment to Xaqq, you said "I need to exchange messages between the two threads." So using a container like a queue is how messages can be passed to another thread for processing. If you don't like the STL containers, Boost does have some. As far as I know, there is no Boost ASIO internal container that can be accessed. Storing and passing the messages around is something you have to do in your code. One last note about the call to io_service::run. It will only block while there is work to do. See this link. In my example code above, a work item is added to the io_service object before the run method is called, so it will block indefinitely - which is what I want. If I really wanted only one thread, then what I might do is set up the worker thread to call the run method with a work object so that it would block indefinitely. This would handle all asynchronous I/O coming from and going to the server. Inside the class, I would write an interface method or two so that the gui can send data to the server. These methods could use the async write .vs. the synch write method and thus would return right away - so your gui won't block long. You would need to write a HandleWrite method. My code does not do much with it - just logs an error if one occurs.
Boost access violation while deleting ssl::stream<ip::tcp::socket>* sslSocket
While deleting an SSLSocket after a successful connection is made and used I am getting an access violation Unhandled exception at 0x770f32d0 in Application_client_example.exe: 0xC0000005: Access violation reading location 0x00000dd3c0c76c48. The access violation is coming from this part of the boost code: engine::~engine() { if (SSL_get_app_data(ssl_)) { delete static_cast<verify_callback_base*>(SSL_get_app_data(ssl_)); SSL_set_app_data(ssl_, 0); } ::BIO_free(ext_bio_); ::SSL_free(ssl_); } This code worked in boost version 1.47. The only changes I have made is I updated the boost libraries to the current version 1.53 and built a 64 bit version of the library and exe. Here is the SSL connection that is created and deleted: // Connect SSLSocket* socket = new SSLSocket(); if ((errorCode = socket->connect((char*)server.c_str(), (char*)port.c_str())) != 0) { Logger::log(log4cpp::Priority::FATAL, "Secure Socket Error"); return errorCode; } delete socket Here is the SSLSocket destructor SSLSocket::~SSLSocket(void) { try { sslSocket->shutdown(); delete sslSocket; } catch (std::exception& e) { std::string exception(e.what()); Logger::log(log4cpp::Priority::FATAL, "[SSLSocket] Error deleting sslSocket. Exception: " + exception); } } Here is the def for SSLSocket. SSLSocket is essentially just a wrapper class for the ssl socket: #ifndef __SSLSOCKET__ #define __SSLSOCKET__ #include <boost/bind.hpp> #include <boost/asio.hpp> #include <boost/array.hpp> #include <boost/asio/ssl.hpp> #include <string> #include "Logger.h" #include "Config.h" using namespace boost::asio; class SSLSocket { private: io_service io_service; ssl::stream<ip::tcp::socket>* sslSocket; public: SSLSocket(void); ~SSLSocket(void); DWORD connect(char* remoteServer, char* remotePort); DWORD sendString(std::string data); std::string receiveString(void); }; #endif
Here is my code that I use to shut down the socket connection in my SSLSockets class, which is also a wrapper around ASIO for an SSL connection. I am using Boost ASIO version 1.52 for Windows with 32 bit libraries. I too used to receive an exception when shutting down the socket until I discovered how to do it properly: void SSLSocket::Stop() { // This method calls the shutdown method on the socket in order to stop reads or writes that might be going on. If this is not done, then an exception will be thrown // when it comes time to delete this object. // boost::system::error_code EC; try { // This method can be called from the handler as well. So once the ShuttingDown flag is set, don't go throught the same code again. if (ShuttingDown) return; LockCode->Acquire(); // Single thread the code. // Only do this once. if (!ShuttingDown) { ShuttingDown = true; pSocket->next_layer().cancel(); pSocket->shutdown(EC); // Note that EC will usually have an error condition, but it does not seem to be a problem. delete pSocket; pSocket = 0; ReqAlive = false; SetEvent(hEvent); IOService->stop(); LobbySocketOpen = false; // Wait until the 2 threads have exited before returning. WorkerThreads.join_all(); } LockCode->Release(); delete LockCode; LockCode = 0; } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::Stop: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } In answer to question about the Lock variable Lock is a class that encapsulates a critical section (specific to Microsoft) so that the code can be single threaded. Here is the definition for it: class Lock { public: Lock() { ::InitializeCriticalSection(&CS); } ~Lock() { ::DeleteCriticalSection(&CS); } void Acquire() { ::EnterCriticalSection(&CS); } void Release() { ::LeaveCriticalSection(&CS); } private: Lock(const Lock&); Lock& operator=(const Lock&); CRITICAL_SECTION CS; }; Socket Creation Code This is the code I use to create the SSL context object and SSL socket object: void SSLSocket::Connect(SSLSocket* psSLS, const string& serverPath, string& port) { // Connects to the server. // serverPath - specifies the path to the server. Can be either an ip address or url. // port - port server is listening on. // try { LockCode->Acquire(); // Single thread the code. // If the user has tried to connect before, then make sure everything is clean before trying to do so again. if (pSocket) { delete pSocket; pSocket = 0; } // If serverPath is a URL, then resolve the address. if ((serverPath[0] < '0') || (serverPath[0] > '9')) // Assumes that the first char of the server path is not a number when resolving to an ip addr. { // Create the resolver and query objects to resolve the host name in serverPath to an ip address. boost::asio::ip::tcp::resolver resolver(*IOService); boost::asio::ip::tcp::resolver::query query(serverPath, port); boost::asio::ip::tcp::resolver::iterator EndpointIterator = resolver.resolve(query); // Set up an SSL context. boost::asio::ssl::context ctx(*IOService, boost::asio::ssl::context::tlsv1_client); // Specify to not verify the server certificiate right now. ctx.set_verify_mode(boost::asio::ssl::context::verify_none); // Init the socket object used to initially communicate with the server. pSocket = new boost::asio::ssl::stream<boost::asio::ip::tcp::socket>(*IOService, ctx); // // The thread we are on now, is most likely the user interface thread. Create a thread to handle all incoming socket work messages. // Only one thread is created to handle the socket I/O reading and another thread is created to handle writing. if (!RcvThreadCreated) { WorkerThreads.create_thread(boost::bind(&SSLSocket::RcvWorkerThread, this)); RcvThreadCreated = true; WorkerThreads.create_thread(boost::bind(&SSLSocket::SendWorkerThread, this)); } // Try to connect to the server. Note - add timeout logic at some point. boost::asio::async_connect(pSocket->lowest_layer(), EndpointIterator, boost::bind(&SSLSocket::HandleConnect, this, boost::asio::placeholders::error)); } else { // serverPath is an ip address, so try to connect using that. // stringstream ss1; boost::system::error_code EC; ss1 << "SSLSocket::Connect: Preparing to connect to game server " << serverPath << " : " << port << ".\n"; Log.LogString(ss1.str(), LogInfo); // Create an endpoint with the specified ip address. const boost::asio::ip::address IP(boost::asio::ip::address::from_string(serverPath)); int iport = atoi(port.c_str()); const boost::asio::ip::tcp::endpoint EP(IP, iport); // Set up an SSL context. boost::asio::ssl::context ctx(*IOService, boost::asio::ssl::context::tlsv1_client); // Specify to not verify the server certificiate right now. ctx.set_verify_mode(boost::asio::ssl::context::verify_none); // Init the socket object used to initially communicate with the server. pSocket = new boost::asio::ssl::stream<boost::asio::ip::tcp::socket>(*IOService, ctx); // // Try to connect to the server. Note - add timeout logic at some point. pSocket->next_layer().connect(EP, EC); if (EC) { // Log an error. This worker thread should exit gracefully after this. stringstream ss; ss << "SSLSocket::Connect: connect failed to " << sClientIp << " : " << uiClientPort << ". Error: " << EC.message() + ".\n"; Log.LogString(ss.str(), LogError); } stringstream ss; ss << "SSLSocket::Connect: Calling HandleConnect for game server " << serverPath << " : " << port << ".\n"; Log.LogString(ss.str(), LogInfo); HandleConnect(EC); } } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::Connect: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } LockCode->Release(); }
Ok so the fix for this after searching around and not really finding anything related was that I was building all the boost libraries as MTd. I was assuming that I would use the MTd build for OpenSSL libraries but come to find out if you use the non MT Openssl libraries it works just fine.
TCP/IP client using Boost::asio
I'm trying to make a TCP/IP client using boost library. This is how I designed my program ->read thread to read from the server ->write thread to send commands ->a function that parses the read data from the server int main() { TCP_IP_Connection router; router.Create_Socket(); boost::thread_group t; t.create_thread(boost::bind(&TCP_IP_Connection::get_status,&router,'i')); t.create_thread(boost::bind(&TCP_IP_Connection::readTCP,&router)); std::string reply="\nend of main()"; std::cout<<reply; t.join_all(); return 0; } void TCP_IP_Connection::Create_Socket() { tcp::resolver resolver(_io);//resolve into TCP endpoint tcp::resolver::query query(routerip,rport); tcp::resolver::iterator endpoint_iterator = resolver.resolve(query); //list of endpoints tcp::resolver::iterator end; boost::asio::streambuf b; _socket = new tcp::socket(_io); //create socket boost::system::error_code error= boost::asio::error::host_not_found; try { while (error && endpoint_iterator != end) //if error go to next endpoint { _socket->close(); _socket->connect(*endpoint_iterator++, error); } if(error) throw boost::system::system_error(error); //else the router is connected } catch (std::exception& e) { std::cerr << e.what() << std::endl; } } void TCP_IP_Connection::get_status(char p) { try { if(p=='i') _socket->send(boost::asio::buffer("llist\n\n")); //sending command for input command else _socket->send(boost::asio::buffer(" sspo l1\n\n")); //sending signal presence for output command } catch (std::exception& e) { std::cerr << e.what() << std::endl; } } void TCP_IP_Connection::readTCP() { this->len=0; boost::system::error_code error= boost::asio::error::host_not_found; try { //loop reading all values from router while(1) { //wait for reply?? _socket->async_read_some(boost::asio::buffer(this- >reply,sizeof(this>reply)),boost::bind(&TCP_IP_Connection::dataProcess,this, boost::asio::placeholders::error,boost::asio::placeholders::bytes_transferred)); _io.run(); if(error==boost::asio::error::eof) //connection closed by router std::cout<<"connection closed by router"; } } catch (std::exception& e) { std::cerr << e.what() << std::endl; } } void TCP_IP_Connection::dataProcess(const boost::system::error_code &er,size_t l) { if(!er) { if(l>0) { for(int i=0;i<l;i++) { this->data[i]=this->reply[i]; //if(data[i]="\n") std::cout<<this->data[i]; } } } } When I run the code all I get is the response from the server that says the client is connected and not the response of the command I send. But when I try debugging I get full output as I need. Am I doing anything wrong in the threading, or in the TCP read buffer.
Your code is creating 2 threads. The first thread created has a thread function called get_status. In get_status, there is no looping so it only executes the code once. It appears to be sending the string "llist\n\n" to the server and this is done synchronously. After that, it does not send anything else. So, are you expecting the server to send other data after the first command is sent? The code in the first thread may or may not execute completely before the code in the second thread executes. The second thread is created and this thread appears to be responsible for processing information coming off of the socket. There is an infinite loop of while(1), but no logic to exit the loop so it will run forever unless an exception is thrown. I believe that the async_read_some method will not cause any data to be transferred until the buffer is full. The size of the buffer is specified by the size of reply. This may be your problem since the dataProcess method won't get called until all of the data specified by the length of reply has been received. In many protocols, the first 4 bytes specifies the length of the message. So, if you are dealing with variable length messages, then your code will have to take this into account. One other item worth mentioning is that the looping code in readTCP to call _io.Run is not really necessary. You can add a work object to your io_service object in order for it to run continuously. For example: void SSLSocket::InitAsynchIO() { // This method is responsible for initiating asynch i/o. boost::system::error_code Err; string s; stringstream ss; // try { ss << "SSLSocket::InitAsynchIO: Worker thread - " << Logger::NumberToString(boost::this_thread::get_id()) << " started.\n"; Log.LogString(ss.str(), LogInfo); // Enable the handlers for asynch i/o. The thread will hang here until the stop method has been called or an error occurs. // Add a work object so the thread will be dedicated to handling asynch i/o. boost::asio::io_service::work work(*IOService); IOService->run(); Log.LogString("SSLSocket::InitAsynchIO: receive worker thread done.\n", LogInfo); } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::InitAsynchIO: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } It is ok to have your first thread do your first async read. Your read handler can be set up to call itself in order to handle the next message. For example: void SSLSocket::HandleRead(const boost::system::error_code& error, size_t bytesTransferred) { // This method is called to process an incomming message. // std::stringstream ss; int ByteCount; try { ss << "SSLSocket::HandleRead: From worker thread " << boost::this_thread::get_id() << ".\n"; Log.LogString(ss.str(), LogInfo); // Set to exit this thread if the user is done. if (!ReqAlive) { // IOService->stop(); return; } if (!error) { // Get the number of bytes in the message. if (bytesTransferred == 4) { ByteCount = BytesToInt(pDataBuf); } else { // Call the C# callback method that will handle the message. ss << "SSLSocket::HandleRead: From worker thread " << boost::this_thread::get_id() << "; # bytes transferred = " << bytesTransferred << ".\n"; Log.LogString(ss.str(), LogDebug2); Log.LogBuf(pDataBuf, (int)bytesTransferred, true, LogDebug3); Log.LogString("SSLSocket::HandleRead: sending msg to the C# client.\n\n", LogDebug2); CallbackFunction(this, bytesTransferred, (void*)pDataBuf); // Prepare to read in the next message length. ByteCount = MsgLenBytes; } pDataBuf = BufMang.GetPtr(ByteCount); boost::system::error_code Err; // boost::asio::async_read(pSocket, boost::asio::buffer(pDataBuf, ByteCount), boost::bind(&SSLSocket::HandleRead, // this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred)); Locking CodeLock(SocketLock); // Single thread the code. boost::asio::async_read(*pSocket, boost::asio::buffer(pDataBuf, ByteCount), boost::bind(&SSLSocket::HandleRead, this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred)); // boost::asio::read(pSocket, boost::asio::buffer(reply_), boost::asio::transfer_exactly(ByteCount), Err); } else { Log.LogString("SSLSocket::HandleRead failed: " + error.message() + "\n", LogError); Stop(); } } catch (std::exception& e) { stringstream ss; ss << "SSLSocket::HandleRead: threw an error - " << e.what() << ".\n"; Log.LogString(ss.str(), LogError); Stop(); } } If none of the above is helpful, then put in some debug code that logs all of the calls to a log file so that you can see what is going on. You might also want to consider downloading Wire Shark in order to see what data is going out and coming in.
Boost async_read
I'm trying to create a TCP server where the Start() method blocks until a connection is accepted, and then begins a series of asynchronous reads. I have the following code, and when I connect using telnet I get this output: Waiting for a new connection Connection accepted terminate called throwing an exceptionAbort trap: 6 Here is the code: void SocketReadThread::Start() { bzero(m_headerBuffer, HEADER_LEN); m_running = true; asio::io_service ios; asio::ip::tcp::acceptor acp (ios, boost::asio::ip::tcp::endpoint(boost::asio::ip::tcp::v4(), GUI_PORT)); asio::ip::tcp::socket sock(ios); std::cout << "Waiting for a new connection" << std::endl; acp.accept(sock); std::cout << "Connection accepted" << std::endl; asio::async_read(sock, asio::buffer(m_headerBuffer, HEADER_LEN), boost::bind(&SocketReadThread::handleReadHeader, shared_from_this(), asio::placeholders::error)); ios.run(); } void SocketReadThread::handleReadHeader(const system::error_code& error) { std::cout << "Read two bytes!" << std::endl; }
You should wrap your main() function in try {...} catch (std::exception& e) { cout << e.what(); } block.
You're probably doing something scary (and awesome) to the stack by declaring your ReadHandler incorrectly. Even if you ignore some parameters, the signature must be: void handler ( const boost::system::error_code& error, // Result of operation. std::size_t bytes_transferred // Number of bytes copied into the // buffers. If an error occurred, // this will be the number of // bytes successfully transferred // prior to the error. );