ZMQ C++ Event Loop Within Class - c++

My overall goal in using ZMQ is to avoid having to get into the weeds of asynchronous message passing; and ZMQ seemed like a portable and practical solution. Most of the ZeroMQ docs, however, like this, and many of the other zmq examples I have Googled upon are based on the helloworld.c format. That is, they are all simple procedural code inside int main(){}.
My problem is that I want to "embed" a zmq "listener" inside a c++ singleton-like class. I want to "listen" for messages and then process them. I'm planning on using zmq's PUSH -> PULL sockets, on the off chance that matters. What I cannot figure out how to do is to have in internal "event loop".
class foomgr {
public:
static foomgr& get_foomgr();
// ...
private:
foomgr();
foomgr(const &foomgr);
// ...
listener_() {
// EVENT LOOP HERE
// RECV and PROCESS ZMQ MSGS
// while(true) DOES NOT WORK HERE
}
// ...
zmq::context_t zmqcntx_;
zmq::socket_t zmqsock_;
const int zmqsock_linger_ = 1000;
// ....
}
I obviously cannot use the while(true) construct in listener, since wherever I call it from will block. Since one of the advantages of using ZMQ is that I do not have to manage "listener" threads myself, it seems silly to have to figure out how create my own thread to wrap listener_ in. I'm lost for solutions.
Note: I'm a c++ newb, so what might be obvious to most is not to me. Also, I'm trying to use generic "words", not library or language specific to avoid confusion. The code is built with -std=c++11, so those
constructs are fine.

The ZMQ C++ library does not implement a listener pattern for message polling. It leaves that task up to you to wrap in your own classes. It does support a non-blocking mode of polling for new messages, however.
So using the right code you can wrap it up in a small loop in a non-blocking fashion.
See this Polling Example here on GitHub written in C++. Note that its polling from 2 sockets, so you'll need to modify it a little to remove the extra code.
The important part that you'll need to wrap inside your own observer implementation is below:
zmq::message_t message;
zmq::poll (&items [0], 2, -1);
if (items [0].revents & ZMQ_POLLIN) {
receiver.recv(&message);
// Process task
}

Zmq is not thread safe by design (versions up to now). In fact, Zmq stresses:
Do not use or close sockets except in the thread that created them.
PERIOD.
Callbacks shouldn't be used because the thread calling the callback, will be for sure different from the thread that created the socket, which is forbidden.
Maybe, you will find useful zmqHelper, a small library (only two classes and a few functions), to make it easier using Zmq in C++ and to enforce (it is guaranteed) that threads can't share sockets.
In the example sections, you will find how to do the most frequent tasks.
Hope it helps.
Code snippet: polling using zmqHelper in a ROUTER-DEALER broker.
zmq::context_t theContext {1}; // 1 thread in the socket
SocketAdaptor< ZMQ_ROUTER > frontend_ROUTER {theContext};
SocketAdaptor< ZMQ_DEALER > backend_DEALER {theContext};
frontend_ROUTER.bind ("tcp://*:8000");
backend_DEALER.bind ("tcp://*:8001");
while (true) {
std::vector<std::string> lines;
//
// wait (blocking poll) for data in any socket
//
std::vector< zmqHelper::ZmqSocketType * > list
= { frontend_ROUTER.getZmqSocket(), backend_DEALER.getZmqSocket() };
zmqHelper::ZmqSocketType * from = zmqHelper::waitForDataInSockets ( list );
//
// there is data, where is it from?
//
if ( from == frontend_ROUTER.getZmqSocket() ) {
// from frontend, read ...
frontend_ROUTER.receiveText (lines);
// ... and resend
backend_DEALER.sendText( lines );
}
else if ( from == backend_DEALER.getZmqSocket() ) {
// from backend, read ...
backend_DEALER.receiveText (lines);
// ... and resend
frontend_ROUTER.sendText( lines );
}
else if ( from == nullptr ) {
std::cerr << "Error in poll ?\n";
}
} // while (true)

Related

Auctions in Boost ASIO

I'm implementing an auctioning system in C++ with Boost.Asio. There is a single centralized auctioneer (the server) and some connecting bidders (the clients). I am implementing this in an asynchronous fashion, and I have implemented the basic communication between the bidder and auctioneer (register, ping, get client list). The skeletal code for the auctioneer would look like follows:
class talkToBidder : public boost::enable_shared_from_this<talkToBidder>
{
// Code for sending and receiving messages, which works fine
};
void on_round_end()
{
// Choose the best bid and message the winner
if (!itemList.empty())
timer_reset();
}
void timer_reset()
{
// Send the item information to the bidders
// When the round ends, call on_round_end()
auction_timer.expires_from_now(boost::posix_time::millisec(ROUND_TIME));
auction_timer.async_wait(boost::bind(on_round_end));
}
void handle_accept(...)
{
// Create new bidder...
acceptor.async_accept(bidder->sock(),boost::bind(handle_accept,bidder,_1));
}
int main()
{
// Create new bidder and handle accepting it
talkToBidder::ptr bidder = talkToBidder::new_();
acceptor.async_accept(bidder->sock(),boost::bind(handle_accept,bidder,_1));
service.run();
}
My issue is, I need to wait for at least one bidder to connect before I can start the auction, so I cannot simply call timer_reset() before I use service.run(). What is the Boost.Asio way to go about doing this?
In asynchronous protocol design, it helps to draw Message Sequence Diagrams. Do include your timers.
The code now becomes trivial. You start your timer when the message arrives that should start your timer. Yes, this is shifting the problem a bit forwards. The real point here is that it's not a Boost Asio coding problem. In your case, that particular message appears to be the login of the first bidder, implemented as a TCP connect (SYN/ACK) which maps to handle_accept in your code.

Am I paranoid while using boost:asio?

I am write an app using boost:asio.
I have a single io_serice::run() thread, and many worker threads. All the worker threads may send msg at any time.
Here is how I implement the send_msg().
// Note: send_msg() could be called from any thread.
// 'msg' must be 'malloc'ed, and its owner ship will be transfered to '_send_q'
//
// NetLibConnection has base classes of tcp::socket and boost::enable_shared_from_this
void NetLibConnection::send_msg(PlainNetLibMsg* msg)
{
AutoLocker __dummy(this->_lock_4_send_q); // _lock_4_send_q is a 'mutex'
bool write_in_progress = ! this->_send_q.empty(); // _send_q is std::deque<PlainNetLibMsg* >,
// the 'send_q' mechansim is learned from boost_asio_example/cpp03/chat
this->_send_q.push_back(msg);
if (write_in_progress)
{
return;
}
this->get_io_service().post( // queue the 'send operation' to a singlton io_serivce::run() thread
boost::bind(&NetLibConnection::async_send_front_of_q
, boost::dynamic_pointer_cast<NetLibConnection>(shared_from_this())
)
);
}
void NetLibConnection::async_send_front_of_q()
{
boost::asio::async_write(*this
, boost::asio::buffer( this->_send_q.front() , _send_q.front()->header.DataSize + sizeof(NetLibChunkHeader) )
, this->_strand.wrap( // this great post https://stackoverflow.com/questions/12794107/why-do-i-need-strand-per-connection-when-using-boostasio/
// convinced me that I should use strand along with Connection
boost::bind( &NetLibConnection::handle_send
, boost::dynamic_pointer_cast<NetLibConnection>(shared_from_this())
, boost::asio::placeholders::error
)
)
);
}
The code works fine. But I am not satisfied with its verbosity. I feel the senq_q acts as the same role of strand.
Since
all real async_write call happen in a single io_service::run() thread
all real async_write are queued one-by-one via the send_q
Do I still need the strand?
Yes, indeed. The documentation details this here:
Threads And Boost Asio
By only calling io_service::run() from a single thread, the user's code can avoid the development complexity associated with synchronisation. For example, a library user can implement scalable servers that are single-threaded (from the user's point of view).
Thinking a bit more broadly, your scenario is the simplest form of having a single logical strand. There are other ways in which you can maintain logical strands (by chaining handlers), see this most excellent answer on the subject: Why do I need strand per connection when using boost::asio?

Boost Asio, async_read/connect timeout

In boost website, there is a good example about timeout of async operations. However, in that example, the socket is closed to cancel operations. There is also socket::cancel(), but in both documentation and as a compiler warning, it is stated as problematic in terms of portability.
Among the stack of Boost.Asio timeout questions in SO, there are several kind of answers. The first one probably is introducing a custom event loop, i.e., loop io_service::run_one() and cancel the event loop on deadline. I am using io_service::run() in a worker thread. That's not the kind of solution I would like to employ, if possible, as I do not want to change my code base.
A second option is directly changing the options of native socket. However, I would like to stick to Boost.Asio if possible and avoid any sort of platform-specific code as much as possible.
The example in the documentation is for an old version of Boost.Asio, but it's working properly, other than being forced to close the socket to cancel the operations. Using the documentation example, I have the following
void check_deadline(const boost::system::error_code &ec)
{
if(!running) {
return;
}
if(timer.expires_at() <= boost::asio::deadline_timer::traits_type::now()) {
// cancel all operations
boost::system::error_code errorcode;
boost::asio::ip::tcp::endpoint endpoint = socket.remote_endpoint();
socket.close(errorcode);
if(errorcode) {
SLOGERROR(mutex, errorcode.message(), "check_deadline()");
}
else {
SLOG(mutex, "timed out", "check_deadline()");
// connect again
Connect(endpoint);
if(errorcode) {
SLOGERROR(mutex, errorcode.message(), "check_deadline()");
}
}
// set timer to infinity, so that it won't expire
// until a proper deadline is set
timer.expires_at(boost::posix_time::pos_infin);
}
// keep waiting
timer.async_wait(std::bind(&TCPClient::check_deadline, this, std::placeholders::_1));
}
This is the only callback function registered to async_wait.The very first solution I could come up is reconnecting after closing the socket. Now my question is, is there a better way? By better way, I mean canceling the operations based on a timer without actually disrupting (i.e., not closing the socket) the connection.

Synchronous and ASynchronous APIs

I am developing a library, which provides some time consuming services. I need to have two versions of each API, one for synchronous function call and the other for asynchronous.
Library user should decide which version to use, a service result might be crucial for continue of system operation (synch call). The same operation might be needed to be done in different worker thread as it result is not needed to continue (asynch call).
What are the problems of this approach?
Is there any better way?
Are there popular libraries which provides both sync/async for the same API (Without using external events or threads) ?
Here is an example of what I am going to provide:
enum StuffStatus
{
SUCCEED,
FAILED,
STILL_RUNNING
};
class IServiceCallback
{
public:
void lengthyStuffCallback(StuffStatus status);
};
class MyServiceClass
{
public:
StuffStatus doSomeLengthStuff(IServiceCallback* callback)
{
if( callback == NULL ) // user wants sync. call
{
// do all operations in caller context
return SUCCEED;
}else{
// save the callback, queue the request in a separate worker thread.
// and after the worker thread finishes the job it calls callback->lengthyStuffCallback(SUCCEED) from its context.
return STILL_RUNNING;
}
}
};
EDIT:
As ' Matthieu M.' mentioned, In my service I need asynchronous with Continuation Passing Style (callback after API finish).
You might want to consider to provide only the synchronous operation and advise users to use std::future<...> (or a similar facility if you can't use C++ 2011) if they want an asynchronous version of the call!
std::future<StuffStatus> async(std::async(&MyServiceClass::doSomeLengthyStuff,
&service));
// do other stuff
StuffStatus status = async.get(); // get the result, possibly using a blocking wait

Boost Asio callback doesn't get called

I'm using Boost.Asio for network operations, they have to (and actually, can, there's no complex data structures or anything) remain pretty low level since I can't afford the luxury of serialization overhead (and the libs I found that did offer well enough performance seemed to be badly suited for my case).
The problem is with an async write I'm doing from the client (in QT, but that should probably be irrelevant here). The callback specified in the async_write doesn't get called, ever, and I'm at a complete loss as to why. The code is:
void SpikingMatrixClient::addMatrix() {
std::cout << "entered add matrix" << std::endl;
int action = protocol::Actions::AddMatrix;
int matrixSize = this->ui->editNetworkSize->text().toInt();
std::ostream out(&buf);
out.write(reinterpret_cast<const char*>(&action), sizeof(action));
out.write(reinterpret_cast<const char*>(&matrixSize), sizeof(matrixSize));
boost::asio::async_write(*connection.socket(), buf.data(),
boost::bind(&SpikingMatrixClient::onAddMatrix, this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
}
which calls the first write. The callback is
void SpikingMatrixClient::onAddMatrix(const boost::system::error_code& error, size_t bytes_transferred) {
std::cout << "entered onAddMatrix" << std::endl;
if (!error) {
buf.consume(bytes_transferred);
requestMatrixList();
} else {
QString message = QString::fromStdString(error.message());
this->ui->statusBar->showMessage(message, 15000);
}
}
The callback never gets called, even though the server receives all the data. Can anyone think of any reason why it might be doing that?
P.S. There was a wrapper for that connection, and yes there will probably be one again. Ditched it a day or two ago because I couldn't find the problem with this callback.
As suggested, posting a solution I found to be the most suitable (at least for now).
The client application is [being] written in QT, and I need the IO to be async. For the most part, the client receives calculation data from the server application and has to render various graphical representations of them.
Now, there's some key aspects to consider:
The GUI has to be responsive, it should not be blocked by the IO.
The client can be connected / disconnected.
The traffic is pretty intense, data gets sent / refreshed to the client every few secs and it has to remain responsive (as per item 1.).
As per the Boost.Asio documentation,
Multiple threads may call io_service::run() to set up a pool of
threads from which completion handlers may be invoked.
Note that all threads that have joined an io_service's pool are considered equivalent, and the io_service may distribute work across them in an arbitrary fashion.
Note that io_service.run() blocks until the io_service runs out of work.
With this in mind, the clear solution is to run io_service.run() from another thread. The relevant code snippets are
void SpikingMatrixClient::connect() {
Ui::ConnectDialog ui;
QDialog *dialog = new QDialog;
ui.setupUi(dialog);
if (dialog->exec()) {
QString host = ui.lineEditHost->text();
QString port = ui.lineEditPort->text();
connection = TcpConnection::create(io);
boost::system::error_code error = connection->connect(host, port);
if (!error) {
io = boost::shared_ptr<boost::asio::io_service>(new boost::asio::io_service);
work = boost::shared_ptr<boost::asio::io_service::work>(new boost::asio::io_service::work(*io));
io_threads.create_thread(boost::bind(&SpikingMatrixClient::runIo, this, io));
}
QString message = QString::fromStdString(error.message());
this->ui->statusBar->showMessage(message, 15000);
}
}
for connecting & starting IO, where:
work is a private boost::shared_ptr to the boost::asio::io_service::work object it was passed,
io is a private boost::shared_ptr to a boost::asio::io_service,
connection is a boost::shared_ptr to my connection wrapper class, and the connect() call uses a resolver etc. to connect the socket, there's plenty examples of that around
and io_threads is a private boost::thread_group.
Surely it could be shortened with some typedefs if needed.
TcpConnection is my own connection wrapper implementation, which sortof lacks functionality for now, and I suppose I could move the whole thread thing into it when it gets reinstated. This snippet should be enough to get the idea anyway...
The disconnecting part goes like this:
void SpikingMatrixClient::disconnect() {
work.reset();
io_threads.join_all();
boost::system::error_code error = connection->disconnect();
if (!error) {
connection.reset();
}
QString message = QString::fromStdString(error.message());
this->ui->statusBar->showMessage(message, 15000);
}
the work object is destroyed, so that the io_service can run out of work eventually,
the threads are joined, meaning that all work gets finished before disconnecting, thus data shouldn't get corrupted,
the disconnect() calls shutdown() and close() on the socket behind the scenes, and if there's no error, destroys the connection pointer.
Note, that there's no error handling in case of an error while disconnecting in this snippet, but it could very well be done, either by checking the error code (which seems more C-like), or throwing from the disconnect() if the error code within it represents an error after trying to disconnect.
I encountered a similar problem (callbacks not fired) but the circumstances are different from this question (io_service had jobs but still would not fire the handlers ). I will post this anyway and maybe it will help someone.
In my program, I set up an async_connect() then followed by io_service.run(), which blocks as expected.
async_connect() goes to on_connect_handler() as expected, which in turn fires async_write().
on_write_complete_handler() does not fire, even though the other end of the connection has received all the data and has even sent back a response.
I discovered that it is caused by me placing program logic in on_connect_handler(). Specifically, after the connection was established and after I called async_write(), I entered an infinite loop to perform arbitrary logic, not allowing on_connect_handler() to exit. I assume this causes the io_service to not be able to execute other handlers, even if their conditions are met because it is stuck here. ( I had many misconceptions, and thought that io_service would automagically spawn threads for each async_x() call )
Hope that helps.