gRPC: How can RPC handlers properly detect if `Server` has been `Shutdown()` - c++

Current, I'm using a hackish way – a global variable – to make RPC handlers able to detect that the Server has been (about to be) called Shutdown().
bool g_ServerIsNotDead = true; // Hack!
Status StreamServiceImpl::GetCurrentTemperature(ServerContext *context_,
const UpdateInterval *request_,
ServerWriter<Temperature> *stream_)
{
auto currentTemp = 100.0f;
while(g_ServerIsNotDead) // Hack!!!
{
qDebug() << QThread::currentThreadId() << currentTemp << "farenheit.";
Temperature message;
message.set_temperature(currentTemp);
stream_->Write(message);
QThread::sleep(2);
currentTemp += 1.0f;
}
return Status::OK;
}
void insideSomeFunction() {
// Testing shutdown 5 seconds later
QTimer::singleShot(std::chrono::seconds(5), this, [=]() {
qDebug() << "Shuting down!";
g_ServerIsNotDead = false; // Hack!!
this->server->Shutdown(); // This method actually blocks until all RPC handlers have exited, believe it or not!
emit shutdown();
qDebug() << "All dead.";
});
}
Ref: https://github.com/C0D1UM/grpc-qt-example/blob/master/rpc_server/hellostream_server.cpp
It would be really nice if I could somehow check that Server has been Shutdown() from grpc::ServerContext, but I didn't see any relevant methods to achieve this.
Even better if someone could propose a way to take out the while loop completely (?). I'm using Qt so everything is event-driven.

So, I think it's worth making clear what Shutdown does. There's a detailed comment about this but basically, server Shutdown doesn't fail, cancel, or kill your existing in-progress calls (unless you use the deadline argument and the gRPC C++ async API).
Rather, it stops listening for new connections, stops accepting new calls, fails requested-but-not-yet-accepted calls. If you want to fail or terminate your calls at shutdown, you can do it at application-level code as you've done above.
I would just recommend that instead of using a global variable, you should use a member function of your StreamServiceImpl class so that you can support multiple services running in the same process if you choose.

We can use ServerContext::IsCancelled as a breaking/termination criteria in streaming APIs. I changed GetCurrentTemperature(...) as follows (just replaced g_ServerIsNotDead with !context_->IsCancelled()) and it worked:
Status StreamServiceImpl::GetCurrentTemperature(ServerContext *context_,
const UpdateInterval *request_,
ServerWriter<Temperature> *stream_) {
auto currentTemp = 100.0f;
while(!context_->IsCancelled) {
qDebug() << QThread::currentThreadId() << currentTemp << "farenheit.";
Temperature message;
message.set_temperature(currentTemp);
stream_->Write(message);
QThread::sleep(2);
currentTemp += 1.0f;
}
return Status::OK;
}

Related

App crashes when it takes too long to reply in a ZMQ REQ/REP pattern

I am writing a plugin that interfaces with a desktop application through a ZeroMQ REQ/REP request-reply communication archetype. I can currently receive a request, but the application seemingly crashes if a reply is not sent quick enough.
I receive the request on a spawned thread and put it in a queue. This queue is processed in another thread, in which the processing function is invoked by the application periodically.
The message is correctly being received and processed, but the response cannot be sent until the next iteration of the function, as I cannot get the data from the application until then.
When this function is conditioned to send the response on the next iteration, the application will crash. However, if I send fake data as the response soon after receiving the request, in the first iteration, the application will not crash.
Constructing the socket
zmq::socket_t socket(m_context, ZMQ_REP);
socket.bind("tcp://*:" + std::to_string(port));
Receiving the message in the spawned thread
void ZMQReceiverV2::receiveRequests() {
nInfo(*m_logger) << "Preparing to receive requests";
while (m_isReceiving) {
zmq::message_t zmq_msg;
bool ok = m_respSocket.recv(&zmq_msg, ZMQ_NOBLOCK);
if (ok) {
// msg_str will be a binary string
std::string msg_str;
msg_str.assign(static_cast<char *>(zmq_msg.data()), zmq_msg.size());
nInfo(*m_logger) << "Received the message: " << msg_str;
std::pair<std::string, std::string> pair("", msg_str);
// adding to message queue
m_mutex.lock();
m_messages.push(pair);
m_mutex.unlock();
}
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
nInfo(*m_logger) << "Done receiving requests";
}
Processing function on seperate thread
void ZMQReceiverV2::exportFrameAvailable()
// checking messages
// if the queue is not empty
m_mutex.lock();
if (!m_messages.empty()) {
nInfo(*m_logger) << "Reading message in queue";
smart_target::SMARTTargetCreateRequest id_msg;
std::pair<std::string, std::string> pair = m_messages.front();
std::string topic = pair.first;
std::string msg_str = pair.second;
processMsg(msg_str);
// removing just read message
m_messages.pop();
//m_respSocket.send(zmq::message_t()); wont crash if I reply here in this invocation
}
m_mutex.unlock();
// sending back the ID that has just been made, for it to be mapped
if (timeToSendReply()) {
sendReply(); // will crash, if I wait for this to be exectued on next invocation
}
}
My research shows that there is no time limit for the response to be sent, so this, seeming to be, timing issue, is strange.
Is there something that I am missing that will let me send the response on the second iteration of the processing function?
Revision 1:
I have edited my code, so that the responding socket only ever exists on one thread. Since I need to get information from the processing function to send, I created another queue, which is checked in the revised the function running on its own thread.
void ZMQReceiverV2::receiveRequests() {
zmq::socket_t socket = setupBindSocket(ZMQ_REP, 5557, "responder");
nInfo(*m_logger) << "Preparing to receive requests";
while (m_isReceiving) {
zmq::message_t zmq_msg;
bool ok = socket.recv(&zmq_msg, ZMQ_NOBLOCK);
if (ok) {
// does not crash if I call send helper here
// msg_str will be a binary string
std::string msg_str;
msg_str.assign(static_cast<char *>(zmq_msg.data()), zmq_msg.size());
NLogger::nInfo(*m_logger) << "Received the message: " << msg_str;
std::pair<std::string, std::string> pair("", msg_str);
// adding to message queue
m_mutex.lock();
m_messages.push(pair);
m_mutex.unlock();
}
std::this_thread::sleep_for(std::chrono::milliseconds(100));
if (!sendQueue.empty()) {
sendEntityCreationMessage(socket, sendQueue.front());
sendQueue.pop();
}
}
nInfo(*m_logger) << "Done receiving requests";
socket.close();
}
The function sendEntityCreationMessage() is a helper function that ultimately calls socket.send().
void ZMQReceiverV2::sendEntityCreationMessage(zmq::socket_t &socket, NUniqueID id) {
socket.send(zmq::message_t());
}
This code seems to be following the thread safety guidelines for sockets. Any suggestions?
Q : "Is there something that I am missing"
Yes,the ZeroMQ evangelisation, called a Zen-of-Zero, since ever promotes never try to share a Socket-instance, never try to block and never expect the world to act as one wishes.
This said, avoid touching the same Socket-instance from any non-local thread, except the one that has instantiated and owns the socket.
Last, but not least, the REQ/REP-Scalable Formal Communication Pattern Archetype is prone to fall into a deadlock, as a mandatory two-step dance must be obeyed - where one must keep the alternating sequence of calling .send()-.recv()-.send()-.recv()-.send()-...-methods, otherwise the principally distributed-system tandem of Finite State Automata (FSA) will unsalvageably end up in a mutual self-deadlock state of the dFSA.
In case one is planning to professionally build on ZeroMQ, the best next step is to re-read the fabulous Pieter HINTJENS' book "Code Connected: Volume 1". A piece of a hard read, yet definitely worth one's time, sweat, tears & efforts put in.

Sleep inside QTConcurrent run method

I'm using QtConcurrent::run to execute some functions in background and not hang the GUI thread. In one function, I read logs from local SQlite database and send them to server by TCP socket.
Now I want to delay the execution after each log so the server has time to save it (TCP response is read in different thread). I'm stuck with Qt4.8 due to implementation limitations (many embeded devices - no chance to upgrade QT on them) and I can't use QThread::sleep(2) because it is protected in 4.8.
Is it possible to somehow pause the execution of thread inside QtConcurrent::run method or should I redesign it to implement my own class inheriting QThread?
void MainWindow::ReportFinishedHUs(bool asyncWork)
{
if(asyncWork == false)
{
QMutexLocker locker(&localDBmutex);
QList<QSqlRecord> HUsToReport = localDB->getHUsForBook();
qDebug() << "HUs to report" << HUsToReport.count();
if(!HUsToReport.isEmpty())
{
Cls_log::WriteDebugLog("HUs to report: " + QString::number(HUsToReport.count()));
foreach (QSqlRecord record, HUsToReport)
{
int _hu = record.indexOf("hu");
int _logTime = record.indexOf("logTime");
QString logTimeString = record.value(_logTime).toString();
QString hu = record.value(_hu).toString();
qDebug() << hu << logTimeString;
// creating message here ...
qDebug() << message;
emit sig_SendTCPMessage(message);
// this is where I need to wait for 2 seconds
QThread::sleep(2);
}
}
}
else
{
QtConcurrent::run(this, &MainWindow::ReportFinishedHUs, false);
}
}
EDIT:
Solved by usleep(2000000) which I somehow discarded for being platform specific... but hey, half of my aplication is platform specific and I only use it in embeded device with constant OS.
Keeping the question open if anyone can suggest more elegand solution using Qt methods. I like to get inspired.

qt waitforreadyread() inconsistent?

Under Qt 4.7.1, OS X 10.6.8
(have to use this -- later versions
of Qt and/or OS X introduce severe
incompatibilities for my users)
The following works. Sometimes. Then sometimes not.
When it doesn't work, it returns "Unknown Error"
hst is good in all cases, qDebug returns same correct
data for hst every time.
The idea is, use ->get to pull a CGI URL; the CGI
returns some data, which I can ignore in this case.
Then I'm done.
hst is a well formed URL,
http://yadda.com/cgi-bin/whatever.py
QString hst;
QNetworkReply *qnr;
QNetworkAccessManager *qqnap = NULL;
qqnap = new(std::nothrow) QNetworkAccessManager(tmw);
if (qqnap != NULL)
{
hst = loaduphst(); // get qstring to send
qnr = qqnap->get(QNetworkRequest(QUrl(hst))); // report in and fetch update info
if (qnr->waitForReadyRead(3000) == FALSE)
{
qDebug() << "waitForReadyRead() returned FALSE -- error or timeout:" << qnr->errorString();
}
}
else
{
qDebug() << "qqnap is NULL";
}
yadda.com is up; the target script is dead simple
and works fine from browser or cmd line every time.
This is running within the context of
MainWindow::closeEvent(QCloseEvent *ce)
before I emit ce->accept() GUI is still up,
etc.
Hints? Tips? Abuse? Thanks!
waitForReadyRead is not implemented in QNetworkReply. The default implementation does nothing:
bool QIODevice::waitForReadyRead(int msecs)
{
Q_UNUSED(msecs);
return false;
}
Use the readyRead signal to find out when there is data available to be read.
More-or-less synchronous use of async networking is very problematic in the context of the main GUI loop. Signals that don't appear (finished OR readyRead), URLs that sometimes send and sometimes don't... and of course, as the kind person above pointed out, unimplemented functions. Zebras!
What we can do, though, is fire up an event loop and a timer on our own, and this will in a more-or-less friendly way act synchronous.
Perhaps some poor soul will need to poke a website CGI as I do; here's the code. It works. At least under Qt 4.7.1 it does!
So anyway, here it is:
QNetworkReply *qnr;
QNetworkAccessManager *qqnap;
QNetworkRequest qnwr;
QEventLoop w;
QTimer arf;
if ((qqnap = new(std::nothrow) QNetworkAccessManager(this)))
{
qnwr.setUrl(myUrl()); // Build web goodness
qnwr.setRawHeader("User-Agent", myUserAgent());
arf.setSingleShot(true);
if (connect(&arf, SIGNAL(timeout()), // timer firing blows...
&w, SLOT(quit()) // ...out event loop
) == FALSE)
{ return(BAD_CONNECT_TOUT); }
if (connect(qqnap, SIGNAL(finished(QNetworkReply*)), // notify we finished...
this, SLOT(qqnapReplyQ(QNetworkReply*)) // ...cuz I need to know
) == FALSE)
{ return(BAD_CONNECT_FINISHED_NOTIFY); }
if (connect(qqnap, SIGNAL(finished(QNetworkReply*)), // finishing blows out...
&w, SLOT(quit()) // ...event loop
) == FALSE)
{ return(BAD_CONNECT_FINISHED_ELOOP); }
if ((qnr = qqnap->get(qnwr))) // Go if qnr is good
{
arf.start(6000); // timeout in ms // Watchdog timer on
w.exec(); // handle all that
if (arf.isActive()) { arf.stop(); } // kill timer if needed
}
else { return(BAD_WWWGET); } // FAIL
}
else
{
return(BAD_NWAM); // FAIL
}
return(ZEN_NETWORKING);

How to trace resource deadlocks?

I've wrote a timer using std::thread - here is how it looks like:
TestbedTimer::TestbedTimer(char type, void* contextObject) :
Timer(type, contextObject) {
this->active = false;
}
TestbedTimer::~TestbedTimer(){
if (this->active) {
this->active = false;
if(this->timer->joinable()){
try {
this->timer->join();
} catch (const std::system_error& e) {
std::cout << "Caught system_error with code " << e.code() <<
" meaning " << e.what() << '\n';
}
}
if(timer != nullptr) {
delete timer;
}
}
}
void TestbedTimer::run(unsigned long timeoutInMicroSeconds){
this->active = true;
timer = new std::thread(&TestbedTimer::sleep, this, timeoutInMicroSeconds);
}
void TestbedTimer::sleep(unsigned long timeoutInMicroSeconds){
unsigned long interval = 500000;
if(timeoutInMicroSeconds < interval){
interval = timeoutInMicroSeconds;
}
while((timeoutInMicroSeconds > 0) && (active == true)){
if (active) {
timeoutInMicroSeconds -= interval;
/// set the sleep time
std::chrono::microseconds duration(interval);
/// set thread to sleep
std::this_thread::sleep_for(duration);
}
}
if (active) {
this->notifyAllListeners();
}
}
void TestbedTimer::interrupt(){
this->active = false;
}
I'm not really happy with that kind of implementation since I let the timer sleep for a short interval and check if the active flag has changed (but I don't know a better solution since you can't interrupt a sleep_for call). However, my program core dumps with the following message:
thread is joinable
Caught system_error with code generic:35 meaning Resource deadlock avoided
thread has rejoined main scope
terminate called without an active exception
Aborted (core dumped)
I've looked up this error and as seems that I have a thread which waits for another thread (the reason for the resource deadlock). However, I want to find out where exactly this happens. I'm using a C library (which uses pthreads) in my C++ code which provides among other features an option to run as a daemon and I'm afraid that this interfers with my std::thread code. What's the best way to debug this?
I've tried to use helgrind, but this hasn't helped very much (it doesn't find any error).
TIA
** EDIT: The code above is actually not exemplary code, but I code I've written for a routing daemon. The routing algorithm is a reactive meaning it starts a route discovery only if it has no routes to a desired destination and does not try to build up a routing table for every host in its network. Every time a route discovery is triggered a timer is started. If the timer expires the daemon is notified and the packet is dropped. Basically, it looks like that:
void Client::startNewRouteDiscovery(Packet* packet) {
AddressPtr destination = packet->getDestination();
...
startRouteDiscoveryTimer(packet);
...
}
void Client::startRouteDiscoveryTimer(const Packet* packet) {
RouteDiscoveryInfo* discoveryInfo = new RouteDiscoveryInfo(packet);
/// create a new timer of a certain type
Timer* timer = getNewTimer(TimerType::ROUTE_DISCOVERY_TIMER, discoveryInfo);
/// pass that class as callback object which is notified if the timer expires (class implements a interface for that)
timer->addTimeoutListener(this);
/// start the timer
timer->run(routeDiscoveryTimeoutInMilliSeconds * 1000);
AddressPtr destination = packet->getDestination();
runningRouteDiscoveries[destination] = timer;
}
If the timer has expired the following method is called.
void Client::timerHasExpired(Timer* responsibleTimer) {
char timerType = responsibleTimer->getType();
switch (timerType) {
...
case TimerType::ROUTE_DISCOVERY_TIMER:
handleExpiredRouteDiscoveryTimer(responsibleTimer);
return;
....
default:
// if this happens its a bug in our code
logError("Could not identify expired timer");
delete responsibleTimer;
}
}
I hope that helps to get a better understanding of what I'm doing. However, I did not to intend to bloat the question with that additional code.

gSoap: how to gracefully shutdown the webservice application?

I'm using gSoap to write a webservice. It's running as a console application. In all gSoap examples I see, that requests are dispatched in infinite loop like for(;;;) even in multi-threaded version.
But how can I make my webservice to terminate gracefully when, say, user presses space on the console?
Preferably:
stop accepting new connections;
Serve existing ones;
Exit from application
The only solution I came up so far is using timeouts
soap->recv_timeout = 20;
soap->send_timeout = 20;
soap->connect_timeout = 5;
soap->accept_timeout = 5;
Then all blocking functions return periodically. But this is not ideal for me, because I want to be able to terminate the app quickly even if there is an ongoing transmission, but at the same time, don't want to compromise reliability on a slow/flaky connection (it's an embedded device connected via GPRS).
The section 7.2.4 How to Create a Multi-Threaded Stand-Alone Service in the documentation has example code for writing an accept loop. You need to write your own accept loop and add signal handling so it responds to Ctrl-C.
stop accepting new connections:
Leave the loop so you stop calling accept.
Serve existing ones:
The threads need to inform you when they are finished, so you can exit when the number of active clients is zero. (boost::thead_group has a join_all which does exactly that.)
Exit from application:
What you need to do is register signal handler so when you terminate your application using Ctrl + C, it calls you registered function where you can gracefully terminates.
e.g
class gsoap_test {
public:
void start() {
running_ = true;
while(running_) {
//gsoap threads
}
//stop and cleanup
}
void stop() {
running_ = false;
}
private:
bool running_;
};
//global variable
gsoap_test gsoap;
void sighandler(int sig)
{
std::cout<< "Signal caught..." << std::endl;
//Stop gracefully here
gsoap.stop();
exit(0);
}
int main(int argc, char** argv) {
//register signal
signal(SIGABRT, &sighandler);
signal(SIGTERM, &sighandler);
signal(SIGINT, &sighandler);
gsoap.start();
return EXIT_SUCCESS;
}