MHD_resume_connection() of libmicrohttpd not working properly with external select - c++

Question

I encountered some problems with MHD_suspend_connection() and MHD_resume_connection() in libmicrohttpd while using the external event loop. Afterwards I have wrote a small example (without error handling) below. My question is: What am I doing wrong? Or is it a bug in the library? It should work as far as I understand the manual. Using external select with suspend/resume is allowed explicitly.
The problem is that connections are not resumed correctly. Processing the connection does not continue right after calling MHD_resume_connection(). In some versions of my program, it did continue after another request was incomming. In other versions later requests was not handled at all (access_handler() was never called). In some of this versions I got a response for the first request while stopping libmicrohttpd. When I enable MHD_USE_SELECT_INTERNALLY and remove my external loop (let it sleep), everything works.
I tested it on Debian (libmicrohttpd 0.9.37) and Arch (libmicrohttpd 0.9.50). The problem exists on both systems but maybe the behavior was a little bit different.
#include <algorithm>
#include <csignal>
#include <cstring>
#include <iostream>
#include <vector>
#include <sys/select.h>
#include <microhttpd.h>
using std::cerr;
using std::cout;
using std::endl;
static volatile bool run_loop = true;
static MHD_Daemon *ctx = nullptr;
static MHD_Response *response = nullptr;
static std::vector<MHD_Connection*> susspended;
void sighandler(int)
{
run_loop = false;
}
int handle_access(void *cls, struct MHD_Connection *connection,
const char *url, const char *method, const char *version,
const char *upload_data, size_t *upload_data_size,
void **con_cls)
{
static int second_call_marker;
static int third_call_marker;
if (*con_cls == nullptr) {
cout << "New connection" << endl;
*con_cls = &second_call_marker;
return MHD_YES;
} else if (*con_cls == &second_call_marker) {
cout << "Suspending connection" << endl;
MHD_suspend_connection(connection);
susspended.push_back(connection);
*con_cls = &third_call_marker;
return MHD_YES;
} else {
cout << "Send response" << endl;
return MHD_queue_response(connection, 200, response);
}
}
void myapp()
{
std::signal(SIGINT, &sighandler);
std::signal(SIGINT, &sighandler);
ctx = MHD_start_daemon(MHD_USE_DUAL_STACK //| MHD_USE_EPOLL
| MHD_USE_SUSPEND_RESUME | MHD_USE_DEBUG,
8080, nullptr, nullptr,
&handle_access, nullptr,
MHD_OPTION_END);
response = MHD_create_response_from_buffer(4, const_cast<char*>("TEST"),
MHD_RESPMEM_PERSISTENT);
while (run_loop) {
int max;
fd_set rs, ws, es;
struct timeval tv;
struct timeval *tvp;
max = 0;
FD_ZERO(&rs);
FD_ZERO(&ws);
FD_ZERO(&es);
cout << "Wait for IO activity" << endl;
MHD_UNSIGNED_LONG_LONG mhd_timeout;
MHD_get_fdset(ctx, &rs, &ws, &es, &max);
if (MHD_get_timeout(ctx, &mhd_timeout) == MHD_YES) {
//tv.tv_sec = std::min(mhd_timeout / 1000, 1ull);
tv.tv_sec = mhd_timeout / 1000;
tv.tv_usec = (mhd_timeout % 1000) * 1000;
tvp = &tv;
} else {
//tv.tv_sec = 2;
//tv.tv_usec = 0;
//tvp = &tv;
tvp = nullptr;
}
if (select(max + 1, &rs, &ws, &es, tvp) < 0 && errno != EINTR)
throw "select() failed";
cout << "Handle IO activity" << endl;
if (MHD_run_from_select(ctx, &rs, &ws, &es) != MHD_YES)
throw "MHD_run_from_select() failed";
for (MHD_Connection *connection : susspended) {
cout << "Resume connection" << endl;
MHD_resume_connection(connection);
}
susspended.clear();
}
cout << "Stop server" << endl;
MHD_stop_daemon(ctx);
}
int main(int argc, char *argv[])
{
try {
myapp();
} catch (const char *str) {
cerr << "Error: " << str << endl;
cerr << "Errno: " << errno << " (" << strerror(errno) << ")" << endl;
return EXIT_FAILURE;
}
return EXIT_SUCCESS;
}

Answer 1

I've compiled and run your sample on Windows and am seeing the same behavior w/ 0.9.51.
It's not a bug in microhttpd. The problem is that you are resuming a connection before queuing a response on it. The only code you have that creates a response relies on more activity on the connection so it's a catch-22.
The point of MHD_suspend_connection/MHD_resume_connection is to not block new connections while long-running work is going on. Thus typically after suspending the connection you need to kick off that work on another thread to continue while the listening socket is maintained. When that thread has queued the response it can resume the connection and the event loop will know it is ready to send back to the client.
I'm not sure of your other design requirements but you may not need to be implementing external select. That is to say, suspend/resume does not require it (I've used suspend/resume just fine with MHD_USE_SELECT_INTERNALLY, e.g.).

Answer 2

I dont know if it's mentioned. But you have a multi-threading bug, and perhaps, "intent bug". As the lib, may or may not use threads, depending on other factors. You can see if you are using threads, by printing the thread id, from the functions. But, your answerToConnection function, sets your vector (without mutex protection), and then you are immediately looking at it, and retrying potentially from another thread. this goes against the intent/purpose of suspend/retry, since suspend is really for something taking "a long time". The gotcha, is that you dont own the calling code, so, you dont know when it's totally done. however, you can age your retry, with a timeval, so, you dont retry too soon. at least a value of tv_usec +1. you need to note, that you are using the vector from two or more threads, without mutex protection.