No difference between curl_easy and curl_multi - c++

Question

I'm performing HTTP requests from my C++ program to my PHP script with libcurl.
The first easy_ version below works good, however it is quite slow (12 requests per second on localhost). Nothing strange - I got similar results using ab -n 1000 -c 1.
On the other hand ab -n 1000 -c 100 performs much more better with 600 request per second.The thing is, using libcurl multi doesn't seem to be concurrent. I used just slightly modified example code and the result is also about 12 req/s.
Do I understand curl_multi right? How can I achieve results similar to ab?
PS. I know that both codes are a bit different, however almost whole time is spent on curl work.
The easy_ way:
CURL *curl;
CURLcode response; // HTTP response
curl = curl_easy_init();
if(curl)
{
curl_easy_setopt(curl, CURLOPT_URL, "http://localhost/process.php");
while(true)
{
if(!requestsQueue.empty())
{
mtx.lock();
string data = requestsQueue.front();
requestsQueue.pop();
mtx.unlock();
const char *post = data.c_str(); //convert string to char used by CURL
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, post);
do
{
response = curl_easy_perform(curl);
} while(response != CURLE_OK);
}
else
{
//there are no request to perform, so wait for them
cout << "Sleeping...\n";
sleep(2);
continue;
}
}
//curl_easy_cleanup(curl);
}
else
{
cout << "CURL init failed!\n";
}
The multi way:
CURLM *multi_handle;
int still_running; /* keep number of running handles */
/* init a multi stack */
multi_handle = curl_multi_init();
/* add the individual transfers */
for(int i=1;i<=300;i++)
{
CURL *handle;
handle = curl_easy_init();
curl_easy_setopt(handle, CURLOPT_URL, "http://localhost/process.php");
curl_multi_add_handle(multi_handle, handle);
}
/* we start some action by calling perform right away */
curl_multi_perform(multi_handle, &still_running);
do {
struct timeval timeout;
int rc; /* select() return code */
fd_set fdread;
fd_set fdwrite;
fd_set fdexcep;
int maxfd = -1;
long curl_timeo = -1;
FD_ZERO(&fdread);
FD_ZERO(&fdwrite);
FD_ZERO(&fdexcep);
/* set a suitable timeout to play around with */
timeout.tv_sec = 1;
timeout.tv_usec = 0;
curl_multi_timeout(multi_handle, &curl_timeo);
if(curl_timeo >= 0) {
timeout.tv_sec = curl_timeo / 1000;
if(timeout.tv_sec > 1)
timeout.tv_sec = 1;
else
timeout.tv_usec = (curl_timeo % 1000) * 1000;
}
/* get file descriptors from the transfers */
curl_multi_fdset(multi_handle, &fdread, &fdwrite, &fdexcep, &maxfd);
/* In a real-world program you OF COURSE check the return code of the
function calls. On success, the value of maxfd is guaranteed to be
greater or equal than -1. We call select(maxfd + 1, ...), specially in
case of (maxfd == -1), we call select(0, ...), which is basically equal
to sleep. */
rc = select(maxfd+1, &fdread, &fdwrite, &fdexcep, &timeout);
switch(rc) {
case -1:
/* select error */
break;
case 0:
default:
/* timeout or readable/writable sockets */
curl_multi_perform(multi_handle, &still_running);
break;
}
} while(still_running);
curl_multi_cleanup(multi_handle);
curl_easy_cleanup(http_handle);
return 0;

Answer 1

curl_multi does indeed work with any amount of transfers in parallel, but it does so using the same single thread for all the work. It has the side-effect that if something anywhere takes a long time, that action blocks all other transfers.
One example of such a blocking operation that sometimes is what causes something like what you're describing, is the name resolver phase if the old blocking name resolver is used. Other explanations include a callback implemented by the application takes time for some reason.
You can build libcurl to instead use c-ares or the threaded-resolver backends that both avoid this blocking behavior and instead much better allow for concurrency. The threaded resolver is default in libcurl since many years now (late 2021).