Curl - Sending hundreds of requests but only four at a time - Programming - c++

Question

How do you proceed to solve this problem? I've hundreds of requests to be sent to Curl but I can send only four at a time.
Thus, I need to make four requests using curl at the same time and processes their responses. However, once one of the curl pointer is available, I need to send another request.
This is because, the server can handle only four requests at a time but I've hundreds of requests to be sent to the server.
Following is the code, I got from curl site
int main(void)
{
const int HANDLECOUNT = 4;
CURL *handles[HANDLECOUNT];
CURLM *multi_handle;
int still_running = 0; /* keep number of running handles */
int i;
CURLMsg *msg; /* for picking up messages with the transfer status */
int msgs_left; /* how many messages are left */
/* Allocate one CURL handle per transfer */
for(i = 0; i<HANDLECOUNT; i++)
handles[i] = curl_easy_init();
/* set the options (I left out a few, you'll get the point anyway) */
curl_easy_setopt(handles[0], CURLOPT_URL, "website");
curl_easy_setopt(handles[0], CURLOPT_POSTFIELDS, XMLRequestToPost.c_str());
curl_easy_setopt(handles[0], CURLOPT_POSTFIELDSIZE, (long)strlen(XMLRequestToPost.c_str()));
curl_easy_setopt(handles[1], CURLOPT_URL, "website");
curl_easy_setopt(handles[2], CURLOPT_URL, "website");
curl_easy_setopt(handles[3], CURLOPT_URL, "website");
/* set the request for other 3 handles too */
/* init a multi stack */
multi_handle = curl_multi_init();
/* add the individual transfers */
for(i = 0; i<HANDLECOUNT; i++)
curl_multi_add_handle(multi_handle, handles[i]);
/* we start some action by calling perform right away */
curl_multi_perform(multi_handle, &still_running);
while(still_running) {
}
}

Answer 1

Create a thread-safe queue to put your requests into.
Start 4 threads, each one with its own CURL object.
Have each thread run a loop that:
pulls the next request from the queue,
sends it
processes/dispatches the response as needed,
and repeats
Until the queue is empty.