LibCurl WriteCallback (Async?) - C++ - c++

I am successfully making an http POST call using the following code:
std::string curlString;
CURL* pCurl = curl_easy_init();
if(!pCurl)
return NULL;
string outgoingUrl = Url;
string postFields = fields;
curl_easy_setopt(pCurl, CURLOPT_TIMEOUT, 0);
curl_easy_setopt(pCurl, CURLOPT_URL, outgoingUrl.c_str());
curl_easy_setopt(pCurl, CURLOPT_POST, 1);
curl_easy_setopt(pCurl, CURLOPT_POSTFIELDS, postFields.c_str());
curl_easy_setopt(pCurl, CURLOPT_POSTFIELDSIZE, (long)postFields.size());
curl_easy_setopt(pCurl, CURLOPT_WRITEFUNCTION, CurlWriteCallback);
curl_easy_setopt(pCurl, CURLOPT_WRITEDATA, &curlString);
curl_easy_perform(pCurl);
curl_easy_cleanup(pCurl);
The write callback has the following prototype:
size_t CurlWriteCallback(char* a_ptr, size_t a_size, size_t a_nmemb, void* a_userp);
Is there a way to do this asynchronously? Currently it waits for the callback to finish before curl_easy_perform returns. This blocking method won't work for a server with many users.

From the libcurl easy documentation:
When all is setup, you tell libcurl to perform the transfer using curl_easy_perform(3). It will then do the entire operation and won't return until it is done (successfully or not).
From the libcurl multi interface docs, one of the features as opposed to the "easy" interface:
Enable multiple simultaneous transfers in the same thread without making it complicated for the application.
Sounds like you want to use the "multi" approach.

Related

What could be reason of getting SSL connect error while posting json request to https url from Windows PC?

I am using libcurl.dll version 7.72.0.0 in Visual Studio for a C++ application.
When I am posting a JSON request to an HTTPS url on one Windows 7 PC with Service Pack 1, the request is posted successfully. But on other PC with same OS as Windows 7 Service Pack 1, I am getting response code 35 from curl_easy_perform(): SSL connect error.
What could be the reason of the same C++ code working on one Windows 7 PC and getting an error on other Windows 7 PC? Both PCs are on the same network.
Following is the C++ code:
curl_easy_setopt(curl, CURLOPT_URL, Url.c_str());
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, header);
// send all data to this function
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
// we pass our 'chunk' struct to the callback function
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)&sResponse);
//some servers don't like requests that are made without a user-agent
// field, so we provide one
curl_easy_setopt(curl, CURLOPT_USERAGENT, "libcurl-agent/1.0");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postthis);
printf("\npostthisData=%s\n", postthis);
if (itsHTTPSRequest)
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0);
// if we don't provide POSTFIELDSIZE, libcurl will strlen() by
// itself
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, (long)strlen(postthis));
curl_easy_setopt( curl, CURLOPT_CONNECTTIMEOUT, m_RWTimeOut );
curl_easy_setopt( curl, CURLOPT_LOW_SPEED_TIME, m_RWTimeOut);
curl_easy_setopt( curl, CURLOPT_LOW_SPEED_LIMIT, 30L);

make libcURL timeout only on connection failure

I'm using some auto-update function in my program. In case of connection failure, I would like the program to keep trying up to 15 seconds, and then announce failure. In order to achieve that, I used the following curl_easy_setopt for cURL easy option:
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 15);
But then I discovered that if the download takes more than 15 seconds, the timeout error is announced.
How can I restrict the 15 seconds only to the case of failure? I.e., if there's no connection for 15 seconds?
More information
The full option list I use is the following:
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); //verify ssl peer
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); //verify ssl hostname
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, CurlWrite_CallbackFunc_StdString);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &s);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, FALSE);
curl_easy_setopt(curl, CURLOPT_PROGRESSDATA, this); //pointer to the current class as it's a GUI program
curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, CurlProgress_CallbackFunc_UpdateProgress);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 20);
and the call to start is done through:
CURLcode res = curl_easy_perform(curl);
If you require more information, please let me know.
Thank you.
Instead of
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 20);
use
curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 15);
curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, 1);
curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, 15);
The first line is the timeout for the connection phase. After the connection is established the timeout becomes irrelevant, but the two following lines make sure that if the average speed in a 15 second time frame drops below 1 byte per second, then the operation is aborted.
Also worth noting that curl will not try to reestablish any connection if it is dropped, because a TCP connection can still be hold if the physical connection is (temporarly) lost up until one of the sides decides to timeout.

Synchronized curl requests

I'm trying to do HTTP requests to multiple targets, and I need to them to run (almost) exactly at the same moment.
I'm trying to create a thread for each request, but I don't know why Curl is crashing when doing the perform. I'm using an easy-handle per thread so in theory everything should be ok...
Has anybody had a similar problem? or Does anyone know if the multi interface allows you to choose when to perform all the requests?
Thanks a lot.
EDIT:
Here is an example of the code:
void Clazz::function(std::vector<std::string> urls, const std::string& data)
{
for (auto it : urls)
{
std::thread thread(&Clazz::DoRequest, this, it, data);
thread->detach();
}
}
int Clazz::DoRequest(const std::string& url, const std::string& data)
{
CURL* curl = curl_easy_init();
curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Expect:");
headers = curl_slist_append(headers, "Content-Type: application/json");
curl_easy_setopt(curl, CURLOPT_POST, 1);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data.c_str());
curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 15);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt (curl, CURLOPT_FAILONERROR, 1L);
//curlMutex.lock();
curl_easy_perform(curl);
//curlMutex.unlock();
long responseCode = 404;
curl_easy_getinfo (curl, CURLINFO_RESPONSE_CODE, &responseCode);
curl_easy_cleanup(curl);
curl_slist_free_all(headers);
}
I hope this can help, thanks!
Are you calling curl_global_init anywhere? Perhaps rather early in your main() method?
Quoting from http://curl.haxx.se/libcurl/c/curl_global_init.html:
This function is not thread safe. You must not call it when any other thread in the program (i.e. a thread sharing the same memory) is running. This doesn't just mean no other thread that is using libcurl. Because curl_global_init calls functions of other libraries that are similarly thread unsafe, it could conflict with any other thread that uses these other libraries.
Quoting from http://curl.haxx.se/libcurl/c/curl_easy_init.html:
If you did not already call curl_global_init, curl_easy_init does it automatically. This may be lethal in multi-threaded cases, since curl_global_init is not thread-safe, and it may result in resource problems because there is no corresponding cleanup.
It sounds like you're not calling curl_global_init, and letting curl_easy_init take care of it for you. Since you're doing it on two threads simultaneously, you're hitting the thread unsafe scenario, with the lethal result that was mentioned.
After being able to debug properly in the device y have found that the problem is an old know issue with curl.
http://curl.haxx.se/mail/lib-2010-11/0181.html
after using CURLOPT_NOSIGNAL in every curl handle the crash has disappeared. :)

c++ curl returns receive error when usng proxies

As the header says, I am trying to load a page with proxy using curl on c++.
I have tried the proxy with curl linux command and got a result without a password, on my code i do:
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_ENCODING, "");
curl_easy_setopt(curl, CURLOPT_AUTOREFERER, true);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curlRead);
curl_easy_setopt(curl, CURLOPT_USERAGENT, agent);
curl_easy_setopt(curl, CURLOPT_IGNORE_CONTENT_LENGTH, 1L);
curl_easy_setopt(curl, CURLOPT_PROXY, "<proxy IP>:80");
res = curl_easy_perform(curl);
but res returns as 56 (CURLE_RECV_ERROR) and the manual says
Failure with receiving network data.
EDIT: I have tried another proxy (also working from terminal) and now I dont get a response at all, it simply stuck.

libcurl http post timeout

I am using curl version 7.15.5 in multi-thread environment. Each thread is initializing and freeing its own curl object. Below is the code, executed for each thread:
CURL* curl = curl_easy_init();
tRespBuffer respBuffer = {NULL, 0};
char errorBuf[CURL_ERROR_SIZE +1];
struct curl_slist *headers=NULL;
headers = curl_slist_append(headers, "Content-Type: text/xml; charset=gbk");
headers = curl_slist_append(headers, "Expect:");
curl_easy_setopt(curl, CURLOPT_URL, url_);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS,encr.c_str());
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE,strlen(encr.c_str()));
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, HttpSmsServer::processHttpResponse);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void*)&respBuffer);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 20); // wait for 20 seconds before aborting the transacttion
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errorBuf); // error returned if any..
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1); // No signals allowed in case of multithreaded apps
res = curl_easy_perform(curl);
curl_slist_free_all(headers);
curl_easy_cleanup(curl);
All the four threads are posting data to http server simultaneously. I see HTTP response timeout for some of the POST requests (~3% of requests). Any idea what could be the reason of timeouts ? I assume http server should not take more than 20 seconds to respond back.
CURLOPT_TIMEOUT includes all the time of http request, have you transferred huge data?
CURLOPT_TIMEOUT:Pass a long as parameter containing the maximum time in seconds that you allow the libcurl transfer operation to take. Normally, name lookups can take a considerable time and limiting operations to less than a few minutes risk aborting perfectly normal operations.