Sending messages to curl via multiple threads, and once a while I get one of the following errors.
curl_easy_perform(): failed ssl connect error. sschannel: next
initializesecuritycontext failed: SEC_E_MESSAGE_ALTERED
curl_easy_perform(): failed ssl connect error. sschannel: next
initializesecuritycontext failed: SEC_E_BUFFER_SMALL
As of now, I'm resolving this by re-sending the request. But why does this error happen ( Same request in the next 40 seconds works) and what can be done to avoid this.
Source code is written in C++. LibCurl was built using Microsoft visual studio 2010.
Following is the code that invokes the curl library.
CURL *curl = curl_easy_init();
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, "connection-page");
curl_easy_setopt(curl, CURLOPT_POST, 1);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, requestToPost.c_str());
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, (long)strlen(requestToPost.c_str()));
curl_easy_setopt(curl, CURLOPT_VERBOSE, 0L);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerInfo);
curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, header_data);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curlErrorbuffer);
std::stringstream resPonseInfo;
std::stringstream headerResponse;
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &resPonseInfo);
curl_easy_setopt(curl, CURLOPT_HEADERDATA, &headerResponse);
curl_easy_setopt(curl, CURLOPT_HTTPAUTH, (long)CURLAUTH_ANY);
res = curl_easy_perform(curl);
if ((res != CURLE_OK))
{
fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
std::cout << "Request === " << std::endl;
std::cout << requestToPost << std::endl;
std::cout << "Error === " << std::endl;
std::cout << curlErrorbuffer << std::endl;
std::cout << "Header == " << std::endl << headerResponse.str() << std::endl;
std::cout << "Response == " << std::endl << resPonseInfo.str() << std::endl;
}
else // if(res == CURLE_OK)
{
std::cout << "Response from the http post was successful " << std::endl;
responseInfo = resPonseInfo.str();
}
curl_easy_cleanup(curl);
curl = NULL;
}
"Sending messages to curl via multiple threads..." - given described symptoms most logical would be to assume a multi-threading related issue
libcurl itself a thread safe, but not the shared data and handles used. You might want to consult this page: https://curl.haxx.se/libcurl/c/threadsafe.html and ensure your threads are not stepping each other toes
one (possibly) easy way to confirm above hypothesis - try running your program in a single thread mode (if you can) and see if the issue reoccurs. If it does then it's definitely not threading.
another way to verify (if above is not an option) put a thread mutex on your curl operation (even before you start setting up curl options) - see if that help avoiding those errors
Related
I set the hosts file on PC like this:
127.0.0.1 www.baidu.com
Then I wrote a simple test:
CURL *curl = curl_easy_init();
for (int i = 0; i < 3; ++i) {
if (curl) {
curl_easy_setopt(curl, CURLOPT_DNS_CACHE_TIMEOUT, 0L);
curl_easy_setopt(curl, CURLOPT_URL, "www.baidu.com");
if (i == 1)
curl_easy_setopt(curl, CURLOPT_DOH_URL, "https://dns.alidns.com/dns-query");
else
curl_easy_setopt(curl, CURLOPT_DOH_URL, NULL);
CURLcode res = curl_easy_perform(curl);
std::cout << std::endl;
std::cout << "==================================================" << std::endl;
std::cout << "Time: " << i << std::endl;
std::cout << "res = " << res << std::endl;
char *url = NULL;
curl_easy_getinfo(curl, CURLINFO_EFFECTIVE_URL, &url);
if (url)
printf("Redirect to: %s\n", url);
std::cout << "==================================================" << std::endl;
curl_easy_reset(curl);
}
}
And the result is like:
==================================================
Time: 0
res = 7
Redirect to: http://www.baidu.com/
==================================================
==================================================
Time: 1
res = 0
Redirect to: http://www.baidu.com/
==================================================
==================================================
Time: 2
res = 0
Redirect to: http://www.baidu.com/
==================================================
As defined in CURLcode, r = 0 means CURLE_OK, and r = 7 means CURLE_COULDNT_CONNECT.
I don't understand the result. When i == 0, I set DOH_URL to NULL, so it is normal that it couldn't connect. But when i == 2, I also set DOH_URL to NULL, and I do reset operation, and set DNS_CACHE_TIMEOUT to 0, why it still could connect?
curl easy uses the existing connection of the 2nd iteration on the 3rd iteration. See Name Resolving
libcurl tries hard to re-use an existing connection rather than to create a new one. The function that checks for an existing connection to use is based purely on the name and is performed before any name resolving is attempted.
See curl_easy_reset()
It does not change the following information kept in the handle: live connections, the Session ID cache, the DNS cache, the cookies, the shares or the alt-svc cache.
I'm trying to build a small multithreading program which takes a subdomains and test them if they are alive on http or https, I've problem that's my program doesn't produce the correct output each time I get different output and also freeze and doesn't continue execution. I followed http://www.cplusplus.com/reference/thread/thread/thread/ when implementing the multithreading.
int main(int argc, char const *argv[] )
{
if (argc < 2){
cout << "Usage httplive <path to subdomains>" << endl;
}
ifstream http(argv[1]);
string line;
vector <std::thread> thread_pool;
while (getline(http, line)){
thread_pool.push_back(thread(httpTest,line, true));
thread_pool.push_back(thread(httpTest, line, false));
}
for (auto& t : thread_pool){
t.join();
}
return 0;
}
void httpTest(string line, bool Flag){
CURL *curl = curl_easy_init();
CURLcode res;
if (curl) {
line = Flag ? "https://" + line : "http://"+ line;
curl_easy_setopt(curl, CURLOPT_URL, const_cast<char*>(line.c_str()));
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
// curl_easy_setopt(curl, CURLOPT_TIMEOUT, 1L);
res = curl_easy_perform(curl);
// cout << res << endl;
if (res == CURLE_OK ) cout << line << endl;
}
curl_easy_cleanup(curl);
}
CURL *curl;
CURLcode res;
curl_global_init(CURL_GLOBAL_DEFAULT);
curl = curl_easy_init();
if (curl) {
string html;
curl_easy_setopt(curl, CURLOPT_URL, "https://www.google.com");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &html);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writer);
curl_easy_setopt(curl, CURLOPT_CAINFO, "D:\\certification\\DigiCertHighAssuranceEVRootCA.crt");
res = curl_easy_perform(curl);
cout << "Error : " << curl_easy_strerror(res) << endl;
cout << html;
The Error message is Unsupported protocol
The http site is okay but The https site doesn't work
What should I add in curl_easy_setopt()
Hey guys I have the following issue. I've been using C++ to scrape website using to find 5 most frequent words in outputHTML which is string. Currently I have following code. Any hint would be awesome.
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "http://example.com");
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &htmlOutput);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
std::cout << htmlOutput << std::endl;
}
Here's some hints for more awesomeness:
std::istringstream awsome_stream(web_text);
std::string word;
std::map<std::string, unsigned int> kewl_words;
while (awsome_stream >> word)
{
kewl_words[word]++;
}
std::cout << "Occurances of 'div': " << kewl_words["div"] << "\n";
I'm at my wit's end on this one. I need to check if a directory exists on a remote FTP server. Here's what I'm thinking:
//ls - lists the names of the files in the remote directory
string query = "ls /public_html/somefolder/";
//prepare to send
headers = curl_slist_append(headers, query.c_str());
curl_easy_setopt(curl, CURLOPT_QUOTE, headers);
//send query to ftp server
res = curl_easy_perform(curl);
//check result
if(res == CURLE_OK) {
cout << "FOLDER EXISTS";
} else {
cout << "FOLDER DOESN'T EXIST";
}
When I check what the res variable contains, it outputs:
CURLE_QUOTE_ERROR (21).
Any ideas on how to do this correctly? I've searched Google profusely.
Don't try to use CURLOPT_QUOTE for this. A LIST is a data transfer, so it is handled as part of the CURLOPT_URL.
CURL *curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_URL, "ftp://ftp#ftp.gnu.org/pub/");
curl_easy_setopt(curl, CURLOPT_NOBODY, 1L);
CURLcode res = curl_easy_perform(curl);
if(res == CURLE_OK) std::cout << "FOLDER EXISTS\n";
else std::cout << "FOLDER DOESN'T EXIST\n";