hopefully someone can answer this for me as I'm beating my head against the wall. I'm using LibCurl to do a simple file post to an HTTPS URL. Overall, it works, however contacting the site receiving the post, they apparently have an issue resolved, and I just don't get it.
Apparently, I prepare my request, put the URL, provide USER and PASSWORD for the site. The post then includes a CURL_FORMADD( for the parameters to add a single file attachment via multi-part form).
I had problems working out the CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST and finally got it working with the following settings.
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false );
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 1);
Here's the problem. Apparently, when I issue the curl_easy_perform(), it tries to send the request, the response comes back and says ... send me the certificate information and then apparently re-submits the entire package WITH the cert info. And then is accepted by the server I'm sending to.
So, how can I tell LibCurl to submit EVERYTHING... User, Password AND Certificate credentials all up front so the server I'm sending to doesn't reject the first request only to get the credentials the subsequent cycle.
Thanks.
---- EDITING via feedback.
While looking into more as mentioned by Eugen...
I've disabled the "CURLOPT_SSL_VERIFYPEER" from other readings that by doing so can open a transaction up to a "man-in-the-middle" compromise. I don't want that. By finding the method of authentication, it is BASIC, so when I am issuing the connection, I am sending along
curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC );
Now, it comes back with CURLE_SSL_CONNECT_ERROR. I'm trying to get the sdterror, but that's crashing via...
FILE *pLocalFile;
if( fopen_s( &pLocalFile, "MyHTTPS.Log", "w" ))
return -1;
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L );
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteFileCallback );
curl_easy_setopt(curl, CURLOPT_STDERR, pLocalFile );
As soon as I do the perform, I get the DLL caused an exception. So, I cant' even see what the connection failure is about .. which I strongly suspect is certificates.
You need to keep things apart to make the solving of this easier.
First you fix your SSL connection/handshake so that you don't get any SSL related problems or errors.
Then you make sure you provide the necessary HTTP authentication stuff and set basic, and you should only need to do a single HTTP request to get the correct response (assuming that is truly what the server wants).
The "DLL caused an exception" seems to be a third problem and is independent of the other stuff. It's hard to tell why you get that as we don't have a complete picture of what you're doing.
Related
With this minimal program, I can download a text file and print it using libcurl. And I do this anonymously, just like any https get request.
Now this URL: https://api.github.com/repos/bitcoin/bitcoin
Is an example of a Restful API case I don't understand but interests me because I need to retrieve the releases of my project anonymously on Github. If you click on it, you get the JSON information anonymously on your web browser with no problems. It works without any authentication. But then when I put that URL in my minimal program, I get error 403!
So any normal https URL works fine in my minimal program, but this Github URL doesn't.
Why am I getting error 403? Do I have to authenticate in some way (and I don't mean with my personal username and password because my requirement is to get this anonymously)? Is this a bug in Github or my code?
OK. I know now what my mistake is. I seem to have confused authentication username/password with the so called "user agent". Adding this line to libcurl configuration:
curl_easy_setopt(curl, CURLOPT_USERAGENT, "Dark Secret Ninja/1.0");
will make it work. No authentication is required.
I'm trying to perform a file upload operation(which is done using multiple HTTP POST requests). Hence I need to save the cookies from the response of first HTTP POST and set those cookies in the request of the second HTTP POST. I save cookies using CURLINFO_COOKIELIST and set them manually using CURLOPT_COOKIELIST.
CURLcode result = curl_easy_setopt(curlHandle, CURLOPT_COOKIELIST, my_cookies)
This works only if I set the cookies on the same curlHandle. If I close the handles and create new ones after each request, it fails.
Is it not possible to use CURLOPT_COOKIELIST option on different curl handles to execute multiple HTTP requests in the same session ?
Any help is much appreciated.
Update:
I'm trying to save and set the cookies like this. Is there anything wrong I might be doing ?
std::string my_cookies;
// Setting other options using curl_easy_setopt
// To start the cookie engine
curl_easy_setopt(curlHandle,CURLOPT_COOKIEFILE,"");
if (!my_cookies.empty())
{
curl_easy_setopt(curlHandle, CURLOPT_COOKIELIST, my_cookies);
}
curl_easy_perform(curlHandle);
// Save cookies from response of first HTTP POST
struct curl_slist* cookies;
curl_easy_getinfo(curlHandle,CURLINFO_COOKIELIST,&cookies);
// Code to copy cookies to my_cookies.
There's nothing in an extracted cookie list that binds it to that particular easy handle so yes, it can be moved over and inserted into another handle.
I am using libcurl with C++ to send an HTTP request to another host (my friend's computer) over the Internet. Now, the other end uses a Node.js script to receive and handle the request.
I have tried the code (both C++ and Node) on my local network between my computers, and everything works just fine. This is the code, in case you wanna look at it:
CURL *handle;
string response;
curl_global_init(CURL_GLOBAL_WIN32);
handle = curl_easy_init();
curl_easy_setopt(handle, CURLOPT_URL, "10.10.10.10:7890"); //<= IP:PORT of the remote host
curl_easy_setopt(handle, CURLOPT_POSTFIELDS, "This is my POST request data!!");
curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, write_to_string);
curl_easy_setopt(handle, CURLOPT_WRITEDATA, &response);
curl_easy_perform(handle);
curl_easy_cleanup(handle);
and the Node.js code:
var http = require("http");
var server = http.createServer(function(request, response) {
request.on('data', function(chunk) {
console.log("We got a connection!");
console.log("Received POST data:");
console.log(chunk.toString());
});
});
server.listen(7890);
console.log("Server is listening");
As I said, on my local network it works very well, but why it doesn't my friend receive the request? I noticed that my code (C++ part) takes some time, before closing, as if it was trying to connect. Is this some kind of firewall problem? We have tried disabling it, but nothing changed. We are almost total noobs with networking, and we're trying to get into it.
If anybody could help us, thanks in advance!
Your friend's home router needs to know what to do with a connection being requested on port 7890 from the internet. It needs to be configured to forward any communication on that port to the specific internal IP address of your friend's computer.
Not all home routers are capable of port forwarding and many manufacturers have different ways of getting to the settings.
Any software firewall on the friend's computer also needs to know that communication over port 7890 is safe.
I am trying to develop the auto mail retrieval application in C++.
I just want to know how can I use libcurl to log in the following webpage.
https://webmail.stanford.edu/
where it is redirected to some log in webpage for the authentication.
I am not sure how I can fill the form and send a request through the libcurl. Here's what I have tried.
I realized that log in web page consists following input fields;
2 hidden input fields named "RT" and "ST" (with value keys) --> I stored them in input_field map.
1 text input field for username
1 password input field for password
So I tried the following methods to send a POST request to the server.
post_field.append("RT=").append(input_field["RT"]).append("&");
post_field.append("ST=").append(input_field["ST"]).append("&");
post_field.append("username=MYUSERNAME&password=MYPASSWORD");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, post_field);
curl_easy_setopt(curl, CURLOPT_POST, 1);
res = curl_easy_perform(curl);
However it seems it is not working (I enabled cookie, SSL connection). All I just want to do is fill the form and press submit button through libcurl.
Edit;
After sending the POST request, the retrieved web page says
Error: The URL given is incomplete.
To try again, please return to the preceding web page (or you may need to start your request for this web service again). For help or to report a service problem, contact...
I want to use Curl to download and parse data from this website:
http://xetra.com/xetra/dispatch/en/xetraCSV/navigation/xetra/100_market_structure_instruments/100_instruments/100_all_tradable_instruments/
I have used my Curl code on different websites before and it works without issue but this site is different in that it returns a redirect response with an actual link containing the data.
I enabled this setting:
curl_easy_setopt(m_pCurl, CURLOPT_FOLLOWLOCATION, TRUE));
but I get caught in an infinite loop of redirects filling the log file.
To avoid this, I then parsed the initial HTTP response to get the redirect location, and attempt to download using that link. However, Curl tells me the headers and body are empty (CURLE_GOT_NOTHING) and throws. When I visit using a browser I can see the data loading so I know that there is something there, Curl just doesn't seem to be able to see it.
Any help on this issue would be greatly appreciated.
Many thanks,
pma07pg
Many thanks to Captain Giraffe for this answer!
If you have have a redirect link and need to store the cookies then add these options:
curl_easy_setopt(m_pCurl, CURLOPT_MAXREDIRS , 5); // Stop redirecting ad infinitum
curl_easy_setopt(m_pCurl, CURLOPT_COOKIEFILE, "");
You need the JSESSIONID cookie to not get redirected.
Add the cookie you receive on the first request (302 Found) to your headers, repeat the request et viola.
Sample dealing with libcurl Cookies here