I am trying to develop the auto mail retrieval application in C++.
I just want to know how can I use libcurl to log in the following webpage.
https://webmail.stanford.edu/
where it is redirected to some log in webpage for the authentication.
I am not sure how I can fill the form and send a request through the libcurl. Here's what I have tried.
I realized that log in web page consists following input fields;
2 hidden input fields named "RT" and "ST" (with value keys) --> I stored them in input_field map.
1 text input field for username
1 password input field for password
So I tried the following methods to send a POST request to the server.
post_field.append("RT=").append(input_field["RT"]).append("&");
post_field.append("ST=").append(input_field["ST"]).append("&");
post_field.append("username=MYUSERNAME&password=MYPASSWORD");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, post_field);
curl_easy_setopt(curl, CURLOPT_POST, 1);
res = curl_easy_perform(curl);
However it seems it is not working (I enabled cookie, SSL connection). All I just want to do is fill the form and press submit button through libcurl.
Edit;
After sending the POST request, the retrieved web page says
Error: The URL given is incomplete.
To try again, please return to the preceding web page (or you may need to start your request for this web service again). For help or to report a service problem, contact...
Related
With this minimal program, I can download a text file and print it using libcurl. And I do this anonymously, just like any https get request.
Now this URL: https://api.github.com/repos/bitcoin/bitcoin
Is an example of a Restful API case I don't understand but interests me because I need to retrieve the releases of my project anonymously on Github. If you click on it, you get the JSON information anonymously on your web browser with no problems. It works without any authentication. But then when I put that URL in my minimal program, I get error 403!
So any normal https URL works fine in my minimal program, but this Github URL doesn't.
Why am I getting error 403? Do I have to authenticate in some way (and I don't mean with my personal username and password because my requirement is to get this anonymously)? Is this a bug in Github or my code?
OK. I know now what my mistake is. I seem to have confused authentication username/password with the so called "user agent". Adding this line to libcurl configuration:
curl_easy_setopt(curl, CURLOPT_USERAGENT, "Dark Secret Ninja/1.0");
will make it work. No authentication is required.
I'm trying to perform a file upload operation(which is done using multiple HTTP POST requests). Hence I need to save the cookies from the response of first HTTP POST and set those cookies in the request of the second HTTP POST. I save cookies using CURLINFO_COOKIELIST and set them manually using CURLOPT_COOKIELIST.
CURLcode result = curl_easy_setopt(curlHandle, CURLOPT_COOKIELIST, my_cookies)
This works only if I set the cookies on the same curlHandle. If I close the handles and create new ones after each request, it fails.
Is it not possible to use CURLOPT_COOKIELIST option on different curl handles to execute multiple HTTP requests in the same session ?
Any help is much appreciated.
Update:
I'm trying to save and set the cookies like this. Is there anything wrong I might be doing ?
std::string my_cookies;
// Setting other options using curl_easy_setopt
// To start the cookie engine
curl_easy_setopt(curlHandle,CURLOPT_COOKIEFILE,"");
if (!my_cookies.empty())
{
curl_easy_setopt(curlHandle, CURLOPT_COOKIELIST, my_cookies);
}
curl_easy_perform(curlHandle);
// Save cookies from response of first HTTP POST
struct curl_slist* cookies;
curl_easy_getinfo(curlHandle,CURLINFO_COOKIELIST,&cookies);
// Code to copy cookies to my_cookies.
There's nothing in an extracted cookie list that binds it to that particular easy handle so yes, it can be moved over and inserted into another handle.
I use curl in C++ to download an html page from a website, then I save it.
After I've saved the html file, with another programm I've to read it, and save it in a string.
This page contain some request (POST) made by JSON-AJAX. If I open it with the broswer I have the right content. If I open it with a text editor I have a bad content because the POST request is not made.
So how can I save the page whit the content obtained after JSON-AJAX request??
Curl will download the HTML code from the page and that's it. When you open the HTML file with a web browser, the browser is taking care of whatever post request is being sent.
You need to find out what the post request contains (i.e., the data and how it's obtained) and send that request separately and save the response.
You might want to look into this question How do you make a HTTP request with C++?
I want to use Curl to download and parse data from this website:
http://xetra.com/xetra/dispatch/en/xetraCSV/navigation/xetra/100_market_structure_instruments/100_instruments/100_all_tradable_instruments/
I have used my Curl code on different websites before and it works without issue but this site is different in that it returns a redirect response with an actual link containing the data.
I enabled this setting:
curl_easy_setopt(m_pCurl, CURLOPT_FOLLOWLOCATION, TRUE));
but I get caught in an infinite loop of redirects filling the log file.
To avoid this, I then parsed the initial HTTP response to get the redirect location, and attempt to download using that link. However, Curl tells me the headers and body are empty (CURLE_GOT_NOTHING) and throws. When I visit using a browser I can see the data loading so I know that there is something there, Curl just doesn't seem to be able to see it.
Any help on this issue would be greatly appreciated.
Many thanks,
pma07pg
Many thanks to Captain Giraffe for this answer!
If you have have a redirect link and need to store the cookies then add these options:
curl_easy_setopt(m_pCurl, CURLOPT_MAXREDIRS , 5); // Stop redirecting ad infinitum
curl_easy_setopt(m_pCurl, CURLOPT_COOKIEFILE, "");
You need the JSESSIONID cookie to not get redirected.
Add the cookie you receive on the first request (302 Found) to your headers, repeat the request et viola.
Sample dealing with libcurl Cookies here
hopefully someone can answer this for me as I'm beating my head against the wall. I'm using LibCurl to do a simple file post to an HTTPS URL. Overall, it works, however contacting the site receiving the post, they apparently have an issue resolved, and I just don't get it.
Apparently, I prepare my request, put the URL, provide USER and PASSWORD for the site. The post then includes a CURL_FORMADD( for the parameters to add a single file attachment via multi-part form).
I had problems working out the CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST and finally got it working with the following settings.
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false );
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 1);
Here's the problem. Apparently, when I issue the curl_easy_perform(), it tries to send the request, the response comes back and says ... send me the certificate information and then apparently re-submits the entire package WITH the cert info. And then is accepted by the server I'm sending to.
So, how can I tell LibCurl to submit EVERYTHING... User, Password AND Certificate credentials all up front so the server I'm sending to doesn't reject the first request only to get the credentials the subsequent cycle.
Thanks.
---- EDITING via feedback.
While looking into more as mentioned by Eugen...
I've disabled the "CURLOPT_SSL_VERIFYPEER" from other readings that by doing so can open a transaction up to a "man-in-the-middle" compromise. I don't want that. By finding the method of authentication, it is BASIC, so when I am issuing the connection, I am sending along
curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC );
Now, it comes back with CURLE_SSL_CONNECT_ERROR. I'm trying to get the sdterror, but that's crashing via...
FILE *pLocalFile;
if( fopen_s( &pLocalFile, "MyHTTPS.Log", "w" ))
return -1;
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L );
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteFileCallback );
curl_easy_setopt(curl, CURLOPT_STDERR, pLocalFile );
As soon as I do the perform, I get the DLL caused an exception. So, I cant' even see what the connection failure is about .. which I strongly suspect is certificates.
You need to keep things apart to make the solving of this easier.
First you fix your SSL connection/handshake so that you don't get any SSL related problems or errors.
Then you make sure you provide the necessary HTTP authentication stuff and set basic, and you should only need to do a single HTTP request to get the correct response (assuming that is truly what the server wants).
The "DLL caused an exception" seems to be a third problem and is independent of the other stuff. It's hard to tell why you get that as we don't have a complete picture of what you're doing.