I followed a tutorial to fetch a webpage. It worked but they manually set the URL.
I tried changing it to use a URL from a var, but that did not work.
I get an error in the terminal
"Couldn't resolve host name"
I tried main(char*) which gave the same error.
I can't seem to find anything online for this.
How can I make it so a user-defined var can be used as the URL?
code below.
#include <iostream>
#include <curl/curl.h>
#include <string>
using namespace std;
int main() {
string website;
getline(cin, website);
CURL* curl = curl_easy_init();
if (!curl) {
fprintf(stderr, "init failed\n");
return EXIT_FAILURE;
}
// set up
curl_easy_setopt(curl, CURLOPT_URL, "$website");
// perform
CURLcode result = curl_easy_perform(curl);
if (result != CURLE_OK) {
fprintf(stderr, "download prob: %s\n", curl_easy_strerror(result));
}
curl_easy_cleanup(curl);
return EXIT_SUCCESS;
}
"$website" is just a string, a piece of text. The variable should be referenced as website, and since the function is expecting a pointer to an array of characters, you use the c_str() or data() method of the class.
curl_easy_setopt(curl, CURLOPT_URL, website.data());
Related
I did write a c++ code which automatically parses a webpage and open and parse some of their links. The point is that in these webpage there are some addresses which were redirected to other webpages. For example, when I try to open:
https://atlas.immobilienscout24.de/property-by-address?districtId=1276001006014
I ended up opening:
https://atlas.immobilienscout24.de/orte/deutschland/baden-württemberg/böblingen-kreis/leonberg
How could I get the url of the second page in C++?
In that particular case, it's given by the Location header in a 301 ("Moved Permanently") response (according to Chrome's Developer Tools).
If you set FOLLOWLOCATION to 0, you can prevent libcurl from following redirects, and then just examine the headers of the original response (or, probably better, query REDIRECT_URL for the information).
(Then you can perform a new request to the alternative URL, if you like.)
The default for this is 0, though, so you must be setting it to 1 yourself currently.
you could use CURLOPT_HEADERFUNCTION to inspect the headers and parse out the Location header, eg
#include <iostream>
#include <cstring>
#include <curl/curl.h>
size_t header_callback(char *buffer, size_t size, size_t nitems, void *userdata){
const std::string needle="Location: ";
if(nitems>needle.size()){
if(std::memcmp(&needle[0],buffer,needle.size()) == 0 ){
//todo: verify that im not off-by-one below.
((std::string*)userdata)->assign(&buffer[needle.size()],nitems-needle.size());
}
}
return nitems;
}
int main(int argc, char *argv[])
{
CURLcode ret;
CURL *hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_URL, "https://atlas.immobilienscout24.de/property-by-address?districtId=1276001006014");
curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(hnd, CURLOPT_NOBODY, 1L);
std::string redirect_url;
curl_easy_setopt(hnd,CURLOPT_HEADERDATA,&redirect_url);
curl_easy_setopt(hnd,CURLOPT_HEADERFUNCTION,header_callback);
ret = curl_easy_perform(hnd);
curl_easy_cleanup(hnd);
hnd = NULL;
std::cout << redirect_url;
return (int)ret;
}
.. but if you want the final url (in case of multiple redirects), rather than just "the second url", you should probably use CURLOPT_FOLLOWLOCATION and CURLINFO_EFFECTIVE_URL instead, eg
#include <iostream>
#include <cstring>
#include <curl/curl.h>
int main(int argc, char *argv[])
{
CURLcode ret;
CURL *hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_URL, "https://atlas.immobilienscout24.de/property-by-address?districtId=1276001006014");
curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(hnd, CURLOPT_NOBODY, 1L);
curl_easy_setopt(hnd,CURLOPT_FOLLOWLOCATION,1L);
ret = curl_easy_perform(hnd);
char *lolc;
curl_easy_getinfo(hnd, CURLINFO_EFFECTIVE_URL, &lolc);
std::string final_url(lolc);
curl_easy_cleanup(hnd);
hnd = NULL;
std::cout << final_url;
return (int)ret;
}
this approach is slower (have to do at least 1 more request upon redirect), but much simpler to implement and works on both redirected urls and non-redirected urls and multiple-redirected-urls alike.
I'm trying to load the contents of this URL in order to send an SMS;
https://app2.simpletexting.com/v1/send?token=[api key]&phone=[phone number]&message=Weather%20Alert!
Using this bit of code implementing libcurl:
std::string sendSMS(std::string smsMessage, std::string usrID) {
std::string simplePath = "debugOld/libDoc.txt";
std::string preSmsURL = "https://app2.simpletexting.com/v1/send?token=";
std::cout << "\n" << getFile(simplePath) << "\n";
std::string fullSmsURL = preSmsURL + getFile(simplePath) + "&phone=" + usrID + "&message=" + smsMessage;
std::cout << fullSmsURL;
//Outputs URL contents into a file
CURL *curl;
FILE *fd;
CURLcode res;
char newFile[FILENAME_MAX] = "debugOld/noSuccess.md";
curl = curl_easy_init();
if (curl) {
fd = fopen(newFile, "wb");
curl_easy_setopt(curl, CURLOPT_URL, fullSmsURL);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, NULL);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fd);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fd);
}
}
I've used pretty much this exact code before to save the JSON contents of a URL to a file, although I'm trying something a bit different here.
This URL will actually send an SMS when visited. When using curl in a cli, I have no problem doing this. Although from C++, it doesn't treat anything as a error, just perhaps the actual function to send the sms isn't being actuated in the same way it would had I visited the URL physically.
I've scoured google for some kind of a solution to no avail. Perhaps I'm too novice to curl to know exactly what to search for.
Edit #1: getFile function
//Read given file
std::string getFile(std::string path) {
std::string nLine;
std::ifstream file_(path);
if (file_.is_open()) {
while (getline(file_, nLine)) {
return nLine;
}
file_.close();
}
else {
std::cout << "file is not open" << "\n";
return "Error 0x000001: inaccesable file location";
}
return "Unknown error in function 'getFile()'"; //This should never happen
}
This line is wrong:
curl_easy_setopt(curl, CURLOPT_URL, fullSmsURL);
CURLOPT_URL expects a char* pointer to null-terminated C string, not a std::string object. You need to use this instead:
curl_easy_setopt(curl, CURLOPT_URL, fullSmsURL.c_str());
Also, you are not performing any error checking on the return values of getFile(), fopen(), or curl_easy_perform() at all. So, your code could be failing in any one of those places and you would never know it.
I'm trying to teach myself C++ by writing a simple program that sends a cURL request to a JSON API, parses the data and then stores it either in a text document or database for a web application to access. I have done this task in PHP and figured C++ wouldn't be much harder but I can't even get cURL to return a string and display it.
I get this to compile with no errors, but the response "JSON data: " doesn't display anything where the JSON data should be.
Where did I go wrong? URL-to-API is the actual URL, so I believe I'm using a wrong setopt function, or not setting one. In PHP, "CURLOPT_RETURNTRANSFER" made it return as a string, but I get an error with it:
error: ‘CURLOPT_RETURNTRANSFER’ was not declared in this scope
curl_easy_setopt(curl, CURLOPT_RETURNTRANSFER, true);
I'm using g++ compiler on Ubuntu and added -lcurl to the command line argument.
#include <cstdio>
#include <cstdlib>
#include <iostream>
#include <string>
#include <curl/curl.h>
//#include "json.hpp"
using namespace std;
//using json = nlohmann::json;
size_t WriteCallback(char *contents, size_t size, size_t nmemb, void *userp) {
((std::string*)userp)->append((char*)contents, size * nmemb);
return size * nmemb;
}
string getJSON(string URL) {
CURL *curl;
CURLcode res;
string readBuffer;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_HEADER, 1);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, true); // follow redirect
//curl_easy_setopt(curl, CURLOPT_RETURNTRANSFER, true); // return as string
curl_easy_setopt(curl, CURLOPT_HEADER, false);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_URL, URL);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
res = curl_easy_perform(curl);
/* always cleanup */
curl_easy_cleanup(curl);
return readBuffer;
}
return 0;
}
int main() {
string data = getJSON("URL-to-api");
cout << "JSON Data: \n" << data;
return 0;
}
When I uncomment the JSON for Modern C++ include and namespace line I get this error:
error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
Along with a bunch of errors for functions in that library. I just downloaded the most recent version of g++ before embarking on this project, so what do I need to do?
I'm using g++ 5.4.0 on Ubuntu.
UPDATE:
So I added a check under res = curl_easy_perform(curl) and it doesn't return the error message, and res gets displayed as 6. This seems to be much more difficult than it should be:
#include <cstdio>
#include <cstdlib>
#include <iostream>
#include <string>
#include <curl/curl.h>
//#include "json.hpp"
using namespace std;
//using json = nlohmann::json;
size_t WriteCallback(char *contents, size_t size, size_t nmemb, void *userp) {
((std::string*)userp)->append((char*)contents, size * nmemb);
return size * nmemb;
}
string getJSON(string URL) {
CURL *curl;
CURLcode res;
string readBuffer;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_HEADER, 1);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, true); // follow redirect
curl_easy_setopt(curl, CURLOPT_HEADER, false);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_URL, URL);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
res = curl_easy_perform(curl);
cout << res << endl;
if (!res) {
cout << "cURL didn't work\n";
}
/* always cleanup */
curl_easy_cleanup(curl);
curl = NULL;
return readBuffer;
}
}
int main() {
string data = getData("");
cout << "JSON Data: \n" << data;
return 0;
}
I get the following output when I run the program:
6
JSON Data:
In PHP "CURLOPT_RETURNTRANSFER" made it return as a string but I get an error:
error: ‘CURLOPT_RETURNTRANSFER’ was not declared in this scope
curl_easy_setopt(curl, CURLOPT_RETURNTRANSFER, true);
There is no CURLOPT_RETURNTRANSFER option documented for curl_easy_setopt(). I think that is an option specify to PHP's curl_exec() function, which doesn't exist in CURL itself. CURLOPT_WRITEFUNCTION is the correct way to go in this situation.
When I uncomment the JSON for Modern C++ include and namespace line I get:
error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
The error is self-explanatory. Your JSON library requires C++11 but you are not compiling with C++11 enabled. Some modern compilers still default to an older C++ version (usually C++98) and require you to explicitly enable C++11 (or later) when invoking the compiler on the command line, or in your project makefile configuration.
In the case of g++, the current version (8.2) defaults to (the GNU dialect of) C17 for C and C++14 for C++, if not specified otherwise via the -std parameter. Your version (5.4) defaults to (the GNU dialect of) C11 and C++98, respectively.
UPDATE: there are other mistakes in your code:
You are passing a std::string object to curl_easy_setopt() where a char* pointer is expected for CURLOPT_URL. You need to change this:
curl_easy_setopt(curl, CURLOPT_URL, URL);
To this instead:
curl_easy_setopt(curl, CURLOPT_URL, URL.c_str());
You are not testing the return value of curl_easy_perform() correctly. Per the documentation, curl_easy_perform() returns 0 (CURLE_OK) on success, and non-zero on error, so you need to change this:
if (!res)
To this instead:
if (res != CURLE_OK)
So I added a check under res = curl_easy_perform(curl) ..., and res gets displayed as 6.
That is CURLE_COULDNT_RESOLVE_HOST, which makes sense as your updated example is passing a blank URL to getJSON():
string data = getJSON(""); // should be "URL-to-api" instead!
I have the following code in C++
#include <cstdlib>
#include <iostream>
#include <curl/curl.h>
#include <string.h>
using namespace std;
int main(int argc, char *argv[])
{
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "http://example.com/mypage.html");
/* example.com is redirected, so we tell libcurl to follow redirection */
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
system("PAUSE");
return EXIT_SUCCESS;
}
When I'm running this program it is showing the source code of mypage.html on the console. Next I updated the source code of mypage.html and again executed the program, but it was printing the previous source code on the console again. Where's the problem? Please Help.
I faced the same problem recently. But it was with WinINet, not with cURL. Simply I changed my .html file to .php on my server and it worked fine! Actually .php files aren't cached by browser. So give it a try.
I think you rewrite source code of the page, but remember update page on web server or page cashed
Here the output is supposed to be anything but 0.
Because 0 according to libcurl documentation is a success. http://curl.haxx.se/libcurl/c/libcurl-errors.html
curl_easy_setopt(curl_handle, CURLOPT_PROXY, "socks5://127.0.0.1:9050");
And clearly curl_easy_setopt is supposed to return 5 i.e CURLE_COULDNT_RESOLVE_PROXY
#include <iostream>
#include <string>
#include <curl/curl.h>
using namespace std;
int main()
{
CURL *curl_handle;
CURLcode err;
curl_global_init(CURL_GLOBAL_ALL);
curl_handle = curl_easy_init();
err = curl_easy_setopt(curl_handle, CURLOPT_PROXY, "socks5://127.0.0.1:9050");
cout << err;
}
Am i missing something here ?
Thanks for the replies guys.
I looked "CURLE_COULDNT_RESOLVE_PROXY" on google and came across a few examples.
Turns out i had to call curl_easy_perform(curl_handle) after every curl_easy_setopt to get the error codes.
err = curl_easy_perform(curl_handle);
cout << err;