I'm trying to teach myself C++ by writing a simple program that sends a cURL request to a JSON API, parses the data and then stores it either in a text document or database for a web application to access. I have done this task in PHP and figured C++ wouldn't be much harder but I can't even get cURL to return a string and display it.
I get this to compile with no errors, but the response "JSON data: " doesn't display anything where the JSON data should be.
Where did I go wrong? URL-to-API is the actual URL, so I believe I'm using a wrong setopt function, or not setting one. In PHP, "CURLOPT_RETURNTRANSFER" made it return as a string, but I get an error with it:
error: ‘CURLOPT_RETURNTRANSFER’ was not declared in this scope
curl_easy_setopt(curl, CURLOPT_RETURNTRANSFER, true);
I'm using g++ compiler on Ubuntu and added -lcurl to the command line argument.
#include <cstdio>
#include <cstdlib>
#include <iostream>
#include <string>
#include <curl/curl.h>
//#include "json.hpp"
using namespace std;
//using json = nlohmann::json;
size_t WriteCallback(char *contents, size_t size, size_t nmemb, void *userp) {
((std::string*)userp)->append((char*)contents, size * nmemb);
return size * nmemb;
}
string getJSON(string URL) {
CURL *curl;
CURLcode res;
string readBuffer;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_HEADER, 1);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, true); // follow redirect
//curl_easy_setopt(curl, CURLOPT_RETURNTRANSFER, true); // return as string
curl_easy_setopt(curl, CURLOPT_HEADER, false);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_URL, URL);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
res = curl_easy_perform(curl);
/* always cleanup */
curl_easy_cleanup(curl);
return readBuffer;
}
return 0;
}
int main() {
string data = getJSON("URL-to-api");
cout << "JSON Data: \n" << data;
return 0;
}
When I uncomment the JSON for Modern C++ include and namespace line I get this error:
error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
Along with a bunch of errors for functions in that library. I just downloaded the most recent version of g++ before embarking on this project, so what do I need to do?
I'm using g++ 5.4.0 on Ubuntu.
UPDATE:
So I added a check under res = curl_easy_perform(curl) and it doesn't return the error message, and res gets displayed as 6. This seems to be much more difficult than it should be:
#include <cstdio>
#include <cstdlib>
#include <iostream>
#include <string>
#include <curl/curl.h>
//#include "json.hpp"
using namespace std;
//using json = nlohmann::json;
size_t WriteCallback(char *contents, size_t size, size_t nmemb, void *userp) {
((std::string*)userp)->append((char*)contents, size * nmemb);
return size * nmemb;
}
string getJSON(string URL) {
CURL *curl;
CURLcode res;
string readBuffer;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_HEADER, 1);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, true); // follow redirect
curl_easy_setopt(curl, CURLOPT_HEADER, false);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_URL, URL);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
res = curl_easy_perform(curl);
cout << res << endl;
if (!res) {
cout << "cURL didn't work\n";
}
/* always cleanup */
curl_easy_cleanup(curl);
curl = NULL;
return readBuffer;
}
}
int main() {
string data = getData("");
cout << "JSON Data: \n" << data;
return 0;
}
I get the following output when I run the program:
6
JSON Data:
In PHP "CURLOPT_RETURNTRANSFER" made it return as a string but I get an error:
error: ‘CURLOPT_RETURNTRANSFER’ was not declared in this scope
curl_easy_setopt(curl, CURLOPT_RETURNTRANSFER, true);
There is no CURLOPT_RETURNTRANSFER option documented for curl_easy_setopt(). I think that is an option specify to PHP's curl_exec() function, which doesn't exist in CURL itself. CURLOPT_WRITEFUNCTION is the correct way to go in this situation.
When I uncomment the JSON for Modern C++ include and namespace line I get:
error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
The error is self-explanatory. Your JSON library requires C++11 but you are not compiling with C++11 enabled. Some modern compilers still default to an older C++ version (usually C++98) and require you to explicitly enable C++11 (or later) when invoking the compiler on the command line, or in your project makefile configuration.
In the case of g++, the current version (8.2) defaults to (the GNU dialect of) C17 for C and C++14 for C++, if not specified otherwise via the -std parameter. Your version (5.4) defaults to (the GNU dialect of) C11 and C++98, respectively.
UPDATE: there are other mistakes in your code:
You are passing a std::string object to curl_easy_setopt() where a char* pointer is expected for CURLOPT_URL. You need to change this:
curl_easy_setopt(curl, CURLOPT_URL, URL);
To this instead:
curl_easy_setopt(curl, CURLOPT_URL, URL.c_str());
You are not testing the return value of curl_easy_perform() correctly. Per the documentation, curl_easy_perform() returns 0 (CURLE_OK) on success, and non-zero on error, so you need to change this:
if (!res)
To this instead:
if (res != CURLE_OK)
So I added a check under res = curl_easy_perform(curl) ..., and res gets displayed as 6.
That is CURLE_COULDNT_RESOLVE_HOST, which makes sense as your updated example is passing a blank URL to getJSON():
string data = getJSON(""); // should be "URL-to-api" instead!
Related
I did write a c++ code which automatically parses a webpage and open and parse some of their links. The point is that in these webpage there are some addresses which were redirected to other webpages. For example, when I try to open:
https://atlas.immobilienscout24.de/property-by-address?districtId=1276001006014
I ended up opening:
https://atlas.immobilienscout24.de/orte/deutschland/baden-württemberg/böblingen-kreis/leonberg
How could I get the url of the second page in C++?
In that particular case, it's given by the Location header in a 301 ("Moved Permanently") response (according to Chrome's Developer Tools).
If you set FOLLOWLOCATION to 0, you can prevent libcurl from following redirects, and then just examine the headers of the original response (or, probably better, query REDIRECT_URL for the information).
(Then you can perform a new request to the alternative URL, if you like.)
The default for this is 0, though, so you must be setting it to 1 yourself currently.
you could use CURLOPT_HEADERFUNCTION to inspect the headers and parse out the Location header, eg
#include <iostream>
#include <cstring>
#include <curl/curl.h>
size_t header_callback(char *buffer, size_t size, size_t nitems, void *userdata){
const std::string needle="Location: ";
if(nitems>needle.size()){
if(std::memcmp(&needle[0],buffer,needle.size()) == 0 ){
//todo: verify that im not off-by-one below.
((std::string*)userdata)->assign(&buffer[needle.size()],nitems-needle.size());
}
}
return nitems;
}
int main(int argc, char *argv[])
{
CURLcode ret;
CURL *hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_URL, "https://atlas.immobilienscout24.de/property-by-address?districtId=1276001006014");
curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(hnd, CURLOPT_NOBODY, 1L);
std::string redirect_url;
curl_easy_setopt(hnd,CURLOPT_HEADERDATA,&redirect_url);
curl_easy_setopt(hnd,CURLOPT_HEADERFUNCTION,header_callback);
ret = curl_easy_perform(hnd);
curl_easy_cleanup(hnd);
hnd = NULL;
std::cout << redirect_url;
return (int)ret;
}
.. but if you want the final url (in case of multiple redirects), rather than just "the second url", you should probably use CURLOPT_FOLLOWLOCATION and CURLINFO_EFFECTIVE_URL instead, eg
#include <iostream>
#include <cstring>
#include <curl/curl.h>
int main(int argc, char *argv[])
{
CURLcode ret;
CURL *hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_URL, "https://atlas.immobilienscout24.de/property-by-address?districtId=1276001006014");
curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(hnd, CURLOPT_NOBODY, 1L);
curl_easy_setopt(hnd,CURLOPT_FOLLOWLOCATION,1L);
ret = curl_easy_perform(hnd);
char *lolc;
curl_easy_getinfo(hnd, CURLINFO_EFFECTIVE_URL, &lolc);
std::string final_url(lolc);
curl_easy_cleanup(hnd);
hnd = NULL;
std::cout << final_url;
return (int)ret;
}
this approach is slower (have to do at least 1 more request upon redirect), but much simpler to implement and works on both redirected urls and non-redirected urls and multiple-redirected-urls alike.
I want to download a file from a dropbox shared link using curl in a c++ program
I found a dropbox api pdf that showed me how to do it
#include <stdio.h>
#include <curl/curl.h>
int main (int argc, char *argv[])
{
CURL *curl;
CURLcode res;
/* In windows, this will init the winsock stuff */
curl_global_init(CURL_GLOBAL_ALL);
/* get a curl handle */
curl = curl_easy_init();
if(curl) {
printf ("Running curl test.\n");
struct curl_slist *headers=NULL; /* init to NULL is important */
headers = curl_slist_append(headers, "Authorization: Bearer
<ACCESS_TOKEN>");
headers = curl_slist_append(headers, "Content-Type:");
headers = curl_slist_append(headers, "Dropbox-API-Arg:
{\"path\":\"/test.txt\"}");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_URL,
"https://content.dropboxapi.com/2/files/download");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "");
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
printf ("\nFinished curl test.\n");
}
curl_global_cleanup();
printf ("Done!\n");
return 0;
}
However, the comments supplied don't offer much explanation for me, and I can't get it to work.
I don't understand these three lines of code:
headers = curl_slist_append(headers, "Authorization: Bearer <ACCESS_TOKEN>");
headers = curl_slist_append(headers, "Content-Type:");
headers = curl_slist_append(headers, "Dropbox-API-Arg:{\"path\":\"/test.txt\"}");
I think I have to replace some stuff but I don't know what
"I think I have to replace some stuff but I don't know what" : Replace <ACCESS_TOKEN> with your actual access token.
You should also set the "Content-Type:" header to an appropriate value for the data you are fetching.
You must also change the value of the "Dropbox-API-Arg" header to match the file you are trying to get.
I finally found the solution to my problem.
Turns out I didn't have to use the Dropbox API
Here is the code
#include <iostream>
#include <curl/curl.h>
using namespace std;
size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream)
{
size_t written;
written = fwrite(ptr, size, nmemb, stream);
return written;
}
int main(int argc, char** argv) {
CURL *curl;
FILE *fp;
const char* destination = "D:\\Desktop\\test.exe";
fp = fopen(destination, "wb");
curl = curl_easy_init();
/* A long parameter set to 1 tells the library to follow any Location: header
* that the server sends as part of an HTTP header in a 3xx response. The
*Location: header can specify a relative or an absolute URL to follow.
*/
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_URL, "https://www.dropbox.com/s/09nd26tdyto23yz/BankAccount.exe?dl=1"); // "dl=0"changed to "dl=1" to force download
// disabe the SSL peer certificate verification allowing the program to download the file from dropbox shared link
// in case it is not used it displays an error message stating "SSL peer certificate or SSH remote key was not OK"
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
CURLcode res;
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
if (res ==CURLE_OK)
cout << "OK";
else
cout << curl_easy_strerror(res);
return 0;
}
Thanks you guys for trying to help me. I appreciate
I am new to c++. I am trying a tiny piece of code to interact with a local instance of InfluxDB.
#include <iostream>
#include <stdio.h>
#include <string.h>
#include <curl/curl.h>
#include <curl/curlver.h>
#include <curl/easy.h>
#include <curl/mprintf.h>
#include <curl/multi.h>
#include <curl/stdcheaders.h>
#include <curl/system.h>
#include <curl/typecheck-gcc.h>
using namespace std;
bool createInfluxDB(char *url, char *data) {
CURL *curl;
curl = curl_easy_init();
if(curl) {
CURLcode res;
/* What Content-type should i use?*/
struct curl_slist* headers = curl_slist_append(headers, "Content-Type: application/json");
/*--data-urlencode*/
char *urlencoded = curl_easy_escape(curl, data, int(strlen(data)));
curl_easy_setopt(curl, CURLOPT_URL, url); // Error here
curl_easy_setopt(curl, CURLOPT_POST, 1L);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, urlencoded);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, (long)strlen(urlencoded));
res = curl_easy_perform(curl);
/*omitted controls*/
curl_free(urlencoded);
curl_slist_free_all(headers);
curl_easy_cleanup(curl);
}
return(true);
}
int main(int argc, char *argv[]){
char *url = "http://localhost:8086/query";
char *data = "q=CREATE DATABASE mydb";
/* should i change data string to json?
data = "{\"q\":\"CREATE DATABASE mydb\" }" */
bool res = createInfluxDB(url, data);
/*control result*/
return(0);
}
When trying to build it, i get the following errors:
expected primary-expression before '__typeof__'
extend list of errors
Any ideas where it might come from ?
I am working on codeblocks 16.01, using GCC compiler on Ubuntu.
That was due to curl library not being linked to the compiler. The correct way to fix this in Codeblocks editor:
Project>Build options..>"Linker settings" tab
Under Link libraries, click on add, type the library name on the popping window, click OK, click OK again. And reboot Codeblocks.
When running my code (pertinent sections pasted below), I periodically get the following error:
program(34010,0x70000e58b000) malloc: *** error for object
0x7fc43d93fcf0: pointer being freed was not allocated set a breakpoint
in malloc_error_break to debug Signal: SIGABRT (signal SIGABRT)
I am running multi-threaded C++ code on a Macbook (OS-10.13) wherein different threads make use of the code in question simultaneously. To my knowledge, libcurl is indeed thread safe as long as I do not utilize the same "curl handle" (which I understand to be an instance of "CURL" aka "CURL *curl = curl_easy_init();") in two different threads at the same time. In my case, since each thread calls the function separately and initializes a new instance of a CURL object, I should be "safe", right? Hopefully there is something obvious that I'm missing that is causing me (or lib curl in this case) to attempt to free memory that has already been freed. If there is any more information I should have included (below) please don't hesitate to let me know.
The function that seg faults is
string http_lib::make_get_request(string url)
on the line that reads
curl_easy_cleanup(curl);
and sometimes (less often) on the line that reads
res = curl_easy_perform(curl);
Below is what I think would be the pertinent sections of my code:
size_t http_lib::CurlWrite_CallbackFunc_StdString(void *contents, size_t size, size_t nmemb, std::string *s)
{
size_t newLength = size*nmemb;
size_t oldLength = s->size();
try
{
s->resize(oldLength + newLength);
}
catch(std::bad_alloc &e)
{
//handle memory problem
return 0;
}
std::copy((char*)contents,(char*)contents+newLength,s->begin()+oldLength);
return size*nmemb;
}
string http_lib::make_post_request(string url, vector<string> headers, string post_params) {
CURL *curl;
CURLcode res;
curl = curl_easy_init();
string s;
if(curl)
{
struct curl_slist *chunk = NULL;
for(int i=0; i<headers.size(); i++){
/* Add a custom header */
chunk = curl_slist_append(chunk, headers[i].c_str());
}
/* set our custom set of headers */
res = curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, post_params.c_str());
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); //only for https
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); //only for https
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, CurlWrite_CallbackFunc_StdString);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &s);
if(networking_debug){
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); //verbose output
}
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
{
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
}
/* always cleanup */
curl_easy_cleanup(curl);
}
// Debug output
if (networking_debug){
cout<<"Response: " << s <<endl;
}
return s;
}
string http_lib::make_get_request(string url) {
//SslCurlWrapper sslObject;
CURL *curl;
CURLcode res;
curl = curl_easy_init();
string s;
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
//tell libcurl to follow redirection
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); //only for https
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); //only for https
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, CurlWrite_CallbackFunc_StdString);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &s);
if(networking_debug){
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); //verbose output
}
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if (res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
if (networking_debug){
cout << "Response: " << s << endl;
}
return s;
}
In main() I have
int main(int argc, char *argv[]){
// Initialize http_lib (curl)
curl_global_init(CURL_GLOBAL_DEFAULT);
... spin up 10 or so threads that make get/post requests to https site (some requests utilize the make_post_request() function and others utilize make_get_requet() function).
}
CMAKE doesn't/didn't seem to want to use anything other than CURL_ROOT_DIR of "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include" for libcurl (aka curl).
Thus it was using the curl lib that mac (and/or Xcode) ships with. I haven't figured out what version that is, but I can say that not using it and instead using CURL version 7.57 is what fixed my issue.
I used "brew" package manager to
brew install curl
Doing so created /usr/local/Cellar/curl/7.57.0 directory and put all libs/includes in there.
Then I added
-I/usr/local/Cellar/curl/7.57.0/include -L/usr/local/Cellar/curl/7.57.0/lib
to my CMAKE CMAKE_CXX_FLAGS.
TLDR; Solution was to ensure I was using the newest version of the curl lib. Now that I am, no problem.
I am currently trying to make an updater for my software project. I need it to be able to download multiple files, I don't mind if they download in sync or one after each other, whatever is easier (file size is not an issue). I followed the example from the libcurl webpage and a few other resources and came up with this:
#include <iostream>
#include <stdio.h>
#include <curl/curl.h>
#include <string.h>
size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
size_t written;
written = fwrite(ptr, size, nmemb, stream);
return written;
}
int main(void){
for (int i = 0; i < 2;){ //download 2 files (loop twice)
CURL *curl;
FILE *fp;
CURLcode res;
char *url = "http://sec7.org/1024kb.txt"; //first file URL
char outfilename[FILENAME_MAX] = "C:\\users\\grant\\desktop\\1024kb.txt";
curl = curl_easy_init();
if (curl){
fp = fopen(outfilename,"wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
}
url = "http://sec7.org/index.html"; //I want to get a new file this time
outfilename[FILENAME_MAX] = "C:\\users\\grant\\desktop\\index.html";
}
return 0;
}
The first issue is if i remove the new file assignments (*url = "http://...") and just try to loop the download code twice, the program simply stops responding. This occurs in any combination of the download being called more than once in the program. The other issue is that I am unable to change the value of the character array outfilename[FILENAME_MAX]. I feel like this is just some silly error I am making but no solution comes to mind. Thank you!
Why not put this in a function and call it twice?
Your syntax for the arrays is all wrong, plus all the variables inside the loop are local, which means they are destroyed after each loop iteration.
What Conspicuous Compiler said. That's what's causing your program to freeze; it's stuck in an infinite loop because i is never > 2.
Put your code into a function like so:
void downloadFile(const char* url, const char* fname) {
CURL *curl;
FILE *fp;
CURLcode res;
curl = curl_easy_init();
if (curl){
fp = fopen(fname, "wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
}
}
And call it twice with the relevant file names and urls:
downloadFile("http://sec7.org/1024kb.txt", "C:\\users\\grant\\desktop\\1024kb.txt");
downloadFile("http://sec7.org/index.html", "C:\\users\\grant\\desktop\\index.html");
The example function is very bad though, it's just an example. You should alter it to return error codes/throw exceptions, and stuff like that.