IDN support not enabled for cURL - c++

I am having some issue getting an image into a dynamic memory buffer with cURL.
The bit of code used is the following:
struct memoryStruct {
char *memory;
size_t size;
};
static void* CURL_realloc(void *ptr, size_t size)
{
/* There might be a realloc() out there that doesn't like reallocing
NULL pointers, so we take care of it here */
if(ptr)
return realloc(ptr, size);
else
return malloc(size);
}
size_t WriteMemoryCallback(void *ptr, size_t size, size_t nmemb, void *data)
{
size_t realsize = size * nmemb;
struct memoryStruct *mem = (struct memoryStruct *)data;
mem->memory = (char *)CURL_realloc(mem->memory, mem->size + realsize + 1);
if (mem->memory)
{
memcpy(&(mem->memory[mem->size]), ptr, realsize);
mem->size += realsize;
mem->memory[mem->size] = 0;
}
return realsize;
}
int main()
{
std::string everything = "https://upload.wikimedia.org/wikipedia/commons/1/1e/Stonehenge.jpg";
CURL *curl; // CURL objects
CURLcode res;
memoryStruct buffer; // memory buffer
curl = curl_easy_init(); // init CURL library object/structure
if(curl) {
// set up the write to memory buffer
// (buffer starts off empty)
buffer.memory = NULL;
buffer.size = 0;
// (N.B. check this URL still works in browser in case image has moved)
curl_easy_setopt(curl, CURLOPT_URL, everything);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1); // tell us what is happening
// tell libcurl where to write the image (to a dynamic memory buffer)
curl_easy_setopt(curl,CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
curl_easy_setopt(curl,CURLOPT_WRITEDATA, (void *) &buffer);
// get the image from the specified URL
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
free(buffer.memory);
}
return 0;
}
As an error output I get the error saying the following:
I am however not sure to understand the error being displayed. My guess is that I should enable IDN support for curl? I am however not sure on how to proceed. Does that mean that I would have to recompile the library with IDN enabled? (If I find how to do it)
Thank you for your help

You're passing an std::string object to curl_easy_setopt, while you should be passing a C string. It's actually trying to resolve god-knows-what.
curl_easy_setopt(curl, CURLOPT_URL, everything.c_str());

Related

cURL write_callback does not pass userdata argument

I am trying to collect some data from a URL. If I do not define any CURLOPT_WRITEFUNCTION and CURLOPT_WRITEDATA I can obviously see the output on console. Then I tried to write that data to memory by copiying the example code, however userdata argument of my callback function returned NULL and I got following exception on line:
char* ptr = (char*)realloc(mem->memory, mem->size + realsize + 1);
Exception thrown: read access violation.
mem was nullptr.
Am I doing something wrong?
Here is my code:
struct MemoryStruct {
char* memory;
size_t size;
};
//-----------------
// Curl's callback
//-----------------
size_t CurlWrapper::curl_cb(char* data, size_t size, size_t nmemb, void* response)
{
size_t realsize = size * nmemb;
std::cout << "CALLBACK CALLED" << std::endl;
MemoryStruct* mem = (struct MemoryStruct*)response;
char* ptr = (char*)realloc(mem->memory, mem->size + realsize + 1);
if (!ptr) {
/* out of memory! */
printf("not enough memory (realloc returned NULL)\n");
return 0;
}
mem->memory = ptr;
memcpy(&(mem->memory[mem->size]), data, realsize);
mem->size += realsize;
mem->memory[mem->size] = 0;
return realsize;
}
//--------------------
// Do the curl
//--------------------
void CurlWrapper::curl_api(
const std::string& url,
std::string& str_result)
{
MemoryStruct chunk;
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &CurlWrapper::curl_cb);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void*)&chunk);
// TODO: enable ssh certificate
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false); // true
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, false); // 2
curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, "zlib");
auto res = curl_easy_perform(curl);
/* Check for errors */
if (res != CURLE_OK) {
// nothing
std::cout << "curl_easy_perform() failed: " << curl_easy_strerror(res) << std::endl;
}
}
}
libcurl version: 7.82.0
Since libcurl is a C library, it does not know anything about C++ member functions or objects. You can overcome this "limitation" with relative ease using for example a static member function that is passed a pointer to the class.
See this example (from the everything curl book).
// f is the pointer to your object.
static size_t YourClass::func(void *buffer, size_t sz, size_t n, void *f)
{
// Call non-static member function.
static_cast<YourClass*>(f)->nonStaticFunction();
}
// This is how you pass pointer to the static function:
curl_easy_setopt(hcurl, CURLOPT_WRITEFUNCTION, YourClass::func);
curl_easy_setopt(hcurl, CURLOPT_WRITEDATA, this);

c++ Libcurl ftp download/upload in memory

I am trying to download/(and later upload it again) a json file from a ftp server(provider: bplaced.net) into memory using LibCurl, but im getting an error.
Please don't be to harsh im pretty new to this. Also feel free to improve my code thanks :p
Memory methods:
struct MemoryStruct {
char* memory;
size_t size;
};
static size_t
WriteMemoryCallback(void* contents, size_t size, size_t nmemb, void* userp)
{
size_t realsize = size * nmemb;
struct MemoryStruct* mem = (struct MemoryStruct*)userp;
char* ptr = (char*)realloc(mem->memory, mem->size + realsize + 1);
if (ptr == NULL) {
/* out of memory! */
printf("not enough memory (realloc returned NULL)\n");
return 0;
}
mem->memory = ptr;
memcpy(&(mem->memory[mem->size]), contents, realsize);
mem->size += realsize;
mem->memory[mem->size] = 0;
return realsize;
}
Download Method:
std::string getFTPFileMem() {
CURL* curl;
CURLcode res;
std::string memory;
struct MemoryStruct chunk;
chunk.memory = (char*)malloc(1); /* will be grown as needed by the realloc above */
chunk.size = 0; /* no data at this point */
curl_global_init(CURL_GLOBAL_DEFAULT);
curl = curl_easy_init();
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL,
"ftp://user:pass#host/jsonLocation/Json.json");
/* send all data to this function */
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
/* we pass our 'chunk' struct to the callback function */
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void*)&chunk);
/* We activate SSL and we require it for both control and data */
curl_easy_setopt(curl, CURLOPT_USE_SSL, CURLUSESSL_ALL);
/* Switch on full protocol/debug output */
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
res = curl_easy_perform(curl);
memory = chunk.memory;
/* always cleanup */
curl_easy_cleanup(curl);
if (CURLE_OK != res) {
/* we failed */
fprintf(stderr, "curl told us %d\n", res);
}
}
free(chunk.memory);
/* we're done with libcurl, so clean it up */
curl_global_cleanup();
return memory;
}
int main() {
std::cout << getFTPFileMem() << "\n";
}
FTP certificate issue:

C++ Object value suddenly becomes NULL with no related prior instruction

NOTE: Reopened question with more details.
So here we have some code
someCode.cpp
(...)
MyStruct sct = performRequest(.... .... ....); //Line 1
std::cout << "Line 2";
std::cout << "Line 3";
(...)
someCode.hpp (for struct MyStruct)
(...)
struct MyStruct {
char** memory;
int* response_code;
};
(...)
performRequest() function
static size_t
WriteMemoryCallback(void* contents, size_t size, size_t nmemb, void* userp)
{
size_t realsize = size * nmemb;
struct MemoryStruct* mem = (struct MemoryStruct*)userp;
char* ptr = (char*) realloc(mem->memory, mem->size + realsize + 1);
if (ptr == NULL) {
/* out of memory! */
printf("not enough memory (realloc returned NULL)\n");
return 0;
}
mem->memory = ptr;
memcpy(&(mem->memory[mem->size]), contents, realsize);
mem->size += realsize;
mem->memory[mem->size] = 0;
return realsize;
}
MyStruct performRequest(std::string requestType, std::string URL, std::string requestParameters, struct curl_slist* headersList, std::string requestFields) {
CURL* curl_handle = curl_easy_init();
CURLcode curl_code;
struct MemoryStruct chunk;
chunk.memory = (char*) malloc(1); /* will be grown as needed by the realloc above */
chunk.size = 0; /* no data at this point */
curl_easy_setopt(curl_handle, CURLOPT_BUFFERSIZE, 102400L);
curl_easy_setopt(curl_handle, CURLOPT_URL, (URL + requestParameters).c_str());
curl_easy_setopt(curl_handle, CURLOPT_NOPROGRESS, true);
curl_easy_setopt(curl_handle, CURLOPT_HEADER, false);
if (!requestFields.empty()) {
curl_easy_setopt(curl_handle, CURLOPT_POSTFIELDS, requestFields.c_str());
}
if (headersList != NULL) {
curl_easy_setopt(curl_handle, CURLOPT_HTTPHEADER, headersList);
}
curl_easy_setopt(curl_handle, CURLOPT_MAXREDIRS, 50L);
curl_easy_setopt(curl_handle, CURLOPT_CUSTOMREQUEST, requestType.c_str());
curl_easy_setopt(curl_handle, CURLOPT_VERBOSE, false);
curl_easy_setopt(curl_handle, CURLOPT_TCP_KEEPALIVE, true);
/* send all data to this function, we pass our 'chunk' struct to the callback function */
curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, (void*)& chunk);
curl_code = curl_easy_perform(curl_handle);
/* check for errors */
if (curl_code != CURLE_OK) {
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(curl_code));
}
int response_code;
curl_easy_getinfo(curl_handle, CURLINFO_RESPONSE_CODE, &response_code);
/* cleanup curl stuff */
curl_easy_cleanup(curl_handle);
MyStruct res;
res.memory = &chunk.memory;
res.response_code = &response_code;
return res;
}
If we debug someCode.cpp and add two breakpoints in lines 2 and 3...
(...)
MyStruct sct = performRequest(···,···,···); //Line 1
[BREAKPOINT (1) HERE] std::cout << "Line 2";
[BREAKPOINT (2) HERE] std::cout << "Line 3";
(...)
...see the following images about the resulting values by the debugger, for the struct sct.
(which corresponds to Breakpoint 1) :)
(which corresponds to Breakpoint 2) :(
As you can see, the values in sct change to NULL without prior instruction. I don't know why this happens, although I suspect it might be something pointer-related...
So to deal with the object lifetime (sct), I changed performRequest() to this:
RequestResponse* performRequest(···) {
(···)
return new RequestResponse(&chunk.memory,&response_code);
}
but still I have the same problem.
Hope you have any idea and thanks!
Your issue is right here:
MyStruct res;
res.memory = &chunk.memory;
res.response_code = &response_code;
return res;
You're supplying res with the address of two local variables. So the second that function ends and you return the struct with those pointers, they're both invalid. Thus leading to this undefined behavior.
You could allocate new pointers on the heap there and seed them with the values of chunk.memory and response_code. Since they're on the heap, they'll persist until you free the memory, so that'd work.
However, I'd strongly suggest moving to a std::vector<std::string> instead of your char** and move to a raw int instead of int* if possible so you don't even need to worry about UB like this.

Save cURL content result into a string in C++

int main(void)
{
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "http://www.google.com");
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
_getch();
return 0;
}
string contents = "";
I would like to save the result of the curl html content in a string, how do I do this?
It's a silly question but unfortunately, I couldn't find anywhere in the cURL examples for C++
thanks!
You will have to use CURLOPT_WRITEFUNCTION to set a callback for writing. I can't test to compile this right now, but the function should look something close to;
static std::string readBuffer;
static size_t WriteCallback(void *contents, size_t size, size_t nmemb, void *userp)
{
size_t realsize = size * nmemb;
readBuffer.append(contents, realsize);
return realsize;
}
Then call it by doing;
readBuffer.clear();
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
// ...other curl options
res = curl_easy_perform(curl);
After the call, readBuffershould have your contents.
Edit: You can use CURLOPT_WRITEDATA to pass the buffer string instead of making it static. In this case I just made it static for simplicity. A good page to look (besides the linked example above) is here for an explanation of the options.
Edit2: As requested, here's a complete working example without the static string buffer;
#include <iostream>
#include <string>
#include <curl/curl.h>
static size_t WriteCallback(void *contents, size_t size, size_t nmemb, void *userp)
{
((std::string*)userp)->append((char*)contents, size * nmemb);
return size * nmemb;
}
int main(void)
{
CURL *curl;
CURLcode res;
std::string readBuffer;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "http://www.google.com");
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
std::cout << readBuffer << std::endl;
}
return 0;
}
On my blog I have published a simple wrapper class to perform this task.
Usage example:
#include "HTTPDownloader.hpp"
int main(int argc, char** argv) {
HTTPDownloader downloader;
std::string content = downloader.download("https://stackoverflow.com");
std::cout << content << std::endl;
}
Here's the header file:
/**
* HTTPDownloader.hpp
*
* A simple C++ wrapper for the libcurl easy API.
*
* Written by Uli Köhler (techoverflow.net)
* Published under CC0 1.0 Universal (public domain)
*/
#ifndef HTTPDOWNLOADER_HPP
#define HTTPDOWNLOADER_HPP
#include <string>
/**
* A non-threadsafe simple libcURL-easy based HTTP downloader
*/
class HTTPDownloader {
public:
HTTPDownloader();
~HTTPDownloader();
/**
* Download a file using HTTP GET and store in in a std::string
* #param url The URL to download
* #return The download result
*/
std::string download(const std::string& url);
private:
void* curl;
};
#endif /* HTTPDOWNLOADER_HPP */
Here's the source code:
/**
* HTTPDownloader.cpp
*
* A simple C++ wrapper for the libcurl easy API.
*
* Written by Uli Köhler (techoverflow.net)
* Published under CC0 1.0 Universal (public domain)
*/
#include "HTTPDownloader.hpp"
#include <curl/curl.h>
#include <curl/easy.h>
#include <curl/curlbuild.h>
#include <sstream>
#include <iostream>
using namespace std;
size_t write_data(void *ptr, size_t size, size_t nmemb, void *stream) {
string data((const char*) ptr, (size_t) size * nmemb);
*((stringstream*) stream) << data;
return size * nmemb;
}
HTTPDownloader::HTTPDownloader() {
curl = curl_easy_init();
}
HTTPDownloader::~HTTPDownloader() {
curl_easy_cleanup(curl);
}
string HTTPDownloader::download(const std::string& url) {
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
/* example.com is redirected, so we tell libcurl to follow redirection */
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1); //Prevent "longjmp causes uninitialized stack frame" bug
curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, "deflate");
std::stringstream out;
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &out);
/* Perform the request, res will get the return code */
CURLcode res = curl_easy_perform(curl);
/* Check for errors */
if (res != CURLE_OK) {
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
}
return out.str();
}
Using the 'new' C++11 lambda functionality, this can be done in a few lines of code.
#ifndef WIN32 #define __stdcall "" #endif //For compatibility with both Linux and Windows
std::string resultBody { };
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &resultBody);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, static_cast<size_t (__stdcall *)(char*, size_t, size_t, void*)>(
[](char* ptr, size_t size, size_t nmemb, void* resultBody){
*(static_cast<std::string*>(resultBody)) += std::string {ptr, size * nmemb};
return size * nmemb;
}
));
CURLcode curlResult = curl_easy_perform(curl);
std::cout << "RESULT BODY:\n" << resultBody << std::endl;
// Cleanup etc
Note the __stdcall cast is needed to comply to the C calling convention (cURL is a C library)
This might not work right away but should give you an idea:
#include <string>
#include <curl.h>
#include <stdio.h>
size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
size_t written;
written = fwrite(ptr, size, nmemb, stream);
return written;
}
int main() {
std::string tempname = "temp";
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
FILE *fp = fopen(tempname.c_str(),"wb");
curl_easy_setopt(curl, CURLOPT_URL, "http://www.google.com");
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
fp = fopen(tempname.c_str(),"rb");
fseek (fp , 0 , SEEK_END);
long lSize = ftell (fp);
rewind(fp);
char *buffer = new char[lSize+1];
fread (buffer, 1, lSize, fp);
buffer[lSize] = 0;
fclose(fp);
std::string content(buffer);
delete [] buffer;
}
}
Came out with useful, yet simple solution, which overloads std::ostream::operator<<
#include <ostream>
#include <curl/curl.h>
size_t curlCbToStream (
char * buffer,
size_t nitems,
size_t size,
std::ostream * sout
)
{
*sout << buffer;
return nitems * size;
}
std::ostream & operator<< (
std::ostream & sout,
CURL * request
)
{
::curl_easy_setopt(request, CURLOPT_WRITEDATA, & sout);
::curl_easy_setopt(request, CURLOPT_WRITEFUNCTION, curlCbToStream);
::curl_easy_perform(request);
return sout;
}
Possible drawback of taken approach could be:
typedef void CURL;
That means it covers all known pointer types.
Based on #JoachimIsaksson answer, here is a more verbose output that handles out-of-memory and has a limit for the maximum output from curl (as CURLOPT_MAXFILESIZE limits only based on header information and not on the actual size transferred ).
#DEFINE MAX_FILE_SIZE = 10485760 //10 MiB
size_t curl_to_string(void *ptr, size_t size, size_t count, void *stream)
{
if(((string*)stream)->size() + (size * count) > MAX_FILE_SIZE)
{
cerr<<endl<<"Could not allocate curl to string, output size (current_size:"<<((string*)stream)->size()<<"bytes + buffer:"<<(size * count) << "bytes) would exceed the MAX_FILE_SIZE ("<<MAX_FILE_SIZE<<"bytes)";
return 0;
}
int retry=0;
while(true)
{
try{
((string*)stream)->append((char*)ptr, 0, size*count);
break;// successful
}catch (const std::bad_alloc&) {
retry++;
if(retry>100)
{
cerr<<endl<<"Could not allocate curl to string, probably not enough memory, aborting after : "<<retry<<" tries at 10s apart";
return 0;
}
cerr<<endl<<"Could not allocate curl to string, probably not enough memory, sleeping 10s, try:"<<retry;
sleep(10);
}
}
return size*count;
}
I use Joachim Isaksson's answer with a modern C++ adaptation of CURLOPT_WRITEFUNCTION.
No nagging by the compiler for C-style casts.
static auto WriteCallback(char* ptr, size_t size, size_t nmemb, void* userdata) -> size_t {
static_cast<string*>(userdata)->append(ptr, size * nmemb);
return size * nmemb;
}

Get page contents in C++

I've created a PHP and C# app that gets the users Xbox 360 Profile ID from their gamertag.
I was just wondering how I do a web request and put the page contents of the url into a string in C++.
Thanks.
Here's a down-and-dirty C++ HTTP fetcher I wrote using libCURL some time back.
It takes a URL and returns a string with the contents of a get request. Modify and flavor to taste:
/*
* HttpFetcher.h
*/
#ifndef _http_fetcher_
#define _http_fetcher_
#include <stdlib.h>
#include <string>
class HttpFetcher {
public:
std::string simpleGetRequest(std::string url);
private:
// Private methods to handle getting
struct MemoryStruct {
char *memory;
size_t size;
};
static size_t WriteMemoryCallback(void *ptr, size_t size, size_t nmemb, void *data);
};
#endif // ifndef _http_fetcher_
/*
* HttpFetcher.cpp
* Much of this is quite similar to examples on the libcURL site
*/
#include "HttpFetcher.h"
#include "curl.h"
#include "easy.h"
#include <iostream>
size_t HttpFetcher::WriteMemoryCallback(void *ptr, size_t size, size_t nmemb, void *data)
{
size_t realsize = size * nmemb;
struct MemoryStruct *mem = (struct MemoryStruct *)data;
mem->memory = (char*)realloc(mem->memory, mem->size + realsize + 1);
if (mem->memory == NULL) {
/* out of memory! */
printf("not enough memory (realloc returned NULL)\n");
exit(EXIT_FAILURE);
}
memcpy(&(mem->memory[mem->size]), ptr, realsize);
mem->size += realsize;
mem->memory[mem->size] = 0;
return realsize;
}
std::string HttpFetcher::simpleGetRequest(std::string url)
{
std::string toReturn = "";
CURL *curl = NULL;
struct MemoryStruct chunk;
chunk.memory = (char*)malloc(1);
chunk.size = 0;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &WriteMemoryCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void*)&chunk);
// Some requests fail without a user agent.
curl_easy_setopt(curl, CURLOPT_USERAGENT, "libcurl-agent/1.0");
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
toReturn = std::string(chunk.memory);
}
free(chunk.memory);
return toReturn;
}