The following functions in libcurl saves a file and returns the http status code. However, when I run this using valgrind, it is reporting 0 bytes for "definitely lost", "indirectly lost", "possibly lost", but it is reporting 47448 bytes for "still reachable". I'm trying to resolve the "still reachable" bytes.
Are there any potential memory leaks in the code below?
size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream){
size_t written = fwrite(ptr, size, nmemb, stream);
return written;
}
void connectAndSaveFile(char* url, char* output_file_name){
CURL *curl;
curl = curl_easy_init();
if (curl) {
FILE *fp = fopen(output_file_name,"wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
}
}
string get_http_status_code(string URL) {
CURL *session;
session = curl_easy_init();
curl_easy_setopt(session, CURLOPT_URL, URL.c_str());
curl_easy_setopt(session, CURLOPT_NOBODY, true);
CURLcode curl_code = curl_easy_perform (session);
long http_code = 0;
curl_easy_getinfo (session, CURLINFO_RESPONSE_CODE, &http_code);
curl_easy_cleanup(session);
std::ostringstream buff;
buff << http_code;
return buff.str();
}
"still reachable" is most frequently not actually a leak
you might get slightly less memory reachable if you use curl_global_init and curl_global_cleanup
The most of the code mention above uses libcurl. So I think we would have to look the documentation and read about API and what are the recommended steps.
However in the below method, client is passing pointer in which fwrite API is writing and returns back to the caller. This memory needs to be released in client(who would be calling this function) code once usage is complete.
size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream)
However in pure C++ ways, we should use std::fstream & std::string so that we need not worry about the memory management. For more informtion you may refer to following link:
https://stackoverflow.com/a/22048298/2724703
Related
I am using following code to download data from an url to memory (stream). Around 2% chance, the size of the stream is zero. I can download proper data from the same failing url if I try it another time. I am not sure if this is a network issue, CPU usage issue, or it's just the code not covering some corner cases. Please advice. Thanks!
static size_t write_data(char *ptr, size_t size, size_t nmemb, void *userdata)
{
std::vector<uchar> *stream = (std::vector<uchar>*)userdata;
size_t count = size * nmemb;
stream->insert(stream->end(), ptr, ptr + count);
return count;
}
static void CurlUrl(const char* img_url, std::vector<uchar>* stream) {
CURL *curl = curl_easy_init(); // curl_global_init is called eleswhere.
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1);
curl_easy_setopt(curl, CURLOPT_URL, img_url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, stream);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 10);
CURLcode res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
If it didn't deliver any download data into the buffer via the callback, it means that the transfer either failed or that there was exactly zero bytes to transfer.
Check the return code from curl_easy_perform() as it might actually tell you exactly what happened.
Use CURLOPT_VERBOSE to see what's going on if (1) is not enough.
Use CURLOPT_ERRORBUFFER to get a better error description if it fails if (2) is not enough.
I am using libcurl to fetch json data using GET request from a webserver.
This is my sample code:
char *DownloadedResponse;
static int writer(char *data, size_t size, size_t nmemb, char *buffer_in)
{
if (buffer_in != NULL)
{
buffer_in = new char[size*nmemb];
strcpy(buffer_in,data);
DownloadedResponse = buffer_in;
return size * nmemb;
}
return 0;
}
char * DownloadJSON(string URL)
{
CURL *curl;
CURLcode res;
struct curl_slist *headers=NULL;
curl_slist_append(headers, "Accept: application/json");
curl_slist_append( headers, "Content-Type: application/json");
curl_slist_append( headers, "charsets: utf-8");
curl = curl_easy_init();
if (curl)
{
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_URL, URL.c_str());
curl_easy_setopt(curl, CURLOPT_HTTPGET,1);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl,CURLOPT_WRITEFUNCTION,writer);
res = curl_easy_perform(curl);
if (CURLE_OK == res)
{
char *ct;
res = curl_easy_getinfo(curl, CURLINFO_CONTENT_TYPE, &ct);
if((CURLE_OK == res) && ct)
{
cout<<"\nresponse received: "<<DownloadedResponse;
}
else
{
curl_slist_free_all(headers);
curl_easy_cleanup(curl);
curl = NULL;
return NULL;
}
}
}
curl_slist_free_all(headers);
curl_easy_cleanup(curl);
curl = NULL;
}
Here I am able to get json data in DownloadedResponse in callback "writer" of CURLOPT_WRITEFUNCTION.
But if I print using custom pointer of CURLOPT_WRITEDATA,
char *dataPointer = NULL;
CURLcode curl_easy_setopt(curl, CURLOPT_WRITEDATA, dataPointer);
cout<<dataPointer;
Output of dataPointer is empty.
What is the issue here since i able to print json data in callback of CURLOPT_WRITEFUNCTION but not in the pointer of CURLOPT_WRITEDATA
You write a function that takes data read from the network, and writes it to where you want it.
static int writer(char *data, size_t size, size_t nmemb, char *buffer_in){
if (buffer_in != NULL) {
// very bad code which is never executed
}
return 0;
}
In order for that function to write the data, it has to know where to write it, so you tell it to write to NULL
char *dataPointer = NULL;
CURLcode curl_easy_setopt(curl, CURLOPT_WRITEDATA, dataPointer);
What value do you tell it to use as buffer_in? You pass it dataPointer, which is NULL, so you just told it buffer_in = NULL. I think instead you meant to say "the address of dataPointer", which would be &dataPointer.
Technically, I have answered your question now. You passed it NULL for the buffer, so the write function exited immediately. But there's more. Now you get to execute that really bad code in writer().
if (buffer_in != NULL)
{
// if buffer_in already has allocated memory then leak it immediately
// create a new buffer of memory to leak later
buffer_in = new char[size*nmemb];
// store the data in buffer_in
// assume it is null terminated (it is not)
// rather than using the length we already know
strcpy(buffer_in,data);
// remember buffer_in? We don't use it so assign that data pointer to a global variable.
DownloadedResponse = buffer_in;
// return size of this particular chunk of data
return size * nmemb;
}
This function MUST use the length of the data, and not assume data is null terminated (see https://curl.haxx.se/libcurl/c/CURLOPT_WRITEFUNCTION.html).
This function MUST be able to handle the data in multiple small pieces by adding them to what it has already read. You can't call new and then discard the new memory. And you can't do that anyway because you just leaked that memory -- every new must be matched with exactly one delete. In fact, you would be very well advised not to use new or delete at all, now that we have the standard library.
This function should use the buffer_in argument you give it rather than a global variable, but you can use a global variable if you want, it's just error prone. It's not literally an error like the other stuff.
The whole point of buffer_in is to give you a persistent data structure where you can accumulate the answers. It probably should be in local scope around the curl_easy_execute, so you can then just return the content from that data structure if you got CURLE_OK. I strongly recommend you write the data to std::vector, so you don't have to keep track of memory allocation. You have trouble with it, but you don't need to do it at all. Modern style says everybody has trouble with it, so just let the standard library handle it.
You claim to follow the example in the docs, which links to https://curl.haxx.se/libcurl/c/getinmemory.html If you look again, you will see what they are doing, and how your code doesn't match. In particular, they pass &chunk (the address of chunk) and then write data into chunk so they keep what was there before.
struct MemoryStruct {
char *memory;
size_t size;
};
static size_t
WriteMemoryCallback(void *contents, size_t size, size_t nmemb, void *userp)
{
// here is where they get access to the buffer
struct MemoryStruct *mem = (struct MemoryStruct *)userp;
In the call to curl, you will find the struct locally defined, then the remote call:
struct MemoryStruct chunk;
curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, (void *)&chunk);
res = curl_easy_perform(curl_handle);
if (stuff)
printf("%lu bytes retrieved\n", (long)chunk.size);
I am trying to read the content of a PHP / HTML file on a remote web server using C++, but haven't found a way to do it. I want to pass GET statements to it, so http://example.com/login.php?user=abc&password=def.
How would I do it?
Your best bet is to use an external library. libcurl is popular and fairly easy to use.
Here's a simple example, you need to add error checking though:
string data;
CURL *curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_URL, url_.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &data);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curlWrite);
curl_easy_perform(curl);
Your callback would look something like this:
size_t curlWrite(void *ptr, size_t size, size_t nmemb, void *usrPtr)
{
size_t bytes = size * nmemb;
string *data = static_cast<string *>(usrPtr);
data->append(static_cast<const char *>(ptr), bytes);
return bytes;
}
You can add your GET parameters on the end of the URL.
Take a look at the following code
static size_t reader(void *ptr, size_t size, size_t nmemb, FILE *stream) {
size_t retcode = fread(ptr, size, nmemb, stream);
cout << "*** We read " << retcode << " bytes from file" << endl;
return retcode;
}
void upload() { //upload() is called from ouside
FILE *pFile;
pFile = fopen("map.txt" , "r");
struct stat file_info;
stat("map.txt", &file_info);
size_t size = (size_t)file_info.st_size;
uploadFile(pFile, size);
}
bool uploadFile(void* data, size_t datasize) {
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if (curl) {
char *post_params = ...;
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, post_params);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, (long) strlen(post_params));
curl_easy_setopt(curl, CURLOPT_READFUNCTION, reader);
curl_easy_setopt(curl, CURLOPT_READDATA, data);
curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, (curl_off_t) datasize);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
return true;
}
When the code is executed, the following is outputed
*** We read 490 bytes from file
*** We read 0 bytes from file
after that the app does nothing (even not exiting).
Can someone point out at what's wrong here?
Will be grateful for any help!!!
There's some serious confusions shown in this code. Let me try to explain:
CURLOPT_UPLOAD - this will ask libcurl to PUT the file when the protocol of choice is HTTP
CURLOPT_POSTFIELDS - tells libcurl to POST the data that is provided in the additional argument (which has the size set with CURLOPT_POSTFIELDSIZE)
CURLOPT_READFUNCTION - provides libcurl an alternative way to get data than CURLOPT_POSTFIELDS to allow a POST that reads the data from a file. When using CURLOPT_UPLOAD this is the only way to provide data.
So in the end the questions left for you are:
Do you want PUT or POST?
Do you want to provide the data as a string or do you want it provided with a callback?
I am currently trying to make an updater for my software project. I need it to be able to download multiple files, I don't mind if they download in sync or one after each other, whatever is easier (file size is not an issue). I followed the example from the libcurl webpage and a few other resources and came up with this:
#include <iostream>
#include <stdio.h>
#include <curl/curl.h>
#include <string.h>
size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
size_t written;
written = fwrite(ptr, size, nmemb, stream);
return written;
}
int main(void){
for (int i = 0; i < 2;){ //download 2 files (loop twice)
CURL *curl;
FILE *fp;
CURLcode res;
char *url = "http://sec7.org/1024kb.txt"; //first file URL
char outfilename[FILENAME_MAX] = "C:\\users\\grant\\desktop\\1024kb.txt";
curl = curl_easy_init();
if (curl){
fp = fopen(outfilename,"wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
}
url = "http://sec7.org/index.html"; //I want to get a new file this time
outfilename[FILENAME_MAX] = "C:\\users\\grant\\desktop\\index.html";
}
return 0;
}
The first issue is if i remove the new file assignments (*url = "http://...") and just try to loop the download code twice, the program simply stops responding. This occurs in any combination of the download being called more than once in the program. The other issue is that I am unable to change the value of the character array outfilename[FILENAME_MAX]. I feel like this is just some silly error I am making but no solution comes to mind. Thank you!
Why not put this in a function and call it twice?
Your syntax for the arrays is all wrong, plus all the variables inside the loop are local, which means they are destroyed after each loop iteration.
What Conspicuous Compiler said. That's what's causing your program to freeze; it's stuck in an infinite loop because i is never > 2.
Put your code into a function like so:
void downloadFile(const char* url, const char* fname) {
CURL *curl;
FILE *fp;
CURLcode res;
curl = curl_easy_init();
if (curl){
fp = fopen(fname, "wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
}
}
And call it twice with the relevant file names and urls:
downloadFile("http://sec7.org/1024kb.txt", "C:\\users\\grant\\desktop\\1024kb.txt");
downloadFile("http://sec7.org/index.html", "C:\\users\\grant\\desktop\\index.html");
The example function is very bad though, it's just an example. You should alter it to return error codes/throw exceptions, and stuff like that.