Downloading multiple files with libcurl in C++ - c++

I am currently trying to make an updater for my software project. I need it to be able to download multiple files, I don't mind if they download in sync or one after each other, whatever is easier (file size is not an issue). I followed the example from the libcurl webpage and a few other resources and came up with this:
#include <iostream>
#include <stdio.h>
#include <curl/curl.h>
#include <string.h>
size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
size_t written;
written = fwrite(ptr, size, nmemb, stream);
return written;
}
int main(void){
for (int i = 0; i < 2;){ //download 2 files (loop twice)
CURL *curl;
FILE *fp;
CURLcode res;
char *url = "http://sec7.org/1024kb.txt"; //first file URL
char outfilename[FILENAME_MAX] = "C:\\users\\grant\\desktop\\1024kb.txt";
curl = curl_easy_init();
if (curl){
fp = fopen(outfilename,"wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
}
url = "http://sec7.org/index.html"; //I want to get a new file this time
outfilename[FILENAME_MAX] = "C:\\users\\grant\\desktop\\index.html";
}
return 0;
}
The first issue is if i remove the new file assignments (*url = "http://...") and just try to loop the download code twice, the program simply stops responding. This occurs in any combination of the download being called more than once in the program. The other issue is that I am unable to change the value of the character array outfilename[FILENAME_MAX]. I feel like this is just some silly error I am making but no solution comes to mind. Thank you!

Why not put this in a function and call it twice?
Your syntax for the arrays is all wrong, plus all the variables inside the loop are local, which means they are destroyed after each loop iteration.
What Conspicuous Compiler said. That's what's causing your program to freeze; it's stuck in an infinite loop because i is never > 2.
Put your code into a function like so:
void downloadFile(const char* url, const char* fname) {
CURL *curl;
FILE *fp;
CURLcode res;
curl = curl_easy_init();
if (curl){
fp = fopen(fname, "wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
}
}
And call it twice with the relevant file names and urls:
downloadFile("http://sec7.org/1024kb.txt", "C:\\users\\grant\\desktop\\1024kb.txt");
downloadFile("http://sec7.org/index.html", "C:\\users\\grant\\desktop\\index.html");
The example function is very bad though, it's just an example. You should alter it to return error codes/throw exceptions, and stuff like that.

Related

Downloading a file to Ubuntu with libcurl C++, simple example doesn't work

I'm trying to use libcurl with C++ to download a single image file to my Ubuntu machine.
I tried copying and pasting the simple example shown in this question: Download file using libcurl in C/C++
#include <stdio.h>
#include <curl/curl.h>
#include <string>
using namespace std;
size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
size_t written = fwrite(ptr, size, nmemb, stream);
return written;
}
int main(void) {
CURL *curl;
FILE *fp;
CURLcode res;
const char *url = "https://i.imgur.com/mWj0yzI.jpg";
char outfilename[FILENAME_MAX] = "/home/my_username/test.jpg";
curl = curl_easy_init();
if (curl)
{
fp = fopen(outfilename,"wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
/* always cleanup */
curl_easy_cleanup(curl);
fclose(fp);
}
return 0;
}
I expected it to download the image file and save it as "test.jpg" on my machine. However, when I run this program, "test.jpg" is 0 bytes in size. Apparently the image didn't write to the file for some reason.
What am I doing wrong?
Your link does not point to an existing file. With a correct link it works for me. Try this:
const char *url = "https://i.imgur.com/oRtvmGT.jpg";

libcurl automatically replacing line feed with line feed + carriage return

As title says when downloading something and saving libcurl replacing all the LF with LF + CR. It is fine for the text documents. But for binary it is a disaster. I already tried
curl_easy_setopt(curl, CURLOPT_CRLF, 0L);
How to disable this thing. I'm running on windows and curl 7.40.0
#include <iostream>
#include <curl/curl.h>
using namespace std;
CURL *curl;
CURLcode res;
size_t file_write_callback(char *ptr, size_t size, size_t nmemb, void *userdata)
{
fwrite(ptr,size,nmemb,(FILE *)userdata);
return nmemb;
}
int main(void)
{
FILE * pFile;
pFile = fopen ("myfile.png","w");
curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_URL, "http://www.dilushan.tk/Media/128px_feed.png");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
if (pFile!=NULL)
{
curl_easy_setopt(curl, CURLOPT_WRITEDATA, pFile);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, file_write_callback);
res = curl_easy_perform(curl);
}
fclose (pFile);
return 0;
}
libburl is not the culprit, but underlying system library is. Because Windows has a notion of binary files where no conversion should occur, and text files where end of lines are represented as CrLf (\r\n) on disk and only \n in C or C++.
And the fix is quite easy : simply use b (for binary) in mode string in open :
pFile = fopen ("myfile.png","wb");

Extract specific data from webpage

Basically this is my code :
int main()
{
CURL *curl;
FILE *fp;
CURLcode res;
std::string readBuffer;
curl = curl_easy_init();
char outfilename[FILENAME_MAX] = "C:\\Users\\admin\\desktop\\test.txt";
if(curl) {
fp = fopen(outfilename,"wb");
curl_easy_setopt(curl, CURLOPT_URL, "http://www.example.com");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "user=123&pass=123");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
Sleep(1000);
curl_easy_cleanup(curl);
fclose(fp);
}
return EXIT_SUCCESS;
}
The output is successfully saved in the text file.
My concern is how to extract specific content in between specific tags.
For example i want only the content between < bla> .............. < /bla> .
Whats the easiest way and thank you.
In your Example, you are dumping the response from the website to a file, libcURL writes the data returned by the webpage that you hit as it is, it does not take efforts for restructuring the returned data.
You can obtain the data in a memory, by defining the write_data function, which needs the following format only:
size_t write_data(char *ptr, size_t size, size_t nmemb, void *userdata);
Once you get the data in a memory, you can parse it and restructure it as required.
See Example Here for using write_data function.
For XML Parsing you may use This sample code

Using libcurl in loop C

I would like to dowload some page content of wikitionary. I use curl in a loop. The first iteration is ok but the others give me the same result as the first. What is missing/wrong?. Thank you. This is the loop:
std::string buffer;
size_t curl_write( void *ptr, size_t size, size_t nmemb, void *stream)
{
buffer.append((char*)ptr, size*nmemb);
return size*nmemb;
}
int main(int argc, char **argv)
{
CURL *curl = curl_easy_init();
string data;
data="http://fr.wiktionary.org/w/api.php?format=json&action=query&titles=";
//Page titles are read from local file. The code is not shown to make short.
while ( not_end_of_file){
//list_of_page_title is pages requested for the current iteration.
data=data+list_of_page_title+"prop=revisions&rvprop=content";
curl_easy_setopt(curl, CURLOPT_URL, data.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curl_write);
curl_easy_perform(curl);
curl_easy_reset(curl);
}
curl_easy_cleanup(curl);
return 0;
}
I am new to curl. May be many things are missed. Thank you for the help.
data=data+list_of_page_title will append the new title onto your previous URL instead of replacing the previous. By the end you'll have a gigantic URL full of garbage. The server is probably paying attention to the first title and ignoring the rest.
And this would be obvious if you just output your URL as the first step of debugging... "Am I requesting what I think I'm requesting?"
One problem is that you are not resetting your buffer variable.
while ( not_end_of_file){
buffer = ""; // reset buffer to empty string
//list_of_page_title is pages requested for the current iteration.
data="http://fr.wiktionary.org/w/api.php?format=json&action=query&titles=" +
list_of_page_title +
"prop=revisions&rvprop=content";
curl_easy_setopt(curl, CURLOPT_URL, data.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curl_write);
curl_easy_perform(curl);
curl_easy_reset(curl);
}
And as Peter points out your handling of the data variable has a very similar problem.

Downloading all files in directory using libcurl

I am new to the libcurl and found a way to download a single file from the ftp server. Now my requirement is to download all files in a directory and i guess it was not supported by libcurl. Kindly suggest on libcurl how to download all files in directory or is there any other library similar to libcurl?
Thanks in advance.
Here is a sample piece of code.
static size_t GetFilesList_response(void *ptr, size_t size, size_t nmemb, void *data)
{
FILE *writehere = (FILE *)data;
return fwrite(ptr, size, nmemb, writehere);
}
bool FTPWithcURL::GetFilesList(char* tempFile)
{
CURL *curl;
CURLcode res;
FILE *ftpfile;
/* local file name to store the file as */
ftpfile = fopen(tempFile, "wb"); /* b is binary, needed on win32 */
curl = curl_easy_init();
if(curl)
{
curl_easy_setopt(curl, CURLOPT_URL, "ftp://ftp.example.com");
curl_easy_setopt(curl, CURLOPT_USERPWD, "username:password");
curl_easy_setopt(curl, CURLOPT_WRITEDATA, ftpfile);
// added to #Tombart suggestion
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, GetFilesList_response);
curl_easy_setopt(curl, CURLOPT_DIRLISTONLY, 1);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
fclose(ftpfile); //
if(CURLE_OK != res)
return false;
return true;
}
You need the list of files on the FTP server. Which isn't straightforward as each FTP server might return a different format of file listing...
Anyway, the ftpgetresp.c example shows a way to do it, I think. FTP Custom CUSTOMREQUEST suggests another way.
Just use CURLOPT_WILDCARDMATCH feature.
Sample code:
https://curl.haxx.se/libcurl/c/ftp-wildcard.html