I have a binary file to be downloaded and using curl in linux terminal, the following command supports download resume on a broken request.
curl -C - -o sample1.bin https://speed.hetzner.de/100MB.bin
The above will resume a download that is canceled.
When i use libcurl in my cpp program to do the same, is there any api's that i can use to achieve the above result on a HTTPS broken request.
Thank you for you help.
NOTE: CURL_RESUME_FROM does not have support on HTTPS.
I think you could implement a retry system by yourself, for example:
CURL *curl;
curl = curl_easy_init();
//Set curl options as needed with curl_easy_setopt()
char* url;
int tries = 0;
bool done = false;
while (tries != 3 && !done) {
res = curl_easy_perform(curl);
curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code);
curl_easy_getinfo(curl, CURLINFO_TOTAL_TIME, &elapsed);
curl_easy_getinfo(curl, CURLINFO_EFFECTIVE_URL, &url);
if (res != CURLE_OK || http_code != 200)
tries++;
else
done = true;
}
// Check if any error has occurred
if (res != CURLE_OK || http_code != 200) {
// Could not perform request "
if (tries == 3) {
//Too many tries, remote host is overloaded or down
} else {
// Cannot perform CURL
}
}
// Curl succeeded
Also you can have a look to CURLOPT_LOW_SPEED_LIMIT and CURLOPT_LOW_SPEED_TIME to avoid any overhead in the server.
How about something "low-level" like passing an extra header? Let s be the size of the partially-downloaded file, just use Range: bytes=s-.
See Requesting_a_specific_range_from_a_server and CURLOPT_HTTPHEADER explained.
#include <curl/curl.h>
#include <string>
#include <sstream>
int dim=... //size of partial download sample1.bin
std::string s=std::to_string(dim); // <-- here s is the string representing the size of the partial download
CURL *curl = curl_easy_init();
struct curl_slist *list = NULL;
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL,"https://speed.hetzner.de/100MB.bin");
list = curl_slist_append(list, "Range: bytes="+s+"-"); //from where it left off to the end (or where it stops again)
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, list);
curl_easy_perform(curl);
curl_slist_free_all(list); /* free the list again */
}
Check out also Making HTTPS GET with libcurl.
Related
Scenario:
Im writing a c++ program which should retrieve files and strings from an expressJS API..
Downloading txt.files with curlRequests works perfectly but as soon as i try to retrieve plain strings, they can only be used for printing..
Problem: When trying to do anything with the response from my GET request (from the expressjs api), my response doesnt get treated as a string.
string myText = curlRequest.GetText("/templates/names");
string myTextB = "react.txt, scss.txt"
cout << myText << endl; // prints"react.txt, scss.txt"
cout << myTextB << endl; // prints "react.txt, scss.txt"
cout << stringHelper.GetSubstringPos(myText, "scss") << endl; // printsstring::npos
cout << stringHelper.GetSubstringPos(myTextB, "scss") << endl; // printsposition of 's' as expected
Here is my GetText function of the curlrequest.h in c++
string GetText(string ACTIONURL) {
CURL* curl;
CURLcode curlRes;
string res;
string url = templateCreator.APIURL + ACTIONURL;
curl_global_init(CURL_GLOBAL_DEFAULT);
curl = curl_easy_init();
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1L);
curlRes = curl_easy_perform(curl);
res = curlRes;
if (curlRes == CURLE_HTTP_RETURNED_ERROR) {
res = "FAILED";
}
else if (curlRes != CURLE_OK) {
fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(curlRes));
res = "FAILED";
}
curl_easy_cleanup(curl);
}
curl_global_cleanup();
return res;
}
Here is the route in expressjs (receiving the request and responding with a string)
router.get('/names', function(req, res, next) {
var str = "react.txt, scss.txt, example.txt";
res.send(str);
});
// I also tried sending a buffer but as its sended as string i face the same problem..
// C++ could decode the buffer ..
router.get('/buf', function(req, res, next) {
let data = 'hello world: (1 + 2 !== 4)';
let buff = new Buffer.from(data);
let base64data = buff.toString('base64');
console.log(base64data);
res.send(base64data);
});
Retrieving textfiles from my expressjs API is not a problem..
void GetFile(string ACTIONURL, string OUTDIR) {
CURL* curl;
FILE* fp;
CURLcode res;
string url = templateCreator.APIURL + ACTIONURL;
curl = curl_easy_init();
if (curl)
{
fopen_s(&fp, OUTDIR.c_str(), "wb");
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
}
return;
}
(After download I can read those line by line and store in a vector).
Still im hoping to get sending actual strings working..
Does anyone have a clue why im facing problems here?
I literally spent days on this unexpected 'small' issue already..
Thank you #n.'pronouns'm.
I Updated my function and realised that res = curlRes was a nobrainer..
Also the checks for valid response is gone for now.
//those 2 lines and a write_to_string function were missing and `res = curlRes` should do their job
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_to_string);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);
The following 2 functions replace my GetText function now, and everything works as expected.
size_t write_to_string(void* ptr, size_t size, size_t count, void* stream) {
((string*)stream)->append((char*)ptr, 0, size * count);
return size * count;
}
string GetText(string ACTIONURL) {
CURL* curl;
CURLcode res;
string response;
curl = curl_easy_init();
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, templateCreator.APIURL + ACTIONURL.c_str());
curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1L);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_to_string);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);
res = curl_easy_perform(curl);
if (res != CURLE_OK) {
fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
}
curl_easy_cleanup(curl);
}
return response;
}
Thank you! I found the fix on 1 or 2 questions earlier too but was not aware that this was the actual problem. Working with strings is possible now!
I want to download a file from a dropbox shared link using curl in a c++ program
I found a dropbox api pdf that showed me how to do it
#include <stdio.h>
#include <curl/curl.h>
int main (int argc, char *argv[])
{
CURL *curl;
CURLcode res;
/* In windows, this will init the winsock stuff */
curl_global_init(CURL_GLOBAL_ALL);
/* get a curl handle */
curl = curl_easy_init();
if(curl) {
printf ("Running curl test.\n");
struct curl_slist *headers=NULL; /* init to NULL is important */
headers = curl_slist_append(headers, "Authorization: Bearer
<ACCESS_TOKEN>");
headers = curl_slist_append(headers, "Content-Type:");
headers = curl_slist_append(headers, "Dropbox-API-Arg:
{\"path\":\"/test.txt\"}");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_URL,
"https://content.dropboxapi.com/2/files/download");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "");
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
printf ("\nFinished curl test.\n");
}
curl_global_cleanup();
printf ("Done!\n");
return 0;
}
However, the comments supplied don't offer much explanation for me, and I can't get it to work.
I don't understand these three lines of code:
headers = curl_slist_append(headers, "Authorization: Bearer <ACCESS_TOKEN>");
headers = curl_slist_append(headers, "Content-Type:");
headers = curl_slist_append(headers, "Dropbox-API-Arg:{\"path\":\"/test.txt\"}");
I think I have to replace some stuff but I don't know what
"I think I have to replace some stuff but I don't know what" : Replace <ACCESS_TOKEN> with your actual access token.
You should also set the "Content-Type:" header to an appropriate value for the data you are fetching.
You must also change the value of the "Dropbox-API-Arg" header to match the file you are trying to get.
I finally found the solution to my problem.
Turns out I didn't have to use the Dropbox API
Here is the code
#include <iostream>
#include <curl/curl.h>
using namespace std;
size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream)
{
size_t written;
written = fwrite(ptr, size, nmemb, stream);
return written;
}
int main(int argc, char** argv) {
CURL *curl;
FILE *fp;
const char* destination = "D:\\Desktop\\test.exe";
fp = fopen(destination, "wb");
curl = curl_easy_init();
/* A long parameter set to 1 tells the library to follow any Location: header
* that the server sends as part of an HTTP header in a 3xx response. The
*Location: header can specify a relative or an absolute URL to follow.
*/
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_URL, "https://www.dropbox.com/s/09nd26tdyto23yz/BankAccount.exe?dl=1"); // "dl=0"changed to "dl=1" to force download
// disabe the SSL peer certificate verification allowing the program to download the file from dropbox shared link
// in case it is not used it displays an error message stating "SSL peer certificate or SSH remote key was not OK"
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
CURLcode res;
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
if (res ==CURLE_OK)
cout << "OK";
else
cout << curl_easy_strerror(res);
return 0;
}
Thanks you guys for trying to help me. I appreciate
I want to retry a curl connection in my C++ program for 5 times. When it fails 5 times in a row, it should stop the execution of the program. However, it stops after the first error at this point. I am able to catch the error, however I don't know how to execute the previous curl connection. E.g., with jQuery I can use something like $.ajax(this);. For LibCurl in C++ I am looking for a similar solution.
My current LibCurl code is shown below, note that I use multiple curl connections which all have other settings, therefore I would like a general approach which I can use for all my LibCurl errors within my LibcurlError function which is also included below.
curl = curl_easy_init();
if (curl) {
CurlResponse = "";
host = "http://google.com";
LibcurlHeaders = curl_slist_append(NULL, "Expect:");
if (ProxyAddress.length() > 0) {
curl_easy_setopt(curl, CURLOPT_PROXY, ProxyAddress.c_str());
}
curl_easy_setopt(curl, CURLOPT_URL, (host).c_str());
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER , 1);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST , 1);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, LibcurlHeaders);
res = curl_easy_perform(curl);
curl_slist_free_all(LibcurlHeaders);
if (res != CURLE_OK) {
//AT THIS POINT I WOULD LIKE TO RETRY FOR 5 TIMES WHICH I WOULD LIKE TO CATCH IN MY LibcurlError FUNCTION.
LibcurlError(curl_easy_strerror(res), host);
}
curl_easy_cleanup(curl);
}
curl_global_cleanup();
void LibcurlError(string error, string host) {
//IF FAILED FOR LESS THEN 5 TIMES IN A ROW -> RETRY CURL
//ELSE I WOULD LIKE TO EXECUTE MY ORIGINAL CODE WHICH IS STATED BELOW
Message = "LibCurl Error: ";
if (error == "Couldn't resolve host name") {
Message.append("Couldn't connect to the server of ");
if (host.find("google.com") != string::npos) {
Message.append("Google");
}
else {
Message.append("'" + host + "'");
}
}
else {
Message.append("'" + error + "'");
}
cout << Message << endl;
system("pause");
exit(0);
}
There is no CURL method that specifically does this because it can be accomplished by repeated calls to curl_easy_perform.
Here is how you would write the code in your question (the relevant part at least) using loops to retry the CURL request repeatedly:
#include <unistd.h>
#include <curl/curl.h>
/*
* This is the maximum number of times CURL will run
*/
const int max_attempts = 5;
curl = curl_easy_init();
if (curl) {
CurlResponse = "";
host = "http://google.com";
LibcurlHeaders = curl_slist_append(NULL, "Expect:");
if (ProxyAddress.length() > 0) {
curl_easy_setopt(curl, CURLOPT_PROXY, ProxyAddress.c_str());
}
curl_easy_setopt(curl, CURLOPT_URL, (host).c_str());
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER , 1);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST , 1);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, LibcurlHeaders);
for (int i = 1; i <= max_attempts &&
(res = curl_easy_perform(curl)) != CURLE_OK; i++) {
/*
* At this point, you would sleep
* for some seconds between requests
*/
const int sleep_secs = 1;
sleep(sleep_secs);
}
// As others have mentioned, you should delete this line:
//curl_slist_free_all(LibcurlHeaders);
if (res != CURLE_OK) {
// The max retries have all failed
LibcurlError(curl_easy_strerror(res), host);
}
else {
// The request has succeeded in the first `max_retries` attempts
// ...
}
curl_easy_cleanup(curl);
}
curl_global_cleanup();
I'm new to NaCl and now I'm starting to have some issues with this. I'm trying to load some data into C++ part of the application from C++ code and I know that I can't access files on HDD from NaCl. So, what I tried to do, was to use curl library to load this files via http from localhost.
So I set up NaCl environment, added curl - everything built fine, then I had an issue that curl couldn't resolve host name, but hopefully I overcame this issue, but then curls says that it can't connect to server (error code 7). I don't know what might be the issue.
I start chrome using this command:
google-chrome 127.0.0.1:5103/myapp --allow-nacl-socket-api=127.0.0.1
--no-sandbox --load-extension="/home/user/Desktop/nacl_sdk/pepper_49/getting_started/web/"
--enable-nacl --user-data-dir="/home/user/Desktop/chrome-dir/"
And code that is trying to load data from localhost is as follows:
static size_t data_write(char* buf, size_t size, size_t nmemb, void* userp) {
printf("start data_write\n");
if (userp) {
std::stringstream& data = *static_cast<std::stringstream*>(userp);
std::streamsize len = size * nmemb;
for (int i = 0; i < len; ++i) {
data << buf[i];
}
return len;
}
printf("end data_write\n");
return 0;
}
CURLcode curl_read(const std::string& url, std::stringstream& os, long timeout = 300) {
CURLcode code(CURLE_FAILED_INIT);
CURL* curl = curl_easy_init();
printf("CURL_READ\n");
if (curl) {
printf("curl is not null\n");
if (CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION,
&data_write)) &&
CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1L)) &&
CURLE_OK ==
(code = curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L)) &&
CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_FILE, &os)) &&
CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_TIMEOUT, timeout)) &&
CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_URL, url.c_str()))) {
printf("START curl perform\n");
code = curl_easy_perform(curl);
printf("CURL ERROR (%d) = %s\n", code, curl_easy_strerror(code));
printf("END curl perform\n");
}
curl_easy_cleanup(curl);
}
return code;
}
And the output from console is:
Init curl
Loading data from: http://127.0.0.1/assets/model.obj
CURL_READ
curl is not null
START curl perform
CURL ERROR (7) = Couldn't connect to server
END curl perform
What might be the issue? Is it even possible to use curl in NaCl?
In windows, this code works file, but now, I want to convert it into ubuntu:
// callback read function to upload file from local to ftp server
size_t read_callback (void* ptr, size_t size, size_t nmemb, FILE *stream){
//return fread(ptr,size,nmemb, (FILE*) stream);
return fread(ptr,size,nmemb,stream);
}
// get file name from a path
string FTPClientConnector::getFileName(string path){
int length = path.size();
for(int i = length - 1; i >= 0; i--){
if(path[i] == '/' || path[i] == '\\'){
return path.substr(i+1, length-i-1);
}
}
}
//function to upload a file to FTP server
int FTPClientConnector::uploadFile(string filePath, string serverPath ){
CURL *curl;
CURLcode res;
FILE *hd_src;
struct stat file_info;
curl_off_t fsize;
char* local_file = new char[filePath.size()+1];
std::copy(filePath.begin(), filePath.end(), local_file);
local_file[filePath.size()] = '\0';
// stat the local file
if(stat(local_file, &file_info)){
printf("couldn't open file\n");
delete local_file;
return -1;
}
// convert URL and username and password to connect to remote server
string urlPath = this->hostName + serverPath;
urlPath += getFileName(filePath);
char *url = new char[urlPath.size() + 1];
std::copy(urlPath.begin(), urlPath.end(), url);
url[urlPath.size()] = '\0';
string userAndPassString = this->userName + ":" + this->password;
char* usernameAndPassword = new char[userAndPassString.size() +1];
std::copy(userAndPassString.begin(), userAndPassString.end(), usernameAndPassword);
usernameAndPassword[userAndPassString.size()] = '\0';
// get the file to open
hd_src = fopen(local_file, "rb");
curl_global_init(CURL_GLOBAL_ALL);
curl = curl_easy_init();
if(curl){
/* specify target */
curl_easy_setopt(curl,CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_PORT, this->port);
curl_easy_setopt(curl, CURLOPT_USERPWD, usernameAndPassword);
/* we want to use our own read function */
curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);
/* enable uploading */
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
/* now specify which file to upload */
curl_easy_setopt(curl, CURLOPT_READDATA, hd_src);
/* Now run off and do what you've been told! */
res = curl_easy_perform(curl);
if(res != CURLE_OK){
printf("Upload file failed!\n");
delete local_file;
delete url;
delete usernameAndPassword;
return -1;
}
curl_easy_cleanup(curl);
}
fclose(hd_src);
delete local_file;
delete url;
delete usernameAndPassword;
return 0;
}
This is what I call in main.cpp:
FTPClientConnector connector(host,user,password,port);
connector.uploadFile("xml/kingfisher.xml", "/xml_test_upload");
The code above doesn't work in Ubuntu with errors:
220 ProFTPD 1.3.4a Server (Debian) [::ffff:10.244.31.244]
500 PUT not understood
500 AUTHORIZATION: not understood
500 HOST: not understood
550 */*: Forbidden command argument
500 TRANSFER-ENCODING: not understood
500 EXPECT: not understood
500 Invalid command: try being more creative
500 2A2 not understood
Edit: This is my Makefile:
uploader:
g++ -o uploader FTPClientConnector.cpp main.cpp -lcurl
The output seems to indicate that you speak HTTP to a FTP server. Make sure your URL properly uses a FTP:// prefix for FTP, as without a protocol prefix libcurl guesses which protocol you want and it defaults to HTTP...
It appears from your comments that you need to use IPv4. Add this to your list of setopt calls:
curl_easy_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4);