I want to download a file from a URL using libcurl in C. I want to save the downloaded file into a specified location (i.e., /mnt/jffs2/ ) in my POS device.
I got code from some one and failed on executing that. Rather I can't save that file in my desired location.
So anyone help me please.
Thanks in advance.
Most easily you use the default CURLOPT_WRITEFUNCTION (which calls fwrite) and just set CURLOPT_WRITEDATA to the FILE * of your target file:
FILE *body = fopen("/mnt/jffs2/storeit", "w");
...
curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, body);'
...
rc = curl_easy_perform(curl_handle);
Related
My Question is closed so I have to update this.
1- My purpose is to send my compressed file google cloud storage URL.
2- To do that I have generated a postman request. I have stored my file to my google cloud storage by using the postman tool and the tool has generated the following code.
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
curl_easy_setopt(curl, CURLOPT_URL, "my URL");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_DEFAULT_PROTOCOL, "https");
struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Content-Type:
application/octet-stream");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl,CURLOPT_POSTFIELDS,"<file contents here>");
res = curl_easy_perform(curl);
}
curl_easy_cleanup(curl);
3- Then I have copied the code above into my c++ project to send my compressed file to the URL.
4- To create CURLOPT_POSTFIELDS content I did implement the following code;
std::ifstream ifs;
ifs.open ("./compressed.gz", std::ios::binary |
std::ios::ate);
PRINT("Size ->", ifs.tellg());
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, &ifs);
And when I compile and run my code, the request returns me the 200 success response.
But when I checked the google storage dashboard, it just contains 6 bytes of data. Actually, the size of my ifstream data is 1090.
So my problem is that why my request uploads all bytes of the compressed file to cloud storage? Whats wrong in my code ?
How to read compressed files c++
Compressed files are generally binary formats, they are not null terminated text. You cannot use strlen to get their length because that requires a null terminated text as input.
You can use any UnformattedInputFunction to read binary data. Don't forget to open any stream in binary mode.
I need to put the compressed file in a char pointer to put it to the server by using libcurl.
You don't need to read the file in order to do that. You can let libcurl take care of reading the file by passing a FILE* to CURLOPT_READDATA.
This way it won't be necessary to store the entire file in memory. Reading the entire file could be a problem if the file is very large.
You can do something like that when you want to read binary data:
std::ifstream file("./compressed.gz", std::ios::binary);
if (!file.is_open())
ERROR();
std::vector<unsigned char> buffer(std::istreambuf_iterator<char>(file), {});
Then you will have it in the vector buffer. Access the size via buffer.size() and to get raw data use buffer.data() Note that since buffer is an std::vector if it goes outside of scope it will be destructed so the data will be deleted.
I have a python test script that performs an API streaming using requests.post(). It looks like so:
response = requests.post(url_events, data="XYZ", stream=True, headers = {"A":"B"})
if (response.ok):
for chunk in response.iter_content(chunk_size=256):
print chunk
I'm trying to figure out how can I have the same logic but using C++. From what I found the curl library may help, however I cannot find how to pass data field. This is the code I have so far:
CURL* connection = curl_easy_init();
// set url
curl_easy_setopt(connection, CURLOPT_URL, url_events);
// set header
struct curl_slist* headers = NULL;
headers = curl_slist_append(headers, "A:B");
code = curl_easy_setopt(connection, CURLOPT_HTTPHEADER, headers);
// set streaming callback that will print every received message
curl_easy_setopt(connection, CURLOPT_WRITEFUNCTION, printCallback);
// start connection
code = curl_easy_perform(connection);
// ...
curl_easy_cleanup(connection);
curl_slist_free_all(headers);
I was looking through the curl.h file trying to find how to specify the data field, but nothing seems to fit (based on the name)?
Am I on the right track? Would using curl be the right approach for my task, or should I be looking into some other C,C++ libraries? An example that does the same task as above request.post() is appreciated, or a suggestion how to achieve the same using curl.
I'm having a bit of a strange problem with libcurl. Essentially, while downloading a file from an HTTP server, it's outputting some garbage bytes at the end of the file. Whereas the file should be 1,710,017 bytes, the library instead writes 1,712,128, i.e. 2,111 more. I suspect it's some sort of buffering issue, as the latter number is a multiple of 2^12 (and 2^13, but it conforms to multiples of 2^12 in other cases). The extra data is either a respective number of bytes read from another part of the file (it only seems to read from one of 4 addresses each time, all towards the end), or in one case the byte CD repeated 2,111 times.
Relevant code:
std::string url; // defined elsewhere
FILE* data; // initialized elsewhere with option "wb"
CURL* query = curl_easy_init();
curl_easy_setopt(query, CURLOPT_WRITEDATA, data);
curl_easy_setopt(query, CURLOPT_URL, url);
curl_easy_setopt(query, CURLOPT_FOLLOWLOCATION, true);
curl_easy_setopt(query, CURLOPT_SSL_VERIFYPEER, false);
CURLcode res = curl_easy_perform(query);
Also: the same issue occurs when using a simple write callback, and the issue occurs with any given remote server, not just this particular one.
Edit #1: I can only replicate this on Windows (tested on two machines with the same library files). It works on Debian.
Edit #2: It also occurs when libcurl is built on my laptop. To provide additional context, I am building from Marc Hörsken's ZIP (available from the official curl downloads page) using a VC14 environment on Windows 10.
I'm chalking this up to some weird VC bug or otherwise anomalous behavior. Rewriting the problem code to use an fstream instead of a FILE did the trick.
I'll preface this by saying I'm still a new C/C++ programmer, so please excuse me for what may be a redundant question.
I'm writing a program in C/C++ to interact with this website: http://www.youtube-mp3.org/.
From what I understand, to get my program to download a link for me I'll have to send a POST request to the server containing the URL I want to convert, then find a way of getting it to follow the URL that is generated allowing me to download the file. I also understand that libcurl is a good way of doing this sort of thing in C/C++.
I've tried using the POST examples on the libcurl website (http://curl.haxx.se/libcurl/c/simplepost.html and one other) but neither seems to work. In addition, I'm not sure how to then get my program to follow the link that appears saying 'Download' . I've tried sending a POST request, then telling my program to get the html source of the page and store this in a file, but that file doesn't seem to contain any download link. When this is done through a browser, the page source definitely includes a working download link.
Would really appreciate some help, as I'm not sure whether I've got completely the wrong idea!
EDIT: My question wasn't very clear at all. Here is the relevant code I'm using for the POST request:
static const char *postthis="http://www.youtube.com/watch?v=KMU0tzLwhbE";
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "http://www.youtube-mp3.org/");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postthis);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, (long)strlen(postthis));
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
}
And for writing the html source to file:
static size_t write_data(void *ptr, size_t size, size_t nmemb, void *stream)
{
int written = fwrite(ptr, size, nmemb, (FILE *)stream);
return written;
}
{
static const char *filename = "head.txt";
FILE *htmlfile;
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
// open the file
htmlfile = fopen(filename,"w");
if (htmlfile == NULL) {
curl_easy_cleanup(curl);
return -1;
}
curl_easy_setopt(curl,CURLOPT_WRITEDATA, htmlfile);
curl_easy_perform(curl);
/* close the header file */
fclose(htmlfile);
/* always clean up */
curl_easy_cleanup(curl);
}
Your code does not work because you are assuming the wrong logic to begin with.
http://www.youtube-mp3.org does NOT use POST, in fact its download form doesn't even submit to a server-side URL at all. When you click on the "Convert Video" button, a client-side JavaScript is invoked to process the input URL, download the relevant information from YouTube, and modify the calling page's HTML to display the actual download link and video preview image. This is why you don't see the download link when you simply retrieve the HTML - you are not invoking the JavaScript that performs the actual work of preparing the download link. And you will not be able to do that from an application (without a LOT of extra work), it has to be done inside of a web browser that has a real JavaScript engine and a real DOM for the script to manipulate.
I want to create a cURL request in some C++ code which will get me the length of a file in a server without downloading the file. For that, I use some cURL options to tell I only want headers in the request response, and then I examine the response to get the file length.
I'm setting the following request options:
curl_easy_setopt(_curl_handle, CURLOPT_HEADER, 1);
curl_easy_setopt(_curl_handle, CURLOPT_NOBODY, 1);
Then processing the request, waiting for the response, which shows a OK=200, and finally enquiring about the file length:
curl_easy_getinfo(_curl_handle, CURLINFO_CONTENT_LENGTH_UPLOAD, &dResult);
But I get a file length of -1. According to cURL documentation, that means size is unknown. How can it happen that cURL doesn't get the file length information from the server?
CURLINFO_CONTENT_LENGTH_UPLOAD is the number of bytes uploaded. You need to use CURLINFO_CONTENT_LENGTH_DOWNLOAD instead.
Note that if the server dynamically generates the data, the length may be different when you actualy download the file versus just downloading its headers.
Also note that if the server sends data as compressed when downloaded, there may not be any size available in the headers (if the Transfer-Encoding header is used instead of the Content-Length header), so CURLINFO_CONTENT_LENGTH_DOWNLOAD would still return -1. The only way to know the size in that situation would be to download it in full.
Have you tried with CURLINFO_CONTENT_LENGTH_DOWNLOAD instead?
need call perform()
curl_easy_setopt(_curl_handle, CURLOPT_HEADER, 1);
curl_easy_setopt(_curl_handle, CURLOPT_NOBODY, 1);
curl_easy_perform(_curl_handle);
curl_easy_getinfo(_curl_handle, CURLINFO_CONTENT_LENGTH_UPLOAD,
&dResult);