I'm trying to download a zip file using curl from a localhost server
http://localhost:8080/zip/json.zip?assetIds=<comma-separated asset ids>
when I type the url on my browser the file starts downloading with no problem.
So when I tried to use curl to an already existing zip file :
RestClient::response RestClient::get(const std::string& url)
{
RestClient::response ret = {};
CURL *curl = NULL;
CURLcode res = CURLE_OK;
FILE *fp
curl = curl_easy_init();
if (curl)
{
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
char outfilename[FILENAME_MAX] = "/Users/stage/Documents/temp/json.zip";
fp = fopen(outfilename,"wb");
curl_easy_setopt(curl, CURLOPT_CAINFO, "./ca-bundle.crt");
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, false);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, RestClient::write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
int i=fclose(fp);
if( i==0)
system("unzip -j json.zip");
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, RestClient::write_callback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &ret);
curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, RestClient::header_callback);
curl_easy_setopt(curl, CURLOPT_HEADERDATA, &ret);
res = curl_easy_perform(curl);
if (res != CURLE_OK)
{
ret.body = "Failed to query.";
ret.code = -1;
return ret;
}
long http_code = 0;
curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code);
ret.code = static_cast<int>(http_code);
curl_easy_cleanup(curl);
curl_global_cleanup();
}
return ret;
}
and the function for writing to the file
size_t RestClient::write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
return fwrite(ptr, size, nmemb, stream);
}
when I run the code I get a message :
Archive: json.zip
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile directory in one of json.zip or
json.zip.zip, and cannot find json.zip.ZIP, period.
The json.zip file that used to contain an image becomes an EMPTY file not even a zip :/
Does somebody know what went wrong?
I was having the exact same issue with the curl command in my script (not C++). I finally looked at the .zip file that was downloaded in a text editor and found it contained an HTML error message instead of being the actual file I thought I downloaded. I then did some debugging in my script and copied the URL I was creating and pasted it into my browser. I got the same error message when I checked the URL.
Long story short, I had a typo in my URL. I would venture to guess that you're having a very similar issue.
In my case however, it was as easy as calling the shell command "curl -o", your C++ code seems a bit more complicated than what I was doing.
I hope this helps in same way.
Related
I am trying to download a zip file from Artifactory using the libcurl library in the below C++ code. But I am getting the following error:
json
{
"errors" : [ {
"status" : 401,
"message" : "Unauthorized"
} ]
}
static size_t WriteCallback(void *ptr, size_t size, size_t nmemb, FILE *stream) {
size_t written;
written = fwrite(ptr, size, nmemb, stream);
return written;
}
int main()
{
CURL *curl;
CURLcode res;
std::string readBuffer;
FILE *fp;
const char outfilename[FILENAME_MAX] = "./Users/shrabana/.myOrg/test_media/Test.zip";
curl = curl_easy_init();
std::string command = "https://artifactory.corp.myOrg.com/artifactory/generic-test-media/Test.zip";
if(curl)
{
fp = fopen(outfilename,"wb");
curl_easy_setopt(curl, CURLOPT_URL, command.c_str());
curl_easy_setopt(curl, CURLOPT_HEADER, "X-JFrog-Art-Api:$ARTIFACTORY_API_KEY -O");
curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
int i = fclose(fp);
if (i == 0)
std::cout<<"All is well"<<std::endl;
system("unzip -j Test.zip");
}
}
As you are getting 401 error while trying to download a zip file from Artifactory, in order to rule out issues with regards to the authentication/API key, please do try retrieving the artifact from the direct curl/wget command using the direct user password instead of the API.
If you are able to retrieve using the direct password then try regenerating the API key from the user profile page and see if that helps in resolving the reported issue.
Basically this is my code :
int main()
{
CURL *curl;
FILE *fp;
CURLcode res;
std::string readBuffer;
curl = curl_easy_init();
char outfilename[FILENAME_MAX] = "C:\\Users\\admin\\desktop\\test.txt";
if(curl) {
fp = fopen(outfilename,"wb");
curl_easy_setopt(curl, CURLOPT_URL, "http://www.example.com");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "user=123&pass=123");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
Sleep(1000);
curl_easy_cleanup(curl);
fclose(fp);
}
return EXIT_SUCCESS;
}
The output is successfully saved in the text file.
My concern is how to extract specific content in between specific tags.
For example i want only the content between < bla> .............. < /bla> .
Whats the easiest way and thank you.
In your Example, you are dumping the response from the website to a file, libcURL writes the data returned by the webpage that you hit as it is, it does not take efforts for restructuring the returned data.
You can obtain the data in a memory, by defining the write_data function, which needs the following format only:
size_t write_data(char *ptr, size_t size, size_t nmemb, void *userdata);
Once you get the data in a memory, you can parse it and restructure it as required.
See Example Here for using write_data function.
For XML Parsing you may use This sample code
I used Curl 7.2.9 and checked connection this way:
Here's example:
curl = curl_easy_init();
bool result = false;
if(curl)
{
curl_easy_setopt(curl, CURLOPT_URL, m_checkConnectionUrl);
CURLcode res = curl_easy_perform(curl);
}
if(res != CURLE_OK)
{
}
else
{
// connection is available
}
Now I switched to curl-7.33.0 and got *CURLE_WRITE_ERROR* error,
and to make it work I must code it like
std::string output;
char* encodedUrl = curl_easy_escape(curl, m_checkConnectionUrl, 0);
curl_easy_setopt(curl, CURLOPT_POST, 0);
curl_easy_setopt(curl, CURLOPT_URL, encodedUrl);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeMemoryCurlCallbackStub);
CURLcode res = curl_easy_perform(curl);
But I don't need to write anything. Any ideas?
Manily the Curl option *CURLOPT_WRITEFUNCTION* is used to have a certain amount of data periodically(at the callback functoin) to handle a large file download. I don't see any reason to use this with your curl purpose, regardless the version.
Remove the *CURLOPT_POST*(by default its 0) and *CURLOPT_WRITEFUNCTION* from the code and it should work. If it doesn't, then you are doing something wrong at other places in your code!
Also, if you are checking whether the URL is ok or not, then using CURL is ok. But to only check for connection, you can only check whether the port 80 of the domain is on or not.
You need to write a writecallback as well
size_t CurlWriteCallback(char* buf, size_t size, size_t nmemb, void* up)
{
TRACE("CURL - Response received:\n%s", buf);
TRACE("CURL - Response handled %d bytes:\n%s", size*nmemb);
// tell curl how many bytes we handled
return size*nmemb;
}
if(curl)
{
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &CurlWriteCallback);
curl_easy_setopt(curl, CURLOPT_URL, m_checkConnectionUrl);
CURLcode res = curl_easy_perform(curl);
}
Old question, but I have just encountered a similar problem. After some more googling this is the solution:
curl_easy_setopt(curl, CURLOPT_CONNECT_ONLY, 1L);
curl_easy_perform(curl);
// OK, now we are connected (if nothing bad happened),
// but it would be nice to communicate with the server:
curl_easy_setopt(curl, CURLOPT_CONNECT_ONLY, 0L);
//now we can do the actual communication
I used this to separate authentication from the actual emails sending.
I am currently trying to make an updater for my software project. I need it to be able to download multiple files, I don't mind if they download in sync or one after each other, whatever is easier (file size is not an issue). I followed the example from the libcurl webpage and a few other resources and came up with this:
#include <iostream>
#include <stdio.h>
#include <curl/curl.h>
#include <string.h>
size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
size_t written;
written = fwrite(ptr, size, nmemb, stream);
return written;
}
int main(void){
for (int i = 0; i < 2;){ //download 2 files (loop twice)
CURL *curl;
FILE *fp;
CURLcode res;
char *url = "http://sec7.org/1024kb.txt"; //first file URL
char outfilename[FILENAME_MAX] = "C:\\users\\grant\\desktop\\1024kb.txt";
curl = curl_easy_init();
if (curl){
fp = fopen(outfilename,"wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
}
url = "http://sec7.org/index.html"; //I want to get a new file this time
outfilename[FILENAME_MAX] = "C:\\users\\grant\\desktop\\index.html";
}
return 0;
}
The first issue is if i remove the new file assignments (*url = "http://...") and just try to loop the download code twice, the program simply stops responding. This occurs in any combination of the download being called more than once in the program. The other issue is that I am unable to change the value of the character array outfilename[FILENAME_MAX]. I feel like this is just some silly error I am making but no solution comes to mind. Thank you!
Why not put this in a function and call it twice?
Your syntax for the arrays is all wrong, plus all the variables inside the loop are local, which means they are destroyed after each loop iteration.
What Conspicuous Compiler said. That's what's causing your program to freeze; it's stuck in an infinite loop because i is never > 2.
Put your code into a function like so:
void downloadFile(const char* url, const char* fname) {
CURL *curl;
FILE *fp;
CURLcode res;
curl = curl_easy_init();
if (curl){
fp = fopen(fname, "wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
}
}
And call it twice with the relevant file names and urls:
downloadFile("http://sec7.org/1024kb.txt", "C:\\users\\grant\\desktop\\1024kb.txt");
downloadFile("http://sec7.org/index.html", "C:\\users\\grant\\desktop\\index.html");
The example function is very bad though, it's just an example. You should alter it to return error codes/throw exceptions, and stuff like that.
I am new to the libcurl and found a way to download a single file from the ftp server. Now my requirement is to download all files in a directory and i guess it was not supported by libcurl. Kindly suggest on libcurl how to download all files in directory or is there any other library similar to libcurl?
Thanks in advance.
Here is a sample piece of code.
static size_t GetFilesList_response(void *ptr, size_t size, size_t nmemb, void *data)
{
FILE *writehere = (FILE *)data;
return fwrite(ptr, size, nmemb, writehere);
}
bool FTPWithcURL::GetFilesList(char* tempFile)
{
CURL *curl;
CURLcode res;
FILE *ftpfile;
/* local file name to store the file as */
ftpfile = fopen(tempFile, "wb"); /* b is binary, needed on win32 */
curl = curl_easy_init();
if(curl)
{
curl_easy_setopt(curl, CURLOPT_URL, "ftp://ftp.example.com");
curl_easy_setopt(curl, CURLOPT_USERPWD, "username:password");
curl_easy_setopt(curl, CURLOPT_WRITEDATA, ftpfile);
// added to #Tombart suggestion
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, GetFilesList_response);
curl_easy_setopt(curl, CURLOPT_DIRLISTONLY, 1);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
fclose(ftpfile); //
if(CURLE_OK != res)
return false;
return true;
}
You need the list of files on the FTP server. Which isn't straightforward as each FTP server might return a different format of file listing...
Anyway, the ftpgetresp.c example shows a way to do it, I think. FTP Custom CUSTOMREQUEST suggests another way.
Just use CURLOPT_WILDCARDMATCH feature.
Sample code:
https://curl.haxx.se/libcurl/c/ftp-wildcard.html