I'm using curl for uploads and downloads and also try to include the provided progress bar from curl. I managed to get the progress bar working when uploading files, but unfortunately the callback function only receives 0 values on the download.
Here are the options that are set for the download:
::curl_easy_reset( m_pimpl->curl ) ;
::curl_easy_setopt( m_pimpl->curl, CURLOPT_SSL_VERIFYPEER, 0L ) ;
::curl_easy_setopt( m_pimpl->curl, CURLOPT_SSL_VERIFYHOST, 0L ) ;
::curl_easy_setopt( m_pimpl->curl, CURLOPT_HEADERFUNCTION, &CurlAgent::HeaderCallback ) ;
::curl_easy_setopt( m_pimpl->curl, CURLOPT_HEADERDATA, this ) ;
::curl_easy_setopt( m_pimpl->curl, CURLOPT_HEADER, 0L ) ;
::curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, method.c_str() ); // "GET" in download
::curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, error ) ;
::curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
::curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &CurlAgent::Receive ) ;
::curl_easy_setopt(curl, CURLOPT_WRITEDATA, this ) ;
//setting the progress callback function
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);
curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, progress_callback);
curl_easy_setopt(curl, CURLOPT_XFERINFODATA, this);
CURLcode curl_code = ::curl_easy_perform(curl);
ANd this is the callback used for the progress bar:
static int progress_callback(void *ptr, curl_off_t TotalDownloadSize, curl_off_t finishedDownloadSize, curl_off_t TotalToUpload, curl_off_t NowUploaded) {
curl_off_t processed = (TotalDownloadSize > TotalToUpload) ? finishedDownloadSize : NowUploaded;
curl_off_t total = (TotalDownloadSize > TotalToUpload) ? TotalDownloadSize : TotalToUpload;
...
return 0;
}
As mentioned when I perform uploads of files the parameters TotalToUpload and NowUploaded contain the correct values. But when downloading all four parameters contain 0!?
Do I have to set another option when downloading files to receive the correct sizes?
Alternative solution
I found an alternative solution, buy using another request that delivers information about the files on the drive which also contains the file size.
In the set callback write function
::curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &CurlAgent::Receive )
the current downloaded size is given as parameter with which it is then possible to create the progress bar.
Here is also the documentation of the used service and the requests:
Per the libcurl documentation:
CURLOPT_XFERINFOFUNCTION explained
Unknown/unused argument values passed to the callback will be set to zero (like if you only download data, the upload size will remain 0). Many times the callback will be called one or more times first, before it knows the data sizes so a program must be made to handle that.
If the callback is never giving you non-zero values during a download, then either:
there is a bug in libcurl (less likely)
libcurl simply doesn't know the sizes (more likely), such as if the download is encoded in such a way that prevents calculating the sizes effectively.
Related
I am using libcurl.dll version 7.72.0.0 in Visual Studio for a C++ application.
When I am posting a JSON request to an HTTPS url on one Windows 7 PC with Service Pack 1, the request is posted successfully. But on other PC with same OS as Windows 7 Service Pack 1, I am getting response code 35 from curl_easy_perform(): SSL connect error.
What could be the reason of the same C++ code working on one Windows 7 PC and getting an error on other Windows 7 PC? Both PCs are on the same network.
Following is the C++ code:
curl_easy_setopt(curl, CURLOPT_URL, Url.c_str());
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, header);
// send all data to this function
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
// we pass our 'chunk' struct to the callback function
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)&sResponse);
//some servers don't like requests that are made without a user-agent
// field, so we provide one
curl_easy_setopt(curl, CURLOPT_USERAGENT, "libcurl-agent/1.0");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postthis);
printf("\npostthisData=%s\n", postthis);
if (itsHTTPSRequest)
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0);
// if we don't provide POSTFIELDSIZE, libcurl will strlen() by
// itself
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, (long)strlen(postthis));
curl_easy_setopt( curl, CURLOPT_CONNECTTIMEOUT, m_RWTimeOut );
curl_easy_setopt( curl, CURLOPT_LOW_SPEED_TIME, m_RWTimeOut);
curl_easy_setopt( curl, CURLOPT_LOW_SPEED_LIMIT, 30L);
I am trying to use libcurl C++ to make REST/HTTP requests. I noticed curl_easy_perform blocks but if I set CURLOPT_READFUNCTION it doesn't. I just want to understand why that is, I am new to libcurl or HTTP/REST protocol.
Here is the code:
m_pCurl = curl_easy_init();
curl_easy_setopt(m_pCurl, CURLOPT_URL, "https://blahblahblah/api/auth/user/login");
curl_easy_setopt(m_pCurl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(m_pCurl, CURLOPT_POST, 1);
curl_easy_setopt(m_pCurl, CURLOPT_COOKIE, "SKEY=BLAHBLAHBLAH");
struct curl_slist *list = NULL;
list = curl_slist_append(list, "Accept: application/json");
list = curl_slist_append(list, "Connection: keep-alive");
list = curl_slist_append(list, "Expect:");
list = curl_slist_append(list, "Content-Type: application/json");
list = curl_slist_append(list, "x-website-parameters: LALALALA");
curl_easy_setopt(m_pCurl, CURLOPT_HTTPHEADER, list);
// Callbacks
readarg_t rarg;
// readcb is a callback function
// Removing the two lines below will cause curl_easy_perform to hang
curl_easy_setopt(m_pCurl, CURLOPT_READFUNCTION, readcb);
curl_easy_setopt(m_pCurl, CURLOPT_READDATA, &rarg);
CURLcode res = curl_easy_perform(m_pCurl);
Note: Some of the encoded data are changed above.
Any help would be greatly appreciated.
Thanks,
K
According to The Manual...
CURLOPT_READFUNCTION explained
...
If you set this callback pointer to NULL, or don't set it at all, the default internal read function will be used. It is doing an fread() on the FILE * userdata set with CURLOPT_READDATA.
However you also don't set CURLOPT_READDATA. So looking again at The manual...
CURLOPT_READDATA explained
...
By default, this is a FILE * to stdin.
So the reason your program "hangs" appears to be because it is waiting for something to arrive on the standard input stdin.
So the way it is supposed to work is this.
1) If you do nothing the data sent to the server comes from the standard input (which is often the keyboard).
2) If you set only CURLOPT_READDATA then it must be a FILE* you opened to an input file that contains the data you want to send.
3) If you set CURLOPT_READFUNCTION then CURLOPT_READDATA can point to anything your function needs to fulfil its task of sending data to the server.
I'm trying to manage the progress of a download with libcurl in C++.
I have managed to do this with curl_easy, but the issue with curl_easy is that it blocks the program until the request has been made.
I need to use curl_mutli so the http request is asynchronous, but when I try changing to curl_multi, my progress function stops working.
I have the following curl_easy request code:
int progressFunc(void* p, double TotalToDownload, double NowDownloaded, double TotalToUpload, double NowUploaded) {
std::cout << TotalToDownload << ", " << NowDownloaded << std::endl;
return 0;
}
FILE* file = std::fopen(filePath.c_str(), "wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, false);
curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, progressFunc);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeData);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, file);
CURLcode res = curl_easy_perform(curl);
which works perfectly and prints to the console the progress of the download.
However, when trying to modify this code to use curl_multi instead, the file does not download correctly (shows 0 bytes) and the download progress callback function shows only 0, 0.
FILE* file = std::fopen(filePath.c_str(), "wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, false);
curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, progressFunc);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeData);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, file);
curl_multi_add_handle(curlm, curl);
int runningHandles;
CURLMcode res = curl_multi_perform(curlm, &runningHandles);
TL; DR: you are supposed to call curl_multi_perform in loop. If you don't use event loop and poll/epoll, you should probably stick with using curl_easy in separate thread.
The whole point of curl_multi API is not blocking: instead of magically downloading entire file in single call, you can use epoll or similar means to monitor curl's non-blocking sockets and invoke curl_multi_perform each time some data arrives from network. When you use it's multi-mode, curl itself does not start any internal threads and does not monitor it's sockets — you are expected to do it yourself. This allows writing highly performant event loops, that run multiple simultaneous curl transfers in the same thread. People, who need that, usually already have the necessary harness or can easily write it themselves.
The first time you invoke curl_multi_perform it will most likely return before the DNS resolution completes and/or before the TCP connection is accepted by remote side. So the amount of payload data transferred in first call will indeed be 0. Depending on server configuration, second call might not transfer any payload either. By "payload" I mean actual application data (as opposed to DNS requests, SSL negotiation, HTTP headers and HTTP2 frame metadata).
To actually complete a transfer you have to repeatedly invoke epoll_wait, curl_multi_perform and number of other functions until you are done. Curl's corresponding example stops after completing one transfer, but in practice it is more beneficial to create a permanently running thread, that handles all HTTP transfers for application's lifetime.
I have a case where I'm using libcurl with c++ to download a 240 MB file, but it takes 15 minutes to do so. I have made sure that my write callback is as fast as possible. It is just writing into an in-memory buffer that is plenty big-enough for the data. When I use the curl command to download this same file from the same server, it takes less than a minute. When I use a browser to download the file, it also takes less than a minute. Is it possible that I'm using libcurl incorrectly? Here's a snippet of my code...
wxString postFields;
postFields += "package_name=" + packageName;
if( desiredVersion != 0 )
postFields += wxString::Format( "&package_version=v%d", desiredVersion );
curl_easy_reset( curlHandleEasy );
curl_slist_free_all( headers );
headers = nullptr;
headers = curl_slist_append( headers, "Content-Type: application/x-www-form-urlencoded" );
headers = curl_slist_append( headers, "Accept: application/x-zip-compressed" );
url = "http://" + packageServer + ":7000/package_download";
urlData = url.c_str();
binResponse = new BinaryResponse( packageSize );
curl_easy_setopt( curlHandleEasy, CURLOPT_HTTPHEADER, headers );
curl_easy_setopt( curlHandleEasy, CURLOPT_POSTFIELDS, postFieldsData );
curl_easy_setopt( curlHandleEasy, CURLOPT_URL, urlData );
curl_easy_setopt( curlHandleEasy, CURLOPT_WRITEFUNCTION, &Response::WriteCallback );
curl_easy_setopt( curlHandleEasy, CURLOPT_WRITEDATA, binResponse );
curlCode = curl_easy_perform( curlHandleEasy );
Is there something wrong with my request setup? If I change my write callback to be a dummy routine that just claims to have written the data, but just throws it away (to be as fast as possible), my download rate is still super slow.
Is it possible that the bottle neck is some sort of security scanning on the network that I'm being subjected to that the browser and curl command aren't?
I had claimed to have tested with a dummy write function, but I actually hadn't. When I tested with a dummy write function, the download speed was fast.
So I investigated why my write function was slow and discovered that I was using an in-memory stream class that wasn't initialized with the required buffer size, so it was growing as needed. The growth of the buffer was probably small, and every time it grew, it probably needed to copy the entire contents of the old buffer into the new one....so, long story short: I'm dumb, and the write stream was slow.
Now I initialize my memory stream to the total size of the file so that it never has to grow. Ugh! Problem solved.
I'm using some auto-update function in my program. In case of connection failure, I would like the program to keep trying up to 15 seconds, and then announce failure. In order to achieve that, I used the following curl_easy_setopt for cURL easy option:
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 15);
But then I discovered that if the download takes more than 15 seconds, the timeout error is announced.
How can I restrict the 15 seconds only to the case of failure? I.e., if there's no connection for 15 seconds?
More information
The full option list I use is the following:
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); //verify ssl peer
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); //verify ssl hostname
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, CurlWrite_CallbackFunc_StdString);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &s);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, FALSE);
curl_easy_setopt(curl, CURLOPT_PROGRESSDATA, this); //pointer to the current class as it's a GUI program
curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, CurlProgress_CallbackFunc_UpdateProgress);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 20);
and the call to start is done through:
CURLcode res = curl_easy_perform(curl);
If you require more information, please let me know.
Thank you.
Instead of
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 20);
use
curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 15);
curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, 1);
curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, 15);
The first line is the timeout for the connection phase. After the connection is established the timeout becomes irrelevant, but the two following lines make sure that if the average speed in a 15 second time frame drops below 1 byte per second, then the operation is aborted.
Also worth noting that curl will not try to reestablish any connection if it is dropped, because a TCP connection can still be hold if the physical connection is (temporarly) lost up until one of the sides decides to timeout.