I am using cocos2d-x game engine to develop a game. Game fetches lot of data from the server. So to reduce the loading time and data consumption , i used gzip encoding.
curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, "gzip,deflate");
But strangely, i see garbage at the end of each http response and when i don't use the gzip , every http response is ok and no garbage in the end of http response.
Please suggest what can be possible reason for this issue. Your help will be appreciated.
Thanks.
Try
curl_easy_cleanup(curl);
And
curl_global_cleanup();
after you finished sending request by curl_easy_perform(), then see if this error still exists.
I have faced the same bug in C language with the same library.
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postString);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, strlen(postString));
You can try to ensure that the length of POSTFIELDS is same as the POSTFIELDSIZE.
Related
I'm trying to manage the progress of a download with libcurl in C++.
I have managed to do this with curl_easy, but the issue with curl_easy is that it blocks the program until the request has been made.
I need to use curl_mutli so the http request is asynchronous, but when I try changing to curl_multi, my progress function stops working.
I have the following curl_easy request code:
int progressFunc(void* p, double TotalToDownload, double NowDownloaded, double TotalToUpload, double NowUploaded) {
std::cout << TotalToDownload << ", " << NowDownloaded << std::endl;
return 0;
}
FILE* file = std::fopen(filePath.c_str(), "wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, false);
curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, progressFunc);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeData);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, file);
CURLcode res = curl_easy_perform(curl);
which works perfectly and prints to the console the progress of the download.
However, when trying to modify this code to use curl_multi instead, the file does not download correctly (shows 0 bytes) and the download progress callback function shows only 0, 0.
FILE* file = std::fopen(filePath.c_str(), "wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, false);
curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, progressFunc);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeData);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, file);
curl_multi_add_handle(curlm, curl);
int runningHandles;
CURLMcode res = curl_multi_perform(curlm, &runningHandles);
TL; DR: you are supposed to call curl_multi_perform in loop. If you don't use event loop and poll/epoll, you should probably stick with using curl_easy in separate thread.
The whole point of curl_multi API is not blocking: instead of magically downloading entire file in single call, you can use epoll or similar means to monitor curl's non-blocking sockets and invoke curl_multi_perform each time some data arrives from network. When you use it's multi-mode, curl itself does not start any internal threads and does not monitor it's sockets — you are expected to do it yourself. This allows writing highly performant event loops, that run multiple simultaneous curl transfers in the same thread. People, who need that, usually already have the necessary harness or can easily write it themselves.
The first time you invoke curl_multi_perform it will most likely return before the DNS resolution completes and/or before the TCP connection is accepted by remote side. So the amount of payload data transferred in first call will indeed be 0. Depending on server configuration, second call might not transfer any payload either. By "payload" I mean actual application data (as opposed to DNS requests, SSL negotiation, HTTP headers and HTTP2 frame metadata).
To actually complete a transfer you have to repeatedly invoke epoll_wait, curl_multi_perform and number of other functions until you are done. Curl's corresponding example stops after completing one transfer, but in practice it is more beneficial to create a permanently running thread, that handles all HTTP transfers for application's lifetime.
I’m using libcurl 7.37.0 in my C++ project to communicate with remote by FTP protocol.
below is the code.
curl_easy_setopt(CurlSessionHandle, CURLOPT_URL, remoteFileUrl);
curl_easy_setopt(CurlSessionHandle, CURLOPT_UPLOAD, ON);
// Set the input local file handle
curl_easy_setopt(CurlSessionHandle, CURLOPT_READDATA, localFileHandle);
// Set on/off all wanted options
// Enable ftp data connection
curl_easy_setopt(CurlSessionHandle, CURLOPT_NOBODY, OFF);
// Create missing directory into FTP path
curl_easy_setopt(CurlSessionHandle, CURLOPT_FTP_CREATE_MISSING_DIRS , ON) ;
// Set the progress function, in order to check the stop transfer request
curl_easy_setopt(CurlSessionHandle, CURLOPT_NOPROGRESS, OFF);
curl_easy_setopt(CurlSessionHandle, CURLOPT_PROGRESSFUNCTION, progressCb);
curl_easy_setopt(CurlSessionHandle, CURLOPT_PROGRESSDATA, this);
CURLcode Result = curl_easy_perform(CurlSessionHandle);
many times i have observed uploading of files is failing due to error code 28.
CURLE_OPERATION_TIMEDOUT (28)
Operation timeout. The specified time-out period was reached according to the conditions.
i didn't set any timeout in the code, after doing a lot of search i came to know we can use CURLOPT_TIMEOUT to set the timeout value, by default it's value is 0 where it doesn't timeout until it finishes the respective operation, in my case i performed file upload operation.
after going through the wireshark logs, i have observed that when data transfer is initiated from port 20, i see libcurl sends [FIN,ACK] without any known reason to port 21, because of that remote sends response code 426(transfer aborted) to libcurl and it returns 28 error code to application.
Please check the image which has wireshark traces.
Source IP: 18 is Linux server & Destination IP: 36 is Windows remote system
This problem is happening randomly.
Could anyone know the reason & a way to avoid this problem.
When this issue occurs, reset the curl context CurlSessionHandle and re initialize it. This might work.
Let's say you have this
curl_easy_setopt(pCurl, CURLOPT_URL, url);
curl_multi_add_handle(pCurlMulti, pCurl);
curl_multi_perform(...)
// now we are waiting for response from the server
// while waiting, could we call
curl_easy_setopt(pCurl, CURLOPT_URL, newUrl);
// without curl_multi_remove_handle & curl_multi_add_handle?
No, you don't change the URL of an active transfer. Instead, enable pipelining on the multi handle and then add one easy handle for each transfer; the multi handle will pipeline the requests over the same connection if possible.
I'm trying to do HTTP requests to multiple targets, and I need to them to run (almost) exactly at the same moment.
I'm trying to create a thread for each request, but I don't know why Curl is crashing when doing the perform. I'm using an easy-handle per thread so in theory everything should be ok...
Has anybody had a similar problem? or Does anyone know if the multi interface allows you to choose when to perform all the requests?
Thanks a lot.
EDIT:
Here is an example of the code:
void Clazz::function(std::vector<std::string> urls, const std::string& data)
{
for (auto it : urls)
{
std::thread thread(&Clazz::DoRequest, this, it, data);
thread->detach();
}
}
int Clazz::DoRequest(const std::string& url, const std::string& data)
{
CURL* curl = curl_easy_init();
curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Expect:");
headers = curl_slist_append(headers, "Content-Type: application/json");
curl_easy_setopt(curl, CURLOPT_POST, 1);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data.c_str());
curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 15);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt (curl, CURLOPT_FAILONERROR, 1L);
//curlMutex.lock();
curl_easy_perform(curl);
//curlMutex.unlock();
long responseCode = 404;
curl_easy_getinfo (curl, CURLINFO_RESPONSE_CODE, &responseCode);
curl_easy_cleanup(curl);
curl_slist_free_all(headers);
}
I hope this can help, thanks!
Are you calling curl_global_init anywhere? Perhaps rather early in your main() method?
Quoting from http://curl.haxx.se/libcurl/c/curl_global_init.html:
This function is not thread safe. You must not call it when any other thread in the program (i.e. a thread sharing the same memory) is running. This doesn't just mean no other thread that is using libcurl. Because curl_global_init calls functions of other libraries that are similarly thread unsafe, it could conflict with any other thread that uses these other libraries.
Quoting from http://curl.haxx.se/libcurl/c/curl_easy_init.html:
If you did not already call curl_global_init, curl_easy_init does it automatically. This may be lethal in multi-threaded cases, since curl_global_init is not thread-safe, and it may result in resource problems because there is no corresponding cleanup.
It sounds like you're not calling curl_global_init, and letting curl_easy_init take care of it for you. Since you're doing it on two threads simultaneously, you're hitting the thread unsafe scenario, with the lethal result that was mentioned.
After being able to debug properly in the device y have found that the problem is an old know issue with curl.
http://curl.haxx.se/mail/lib-2010-11/0181.html
after using CURLOPT_NOSIGNAL in every curl handle the crash has disappeared. :)
I have a HTTP streaming video server that I access using a url like so:
https://192.168.50.23:8011/livevideo/8
If I paste this url in to my web browser address bar I can see live video but after a few minutes the video stops streaming. If I kill the browser and repeat the process I can get video streaming for another few minutes before it stops again.
I figured that it must be something to do with the web browser caching the MJPEG frames and running out of memory so as an experiment I mocked up a simple HTML page like so:
<!DOCTYPE html>
<html>
<body>
<img src="https://192.168.50.23:8011/livevideo/8" width="500" height="500">
</body>
</html>
And the result is that the vieo streams constantly and never stops. So I guess the tag is dealing with disposing of the MJPEG frames and not causing a crash like before.
I used FireFox to analyse the HTTP requests and responses for both scenarios above to see if there is anything different and here is the results:
URL pasted into webbrowser address bar:
URL embedded into web page:
The only differences seems to be the Accept: Parameters.
Now to move onto my real problem. I am using the same url in my C++ curl program and I am seeing the exact same behaviour where I receive video data for a few mins and then all of a sudden the curl callbacks stop.
I have used the following headers in my curl program:
CURL *pEasy = curl_easy_init ();
curl_easy_setopt ( pEasy, CURLOPT_USERNAME, user.c_str() );
curl_easy_setopt ( pEasy, CURLOPT_PASSWORD, pass.c_str() );
curl_easy_setopt ( pEasy, CURLOPT_URL, urlToConnectTo.c_str() );
//Set authentication
curl_easy_setopt ( pEasy, CURLOPT_HTTPAUTH, CURLAUTH_BASIC );
curl_easy_setopt ( pEasy, CURLOPT_SSL_VERIFYPEER, false );
curl_easy_setopt ( pEasy, CURLOPT_SSL_VERIFYHOST, false );
curl_easy_setopt ( pEasy, CURLOPT_HEADER, TRUE );
curl_easy_setopt ( pEasy, CURLOPT_NOBODY, FALSE );
curl_easy_setopt ( pEasy, CURLOPT_WRITEFUNCTION, OnReceiveHttpBodyResponse );
curl_easy_setopt ( pEasy, CURLOPT_WRITEDATA, pEasy );
struct curl_slist *headers=NULL;
curl_slist_append( headers, "User-Agent: MyCurlDll");
curl_slist_append( headers, "Content-Type: text/xml");
curl_slist_append( headers, "Connection: Keep-Alive");
curl_slist_append( headers, "Accept: image/png, text/xml, text/html, application/xml");
curl_slist_append( headers, "Cache-Control: max-age=0");
curl_easy_setopt(pEasy, CURLOPT_HTTPHEADER, headers);
curl_multi_add_handle(m_curlMulti, pEasy);
//Process this curl handle in another function
What can I do to stop this behaviour in CURL? I assume it must be caching is the same way as the browser was doing it.
Sorry, but you've ended up with the wrong conclusion and thus you're sort of barking up the wrong tree here.
curl doesn't cache anything, it simply sends a HTTP request to the server (and with CURLOPT_VERBOSE you can easily inspect it) and then it pipes all data it receives on to the write callback that you provide. There's no caching, no middle layers, no magic.
If you stop getting traffic before it should've ended, it is because there's no more data being delivered or rather being received by libcurl. It could be the server that stopped sending or it can be something in your network that interferes. libcurl sent the request and it'll keep waiting for data to arrive until the entire thing has been delivered.