How to disable Expect 100 continue in libcurl - libcurl

I am using CURLOPT_POST to send an https message. During running, my application stuck at:
Expect: 100-continue
Done waiting for 100-continue

From George's Log -- When curl sends 100-continue, you can set the Expect header to the empty string:
curl -X POST -H "Expect:" http://mywebsite.com/an/endpoint -F data=#myfile
Explanation
In particular you can set an empty "Expect:" header in your put / post request. i found some sample code in the post-callback tutorial for curl that contains the following snippet with a DISABLE_EXPECT "sneeze" guard:
#ifdef DISABLE_EXPECT
/*
Using POST with HTTP 1.1 implies the use of a "Expect: 100-continue"
header. You can disable this header with CURLOPT_HTTPHEADER as usual.
NOTE: if you want chunked transfer too, you need to combine these two
since you can only set one list of headers with CURLOPT_HTTPHEADER. */
/* A less good option would be to enforce HTTP 1.0, but that might also
have other implications. */
{
struct curl_slist *chunk = NULL;
chunk = curl_slist_append(chunk, "Expect:");
res = curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
/* use curl_slist_free_all() after the *perform() call to free this
list again */
}
#endif
i keep an slist of headers to use for put / post requests. adding the equivalent of the above to that list works as advertised:
// Disable Expect: 100-continue
vc->slist = curl_slist_append(vc->slist, "Expect:");
...
curl_easy_setopt(vc->curl, CURLOPT_HTTPHEADER, vc->slist);

Related

Using libcurl "multi" interface for single file download - C++

I am creating a C++ dll and I need to make a single non-blocking http request. What I mean is that at a certain point in the dll, I need to kick off a download of a file in the background, but resume the program after the single download is kicked off.
I am currently using libcurl as my http client. I read that the multi-interface (https://curl.haxx.se/libcurl/c/libcurl-multi.html) will give me non-blocking functionality, but all of the examples I see are for making multiple http requests simultaneously, and waiting for all of them to finish w/a while loop for example. Am I able to use this interface to make a single http request, and resume my program while it downloads? And if so, can it fire a callback upon completion? Or do I have to continually check the handle to see if the download is complete?
Example of single request with libcurl multi interface:
int main() {
curl_global_init(CURL_GLOBAL_DEFAULT);
// init easy curl session
auto curl_easy = curl_easy_init();
// set url
auto url = "https://[foo].exe";
curl_easy_setopt(curl_easy, CURLOPT_URL, url);
// set curl to write file to directory
auto path = "C:\\foo\\app.exe";
auto filePtr = std::fopen(path, "wb");
curl_easy_setopt(curl_easy, CURLOPT_WRITEFUNCTION, writeFile);
curl_easy_setopt(curl_easy, CURLOPT_WRITEDATA, filePtr);
// init multi curl session
auto multi_handle = curl_multi_init();
// add easy handle to multi handle
curl_multi_add_handle(multi_handle, curl_easy);
auto res = curl_multi_perform(multi_handle, &running);
auto running = 0;
while (running) {
res = curl_multi_perform(multi_handle, &running);
}
I'd like to not do the while (running) loop and go do other things in main while the file downloads. I may be misunderstanding the "non-blocking" nature of curl_multi_perform(). Is it non-blocking only in the sense that many requests can happen simultaneously? If so, I don't think using it to make a single request gets me anything I wouldn't have with curl_easy_perform. Apologies if this is veering outside of stack overflow, I don't want a libcurl tutorial. But should I instead be using something like a blocking libcurl call (curl_easy_perform) inside of std::async()?

libcurl: curl_easy_perform blocks unless CURLOPT_READFUNCTION is set

I am trying to use libcurl C++ to make REST/HTTP requests. I noticed curl_easy_perform blocks but if I set CURLOPT_READFUNCTION it doesn't. I just want to understand why that is, I am new to libcurl or HTTP/REST protocol.
Here is the code:
m_pCurl = curl_easy_init();
curl_easy_setopt(m_pCurl, CURLOPT_URL, "https://blahblahblah/api/auth/user/login");
curl_easy_setopt(m_pCurl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(m_pCurl, CURLOPT_POST, 1);
curl_easy_setopt(m_pCurl, CURLOPT_COOKIE, "SKEY=BLAHBLAHBLAH");
struct curl_slist *list = NULL;
list = curl_slist_append(list, "Accept: application/json");
list = curl_slist_append(list, "Connection: keep-alive");
list = curl_slist_append(list, "Expect:");
list = curl_slist_append(list, "Content-Type: application/json");
list = curl_slist_append(list, "x-website-parameters: LALALALA");
curl_easy_setopt(m_pCurl, CURLOPT_HTTPHEADER, list);
// Callbacks
readarg_t rarg;
// readcb is a callback function
// Removing the two lines below will cause curl_easy_perform to hang
curl_easy_setopt(m_pCurl, CURLOPT_READFUNCTION, readcb);
curl_easy_setopt(m_pCurl, CURLOPT_READDATA, &rarg);
CURLcode res = curl_easy_perform(m_pCurl);
Note: Some of the encoded data are changed above.
Any help would be greatly appreciated.
Thanks,
K
According to The Manual...
CURLOPT_READFUNCTION explained
...
If you set this callback pointer to NULL, or don't set it at all, the default internal read function will be used. It is doing an fread() on the FILE * userdata set with CURLOPT_READDATA.
However you also don't set CURLOPT_READDATA. So looking again at The manual...
CURLOPT_READDATA explained
...
By default, this is a FILE * to stdin.
So the reason your program "hangs" appears to be because it is waiting for something to arrive on the standard input stdin.
So the way it is supposed to work is this.
1) If you do nothing the data sent to the server comes from the standard input (which is often the keyboard).
2) If you set only CURLOPT_READDATA then it must be a FILE* you opened to an input file that contains the data you want to send.
3) If you set CURLOPT_READFUNCTION then CURLOPT_READDATA can point to anything your function needs to fulfil its task of sending data to the server.

How to stop Curl from caching data

I have a HTTP streaming video server that I access using a url like so:
https://192.168.50.23:8011/livevideo/8
If I paste this url in to my web browser address bar I can see live video but after a few minutes the video stops streaming. If I kill the browser and repeat the process I can get video streaming for another few minutes before it stops again.
I figured that it must be something to do with the web browser caching the MJPEG frames and running out of memory so as an experiment I mocked up a simple HTML page like so:
<!DOCTYPE html>
<html>
<body>
<img src="https://192.168.50.23:8011/livevideo/8" width="500" height="500">
</body>
</html>
And the result is that the vieo streams constantly and never stops. So I guess the tag is dealing with disposing of the MJPEG frames and not causing a crash like before.
I used FireFox to analyse the HTTP requests and responses for both scenarios above to see if there is anything different and here is the results:
URL pasted into webbrowser address bar:
URL embedded into web page:
The only differences seems to be the Accept: Parameters.
Now to move onto my real problem. I am using the same url in my C++ curl program and I am seeing the exact same behaviour where I receive video data for a few mins and then all of a sudden the curl callbacks stop.
I have used the following headers in my curl program:
CURL *pEasy = curl_easy_init ();
curl_easy_setopt ( pEasy, CURLOPT_USERNAME, user.c_str() );
curl_easy_setopt ( pEasy, CURLOPT_PASSWORD, pass.c_str() );
curl_easy_setopt ( pEasy, CURLOPT_URL, urlToConnectTo.c_str() );
//Set authentication
curl_easy_setopt ( pEasy, CURLOPT_HTTPAUTH, CURLAUTH_BASIC );
curl_easy_setopt ( pEasy, CURLOPT_SSL_VERIFYPEER, false );
curl_easy_setopt ( pEasy, CURLOPT_SSL_VERIFYHOST, false );
curl_easy_setopt ( pEasy, CURLOPT_HEADER, TRUE );
curl_easy_setopt ( pEasy, CURLOPT_NOBODY, FALSE );
curl_easy_setopt ( pEasy, CURLOPT_WRITEFUNCTION, OnReceiveHttpBodyResponse );
curl_easy_setopt ( pEasy, CURLOPT_WRITEDATA, pEasy );
struct curl_slist *headers=NULL;
curl_slist_append( headers, "User-Agent: MyCurlDll");
curl_slist_append( headers, "Content-Type: text/xml");
curl_slist_append( headers, "Connection: Keep-Alive");
curl_slist_append( headers, "Accept: image/png, text/xml, text/html, application/xml");
curl_slist_append( headers, "Cache-Control: max-age=0");
curl_easy_setopt(pEasy, CURLOPT_HTTPHEADER, headers);
curl_multi_add_handle(m_curlMulti, pEasy);
//Process this curl handle in another function
What can I do to stop this behaviour in CURL? I assume it must be caching is the same way as the browser was doing it.
Sorry, but you've ended up with the wrong conclusion and thus you're sort of barking up the wrong tree here.
curl doesn't cache anything, it simply sends a HTTP request to the server (and with CURLOPT_VERBOSE you can easily inspect it) and then it pipes all data it receives on to the write callback that you provide. There's no caching, no middle layers, no magic.
If you stop getting traffic before it should've ended, it is because there's no more data being delivered or rather being received by libcurl. It could be the server that stopped sending or it can be something in your network that interferes. libcurl sent the request and it'll keep waiting for data to arrive until the entire thing has been delivered.

error 411 Length Required c++, libcurl PUT request

Even though I set in header Content-Lenght I'm getting 411 error. I'm trying to send PUT request.
struct curl_slist *headers = NULL;
curl = curl_easy_init();
std::string paramiters =
"<data_file><archive>false</archive><data_type_id>0a7a184a-dcc6-452a-bcd3-52dbd2a83ea2</data_type_id><data_file_name>backwardstep.stt</data_file_name><description>connectionfile</description><job_id>264cf297-3bc7-42e1-8edc-5e2948ee62b6</job_id></data_file>";
if (curl) {
headers = curl_slist_append(headers, "Accept: */*");
headers = curl_slist_append(headers, "Content-Length: 123");
headers = curl_slist_append(headers, "Content-Type: application/xml");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_VERBOSE, true);
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
curl_easy_setopt(curl, CURLOPT_URL,
"..url/data_files/new/link_upload.xml");
curl_easy_setopt(curl, CURLOPT_USERPWD, "kolundzija#example.ch:PASS");
curl_easy_setopt(curl, CURLOPT_HEADER, 1L);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, paramiters.c_str());
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE,
strlen(paramiters.c_str()));
curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1L);
res = curl_easy_perform(curl);
and this is response from SERVER:
Host: cloud...
Transfer-Encoding: chunked
Accept: */*
Content-Length: 123
Content-Type: application/xml
Expect: 100-continue
* The requested URL returned error: 411 Length Required
* Closing connection #0
Ok, I honestly can not find your error. But you should have an example from the curl website (first google hit for "curl put c code"): http://curl.haxx.se/libcurl/c/httpput.html
Maybe mixing the easy and advanced interface confuses curl.
What confuses me are the options CURLOPT_POSTFIELDS and CURLOPT_POSTFIELDSIZE. This is a put request, so why are they even there? With PUT the arguments are in the URL. The body is opaque, at least from the perspective of HTTP.
You DON'T need to use a file and do NOT use custom requests, INstead set the UPLOAD and PUT options as it is specified in the documentation here:
http://curl.haxx.se/libcurl/c/httpput.html
Unlike the example above where they use a file as your data structure you can USE ANYTHING to hold your data.It's all on using a callback function with this option:
CURLOPT_READFUNCTION
The difference is made on how you set your callback function which only has to do two things:
1.-measure the size of your payload (your data) in bytes
2.-copy the data to the memory address that curl passes to the callback (that is the first argument on your call back function, the FIRST void pointer in this definition)
static size_t read_callback(void *ptr, size_t size, size_t nmemb, void *stream)
That is the ptr argument.
Use memcpy to copy the data.
Take a look at this link. I ran into the same problem as you and was able to solve it using this approach,one thing YOU need to keep in mind is that you ALSO need to set the file size before sending the curl request.
How do I send long PUT data in libcurl without using file pointers?
Use CURLOPT_INFILESIZE or CURLOPT_INFILESIZE_LARGE for that.

libcurl 7.26.0 : garbage at the end of every http response

I am using cocos2d-x game engine to develop a game. Game fetches lot of data from the server. So to reduce the loading time and data consumption , i used gzip encoding.
curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, "gzip,deflate");
But strangely, i see garbage at the end of each http response and when i don't use the gzip , every http response is ok and no garbage in the end of http response.
Please suggest what can be possible reason for this issue. Your help will be appreciated.
Thanks.
Try
curl_easy_cleanup(curl);
And
curl_global_cleanup();
after you finished sending request by curl_easy_perform(), then see if this error still exists.
I have faced the same bug in C language with the same library.
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postString);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, strlen(postString));
You can try to ensure that the length of POSTFIELDS is same as the POSTFIELDSIZE.