error 411 Length Required c++, libcurl PUT request - c++

Even though I set in header Content-Lenght I'm getting 411 error. I'm trying to send PUT request.
struct curl_slist *headers = NULL;
curl = curl_easy_init();
std::string paramiters =
"<data_file><archive>false</archive><data_type_id>0a7a184a-dcc6-452a-bcd3-52dbd2a83ea2</data_type_id><data_file_name>backwardstep.stt</data_file_name><description>connectionfile</description><job_id>264cf297-3bc7-42e1-8edc-5e2948ee62b6</job_id></data_file>";
if (curl) {
headers = curl_slist_append(headers, "Accept: */*");
headers = curl_slist_append(headers, "Content-Length: 123");
headers = curl_slist_append(headers, "Content-Type: application/xml");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_VERBOSE, true);
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
curl_easy_setopt(curl, CURLOPT_URL,
"..url/data_files/new/link_upload.xml");
curl_easy_setopt(curl, CURLOPT_USERPWD, "kolundzija#example.ch:PASS");
curl_easy_setopt(curl, CURLOPT_HEADER, 1L);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, paramiters.c_str());
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE,
strlen(paramiters.c_str()));
curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1L);
res = curl_easy_perform(curl);
and this is response from SERVER:
Host: cloud...
Transfer-Encoding: chunked
Accept: */*
Content-Length: 123
Content-Type: application/xml
Expect: 100-continue
* The requested URL returned error: 411 Length Required
* Closing connection #0

Ok, I honestly can not find your error. But you should have an example from the curl website (first google hit for "curl put c code"): http://curl.haxx.se/libcurl/c/httpput.html
Maybe mixing the easy and advanced interface confuses curl.
What confuses me are the options CURLOPT_POSTFIELDS and CURLOPT_POSTFIELDSIZE. This is a put request, so why are they even there? With PUT the arguments are in the URL. The body is opaque, at least from the perspective of HTTP.

You DON'T need to use a file and do NOT use custom requests, INstead set the UPLOAD and PUT options as it is specified in the documentation here:
http://curl.haxx.se/libcurl/c/httpput.html
Unlike the example above where they use a file as your data structure you can USE ANYTHING to hold your data.It's all on using a callback function with this option:
CURLOPT_READFUNCTION
The difference is made on how you set your callback function which only has to do two things:
1.-measure the size of your payload (your data) in bytes
2.-copy the data to the memory address that curl passes to the callback (that is the first argument on your call back function, the FIRST void pointer in this definition)
static size_t read_callback(void *ptr, size_t size, size_t nmemb, void *stream)
That is the ptr argument.
Use memcpy to copy the data.
Take a look at this link. I ran into the same problem as you and was able to solve it using this approach,one thing YOU need to keep in mind is that you ALSO need to set the file size before sending the curl request.
How do I send long PUT data in libcurl without using file pointers?
Use CURLOPT_INFILESIZE or CURLOPT_INFILESIZE_LARGE for that.

Related

libcurl: curl_easy_perform blocks unless CURLOPT_READFUNCTION is set

I am trying to use libcurl C++ to make REST/HTTP requests. I noticed curl_easy_perform blocks but if I set CURLOPT_READFUNCTION it doesn't. I just want to understand why that is, I am new to libcurl or HTTP/REST protocol.
Here is the code:
m_pCurl = curl_easy_init();
curl_easy_setopt(m_pCurl, CURLOPT_URL, "https://blahblahblah/api/auth/user/login");
curl_easy_setopt(m_pCurl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(m_pCurl, CURLOPT_POST, 1);
curl_easy_setopt(m_pCurl, CURLOPT_COOKIE, "SKEY=BLAHBLAHBLAH");
struct curl_slist *list = NULL;
list = curl_slist_append(list, "Accept: application/json");
list = curl_slist_append(list, "Connection: keep-alive");
list = curl_slist_append(list, "Expect:");
list = curl_slist_append(list, "Content-Type: application/json");
list = curl_slist_append(list, "x-website-parameters: LALALALA");
curl_easy_setopt(m_pCurl, CURLOPT_HTTPHEADER, list);
// Callbacks
readarg_t rarg;
// readcb is a callback function
// Removing the two lines below will cause curl_easy_perform to hang
curl_easy_setopt(m_pCurl, CURLOPT_READFUNCTION, readcb);
curl_easy_setopt(m_pCurl, CURLOPT_READDATA, &rarg);
CURLcode res = curl_easy_perform(m_pCurl);
Note: Some of the encoded data are changed above.
Any help would be greatly appreciated.
Thanks,
K
According to The Manual...
CURLOPT_READFUNCTION explained
...
If you set this callback pointer to NULL, or don't set it at all, the default internal read function will be used. It is doing an fread() on the FILE * userdata set with CURLOPT_READDATA.
However you also don't set CURLOPT_READDATA. So looking again at The manual...
CURLOPT_READDATA explained
...
By default, this is a FILE * to stdin.
So the reason your program "hangs" appears to be because it is waiting for something to arrive on the standard input stdin.
So the way it is supposed to work is this.
1) If you do nothing the data sent to the server comes from the standard input (which is often the keyboard).
2) If you set only CURLOPT_READDATA then it must be a FILE* you opened to an input file that contains the data you want to send.
3) If you set CURLOPT_READFUNCTION then CURLOPT_READDATA can point to anything your function needs to fulfil its task of sending data to the server.

Synchronized curl requests

I'm trying to do HTTP requests to multiple targets, and I need to them to run (almost) exactly at the same moment.
I'm trying to create a thread for each request, but I don't know why Curl is crashing when doing the perform. I'm using an easy-handle per thread so in theory everything should be ok...
Has anybody had a similar problem? or Does anyone know if the multi interface allows you to choose when to perform all the requests?
Thanks a lot.
EDIT:
Here is an example of the code:
void Clazz::function(std::vector<std::string> urls, const std::string& data)
{
for (auto it : urls)
{
std::thread thread(&Clazz::DoRequest, this, it, data);
thread->detach();
}
}
int Clazz::DoRequest(const std::string& url, const std::string& data)
{
CURL* curl = curl_easy_init();
curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Expect:");
headers = curl_slist_append(headers, "Content-Type: application/json");
curl_easy_setopt(curl, CURLOPT_POST, 1);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data.c_str());
curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 15);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt (curl, CURLOPT_FAILONERROR, 1L);
//curlMutex.lock();
curl_easy_perform(curl);
//curlMutex.unlock();
long responseCode = 404;
curl_easy_getinfo (curl, CURLINFO_RESPONSE_CODE, &responseCode);
curl_easy_cleanup(curl);
curl_slist_free_all(headers);
}
I hope this can help, thanks!
Are you calling curl_global_init anywhere? Perhaps rather early in your main() method?
Quoting from http://curl.haxx.se/libcurl/c/curl_global_init.html:
This function is not thread safe. You must not call it when any other thread in the program (i.e. a thread sharing the same memory) is running. This doesn't just mean no other thread that is using libcurl. Because curl_global_init calls functions of other libraries that are similarly thread unsafe, it could conflict with any other thread that uses these other libraries.
Quoting from http://curl.haxx.se/libcurl/c/curl_easy_init.html:
If you did not already call curl_global_init, curl_easy_init does it automatically. This may be lethal in multi-threaded cases, since curl_global_init is not thread-safe, and it may result in resource problems because there is no corresponding cleanup.
It sounds like you're not calling curl_global_init, and letting curl_easy_init take care of it for you. Since you're doing it on two threads simultaneously, you're hitting the thread unsafe scenario, with the lethal result that was mentioned.
After being able to debug properly in the device y have found that the problem is an old know issue with curl.
http://curl.haxx.se/mail/lib-2010-11/0181.html
after using CURLOPT_NOSIGNAL in every curl handle the crash has disappeared. :)

libcurl http post timeout

I am using curl version 7.15.5 in multi-thread environment. Each thread is initializing and freeing its own curl object. Below is the code, executed for each thread:
CURL* curl = curl_easy_init();
tRespBuffer respBuffer = {NULL, 0};
char errorBuf[CURL_ERROR_SIZE +1];
struct curl_slist *headers=NULL;
headers = curl_slist_append(headers, "Content-Type: text/xml; charset=gbk");
headers = curl_slist_append(headers, "Expect:");
curl_easy_setopt(curl, CURLOPT_URL, url_);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS,encr.c_str());
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE,strlen(encr.c_str()));
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, HttpSmsServer::processHttpResponse);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void*)&respBuffer);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 20); // wait for 20 seconds before aborting the transacttion
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errorBuf); // error returned if any..
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1); // No signals allowed in case of multithreaded apps
res = curl_easy_perform(curl);
curl_slist_free_all(headers);
curl_easy_cleanup(curl);
All the four threads are posting data to http server simultaneously. I see HTTP response timeout for some of the POST requests (~3% of requests). Any idea what could be the reason of timeouts ? I assume http server should not take more than 20 seconds to respond back.
CURLOPT_TIMEOUT includes all the time of http request, have you transferred huge data?
CURLOPT_TIMEOUT:Pass a long as parameter containing the maximum time in seconds that you allow the libcurl transfer operation to take. Normally, name lookups can take a considerable time and limiting operations to less than a few minutes risk aborting perfectly normal operations.

Determine extension when downloading a file with curl?

I've created a program that downloads subtitles from INTERNET using curl
How to know the extension of the file I downloaded (.zip or .rar)?
here is my code (it is a part of function)
FILE* download=fopen("download.zip","wb");//i assume it's a zip
curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_POST,1);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS,post_data.c_str());
curl_easy_setopt(curl, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)");
curl_easy_setopt(curl, CURLOPT_URL,url.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write2file);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, download);
res = curl_easy_perform(curl);
if(res != CURLE_OK)
std::cout<<"Download Failed "<<std::endl;
curl_easy_cleanup(curl);
fclose(download);
You can use the curl_easy_getinfo() function, it has the content-type info:
CURLINFO_CONTENT_TYPE
Pass a pointer to a char pointer to receive the content-type of the
downloaded object. This is the value read from the Content-Type:
field. If you get NULL, it means that the server didn't send a valid
Content-Type header or that the protocol used doesn't support this.
Using the info you can give the right extension to the downloaded file by renaming it on disk.

How (i.e. what tool to use) to monitior headers sent by Curl (Cookie problem)

I am using Curl (libcurl) in a C++ aplication, and am unable to send cookies (I think).
I have Fiddler, TamperData and LiveHTTP Headers installed, but they are only useful for viewing browser traffic, and are (it would seem) unable of monitoring general network traffic on a machine, so when I run my machine, I cant see the header information being sent. However, when I view the page in a browser, when succesfully logged on, I can see that cookie information is being sent.
When running my app, I succesfully log onto the page, when I subsequently, try to fetch another page, the (page) data suggests that I am not logged on - i.e. "state" has somehow being lost.
My C++ code looks alright, so I dont know what is going wrong - this is why I need to:
First be able to view my machines network traffic (not just browser traffic) - which (free) tool?
Assuming I am using Curl incorrectly, whats wrong with my code? (the cookies are being retrieved and stored ok, it seems they are just not being sent with requests for some reason.
Here is the section of my class that deals with the cookie side of Http requests:
curl_easy_setopt(curl, CURLOPT_TIMEOUT, long(m_timeout));
curl_easy_setopt(curl, CURLOPT_USERAGENT,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; WOW64; SV1; .NET CLR 2.0.50727)");
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "cookies.txt");
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, "cookies.txt");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_AUTOREFERER, 1L);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, CurlCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, this);
Is there anything wrong with the above code?
You can use Wireshark (the former Ethereal) to view all the network traffic a machine is sending and receiving.
As Sean Carpenter said, Wireshark is the right tool to view network traffic. Start a capture and use http as a filter to see only HTTP traffic. If you just want to see HTTP requests/responses sent/received by Curl, set the CURL_VERBOSE option and look at stderr: curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L).
I believe you are using Curl correctly. Compile and run the following (complete) example; you will see that, the second time you run it (when cookies.txt exists) cookies are sent to the server.
Example code:
#include <stdio.h>
#include <curl/curl.h>
int main()
{
CURL *curl;
CURLcode success;
char errbuf[CURL_ERROR_SIZE];
int m_timeout = 15;
if ((curl = curl_easy_init()) == NULL) {
perror("curl_easy_init");
return 1;
}
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errbuf);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, long(m_timeout));
curl_easy_setopt(curl, CURLOPT_URL, "http://www.google.com/");
curl_easy_setopt(curl, CURLOPT_USERAGENT,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; WOW64; SV1; .NET CLR 2.0.50727)");
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "cookies.txt");
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, "cookies.txt");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_AUTOREFERER, 1L);
if ((success = curl_easy_perform(curl)) != 0) {
fprintf(stderr, "%s: %s\n", "curl_easy_perform", errbuf);
return 1;
}
curl_easy_cleanup(curl);
return 0;
}