Empty Content-Type in curl_easy_perform GET request - libcurl

Kindly disregard this issue, the problems seems to lie in werkzeug.
I believe I have detected a problem with Content-Type for GET requests sent using curl_easy_perform, in libcurl version 7.29 (CentOS 7).
The problem is that the HTTP header Content-Type is present (with empty value) in the request:
Content-Type:
In Ubuntu 18.04, with libcurl 7.58, this problem doesn't occur (as expected the header is not present in the request). I've searched Changlogs but without finding anything about this assumed fix. I did find some mention of the very same bug fixed in the PHP-curl library.
So, my question is simply: is it true that this behaviour has been fixed somewhere between 7.29 and 7.58 ?
I just made a test implementation, as short as possible, to see the problem.
This is the entire code:
#include <stdio.h> // fprintf, ...
#include <stdlib.h> // exit
#include <curl/curl.h> // curl
#include <curl/curlver.h> // VERSION
int main(int argC, char* argV[])
{
CURL* curlP;
CURLcode cc;
printf("curl version: %s\n", LIBCURL_VERSION);
curlP = curl_easy_init();
if (curlP == NULL)
{
fprintf(stderr, "curl_easy_init failed: %s\n", curl_easy_strerror(cc));
exit(1);
}
curl_easy_setopt(curlP, CURLOPT_URL, "localhost:1028/accumulate");
cc = curl_easy_perform(curlP);
if (cc != CURLE_OK)
{
fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(cc));
exit(2);
}
curl_easy_cleanup(curlP);
return 0;
}
The code is edited in a file called get.c and I compile it like this:
gcc get.c -o get -lcurl
The test runs in CentOS 7 with libcurl 7.58 (that I installed manually under /usr/local).
I have a simple test program, an accumulator, written in python, listening to port 1028, recording all requests that enters, and that spits it out on receiving a request with the URL path /dump.
Here is the entire sequence:
kz#centos7:curlTest> gcc get.c -o get -lcurl
kz#centos7:curlTest> ./get
curl version: 7.58.0
kz#centos7:curlTest> curl localhost:1028/dump
GET http://localhost:1028/accumulate
Content-Length:
Host: localhost:1028
Accept: */*
Content-Type:
=======================================
kz#centos7:curlTest>
As can be seen in this test, the GET request (sent by get to the accumulator) includes HTTP headers Content-Length and Content-Type without value.
The exact same test in Ubuntu 18.04 looks like this:
kz#xps:libcurlTest> gcc get.c -o get -lcurl
kz#xps:libcurlTest> ./get
curl version: 7.58.0
kz#xps:libcurlTest> curl localhost:1028/dump
GET http://localhost:1028/accumulate
Accept: */*
Host: localhost:1028
=======================================
kz#xps:libcurlTest>
Note the absence of Content-Lengthand Content-Type.
Before implementing this little test program, I made numerous attempts to avoid the Content-Type, by setting it in its slist to "Content-Type:", without space, with one space and with two spaces after the colon. I also tried without colon and of course without adding it to the slist (in which case it comes with the default content-type). I also tried to tell libcurl that the request is a GET, don't remember the details of that call right now.
Now, the error is probably mine, as I have very little experience using libcurl. It's just strange that all is OK in Ubuntu but not in CentOS ...

Related

libcpr not working properly when using it for HTTP request

I am using libcpr for http requests in Visual Studio 2019 IDE. I downloaded it using vcpkg from microsoft. The sample code below is from cpr github page https://github.com/libcpr/cpr#:~:text=%23include%20%3C,return%200%3B%0A%7D
#include <cpr/cpr.h>
int main(int argc, char** argv) {
cpr::Response r =
cpr::Get(cpr::Url{"https://api.github.com/repos/whoshuu/cpr/contributors"},
cpr::Authentication{"user", "pass", cpr::AuthMode::BASIC},
cpr::Parameters{{"anon", "true"}, {"key", "value"}});
r.status_code; // 200
r.header["content-type"]; // application/json; charset=utf-8
r.text; // JSON text string
return 0;
}
This doesn't work! It is giving error "namespace "cpr" has no member "AuthMode". This problem is not with this only. There was some other stuff that gives similar error e.g. https://docs.libcpr.org/advanced-usage.html#https-options:~:text=cpr%3A%3ASslOptions%20sslOpts%20%3D%20cpr%3A%3ASsl(ssl%3A%3ACaBuffer%7B%22%2D%2D%2D%2D%2DBEGIN%20CERTIFICATE%2D%2D%2D%2D%2D%5B...%5D%22%7D)%3B%0Acpr%3A%3AResponse%20r%20%3D%20cpr%3A%3AGet(cpr%3A%3AUrl%7B%22https%3A//www.httpbin.org/get%22%7D%2C%20sslOpts)%3B in this case "CaBuffer" has same issue.
Any help would be appreciated!!
Thanks
Looks to me like a versioning issue. AuthMode exists in the latest header file, but does not exist in the version 1.8 header file, which is presumably what you have.
So, either downgrade your code, or upgrade your installation.
Sample code from version 1.8 is here

46: regex error 17 for `(dryad-bibo/v)[0-9].[0-9]', (match failed)

I'm trying to determine the mime-type for several types of files using libmagic and the following bit of code:
auto handle = ::magic_open(MAGIC_MIME_TYPE);
::magic_load(handle, NULL);
// Both of these fail with the same error
// file_path being a const char* with the path to the file.
auto type2 = ::magic_file(handle, file_path);
// svg_content being an std::vector<char> with the contents of the file.
//auto type2 = ::magic_buffer(handle, svg_content.data(), svg_content.size());
if(!type2)
{
std::cout << magic_error(handle) << std::endl;
}
::magic_close(handle);
But for any file or buffer I try I receive regex error, either being or similar to:
46: regex error 17 for `(dryad-bibo/v)[0-9].[0-9]', (match failed)
For example with this .svg file:
<svg xmlns="http://www.w3.org/2000/svg" id="flag-icon-css-no" viewBox="0 0 640 480">
<path fill="#ed2939" d="M0 0h640v480H0z"/>
<path fill="#fff" d="M180 0h120v480H180z"/>
<path fill="#fff" d="M0 180h640v120H0z"/>
<path fill="#002664" d="M210 0h60v480h-60z"/>
<path fill="#002664" d="M0 210h640v60H0z"/>
</svg>
What I've tried so far:
libmagic 5.35
libmagic 5.39
libmagic 5.40
libmagic from opensource.apple
setting LC_TYPE and LANG to "C"
I'm linking against a version of libmagic that was built locally, could there be anything I have missed while building? Are any of the calls incorrect or is there something I'm missing?
I get similar errors when trying to run the related file binary that was compiled locally. Whereas when I use the file command that is available by default I do get image/svg+xml as output.
Edit
To build libmagic (for macOS and Ubuntu), I followed these steps:
Downloaded relevant release from Github
autoreconf --install
./configure
make
make install
Update
It looks like the regex at the bottom of this file is causing issues (at least for the svg):
https://github.com/file/file/blob/b56b58d499dbe58f2bed28e6b3c297fe7add992e/magic/Magdir/dataone
Update 2
Something strange going on; On the system where I've got it working, magic_version() reports 540, as expected. But on the systems where it fails with this error, magic_version() reports 538.
This makes little sense to me as I can't find that version on the system itself anywhere and when I run ./file --version in the build library, it reports file-5.40.
Very dissatisfying answer, but it was linking against GoogleTest causing this error somehow, not even running any tests, just linking against it.
I switched to using Catch2 instead and the issue was resolved.
Tested on Ubuntu 20.04.:
Clone the repo
git clone git#github.com:file/file.git
cd file/
Try this in fresh clone of the repo:
autoreconf -f -i
./configure --disable-silent-rules
make -j4
make -C tests check
And see whether there are any errors reported. After installation with make install, grab some valid xml file named as "test.xml" and put it to some folder together with this main.c:
#include <stdio.h>
#include <magic.h>
int main(void)
{
char *actual_file = "test.xml";
const char *magic_full;
magic_t magic_cookie;
/* MAGIC_MIME tells magic to return a mime of the file,
but you can specify different things */
magic_cookie = magic_open(MAGIC_MIME);
if (magic_cookie == NULL) {
printf("unable to initialize magic library\n");
return 1;
}
printf("Loading default magic database\n");
if (magic_load(magic_cookie, NULL) != 0) {
printf("cannot load magic database - %s\n", magic_error(magic_cookie));
magic_close(magic_cookie);
return 1;
}
magic_full = magic_file(magic_cookie, actual_file);
printf("%s\n", magic_full);
magic_close(magic_cookie);
return 0;
}
(Courtesy to vivithemage.)
Compile and try:
$ gcc main.c -lmagic
$ ./a.out
Loading default magic database
text/xml; charset=us-ascii
If it does not work on your system, report bug on project's bugtracker with specification of your OS and architecture. You can try to hotfix your problem by removing offending record from the file you have found in your update.

libcurl half as fast as curl

I am using libcurl to download files using the SFTP (protocol), it takes libcurl about twice as much times as it does using the curl executable.
This is the curl example I followed when designing this:
size_t my_fwrite(char* buffer, size_t size, size_t nmemb, File* file)
{
return size * nmemb; //Just return the size to prove that it's not the write
}
curl_global_init(CURL_GLOBAL_DEFAULT);
CURL* curl = curl_easy_init();
/** Setup URL/PORT/USERNAME/PASSWORD*/
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, my_fwrite);
CURLcode res = curl_easy_perform(curl);
if (CURLE_OK != res) {
// we failed
std::cout << "we failed due to '" << curl_easy_strerror(res) << "'\n";
}
// Close the file
curl_global_cleanup();
When running, I measured the time using chrono steady_clock, and it used about 39 seconds to download 119mb. Whereas using the curl command it took 16 seconds. I tried with lager files as well, and the same applied. When downloading 500mb libcurl used around 3 minutes, where curl would use 1,5 minutes. The curl binary and the API is compiled using CMake.
The curl command was:
curl -k sftp://username:password#host.com:port/path/to/file.zip --output file.zip
Do note that I used the compiled curl and not the system downloaded curl, as that one did not support sftp
EDIT:
It is compiled as release.
Running the very sleepy profiler on a debug compile executable gave me that most of the time was spent in Curl_multi_wait screeshot
CMAKE flags:
set(CMAKE_USE_LIBSSH2 ON)
set(CURL_ZLIB ON)
set(CMAKE_USE_OPENSSL ON)
set(BUILD_CURL_EXE ON)
set(CURL_STATICLIB ON)
EDIT2:
I compiled my code on both Ubuntu and Windows (GCC 10.1.0 and VS 2019 16.6.3), and both of them suffer from the same issue that using Libcurl is twice as slow than using the Curl binary I compile.

Compile libcurl with HTTPS/SSL

I'm using cURL 7.42.1 and MinGW. I'm having problems compiling libcurl with ssl support (for HTTPS). Ive tried defining USE_OPENSSL on lib\curl_setup.h but it still didn't work.
Ive been trying to find a solution to this for hours but i couldn't find one.
You must build libcurl with SSL support enabled, adding a define to curl_setup.h won't have the desired effect.
i.e.:
./configure --with-winssl
Then init curl the following way in your application:
if (curl_global_init(CURL_GLOBAL_ALL) != CURLE_OK)
/* handle error */

Initializing SSL and libcurl and getting "out of memory"

I intend to do https requests with libcurl and openssl with a C++ program.
I initialized libcurl with curl_global_init(CURL_GLOBAL_ALL) as described in the documentation. Then I use an curl_easy handle that I initialize, populate with headers and a body, and send everything out to ´https://example.com:443/foo´. It works for non-https connections.
Looking around I find that there may be other libraries that are already getting an SSL context which is what causes libcurl to fail in doing precisely that. I get the following error message:
curl_easy_perform failed: Out of memory
In my case I am using libmicrohttpd which I initialize with
mhdDaemon = MHD_start_daemon(MHD_USE_THREAD_PER_CONNECTION | MHD_USE_SSL,
htons(port),
NULL,
NULL,
connectionTreat, NULL,
MHD_OPTION_HTTPS_MEM_KEY, httpsKey,
MHD_OPTION_HTTPS_MEM_CERT, httpsCertificate,
MHD_OPTION_CONNECTION_MEMORY_LIMIT, memoryLimit,
MHD_OPTION_SOCK_ADDR, (struct sockaddr*) &sad,
MHD_OPTION_NOTIFY_COMPLETED, requestCompleted, NULL,
MHD_OPTION_END);
So I am indeed using openSSL somewhere else. The thing is, if I take out the MHD_USE_SSL part it does not fix the problem.
This is the list of libraries that are linked to the application (I'm using cmake):
-lmicrohttpd
-lmongoclient
-lboost_thread
-lboost_filesystem
-lboost_system
-lpthread
Is there any of the others that could be loading SSL? Is microhttpd loading it anyways even if I comment out the MHD_USE_SSL flag (plus all other related flags)? Could there be any other reason for this error?
I'm not aware of any problem in libcurl that would cause this error code to get returned if indeed a memory allocation function doesn't fail. Using OpenSSL in multiple modules does not cause such a failure. (I am the lead developer of libcurl.)
So, run your app with VERBOSE set to libcurl, or even strace to see which syscall that fails and it should give you more clues.
As the descibed in this answer, you might need to disable SSLv3 if you are on Ubuntu 16.04 like so
curl_easy_setopt(curl_, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1_2));
It was disabled on Ubuntu 16.04 for security reasons, see more here.