libcurl half as fast as curl - c++

Question

I am using libcurl to download files using the SFTP (protocol), it takes libcurl about twice as much times as it does using the curl executable.
This is the curl example I followed when designing this:
size_t my_fwrite(char* buffer, size_t size, size_t nmemb, File* file)
{
return size * nmemb; //Just return the size to prove that it's not the write
}
curl_global_init(CURL_GLOBAL_DEFAULT);
CURL* curl = curl_easy_init();
/** Setup URL/PORT/USERNAME/PASSWORD*/
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, my_fwrite);
CURLcode res = curl_easy_perform(curl);
if (CURLE_OK != res) {
// we failed
std::cout << "we failed due to '" << curl_easy_strerror(res) << "'\n";
}
// Close the file
curl_global_cleanup();
When running, I measured the time using chrono steady_clock, and it used about 39 seconds to download 119mb. Whereas using the curl command it took 16 seconds. I tried with lager files as well, and the same applied. When downloading 500mb libcurl used around 3 minutes, where curl would use 1,5 minutes. The curl binary and the API is compiled using CMake.
The curl command was:
curl -k sftp://username:password#host.com:port/path/to/file.zip --output file.zip
Do note that I used the compiled curl and not the system downloaded curl, as that one did not support sftp
EDIT:
It is compiled as release.
Running the very sleepy profiler on a debug compile executable gave me that most of the time was spent in Curl_multi_wait screeshot
CMAKE flags:
set(CMAKE_USE_LIBSSH2 ON)
set(CURL_ZLIB ON)
set(CMAKE_USE_OPENSSL ON)
set(BUILD_CURL_EXE ON)
set(CURL_STATICLIB ON)
EDIT2:
I compiled my code on both Ubuntu and Windows (GCC 10.1.0 and VS 2019 16.6.3), and both of them suffer from the same issue that using Libcurl is twice as slow than using the Curl binary I compile.