To donwload the file i am currently using
curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, Read_Cb);
curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, &ReadBuffer);
curl_easy_perform(curl_handle)
Can CURLOPT_RESUME_FROM be used for download ? So that i can resume the next transfer from that location.
Also does it internally send range request ?
The option CURLOPT_RESUME_FROM is more suitable for uploads, HTTP POST and HTTP PUT requests. You need to use CURLOPT_RANGE for downloads, HTTP GET and HTTP HEAD request. If you use CURLOPT_RESUME_FROM for downloads, it sets Range: <From>- HTTP header.
Related
I am not very familiar with curl or HTTP requests, but trying to learn.
In my case, I am trying to use libcurl in C++ (using Visual Studio 2019 on Windows 10) to perform a GET request. I tried solutions from Curl in C++ - Can't get data from HTTPS, but the only thing that worked for me was to disable SSL peer verification using:
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false);
Here is my code:
void getPairPrice(string & pair) {
string url(BINANCE_HOST);
url += "/api/v3/ticker/price?symbol=";
url += pair;
CURL* curl;
CURLcode res;
std::string readBuffer;
curl_global_init(CURL_GLOBAL_DEFAULT);
curl = curl_easy_init();
if (curl) {
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "GET");
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false);
curl_easy_setopt(curl, CURLOPT_ENCODING, "gzip");
struct curl_slist* headers = NULL;
headers = curl_slist_append(headers, PUBLIC_KEY_HEADER);
headers = curl_slist_append(headers, CONTENT_TYPE_HEADER);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
res = curl_easy_perform(curl);
//cout << "res : " << res << endl;
std::cout << "GET symbol " << pair << " price : " << readBuffer << std::endl;
const string jsonKey = "price";
cout << "Price : " << extractJsonValue(readBuffer, jsonKey) << endl;
}
curl_easy_cleanup(curl);
curl_global_cleanup();
}
Without disabling the SSL_VERIFYPEER option, the response is always 77. This is fine for testing, but I would like to know how to solve that when releasing my software. It seems that I should somehow download the host's SSL certificate in PEM format and point libcurl to it.
Can anyone help?
Error 77 is CURLE_SSL_CACERT_BADFILE
Problem with reading the SSL CA cert (path? access rights?)
The CURLOPT_SSL_VERIFYPEER documentation says:
When negotiating a TLS or SSL connection, the server sends a certificate indicating its identity. Curl verifies whether the certificate is authentic, i.e. that you can trust that the server is who the certificate says it is. This trust is based on a chain of digital signatures, rooted in certification authority (CA) certificates you supply. curl uses a default bundle of CA certificates (the path for that is determined at build time) and you can specify alternate certificates with the CURLOPT_CAINFO option or the CURLOPT_CAPATH option.
When CURLOPT_SSL_VERIFYPEER is enabled, and the verification fails to prove that the certificate is authentic, the connection fails. When the option is zero, the peer certificate verification succeeds regardless.
So yes, if you intend to validate the server's certificate, you need to provide libcurl with the Certificate Authority (CA) that was used to sign the server's certificate. If you don't have the CA available, you can't validate the certificate.
The documentation also says:
Authenticating the certificate is not enough to be sure about the server. You typically also want to ensure that the server is the server you mean to be talking to. Use CURLOPT_SSL_VERIFYHOST for that. The check that the host name in the certificate is valid for the host name you're connecting to is done independently of the CURLOPT_SSL_VERIFYPEER option.
WARNING: disabling verification of the certificate allows bad guys to man-in-the-middle the communication without you knowing it. Disabling verification makes the communication insecure. Just having encryption on a transfer is not enough as you cannot be sure that you are communicating with the correct end-point.
NOTE: even when this option is disabled, depending on the used TLS backend, curl may still load the certificate file specified in CURLOPT_CAINFO. curl default settings in some distributions might use quite a large file as a default setting for CURLOPT_CAINFO, so loading the file can be quite expensive, especially when dealing with many connections. Thus, in some situations, you might want to disable verification fully to save resources by setting CURLOPT_CAINFO to NULL - but please also consider the warning above!
I have a c++ application, which has to upload files. When not on a proxy, the FTP-upload works great. However, when the client is uploading using a proxy, LibCurl fails to build a good request. At least, I don't know how to give it the right information. Of course I tell LibCurl the proxy-address, which works great with HTTP-requests. However, the FTP-upload fails. The code I'm using:
struct curl_slist *LibcurlHeaders = NULL;
curl = curl_easy_init();
string ProxyAddress = "ProxyURL";
string JSONFileName = "dataonserver.json";
FILE* JSONFile = fopen("data.json", "rb");
if (curl) {
CurlResponse = "";
host = "ftp://host.com/" + JSONFileName;
if (ProxyAddress.length() > 0) {
curl_easy_setopt(curl, CURLOPT_PROXY, ProxyAddress.c_str());
}
curl_easy_setopt(curl, CURLOPT_URL, host.c_str());
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER , 1);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST , 1);
LibcurlHeaders = curl_slist_append(LibcurlHeaders, "Expect:");
curl_easy_setopt(curl, CURLOPT_USERPWD, (FTPUsername + ":" + FTPPassword).c_str());
curl_easy_setopt(curl, CURLOPT_CAINFO, FilePathSSLCertificate.c_str());
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, LibcurlHeaders);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1);
curl_easy_setopt(curl, CURLOPT_READDATA, JSONFile);
res = curl_easy_perform(curl);
if (res != CURLE_OK) {
LibcurlError(curl_easy_strerror(res), host);
}
curl_slist_free_all(LibcurlHeaders);
curl_easy_cleanup(curl);
}
fclose(JSONFile);
The response I get:
* timeout on name lookup is not supported
* Trying host IP...
* TCP_NODELAY set
* Connected to host (host IP) port 21 (#0)
* Server auth using Basic with user 'username#domain.com'
> PUT ftp://username#domain.com:password#domain.com/FileName HTTP/1.1
Host: domain.com:21
Authorization: Basic Q2xpZW50VXBsb2FkQGdvZmlsZXIub3JnOkNsaWVudFVwbG9hZDE=
Accept: */*
Proxy-Connection: Keep-Alive
Transfer-Encoding: chunked
220- This is the xxxx FTP proxy server.
220- My external IP address is xxxx and I have
220- a backup ftp proxy in xxxx. When setting up
220- access to 3rd parties, be sure to allow both source
220- addresses.
220-
220- All requests will come from one of the sources IP's below:
220- xxxxx (xxx Proxy)
220- xxxxx (xxx Proxy)
220-
220- To connect to a site:
220-
220- Enter the username and site name at the User: prompt.
220- For example, to log into ftp.uu.net as anonymous, you
220- would give anonymous#ftp.uu.net at the User: prompt.
220- Enter the site password at the Password: prompt.
220- If you are connecting as anonymous, give your email address.
220-
220 Type quit to disconnect.
500 Syntax error, command unrecognized.
500 Syntax error, command unrecognized.
500 Syntax error, command unrecognized.
500 Syntax error, command unrecognized.
500 Syntax error, command unrecognized.
500 Syntax error, command unrecognized.
500 Syntax error, command unrecognized.
500 Syntax error, command unrecognized.
500 Syntax error, command unrecognized.
You can normally only do FTP uploads over a HTTP proxy by "tunneling through" it. You ask libcurl to do this by setting the CURLOPT_HTTPPROXYTUNNEL option.
Without that option set, libcurl will convert your FTP upload to a HTTP PUT over the proxy.
With that option set, libcurl will issue a CONNECT to the proxy and ask for a tunnel to the destination server and then issue "ordinary FTP" commands to the server. Do note however that many HTTP proxies are not set up to allow CONNECT through proxies to other ports than 443 or perhaps a few more. Port 21 that is needed for a typical FTP transfer is rarely accepted.
I am writing an application in C ++ under Windovs to work with cloud services Amazon. I need to be authorized on the site to work with specific data and I want to get Access Token. The problem is that I want, I get authorization without using a browser, but I can not understand how. I use cURL in the project and send a request that looks like this:
int main(void) {
CURL *curl;
CURLcode res;
struct curl_slist *list = NULL;
FILE *Response = fopen("Response.txt", "wb+");
curl = curl_easy_init();
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, "https://www.amazon.com/ap/oa?client_id=%MY CLIENT ID%scope=clouddrive:read_all&response_type=token&redirect_uri=http://localhost");
list = curl_slist_append(list, "Accept: text/html");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, list);
curl_easy_setopt(curl, CURLOPT_HTTPGET, 1L);
curl_easy_setopt(curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, SaveData);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, Response);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if (res != CURLE_OK)
fprintf(Response, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
curl_slist_free_all(list);
/* always cleanup */
curl_easy_cleanup(curl);
}
When I get response I have a copy of the file with HTML Authorization Page. /Below is a part of the answer/
<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">
<html xmlns=\"http://www.w3.org/1999/xhtml\">
<head>
<script type='text/javascript'>var ue_t0=ue_t0||+new Date();</script>
<script type='text/javascript'>
var ue_csm = window,
ue_hob = +new Date();
If I can not use the browser, while I can open a small window with a Web browser (no frame) and to receive data from the server (Access Token). There is nothing in the documents it (Amazon Documentation).
If you want to make you application "talk" to AWS, you are supposed to use their APIs, and not the web console. API requests are stateless and signed with credentials, and don't depend on authentication sessions like the web console.
They have official SDKs for many popular languages, but not for C++, unfortunately. So, unless you find one somewhere, you'll have to implement your own client(s), but it's all HTTP(S)-based.
Another thing worth to mention is that every product has a separate and independent API (EC2, S3, RDS, etc). As far as I know, you won't find an all-in-one API for all AWS products. Documentation may be a bit sparse too, but most products are well-covered.
Good luck. ;)
I am using libcurl with C++ to send an HTTP request to another host (my friend's computer) over the Internet. Now, the other end uses a Node.js script to receive and handle the request.
I have tried the code (both C++ and Node) on my local network between my computers, and everything works just fine. This is the code, in case you wanna look at it:
CURL *handle;
string response;
curl_global_init(CURL_GLOBAL_WIN32);
handle = curl_easy_init();
curl_easy_setopt(handle, CURLOPT_URL, "10.10.10.10:7890"); //<= IP:PORT of the remote host
curl_easy_setopt(handle, CURLOPT_POSTFIELDS, "This is my POST request data!!");
curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, write_to_string);
curl_easy_setopt(handle, CURLOPT_WRITEDATA, &response);
curl_easy_perform(handle);
curl_easy_cleanup(handle);
and the Node.js code:
var http = require("http");
var server = http.createServer(function(request, response) {
request.on('data', function(chunk) {
console.log("We got a connection!");
console.log("Received POST data:");
console.log(chunk.toString());
});
});
server.listen(7890);
console.log("Server is listening");
As I said, on my local network it works very well, but why it doesn't my friend receive the request? I noticed that my code (C++ part) takes some time, before closing, as if it was trying to connect. Is this some kind of firewall problem? We have tried disabling it, but nothing changed. We are almost total noobs with networking, and we're trying to get into it.
If anybody could help us, thanks in advance!
Your friend's home router needs to know what to do with a connection being requested on port 7890 from the internet. It needs to be configured to forward any communication on that port to the specific internal IP address of your friend's computer.
Not all home routers are capable of port forwarding and many manufacturers have different ways of getting to the settings.
Any software firewall on the friend's computer also needs to know that communication over port 7890 is safe.
hopefully someone can answer this for me as I'm beating my head against the wall. I'm using LibCurl to do a simple file post to an HTTPS URL. Overall, it works, however contacting the site receiving the post, they apparently have an issue resolved, and I just don't get it.
Apparently, I prepare my request, put the URL, provide USER and PASSWORD for the site. The post then includes a CURL_FORMADD( for the parameters to add a single file attachment via multi-part form).
I had problems working out the CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST and finally got it working with the following settings.
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false );
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 1);
Here's the problem. Apparently, when I issue the curl_easy_perform(), it tries to send the request, the response comes back and says ... send me the certificate information and then apparently re-submits the entire package WITH the cert info. And then is accepted by the server I'm sending to.
So, how can I tell LibCurl to submit EVERYTHING... User, Password AND Certificate credentials all up front so the server I'm sending to doesn't reject the first request only to get the credentials the subsequent cycle.
Thanks.
---- EDITING via feedback.
While looking into more as mentioned by Eugen...
I've disabled the "CURLOPT_SSL_VERIFYPEER" from other readings that by doing so can open a transaction up to a "man-in-the-middle" compromise. I don't want that. By finding the method of authentication, it is BASIC, so when I am issuing the connection, I am sending along
curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC );
Now, it comes back with CURLE_SSL_CONNECT_ERROR. I'm trying to get the sdterror, but that's crashing via...
FILE *pLocalFile;
if( fopen_s( &pLocalFile, "MyHTTPS.Log", "w" ))
return -1;
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L );
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteFileCallback );
curl_easy_setopt(curl, CURLOPT_STDERR, pLocalFile );
As soon as I do the perform, I get the DLL caused an exception. So, I cant' even see what the connection failure is about .. which I strongly suspect is certificates.
You need to keep things apart to make the solving of this easier.
First you fix your SSL connection/handshake so that you don't get any SSL related problems or errors.
Then you make sure you provide the necessary HTTP authentication stuff and set basic, and you should only need to do a single HTTP request to get the correct response (assuming that is truly what the server wants).
The "DLL caused an exception" seems to be a third problem and is independent of the other stuff. It's hard to tell why you get that as we don't have a complete picture of what you're doing.