I'm trying to download an internet page using curl c++.
Using http analyzer I can see the header and use it in cURL.
When I execute my GET request by curl I have a crypted response content, something like this:
‹í=ksÛ8’Ÿ“ªùíÞÊ©1%‘Ô3‰=å8NÆ»ñccMfo§æX IH(BÇmewÿÏÝ߸ûc×
ð%‰’(Kžä®&›ÐÝèn¼^~ûúê´ÿï×gdNÜã§/ñq©7:ªðP;ïWÈýÄõ‚£Ê8§Ïëõ»»»ÚYþ¨®÷z½ú=f¨`FFã§O_NXH‰=¦~À£Jµ.|†02öÙp÷†"}êÐë
R¢ðè„UFÌc>
…_!¶ðBæN ÷Ê$ÍZ‡œ^Ü`Ñ9x€åÞ(¬¸‹`ì–P?:Yu‡8t /]î}">s*A8sY0f,¬p6…ì!»ëvTâú#ª0C6™Öm1™ú,„_Ÿ0ÄÍh;–ÑèÒv‡1Çh9tØëtMÖ`FË6›5Àó½Þlµ›ÝN§Õ¨ s8=ªPW½
CîB'L¸WGZ\²?ק>÷BUŽÙ6;³aši9òã>JúÔ‡Ô³gq_ÿøkÄüY-y×ôšYKjj6M½g6Ú)í3æí«²»’º¶;F«?RI~Èâ‚:Íf·e˜’l¼8¬ã‡»¡ó%sÙ~A‰Ü¦øßàeuà´Å´¸‘F+ßÝ…æÉ {‰ž;—Z®qÏrøˆ‡Ô•©”©˜FӤ˻V«Û4öGMAãÌDFf’ˆô-&A‡ÆÐÛ9®{asêZˆ3Täò„t½G%$¤ƒs’d‹àé3•TüšØêšFÓ0¹ÂPØ L#ásºHâʯ ‰ºÙê5ÌîoH¢±žFc‘Η¥Ԙ¦&
A»¹Ç¾æ[SN=(Å÷ùpê(p¹Ãüú{ˆÈ·œëhàr»~
j³oýY®1GŸÆÎndŽd0îuÑùrD
#Ÿi [äu¿Âî¸çUСb$¾ª‚´AÀ _
e('Cá ‹{¶×dE¢9
¨Â¹¡…V‘؇|KFÇv»ÛíöZ_”kœÖz”DKKtB=MÙʼn:2ÛF¯Ñx³êY`&Fؾ–ÙyŒ¢†B
If I execute my request when the sniffer is enabled all works correctly.
I can't understand why... The website is in http and not in https.
This is my code:
curl_easy_reset(curl);
/* Headers ed altri parametri validi per ogni cURL*/
struct curl_slist *chunk = NULL;
chunk = curl_slist_append(chunk, "Host: www.*****.it");
chunk = curl_slist_append(chunk, "User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:43.0) Gecko/20100101 Firefox/43.0");
chunk = curl_slist_append(chunk, "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
chunk = curl_slist_append(chunk, "Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3");
chunk = curl_slist_append(chunk, "Accept-Encoding: gzip, deflate");
chunk = curl_slist_append(chunk, "Connection: Keep-Alive");
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "cookie.txt");
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, "cookie.txt");
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 90);
curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 90);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
curl_easy_setopt(curl,CURLOPT_FOLLOWLOCATION, true);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
curl_easy_setopt(curl, CURLOPT_URL, "http://www.******.it/");
res = curl_easy_perform(curl);
writeFile("a.html", readBuffer);
What can change the sniffer when it is enabled?
Why I receive a strange response content?
With this line:
chunk = curl_slist_append(chunk, "Accept-Encoding: gzip, deflate");
You're saying that you accept gzip, deflate, so the server is sending it back compressed. Try removing that line.
Related
when posting data to zendesk, all of my umlauts are saved as "?" characters.
the same post from the postmann app works fine.
i even copied the whole postman C - libcurl code, compiled and fired it up:
same result, umlauts are shown as questionsmarks.
Whats the difference between postman posting the json and visual studio posting it?
here is the code:
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "POST");
curl_easy_setopt(curl, CURLOPT_URL, "https://xx.zendesk.com/api/v2/users/create_or_update.json");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_DEFAULT_PROTOCOL, "https");
struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "id: 1");
headers = curl_slist_append(headers, "Authorization: Basic eXaMpLe1234");
headers = curl_slist_append(headers, "Content-Type: application/json");
headers = curl_slist_append(headers, "Cookie: __cfruid=2c9eb500bc16cc34cd7390604b24ece125e7e8ad-187187187");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
const char *data = "{ \"user\": {\"external_id\": \"160102\", \"name\" : \"TestÜÄÖ\", \"phone\": \"014567-0\", \"user_fields\": { \"anrede\": \"Herr\"}, \"organization\":{\"name\":\"testorg\"}, \"email\" : \"\"} }";
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data);
res = curl_easy_perform(curl);
}
curl_easy_cleanup(curl);
is there a configuration in VS that i have to do to tell him i want utf-8 encoding when posting data?
"charset utf-8" in header yields the same results
Edit: sending a basic curl with the same data using windows cmd behaves the same and saved umlauts as questionmarks, even tho postman generated that curl statement which works when postman sends it.
for people googling:
the u8 prefix as commented by Botje worked
Just a simple file download test. Not working. Probaby because of the proxy at a Company. How can I configure the proxy?
Before adding "127.0.0.1:8080" I always got error 7: Could not connect to server. Now I always get return code 0, but the test file json.txt contains only a 404. But the url is obviously working in my browser. I also tried to get proxy settings from system via this, but I failed.
HTTP/1.1 404 Not Found
Connection: Keep-Alive
Server: Embedthis-http
ETag: "0-0-56d6b6ea"
Cache-Control: no-cache
Last-Modified: Wed, 02 Mar 2016 09:48:26 GMT
Date: Fri, 19 May 2017 14:33:57 GMT
Content-Length: 167
Keep-Alive: timeout=60, max=199
<!DOCTYPE html>
<html><head><title>Not Found</title></head>
<body>
<h2>Access Error: 404 -- Not Found</h2>
<pre>Cannot locate document: /</pre>
</body>
</html>
Here is my cURL attempt:
CURL *curl;
FILE *fp;
char url[] = "http://www.carqueryapi.com/api/0.3/?callback=?&cmd=getMakes&year=2010&sold_in_us=1";
char outfilename[FILENAME_MAX] = "C:\\temp\\json.txt";
curl = curl_easy_init();
if (curl) {
fp = fopen(outfilename,"wb");
CURLcode res;
curl_easy_setopt(curl, CURLOPT_URL, url );
curl_easy_setopt(curl, CURLOPT_NOBODY, true);
curl_easy_setopt(curl, CURLOPT_HEADER, true);
curl_easy_setopt(curl, CURLOPT_HTTPGET, true);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1 );
curl_easy_setopt(curl, CURLOPT_PROXY, "127.0.0.1:8080" );
res = curl_easy_perform(curl);
cout << "Curl response code " << res << ": " << curl_easy_strerror( res ) << endl;
curl_easy_cleanup(curl);
fclose(fp);
}
UPDATE
The internet settings are configured to use the wpad.dat file to detect proxies. If I enter one of those, I get a response from the server, but it says the access is denied because the website is blocked. But my browser can display this site.
On the contrary, if I enter netsh.exe winhttp show proxy it says that the current winhttp proxy settings are "direct access (no proxy server)".
UPDATE 2
I finally got it working with those options: Used Verbose to get more helpful output; output showed some auth method like negotiate and ntlm; the ntlm option did not work but the negotiateone together with the userpwd : which takes the credentials from the system, if libcurl has the sspi module.
curl_easy_setopt(curl, CURLOPT_URL, url );
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1 );
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1 );
curl_easy_setopt(curl, CURLOPT_HEADER, 1);
curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY );
curl_easy_setopt(curl, CURLOPT_PROXY, <"host:port"> );
curl_easy_setopt(curl, CURLOPT_PROXYAUTH, CURLAUTH_NEGOTIATE );
curl_easy_setopt(curl, CURLOPT_PROXYUSERPWD, ":" );
Try user agent and define header.
#define HEADER_ACCEPT "Accept:text/html,application/xhtml+xml,application/xml"
#define HEADER_USER_AGENT "User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.70 Safari/537.17"
struct curl_slist *list = NULL;
list = curl_slist_append(list, HEADER_USER_AGENT);
list = curl_slist_append(list, HEADER_ACCEPT);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, list);
You may need a real proxy, find a proxy sites, and retry with another ip
What i want to achieve is to be able to upload a file of any size and also send post data in the same request. The server is guaranteed to be able to handle chunked encoding and also support unlimited size of size.
To achieve that I used CURLOPT_HTTPOST and set the header Transfer-Encoding: chunked, The problem is that I'm am not sure if libcurl is actually using chunked encoding at all. In my attempts to try and figure out I set CURLOPT_VERBOSE to 1 and implemented a custom debug callback to monitor the http headers. From curl I can only see 1 HTTP Request and only 1 HTTP Response (code 200). I didn't see any http code 201 responses.
My question is how do I enable chunk encoding for file uploads in multipart POST requests? If what I'm doing is correct why does libcurl not show me all the requests, how do I make sure it works like it is required?
Bellow is a short example of what I am doing in code.
//Assume `curl` has been initialised
struct curl_httppost *formpost=NULL;
struct curl_httppost *lastptr=NULL;
struct curl_slist *headerlist=NULL;
curl_formadd(&formpost,
&lastptr,
CURLFORM_COPYNAME,
"foo",
CURLFORM_COPYCONTENTS,
"bar",
CURLFORM_END);
curl_formadd(&formpost,
&lastptr,
CURLFORM_COPYNAME,
"file",
CURLFORM_FILE,
"filepath",
CURLFORM_FILENAME,
"filename",
CURLFORM_END);
headerlist = curl_slist_append(headerlist, "Transfer-Encoding: chunked");
curl_easy_setopt(curl, CURLOPT_URL, "https://foo/uploadfile");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curl_write_to_string);
curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, curl_progress_cb);
curl_easy_setopt(curl, CURLOPT_XFERINFODATA, &progress_id);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1);
CURLcode res = curl_easy_perform(curl);
Thanks in advance
First of, I'm very new to HTTP commands and the libcurl library, so there is a good chance that I'm not understanding something fundamental. That said, I'm attempting to replicate an HTTP POST command sent to a device via an internal server on a windows based MFC app. Essentially I'm sending a small bitmap image along with a command. I captured the command using Fiddler, and it looks something like:
POST /Service/MyCommand HTTP/1.1
Authorization: MyAuth
Content-Type: image/bmp
User-Agent: Mozilla/4.0 (Windows 7 6.1) Java/1.7.0_51
Host: MyHost:MyPort
Accept: MyAccept
Connection: MyConnection
Content-Length: 15606
/* BMP Data */
I'm having two problems replicating this (using libcurl). First, my post command of 'Service/MyCommand' appears at the very end of the header, rather than after the 'POST /'. I have attempted to move it around, but it will cease to appear on my WireShark filter window. Second, when I attempt to set my content length to 15606, as with the original, the protocol on WireShark switches from "POST" to "TCP". I've attached the code below.
int CHttpPost::fnSendContent()
{
using namespace std;
int Error = 0;
CString str;
CURL* curl;
CURLcode res;
struct curl_httppost *formpost=NULL;
struct curl_httppost *lastptr=NULL;
struct curl_slist *headerlist=NULL;
static const char buf[] = "a"; // not sure what to do with this
curl_global_init(CURL_GLOBAL_ALL);
curl_formadd(&formpost, &lastptr, CURLFORM_COPYNAME, "importfile", CURLFORM_FILE, "MyImage.bmp", CURLFORM_CONTENTTYPE, "image/bmp", CURLFORM_END);
curl_formadd(&formpost, &lastptr, CURLFORM_COPYNAME, "action", CURLFORM_COPYCONTENTS, "upload", CURLFORM_END);
curl = curl_easy_init();
headerlist = curl_slist_append(headerlist, buf);
headerlist = curl_slist_append(headerlist, "Authorization: MyAuth");
headerlist = curl_slist_append(headerlist, "Content-Type: image/bmp");
headerlist = curl_slist_append(headerlist, "User-Agent: Mozilla/4.0 (Windows 7 6.1) Java/1.7.0_51");
headerlist = curl_slist_append(headerlist, "Accept: MyAccept");
headerlist = curl_slist_append(headerlist, "Connection: MyConnection");
headerlist = curl_slist_append(headerlist, "Content-Length: 15606");
//Set URL to recevie POST
curl_easy_setopt(curl, CURLOPT_VERBOSE, true);
curl_easy_setopt(curl,CURLOPT_POST, true);
curl_easy_setopt(curl, CURLOPT_HEADER, true);
curl_easy_setopt(curl, CURLOPT_URL, "MyHost:MyPort");
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "Service/MyCommand");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
curl_global_cleanup();
return Error;
}
Any other suggestions or corrections you have are much appreciated as well.
EDIT: I was being an idiot by placing the "Service/MyCommand" in the postfield rather than the URL. I apparently misunderstood one of the tutorials. My content-length question still stands however.
You can do this much simpler. You, just use CURLOPT_POSTFIELDS to point to the BMP data, and make CURLOPT_POSTFIELDSIZE_LARGE be the full size of that data (15606 I would guess in this particular case).
You do not want curl_formadd() and the formpost stuff you use. That's for the CURLOPT_HTTPPOST option, which is for multipart formposts but you don't want that.
char *bmp_array = [your data];
long bmp_size = 15606;
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, bmp_array);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, bmp_size);
I am currently using pre-compiled minGW32 libraries for libcurl-7.21.6, C++, QT libraries, minGW32 windows compiler (in QT creator) and QT creator as my IDE.
I am trying to post some http information but keep running into a problem. I need to remove Expect: 100... from my header. From everything i have seen online,
headerlist = curl_slist_append(headerlist, "Expect:");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
should do it, but it seems to get nullified by:
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
If i call the HTTPPOST before the HTTPHEADER, it seems like the HTTPHEADER nullifies everything from the HTTPPOST, and vice versa.
Am I doing something wrong? Should the Expect: be somehow included in the HTTPPOST instead of separately in HTTPHEADER?
I am trying to get rid of the Expect header because the server keeps responding to my request with Expectation Failed.
Here is the curl command used with curl.exe to do the samething i'm trying to do with libcurl:
system("curl --referer http://192.168.16.23/upthefile.html -F uploadfile=#instructions.xml -F config=on http://192.168.16.23/cgi-bin/upload.cgi -H \"Expect:\">nul");
Any help will be appreciated.
Using cURL 7.25.0 and this snippet:
curl_global_init(CURL_GLOBAL_ALL);
struct curl_httppost *formpost = NULL;
struct curl_httppost *lastptr = NULL;
struct curl_slist *headerlist = curl_slist_append(NULL, "Expect:");
curl_formadd(&formpost, &lastptr, CURLFORM_COPYNAME, "field", CURLFORM_COPYCONTENTS, "value", CURLFORM_END);
CURL *curl = curl_easy_init();
if (curl)
{
curl_easy_setopt(curl, CURLOPT_URL, "http://www.example.com/");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
CURLcode res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
curl_formfree(formpost);
curl_slist_free_all(headerlist);
}
I obtain the following request (Wireshark'd):
POST / HTTP/1.1
Host: www.example.com
Accept: */*
Content-Length: 145
Content-Type: multipart/form-data; boundary=----------------------------523d686b5061
------------------------------523d686b5061
Content-Disposition: form-data; name="field"
value
------------------------------523d686b5061--
While omitting the line curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist); the header contains also:
HTTP/1.1 100 Continue