libcurl, enabling chunk encoding with multipart form data - c++

What i want to achieve is to be able to upload a file of any size and also send post data in the same request. The server is guaranteed to be able to handle chunked encoding and also support unlimited size of size.
To achieve that I used CURLOPT_HTTPOST and set the header Transfer-Encoding: chunked, The problem is that I'm am not sure if libcurl is actually using chunked encoding at all. In my attempts to try and figure out I set CURLOPT_VERBOSE to 1 and implemented a custom debug callback to monitor the http headers. From curl I can only see 1 HTTP Request and only 1 HTTP Response (code 200). I didn't see any http code 201 responses.
My question is how do I enable chunk encoding for file uploads in multipart POST requests? If what I'm doing is correct why does libcurl not show me all the requests, how do I make sure it works like it is required?
Bellow is a short example of what I am doing in code.
//Assume `curl` has been initialised
struct curl_httppost *formpost=NULL;
struct curl_httppost *lastptr=NULL;
struct curl_slist *headerlist=NULL;
curl_formadd(&formpost,
&lastptr,
CURLFORM_COPYNAME,
"foo",
CURLFORM_COPYCONTENTS,
"bar",
CURLFORM_END);
curl_formadd(&formpost,
&lastptr,
CURLFORM_COPYNAME,
"file",
CURLFORM_FILE,
"filepath",
CURLFORM_FILENAME,
"filename",
CURLFORM_END);
headerlist = curl_slist_append(headerlist, "Transfer-Encoding: chunked");
curl_easy_setopt(curl, CURLOPT_URL, "https://foo/uploadfile");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curl_write_to_string);
curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, curl_progress_cb);
curl_easy_setopt(curl, CURLOPT_XFERINFODATA, &progress_id);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1);
CURLcode res = curl_easy_perform(curl);
Thanks in advance

Related

Ignore curl response body in c++

I am doing a post request using CURL in c++ and I wanted to know if theres a way to ignore the response body that gets printed to the output. This is what my code looks like currently
if (curl) {
// Set url
curl_easy_setopt(curl, CURLOPT_URL, str_url);
// Set header
list = curl_slist_append(list, header);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, list);
// Set request type and data
curl_easy_setopt(curl, CURLOPT_POST, 1);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, body);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, (long)strlen(body));
res = curl_easy_perform(curl);
everything works but I keep getting the response body posted to the output and I just want to ignore it since all I need is the response code. With the command line I can make a curl request with the following curl --silent --output to ignore it but is there a way to do this in c++?

Umlauts problem when sending Json Post with VS2017 C++ using libcurl

when posting data to zendesk, all of my umlauts are saved as "?" characters.
the same post from the postmann app works fine.
i even copied the whole postman C - libcurl code, compiled and fired it up:
same result, umlauts are shown as questionsmarks.
Whats the difference between postman posting the json and visual studio posting it?
here is the code:
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "POST");
curl_easy_setopt(curl, CURLOPT_URL, "https://xx.zendesk.com/api/v2/users/create_or_update.json");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_DEFAULT_PROTOCOL, "https");
struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "id: 1");
headers = curl_slist_append(headers, "Authorization: Basic eXaMpLe1234");
headers = curl_slist_append(headers, "Content-Type: application/json");
headers = curl_slist_append(headers, "Cookie: __cfruid=2c9eb500bc16cc34cd7390604b24ece125e7e8ad-187187187");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
const char *data = "{ \"user\": {\"external_id\": \"160102\", \"name\" : \"TestÜÄÖ\", \"phone\": \"014567-0\", \"user_fields\": { \"anrede\": \"Herr\"}, \"organization\":{\"name\":\"testorg\"}, \"email\" : \"\"} }";
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data);
res = curl_easy_perform(curl);
}
curl_easy_cleanup(curl);
is there a configuration in VS that i have to do to tell him i want utf-8 encoding when posting data?
"charset utf-8" in header yields the same results
Edit: sending a basic curl with the same data using windows cmd behaves the same and saved umlauts as questionmarks, even tho postman generated that curl statement which works when postman sends it.
for people googling:
the u8 prefix as commented by Botje worked

Curl works only if the sniffer is enabled - encrypted response content

I'm trying to download an internet page using curl c++.
Using http analyzer I can see the header and use it in cURL.
When I execute my GET request by curl I have a crypted response content, something like this:
‹í=ksÛ8’Ÿ“ªùíÞÊ©1%‘Ô3‰=å8NÆ»ñccMfo§æX IH(BÇmewÿÏÝ߸ûc×
ð%‰’(Kžä®&›ÐÝèn¼^~ûúê´ÿï×gdNÜã§/ñq©7:ªðP;ïWÈýÄõ‚£Ê8§Ïëõ»»»ÚYþ¨®÷z½ú=f¨`FFã§O_NXH‰=¦~À£Jµ.|†02öÙp÷†"}êÐë
R¢ðè„UFÌc>
…_!¶ðBæN ÷Ê$ÍZ‡œ^Ü`Ñ9x€åÞ(¬¸‹`ì–P?:Yu‡8t /]î}">s*A8sY0f,¬p6…ì!»ëvTâú#ª0C6™Öm1™ú,„_Ÿ0ÄÍh;–ÑèÒv‡1Çh9tØëtMÖ`FË6›5Àó½Þlµ›ÝN§Õ¨ s8=ªPW½
CîB'L¸WGZ\²?ק>÷BUŽÙ6;³aši9òã>JúÔ‡Ô³gq_ÿøkÄüY-y×ôšYKjj6M½g6Ú)í3æí«²»’º¶;F«­?RI~Èâ‚:Íf·e˜­’l¼8¬ã‡»¡ó%sÙ~A‰Ü¦øßàeuà´Å´¸‘F+ßÝ…æÉ {‰ž;—Z®qÏrøˆ‡Ô•©”©˜FӤ˻­V«Û4öGMAãÌDFf’ˆô-&A‡ÆÐÛ9®{asêZˆ3Täò„t½G%$¤ƒs’d‹àé3•T­üšØêšFÓ0¹­ÂPØ L#ásºHâʯ ‰ºÙê5ÌîoH¢±žFc‘Η¥Ԙ¦&
A»¹Ç¾æ[SN=(Å÷ùpê(p¹Ãüú{ˆÈ·œëhàr»~
j³oýY®1GŸÆÎndŽd0îuÑùrD
#Ÿi [äu¿Âî¸çUСb$¾ª‚´AÀ _
e('Cá ‹{¶×dE¢9
¨Â¹¡…V‘؇|KFÇv»ÛíöZ_”kœÖz”DKKtB=MÙʼn:2ÛF¯Ñx³êY`&Fؾ­–ÙyŒ¢†B
If I execute my request when the sniffer is enabled all works correctly.
I can't understand why... The website is in http and not in https.
This is my code:
curl_easy_reset(curl);
/* Headers ed altri parametri validi per ogni cURL*/
struct curl_slist *chunk = NULL;
chunk = curl_slist_append(chunk, "Host: www.*****.it");
chunk = curl_slist_append(chunk, "User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:43.0) Gecko/20100101 Firefox/43.0");
chunk = curl_slist_append(chunk, "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
chunk = curl_slist_append(chunk, "Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3");
chunk = curl_slist_append(chunk, "Accept-Encoding: gzip, deflate");
chunk = curl_slist_append(chunk, "Connection: Keep-Alive");
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "cookie.txt");
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, "cookie.txt");
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 90);
curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 90);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
curl_easy_setopt(curl,CURLOPT_FOLLOWLOCATION, true);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
curl_easy_setopt(curl, CURLOPT_URL, "http://www.******.it/");
res = curl_easy_perform(curl);
writeFile("a.html", readBuffer);
What can change the sniffer when it is enabled?
Why I receive a strange response content?
With this line:
chunk = curl_slist_append(chunk, "Accept-Encoding: gzip, deflate");
You're saying that you accept gzip, deflate, so the server is sending it back compressed. Try removing that line.

HTTP Post Header Fields with libcurl

First of, I'm very new to HTTP commands and the libcurl library, so there is a good chance that I'm not understanding something fundamental. That said, I'm attempting to replicate an HTTP POST command sent to a device via an internal server on a windows based MFC app. Essentially I'm sending a small bitmap image along with a command. I captured the command using Fiddler, and it looks something like:
POST /Service/MyCommand HTTP/1.1
Authorization: MyAuth
Content-Type: image/bmp
User-Agent: Mozilla/4.0 (Windows 7 6.1) Java/1.7.0_51
Host: MyHost:MyPort
Accept: MyAccept
Connection: MyConnection
Content-Length: 15606
/* BMP Data */
I'm having two problems replicating this (using libcurl). First, my post command of 'Service/MyCommand' appears at the very end of the header, rather than after the 'POST /'. I have attempted to move it around, but it will cease to appear on my WireShark filter window. Second, when I attempt to set my content length to 15606, as with the original, the protocol on WireShark switches from "POST" to "TCP". I've attached the code below.
int CHttpPost::fnSendContent()
{
using namespace std;
int Error = 0;
CString str;
CURL* curl;
CURLcode res;
struct curl_httppost *formpost=NULL;
struct curl_httppost *lastptr=NULL;
struct curl_slist *headerlist=NULL;
static const char buf[] = "a"; // not sure what to do with this
curl_global_init(CURL_GLOBAL_ALL);
curl_formadd(&formpost, &lastptr, CURLFORM_COPYNAME, "importfile", CURLFORM_FILE, "MyImage.bmp", CURLFORM_CONTENTTYPE, "image/bmp", CURLFORM_END);
curl_formadd(&formpost, &lastptr, CURLFORM_COPYNAME, "action", CURLFORM_COPYCONTENTS, "upload", CURLFORM_END);
curl = curl_easy_init();
headerlist = curl_slist_append(headerlist, buf);
headerlist = curl_slist_append(headerlist, "Authorization: MyAuth");
headerlist = curl_slist_append(headerlist, "Content-Type: image/bmp");
headerlist = curl_slist_append(headerlist, "User-Agent: Mozilla/4.0 (Windows 7 6.1) Java/1.7.0_51");
headerlist = curl_slist_append(headerlist, "Accept: MyAccept");
headerlist = curl_slist_append(headerlist, "Connection: MyConnection");
headerlist = curl_slist_append(headerlist, "Content-Length: 15606");
//Set URL to recevie POST
curl_easy_setopt(curl, CURLOPT_VERBOSE, true);
curl_easy_setopt(curl,CURLOPT_POST, true);
curl_easy_setopt(curl, CURLOPT_HEADER, true);
curl_easy_setopt(curl, CURLOPT_URL, "MyHost:MyPort");
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "Service/MyCommand");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
curl_global_cleanup();
return Error;
}
Any other suggestions or corrections you have are much appreciated as well.
EDIT: I was being an idiot by placing the "Service/MyCommand" in the postfield rather than the URL. I apparently misunderstood one of the tutorials. My content-length question still stands however.
You can do this much simpler. You, just use CURLOPT_POSTFIELDS to point to the BMP data, and make CURLOPT_POSTFIELDSIZE_LARGE be the full size of that data (15606 I would guess in this particular case).
You do not want curl_formadd() and the formpost stuff you use. That's for the CURLOPT_HTTPPOST option, which is for multipart formposts but you don't want that.
char *bmp_array = [your data];
long bmp_size = 15606;
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, bmp_array);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, bmp_size);

libcurl specifying the header

I am currently using pre-compiled minGW32 libraries for libcurl-7.21.6, C++, QT libraries, minGW32 windows compiler (in QT creator) and QT creator as my IDE.
I am trying to post some http information but keep running into a problem. I need to remove Expect: 100... from my header. From everything i have seen online,
headerlist = curl_slist_append(headerlist, "Expect:");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
should do it, but it seems to get nullified by:
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
If i call the HTTPPOST before the HTTPHEADER, it seems like the HTTPHEADER nullifies everything from the HTTPPOST, and vice versa.
Am I doing something wrong? Should the Expect: be somehow included in the HTTPPOST instead of separately in HTTPHEADER?
I am trying to get rid of the Expect header because the server keeps responding to my request with Expectation Failed.
Here is the curl command used with curl.exe to do the samething i'm trying to do with libcurl:
system("curl --referer http://192.168.16.23/upthefile.html -F uploadfile=#instructions.xml -F config=on http://192.168.16.23/cgi-bin/upload.cgi -H \"Expect:\">nul");
Any help will be appreciated.
Using cURL 7.25.0 and this snippet:
curl_global_init(CURL_GLOBAL_ALL);
struct curl_httppost *formpost = NULL;
struct curl_httppost *lastptr = NULL;
struct curl_slist *headerlist = curl_slist_append(NULL, "Expect:");
curl_formadd(&formpost, &lastptr, CURLFORM_COPYNAME, "field", CURLFORM_COPYCONTENTS, "value", CURLFORM_END);
CURL *curl = curl_easy_init();
if (curl)
{
curl_easy_setopt(curl, CURLOPT_URL, "http://www.example.com/");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
CURLcode res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
curl_formfree(formpost);
curl_slist_free_all(headerlist);
}
I obtain the following request (Wireshark'd):
POST / HTTP/1.1
Host: www.example.com
Accept: */*
Content-Length: 145
Content-Type: multipart/form-data; boundary=----------------------------523d686b5061
------------------------------523d686b5061
Content-Disposition: form-data; name="field"
value
------------------------------523d686b5061--
While omitting the line curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist); the header contains also:
HTTP/1.1 100 Continue