Curl vs CPPREST - c++

I am trying to access an URL, using CPPREST http_client :
http://www.20min.ch/rss/rss.tmpl?type=channel&get=68
I am receiving response code 302 for URL- redirection.
But when i try to access the same URL using CURL, I am receiving CURLE_OK.
Below are the 2 piece of code :
using CURL :
CURL *curl;
CURLcode res;
curl_global_init(CURL_GLOBAL_ALL);
curl = curl_easy_init();
if(curl){
curl_easy_setopt(curl, CURLOPT_URL, "http://www.20min.ch/rss/rss.tmpl?type=channel&get=68");
res = curl_easy_perform(curl);
if(res != CURLE_OK) {
cout<<"failed";
}
else {
cout<<"success";
}
curl_easy_cleanup(curl);
}
curl_global_cleanup();
The output is : success
using CPPREST :
std::string url_= "http://www.20min.ch/rss/rss.tmpl?type=channel&get=68";
try
{
http_client client1(U(url_));
uri_builder builder1(U(""));
client1.request(methods::GET, builder1.to_string()).then([=](http_response response)
{
cout<<"Response code is : "<<response.status_code();
});
}
catch(std::exception& e)
{
cout<<"response :"<<e.what();
}
The output is :: Response code is : 302
I do not understand why the two libs are behaving differently for same URL??
UPDATE :
I have also tried with :
http_client client1(utility::conversions::to_string_t(url_));
and
http_client client1(U("http://www.20min.ch/rss/rss.tmpl?type=channel&get=68"));
and
http_client client1(U("http://www.20min.ch/"));
but the response is same 302 with cpp rest. [ for cross checking bing example
is working fine]
UPDATE 2:
The method as explained by #Matt Weber seems very helpful and legit but i am continuously getting error code : 400 for that, So I tried the below things:
I tried to set the host and port for the URL in uri_builder.
http_client client(U("http://www.20min.ch/rss/"));
uri_builder builder(U("/rss.tmpl"));
builder.append_query(U("type"), U("channel"));
builder.append_query(U("get"), U("68"));
builder.set_host(U("www.20min.ch"));
builder.set_port(U("80"));
client.request(methods::GET, builder.to_string()).then([=](http_response response)
{
cout<<"Received response status code: "<<response.status_code();
});
but still same 302.

The problem with the Rest SDK code is the http_client initialization:
http_client client1(U(url_));
The U macro is for use with string literals to produce something from which a uri can be constructed. If you're on Windows, this shouldn't compile, because the macro expansion results in Lurl_. Apparently whatever this results in on your system leads to a request for something that responds with a 302.
There are a couple of options. One would be to simply use the literal directly:
http_client client1(U("http://www.20min.ch/rss/rss.tmpl?type=channel&get=68"));
If you want to keep the std::string and initialize the client from that, you can convert to a utility::string_t from which the uri can be constructed.
std::string url_= "http://www.20min.ch/rss/rss.tmpl?type=channel&get=68";
http_client client1(utility::conversions::to_string_t(url_));
Once that is done, you'll likely find that you need to call the wait function on the continuation from request in order to actually see the expected output:
client1.request(methods::GET, builder1.to_string()).then([](http_response response)
{
cout<<"Response code is : "<<response.status_code();
}).wait(); // ensure that the response gets processed
EDIT:
The above is relevant for building on Windows, but has nothing to do with the 302 response.
On Linux, the request results in a 302 consistently. Looking at the request and response on the wire, a request from a Windows host gets a 200 and a request from a Linux host gets a 302. The reason is that in the Linux version, the host header includes a port number, which is what triggers the server to respond with a 302.
Windows request:
GET /rss/rss.tmpl?type=channel&get=68 HTTP/1.1\r\n
Connection: Keep-Alive\r\n
User-Agent: cpprestsdk/2.8.0\r\n
Host: www.20min.ch\r\n
\r\n
Linux request:
GET /rss/rss.tmpl?type=channel&get=68 HTTP/1.1\r\n
Host: www.20min.ch:80\r\n
User-Agent:cpprestsdk/2.8.0\r\n
Connection: Keep-Alive\r\n
\r\n
You can verify that this is the cause with wget:
$ wget --header="Host: www.20min.ch" -S "http://www.20min.ch/rss/rss.tmpl?type=channel&get=68"
HTTP/1.1 200 OK
$ wget --header="Host: www.20min.ch:80" -S "http://www.20min.ch/rss/rss.tmpl?type=channel&get=68" --max-redirect 0
HTTP/1.1 302 Found
The difference in the header is due to the different implementations. The WinHTTP client implementation does not add the Host header explicitly, presumably because it relies on WinHTTP to do that internally. The asio client implementation does add it, though.
// Add the Host header if user has not specified it explicitly
if (!ctx->m_request.headers().has(header_names::host))
{
request_stream << "Host: " << host << ":" << port << CRLF;
}
So to get the expected behavior, the header can be set explicitly to avoid adding the port information:
std::string url_= "http://www.20min.ch/rss/rss.tmpl?type=channel&get=68";
http_client client1(utility::conversions::to_string_t(url_));
http_request request;
request.set_method(methods::GET);
request.headers().add(U("Host"), U("www.20min.ch"));
client1.request(request).then([](http_response response)
{
std::cout<<"Response code is : "<<response.status_code();
}).wait();
With this change, I get a 200 OK on both Windows and Linux.

Related

To change the status code from 200 to 404 when stepfunction not found

I'm trying to Execute the AWS step function from API Gateway, It's working as expected.
Whenever I'm passing the input, statemachinearn(stepfunction name to execute) It's triggering the step function.
But It's still returning the status code 200, whenever it's not able to find the stepfunction, I want to return the status code 404 if the apigateway not found that stepfunction.
Could you please help me on that
Response:
Status: 200ok
Expected:
Status: 404
Thanks,
Harika.
As per the documentation StartExecution API call do return 400 Bad Request for non existent statemachine which is correct as RESTful API standard.
StateMachineDoesNotExist
The specified state machine does not exist.
HTTP Status Code: 400
From the RESTful API point of view, endpoint /execution/(which I created in API Gateway for the integration setup) is a resource, no matter it accepts GET or POST or something else. 404 is only appropriate when the resource /execution/ itself does not exist. If /execution/ endpoint exists, but its invocation failed (no matter what the reasons), the response status code must be something other than 404.
So in the case of the returned response(200) for POST call with non-existent statemachine it is correct. But when API Gateway tried to make the call to non-existent statemachine it got 404 from StartExecution api call which it eventually wrapped into a proper message instead of returning 404 http response.
curl -s -X POST -d '{"input": "{}","name": "MyExecution17","stateMachineArn": "arn:aws:states:eu-central-1:1234567890:stateMachine:mystatemachine"}' https://123456asdasdas.execute-api.eu-central-1.amazonaws.com/v1/execution|jq .
{
"__type": "com.amazonaws.swf.service.v2.model#StateMachineDoesNotExist",
"message": "State Machine Does Not Exist: 'arn:aws:states:eu-central-1:1234567890:stateMachine:mystatemachine1'"
}
Let's say you create another MethodResponse where you can provide an exact HTTP Status Code in your case 404 which you want to return and you do an Integration Response where you have to choose the Method Response by providing either Exact HTTP Responce Code(400 -> Upstream response from the **StartExecution** API Call) OR a Regex -> (4\{d}2) matching all the 4xx errors.
In that case you will be giving 404 for all the responses where the upstream error 4xx StartExecution Errors
ExecutionAlreadyExists -> 400
ExecutionLimitExceeded -> 400
InvalidArn -> 400
InvalidExecutionInput -> 400
InvalidName -> 400
StateMachineDeleting -> 400
StateMachineDoesNotExist -> 400
Non Existent State Machine:
curl -s -X POST -d '{"input": "{}","name": "MyExecution17","stateMachineArn": "arn:aws:states:eu-central-1:1234567890:stateMachine:mystatemachine1"}' https://123456asdasdas.execute-api.eu-central-1.amazonaws.com/v1/execution|jq .
< HTTP/2 404
< date: Sat, 30 Jan 2021 14:12:16 GMT
< content-type: application/json
...
{
"__type": "com.amazonaws.swf.service.v2.model#StateMachineDoesNotExist",
"message": "State Machine Does Not Exist: 'arn:aws:states:eu-central-1:1234567890:stateMachine:mystatemachine1'"
}
Execution Already Exists
curl -s -X POST -d '{"input": "{}","name": "MyExecution17","stateMachineArn": "arn:aws:states:eu-central-1:1234567890:stateMachine:mystatemachine"}' https://123456asdasdas.execute-api.eu-central-1.amazonaws.com/v1/execution|jq .
* We are completely uploaded and fine
< HTTP/2 404
< date: Sat, 30 Jan 2021 14:28:27 GMT
< content-type: application/json
{
"__type": "com.amazonaws.swf.service.v2.model#ExecutionAlreadyExists",
"message": "Execution Already Exists: 'arn:aws:states:eu-central-1:1234567890:execution:mystatemachine:MyExecution17'"
}
Which I think will be misleading.

libcurl HTTP request set content-disposition and content-type in MIME data

I'm trying to upload an image on twitter using libcurl, I used the twurl command line tool to generate an HTTP request and see how it should look like, what I get is this:
POST /1.1/media/upload.json HTTP/1.1
Accept: */
Content-Type: multipart/form-data, boundary="00Twurl342528555775455418lruwT99"
Authorization: OAuth oauth_body_hash="XXX", oauth_consumer_key="XXX", oauth_nonce="XXX", oauth_signature="XXX", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1603308767", oauth_token="XXX", oauth_version="1.0"
Connection: close
Host: upload.twitter.com
Content-Length: 612739
--00Twurl342528555775455418lruwT99
Content-Disposition: form-data; name="media"; filename="image.png"
Content-Type: application/octet-stream
binary data of image.png
--00Twurl342528555775455418lruwT99--
The request that I can generate via libcurl (got it using curl verbose) for the moment is this one:
POST /1.1/media/upload.json HTTP/2
Host: upload.twitter.com
accept: */*
authorization: OAuth oauth_consumer_key="XXX",oauth_nonce="XXX",oauth_signature="XXX",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1603372043",oauth_token="XXX",oauth_version="1.0"
content-length: 268
content-type: multipart/form-data; boundary=------------------------d1b0fc28e693c24a
Using the following code:
curl_mime *mime = nullptr;
curl_mimepart *part = nullptr;
mime = curl_mime_init(request_handle);
part = curl_mime_addpart(mime);
curl_mime_name(part, "media");
curl_mime_filename(part, "image.png");
curl_easy_setopt(request_handle, CURLOPT_MIMEPOST, mime);
The problem is that I don't know how to make my request similar to the first one with libcurl, how do I specify Content-Type and Content-Disposition ?
Edit: solution
Full code
curl_mime* mime = nullptr;
curl_mimepart* part = nullptr;
/* initialize mime part */
mime = curl_mime_init(request_handle);
part = curl_mime_addpart(mime);
/* content-disposition: form-data; name="media"; filename="image.png" */
curl_mime_name(part, "media");
curl_mime_filename(part, "image.png");
/* add file content */
curl_mime_filedata(part, "image.png");
/* content-type: application/octet-stream */
curl_mime_type(part, "application/octet-stream");
/* link the MIME data to your curl handle */
curl_easy_setopt(request_handle, CURLOPT_MIMEPOST, mime);
I didn't do it to highlight the functions to use, but check function return.
how do I specify Content-Type and Content-Disposition ?
Just read the fine manual (which you can navigate to from the fine example postit2.c)
CURLcode curl_mime_type(curl_mimepart * part, const char * mimetype);
curl_mime_type sets a mime part's content type.
CURLcode curl_mime_filename(curl_mimepart * part, const char * filename);
curl_mime_filename sets a mime part's remote file name. When remote file name is set, content data is processed as a file, whatever is the part's content source. A part's remote file name is transmitted to the server in the associated Content-Disposition generated header.
The official libcurl tutorial is also a nice read.

How do I correctly setup the multipart formdata with libcurl (in C++) to upload a binary file

To provide a little background. I am not very experienced with Ethernet communications so I apologize in advance for that. I'm working on a project where I need to figure out how to upload a binary file.
I'm trying to upload a large binary file (~34MB) to an embedded device. I have a python code snippet that works but I'm trying to implement the same capability in a different application using C++. Using WireShark, I've captured the header from the python program that works as well as the header that I end up with in my C++ code which doesn't work as needed.
Success Uploading with Python
Here is the python code that works:
response = session.post('http://10.42.42.1:81/__FileUpload',
files={"upfile": open(filename, 'rb')},
stream=False)
Here is the header information extracted from the message which initiates the successful file upload:
POST /__FileUpload HTTP/1.1
Host: 10.42.42.1:81
Content-Length: 34112690
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.7.0 CPython/2.7.13 Windows/10
Connection: keep-alive
Content-Type: multipart/form-data; boundary=6a659e345a35419e99b66546c1bd9d4e
--6a659e345a35419e99b66546c1bd9d4e
Content-Disposition: form-data; name="upfile"; filename="TestFile.bin"
No Success with C++ Code
Here is the essence of the code that I'm using to upload the file in C++:
curl_mime *multipart;
curl_mimepart *part;
// Specify the target URL
std::string str(comms.BaseURL() + kFileUploadEndpoint);
curl_easy_setopt(pCurl, CURLOPT_URL, str.c_str());
multipart = curl_mime_init(pCurl);
part = curl_mime_addpart(multipart);
curl_mime_name(part, "upfile");
curl_mime_data(part, ("filename=\"" + FileName + "\"").c_str(), CURL_ZERO_TERMINATED);
part = curl_mime_addpart(multipart);
curl_mime_data_cb(part, fileSize, ReadCallback, SeekCallback, NULL, pFile);
curl_easy_setopt(pCurl, CURLOPT_MIMEPOST, multipart);
curl_easy_setopt(pCurl, CURLOPT_TIMEOUT, 90L);
res = curl_easy_perform(pCurl);
...
Here is the header information from running the C++ code:
--------------------------4977715f070a13da
Content-Disposition: form-data; name="upfile"
filename="TestFile.bin"
--------------------------4977715f070a13da
Content-Disposition: form-data
I realized that the header above does not contain the URL endpoint and such and noticed that apparently the message was split into two pieces. Here is the header content from the message sent before the message above.
POST /__FileUpload HTTP/1.1
Host: 10.42.42.1:81
Accept: */*
Content-Length: 31546130
Content-Type: multipart/form-data; boundary=------------------------4977715f070a13da
Expect: 100-continue
I can check the status from the embedded device during the upload and the one thing that I notice in particular is that when the upload is successful with python then the embedded device reports the filename being uploaded in the status content reply. However, when I run the C++ code the filename is blank when I check the status. Therefore the embedded device is obviously not able to extract the filename from the C++ message.
What the Embedded Device is Looking For
While I don't have access to the source code of the device I did get the following information from someone who does. He indicated that this is what the embedded device is looking for. It didn't help me to figure out how to get things working but it might help someone else more knowledgeable in this area.
<FORM METHOD=POST name="install" enctype="multipart/form-data" target="HiddenFrame" action="/__FileUpload" onsubmit="InstallAction(); return true;">
File to upload: <INPUT TYPE=FILE NAME="upfile" size=50><p>
<INPUT TYPE=SUBMIT VALUE="Submit" >
</FORM>
I would prefer using the libcurl 'curl_mime_...' methods to setup the file upload only because that approach is recommended over using the older HTTP post methods. However, I'm perfectly okay with using the older HTTP post methods if that is easier to do. I just want to get it working.
Thanks in advance for your time.
I ended up finding the solution.
It is posted here for anyone who could use it:
curl_mime *multipart;
curl_mimepart *part;
multipart = curl_mime_init(pCurl);
part = curl_mime_addpart(multipart);
curl_mime_name(part, "upfile");
curl_mime_filename(part, FileName.c_str());
curl_mime_data_cb(part, lSize, ReadCallback, SeekCallback, NULL, pFile);
part = curl_mime_addpart(multipart);
curl_easy_setopt(pCurl, CURLOPT_MIMEPOST, multipart);
// Now send the message
res = curl_easy_perform(pCurl);
// Free the post data
curl_mime_free(multipart);
...
curl_mime_data(part, ("filename=\"" + FileName + "\"").c_str(), CURL_ZERO_TERMINATED);
I presume this wants to set the mime part's file name, and then you should rather use curl_mime_filename, because the file name is not data. Each part has a name and data, but also meta-data such as file name. Setting the file name only of course then requires that you set the data separately.
If you rather want to set the data as well as the file name, then instead do it with curl_mime_filedata.
Also, take a look at the official libcurl example postit2.c.

Send POST multipart/form-data request using restbed C++

I am working on a C++ rest client using restbed lib that will send a base64 encoded image using a POST request.
The code I wrote so far is :
auto request = make_shared< Request >(Uri("http://127.0.0.1:8080/ProcessImage"));
request->set_header("Accept", "*/*");
request->set_header("Content-Type", "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW");
request->set_header("Cache-Control", "no-cache");
request->set_method("POST");
string test = "------WebKitFormBoundary7MA4YWxkTrZu0gW"
"Content-Disposition:form-data;name=\"image\""
""
"testMessage"
"------WebKitFormBoundary7MA4YWxkTrZu0gW--";
request->set_body(imgContent);
auto response = Http::sync(request)
I am not sure how I should set the request body. I tried with simple image="blabla" and also with this long version message I took from postman.
But in every case I received a "error 400 Bad request" answer.
Update:
Tested also with this version of code but with no success:
auto request = make_shared< Request >(Uri("http://127.0.0.1:8080/ProcessImage"));
request->set_header("Accept", "*/*");
request->set_header("Host","127.0.0.1:8080");
request->set_method("POST");
request->set_header("Content-Type", "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW");
request->set_header("Cache-Control", "no-cache");
string imgContent = "------WebKitFormBoundary7MA4YWxkTrZu0gW\r\n"
"Content-Disposition: form-data; name=\"image\"\r\n"
"\r\n"
"test\r\n"
"------WebKitFormBoundary7MA4YWxkTrZu0gW--\r\n";
request->set_body(imgContent
auto response = Http::sync(request);
The response I get from the server:
*** Response ***
Status Code: 400
Status Message: BAD REQUEST
HTTP Version: 1.0
HTTP Protocol: HTTP
Header 'Content-Length' > '192'
Header 'Content-Type' > 'text/html'
Header 'Date' > 'Sun, 04 Feb 2018 21:09:45 GMT'
Header 'Server' > 'Werkzeug/0.14.1 Python/3.5.4'
Body:<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
²²²²∩x...
Also on the server side (which is using python flask) I added:
encoded_img = request.form.get('image') and printed the string. The print result was: "None"
Your body content is missing explicit line break characters at the end of each line. C++ does not insert them automatically for you.
Also, if you are going to send base64 data, you should include a Content-Transfer-Encoding header, too.
Try this:
string imgContent = "------WebKitFormBoundary7MA4YWxkTrZu0gW\r\n"
"Content-Disposition: form-data; name=\"image\"\r\n"
"Content-Transfer-Encoding: base64\r\n"
"\r\n"
"<base64 image data here>\r\n"
"------WebKitFormBoundary7MA4YWxkTrZu0gW--\r\n";
request->set_body(imgContent);

curl request through c++ for indexing solr document

I have used following code for indexing document in solr
CURL *curl = curl_easy_init();
CURLcode res;
if(curl) {
/* First set the URL that is about to receive our POST. This URL can
just as well be a https:// URL if that is what should receive the
data. */
curl_easy_setopt(curl, CURLOPT_URL, "http://192.168.0.164:8983/solr/collection1/update?replacefields=false -H 'Content-type: application/json' -d '[{\"id\":\"4000\", \"to\":\"Life is to.\", \"cc\":\"unknown \", \"subject\":\"Life\"}]'");
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
if(CURLE_OK == res){
logger.LogError("res value CURLE_OK");
}
/* always cleanup */
curl_easy_cleanup(curl);
}
and the return value of curl_easy_perform(curl) i.e. res is CURLE_OK but the record is not indexing in collection1 of solr and while posting following command from terminal record is getting indexed
curl http://192.168.0.164:8983/solr/collection1/update?replacefields=false -H 'Content-type: application/json' -d '[{"id":"4000", "to":"Life is to.", "cc":"unknown ", "subject":"Life"}]'
You're not supplying cURL with your proper URL. Everything after the actual URL, -H 'Content-type: application/json' -d '[{\"id\":\"4000\", \"to\":\"Life is to.\", \"cc\":\"unknown \", \"subject\":\"Life\"}]', is just arguments to the cURL command line tool, and is not part of the URL.
The URL option should be only http://192.168.0.164:8983/solr/collection1/update?replacefields=false. The rest of the parameters has to be set as their own curl_easy_setopt calls.
To set the data to POST, use CURLOPT_POSTFIELDS.
Pass a char * as parameter, pointing to the full data to send in a HTTP POST operation. You must make sure that the data is formatted the way you want the server to receive it.
To set the proper request content type, use CURLOPT_HTTPHEADER.
Pass a pointer to a linked list of HTTP headers to pass to the server and/or proxy in your HTTP request. The same list can be used for both host and proxy requests!
After setting all the options call curl_easy_perform. You may also want to watch the log on the Solr server to see if Solr generates an exception. You can also set CURLOPT_ERRORBUFFER and CURLOPT_VERBOSE to get more information about any failure internally in cURL (CURLE_OK will be returned as long as cURL is able to make the request - but does not change if the server returns 400 or 404 or any actual error code on the server side (except if CURLOPT_FAILONERROR is set).