cpprestsdk: handle chunked response - c++

How should I handle chunked response using cpprestsdk? How to request the next chunk? Is there required functionality at all there?
Here is how we are performing http requests:
web::http::http_request request(web::http::methods::GET);
request.headers().add(LR"(User-Agent)", LR"(ExchangeServicesClient/15.00.0847.030)");
request.headers().add(LR"(Accept)", LR"(text/xml)");
request.set_body(L"request body", L"text/xml");
web::http::client::http_client_config clientConfig;
clientConfig.set_credentials(web::credentials(L"username", L"pass"));
clientConfig.set_validate_certificates(true);
web::http::client::http_client client(L"serviceurl", clientConfig);
auto bodyTask = client.request(request)
.then([](web::http::http_response response) {
auto str = response.extract_string().get();
return str;
});
auto body = bodyTask.get();
If I'm trying naively to perform another request just after this one then I got an error:
WinHttpSendRequest: 5023: The group or resource is not in the correct state to perform the
requested operation.

In order to read received data in chunks, one needs to get the input stream from the server response
concurrency::streams::istream bodyStream = response.body();
then read continuously from that stream until a given char is found or the number of specified bytes is read
pplx::task<void> repeat(Concurrency::streams::istream bodyStream)
{
Concurrency::streams::container_buffer<std::string> buffer;
return pplx::create_task([=] {
auto t = bodyStream.read_to_delim(buffer, '\n').get();
std::cout << buffer.collection() << std::endl;
return t;
}).then([=](int /*bytesRead*/) {
if (bodyStream.is_eof()) {
return pplx::create_task([]{});
}
return repeat(bodyStream);
});
}
Here is the full sample: https://github.com/cristeab/oanda_stream

Related

Write a cpprestsdk json value object into a file

I receive a JSON response body as part of a REST request. I would like to write the body of the JSON into a file. A quick way to do this would be to use the web::http::response object itself..
pplx::task<void> requestTask = fstream::open_ostream(U("testResults.json")).then([=](ostream outFile)
{
*fileStream = outFile;
//some work
})
.then([=](http_response response)
{
printf("Received response status code:%u\n", response.status_code());
// Write response body into the file.
return response.body().read_to_end(fileStream->streambuf());
})
.then([=](size_t s)
{
return fileStream->close();
});
However, after receiving the response body, I extract the JSON from it to calculate some values, after which, as documented, the json cannot be extracted or used again from the response body, so i have to use the web::json::value object.
http::json::value someValue = response.extract_json().get();
I'd like to write this json::value object, someValue, to a JSON file, but not sure how to get it done. Writing to an ostream object with serialize.c_cstr() on someValue gives weird output.
ostream o("someFile.json");
o << setw(4) << someValue.serialize().c_str() << std<<endl;
If your system typdef std::string as a wide string, then outputting via ostream would lead to weird output.
web::json::value::serialize() returns a utility::string_t
https://microsoft.github.io/cpprestsdk/classweb_1_1json_1_1value.html#a5bbd4bca7b13cd912ffe76af825b0c6c
utility::string_t is typedef as a std::string
https://microsoft.github.io/cpprestsdk/namespaceutility.html#typedef-members
So something like this would work:
std::filesystem::path filePath(L"c:\\someFile.json"); // Assuming wide chars here. Could be U instead of L depending on your setup
std::wofstream outputFile(filePath);
outputFile << someJSONValue.serialize().c_str();
outputFile.close();

S3Stream is getting closed before processing the entire payload

I am processing the bulk json payload from s3.
code as follows:
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.core.JsonParser;
import com.amazonaws.services.s3.model.S3Object;
import static com.fasterxml.jackson.core.JsonToken;
import com.google.common.util.concurrent.Futures;
import com.google.common.util.concurrent.ListenableFuture;
public boolean sync(Job job)
throws IOException
//validating the json payload from s3.
try(InputStream s3Stream = readStreamFromS3())
{
validationService.validate(s3Stream);
}
catch (S3SdkInteractionException e) {
{
logger.error(e.getLocalizedMessage();
}
//process the json payload from s3.
try (InputStream s3Stream = readStreamFromS3())
{
syncService.process(s3Stream);
}
catch (S3SdkInteractionException e) {
{
logger.error(e.getLocalizedMessage();
}
}
public InputSteam readStreamFromS3()
{
return S3Object.getObjectContent();
}
// Process will sync the user data in the s3 stream.
// I am not closing the stream till the entire stream is processed. I
// need to handle as a stream processing.
// I dont want keep the contents in memory for processing, not
feasible for my use case.
public boolean process(InputStream s3Stream)
{
jsonFactory = objectMapper.getFactory();
try(JsonParser jsonParser = jsonFactory.createParser(s3Stream) {
JsonToken jsonToken = jsonParser.nextToken();
List<HttpResponseFuture<UserResponse> userFutures = new ArrayLsit<>(20);
while(true) {
for(int i = 0; i < 20; i++)
{
try {
// stream is processed fully
if (jsonToken == null || jsonToken == JSONTOKEN.END_OBJECT) { break; }
while (!jsonToken.isStructStart()) {
jsonToken = jsonParser.nextToken();
}
// Fetch the user record from the stream
if (jsonTokenn.isStructStart()) {
Map<String,Object> userNode = jsonParser.readValueAs(Map.class);
// calling an external service and adding future response
userFutures.add(executeAsync(httpClient, userNode);
//Move to the next user record
if (jsonToken == JSONTOKEN.START_OBJECT) {
jsonToken = jsonParser.nextToken();
}
}
}
catch (JsonParseException jpe) {
logger.error(jpe.getLocalizedMessage());
break;
}
}
for(ListenableFuture<UserResponse> responseFuture : Futures.inCompletionOrder(userFutures)) {
JsonResponse response = responseFuture.get();
}
}
}
return false;
}
There is serviceA through which we are ingesting data (json payload) to S3.
Another serviceB (the pseudocode shown above) will process the s3 data and call another serviceC to sync the data (json payload) in underlying store.
Problem:
I am seeing repeated s3 warning in our code. com.amazonaws.services.s3.internal.S3AbortableInputStream Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use
The validation phase is executing as expected without any issues.
However on syncing the data(ie. syncService.process()), the s3Stream is getting closed before the entire payload is processed.
Since the stream is getting the closed before i process the entire stream, i am in inconsistent state.
Dependency information as follows
aws-java-sdk-s3:1.11.411
guava:guava-25.0-jre
jackson-core:2.9.6
Json payload could vary between few MB's to 2 GB.
Any help would be appreciated.

C++ Consume API REST with Casablanca and POST XML

The question is how to consume API REST that needs to POST content on body type XML and on headers parameters. I found only solutions with JSON.
return pplx::create_task([]
{
json::value postData;
postData[L"name"] = json::value::string(L"Joe Smith");
postData[L"hobby"] = json::value::string(L"Baseball");
http_client client(L"http://localhost:5540/api/values");
return client.request(methods::POST, L"",
postData.to_string().c_str(), L"application/xml");
}).then([](http_response response)
{
if (response.status_code() == status_codes::OK)
{
auto body = response.extract_string();
std::wcout << L"Added new Id: " << body.get().c_str() << std::endl;
return std::stoi(body.get().c_str());
}
return 0;
});
Thanks for all!
I changed the library. Using libcurl with no problem and it's easier.

How can I issue a POST request that contains a basic authentication header, and a JSON body?

I am trying to use the CPPRESTSDK (a.k.a. Casablanca) to POST data to a RESTful server. To do this, I create a request, and assign a header:
// create request, and add header information
web::http::http_request req(methods::POST);
req.headers().add(header_names::authorization, authStr); // authStr is base64 representation of username & password
req.headers().add(header_names::content_type, http::details::mime_types::application_json);
Next, I make a web::json::value object that contains all the key-value pairs:
web::json::value obj = json::value::object();
obj[U("Key1")] = web::json::value::string(U("Val1")];
obj[U("Key2")] = web::json::value::string(U("Val2")];
obj[U("Key3")] = web::json::value::string(U("Val3")];
I then store this object in the request's body by calling:
req.set_body(obj);
Finally, I send the request to the server using an http_client:
// create http client
web::http::client::http_client client(addr); // addr is wstring
return client.request(req).then([](http_response response) {
return response;
});
The problem is that this doesn't do anything. If I place a breakpoint on this line, I get information about "400 Bad Request." I would assume that the request's body is somehow malformed, but it could also be that I am missing some information in the header. This error does not happen when I issue a GET request on the same URL, so it is definitely a problem with POSTs specifically. What do you think?
Here is a working example:
// create a new channel
pplx::task<web::http::http_response> postChannel(http_client client, std::wstring authStr, std::wstring cDesc, std::wstring cName, std::string cDiagCap, int cNormFloat, int cWriteDuty,
int cWriteMeth, std::string cItemPersist, std::wstring cItemPersistDat) {
// create request
http_request req(methods::POST);
req.headers().add(header_names::authorization, authStr);
std::wstring url = L"/config/v1/project/channels";
req.set_request_uri(url);
json::value obj = json::value::object();
obj[U("common.ALLTYPES_DESCRIPTION")] = json::value::string(cDesc);
obj[U("common.ALLTYPES_NAME")] = json::value::string(cName);
obj[U("servermain.CHANNEL_DIAGNOSTICS_CAPTURE")] = json::value(cDiagCap == "true" || cDiagCap == "t");
obj[U("servermain.CHANNEL_NON_NORMALIZED_FLOATING_POINT_HANDLING")] = json::value(cNormFloat);
obj[U("servermain.CHANNEL_WRITE_OPTIMIZATIONS_DUTY_CYCLE")] = json::value(cWriteDuty);
obj[U("servermain.CHANNEL_WRITE_OPTIMIZATIONS_METHOD")] = json::value(cWriteMeth);
obj[U("servermain.MULTIPLE_TYPES_DEVICE_DRIVER")] = json::value::string(U("Simulator")); // right now, Simulator channels are the only option
obj[U("simulator.CHANNEL_ITEM_PERSISTENCE")] = json::value(cItemPersist == "true" || cItemPersist == "t");
obj[U("simulator.CHANNEL_ITEM_PERSISTENCE_DATA_FILE")] = json::value::string(cItemPersistDat);
req.set_body(obj);
return client.request(req).then([](http_response response) {
return response;
});
}

C++ rest sdk POST form data json

Is it possible to post "form data" whith C++ rest SDK (Casablanca)? I have a given web service which looking for post data in "form data", not in the body.
This is the C++ code:
http_client client(L"http://localhost/posttest/jsontest.php");
// Manually build up an HTTP request with header and request URI.
http_request request(methods::POST);
request.headers().add(L"Content-Type", L"application/json");
request.headers().add(L"Content-Length", L"100");
request.headers().add(L"Host", L"example.com");
request.headers().add(L"X-Requested-With", L"XMLHttpRequest");
request.set_body(obj);
return client.request(request).then([id](http_response response)
{
if (response.status_code() == status_codes::OK)
{
return response.extract_json();
}
else {
/* Print bad status code */
wcout << L"Server returned returned status code " << response.status_code() << L'.' << std::endl;
}
return pplx::task_from_result(json::value());
})
The web service can only use data like this (I can't modify it):
$arr = [$_POST['code']];
header('Content-Type: application/json');
echo json_encode($arr);
(This is just a sample PHP code, what I use for testing)
That is the way:
utility::string_t Lreq = L"code=" + Lcode;
http_client client(L"http://localhost/posttest/jsontest.php");
// Manually build up an HTTP request with header and request URI.
http_request request(methods::POST);
request.headers().add(L"Content-Type", L"application/x-www-form-urlencoded; charset=UTF-8");
request.headers().add(L"Content-Length", L"100");
request.headers().add(L"Host", L"testhost.com");
request.headers().add(L"X-Requested-With", L"XMLHttpRequest");
request.set_body(Lreq);