Boost.Beast: Is it possible to construct an HTTP batch request? - c++

I've been using Boost.Beast for awhile now to send HTTP requests and I think it's great and works perfectly. But does anyone know if it's possible to construct an HTTP batch request using Beast? I'm thinking something like creating several subrequests:
boost::beast::http::request<boost::beast::http::string_body> subrequest_1;
subrequest_1.set(boost::beast::http::field::content_type, "application/http");
...
boost::beast::http::request<boost::beast::http::string_body> subrequest_2;
subrequest_2.set(boost::beast::http::field::content_type, "application/http");
...
and then somehow group them together and send them all in one request.
I've been trying to create a vector, add the subrequests to it and then assign the vector to the body of the request I'd like to send, but that hasn't been successful.
/*
std::vector<boost::beast::http::request<boost::beast::http::string_body>> v;
v.push_back(subrequest_1);
v.push_back(subrequest_2);
boost::beast::http::request<boost::beast::http::string_body> request;
request.method(boost::beast::http::verb::post);
...
request.body() = v;
*/
Thanks in advance!

Well this one turned out to be very simple. I'll just post my solution here in case anyone else feels as lost as I did a couple of days ago.
// Set up the request with a new content type
boost::beast::http::request<boost::beast::http::string_body> request;
request.set(field::content_type, "multipart/mixed; boundary='subrequest_boundary'");
// A string to hold the entire batch request
std::string body;
// Add all messages in e.g. a queue to the string
while ( messages.size() )
{
auto message = messages.front();
// Add a boundary to separate the messages
body.append("--subrequest_boundary");
body.append("Content-Type: application/http\n");
/*
And all more headers you need
*/
// I used the nlohmann library to add json docs
nlohmann::json payload = {
{"message",
}
};
body.append( payload.dump() );
messages.pop();
}
// Add a boundary as wrap up
body.append("--subrequest_boundary--");
// Add the batch request to the request
request.body() = body;
request.prepare_payload();

Related

Read request body without deleting it

So I have a MyHandler which has to know what's inside the request body:
class MyHandler
include HTTP::Handler
def call(context)
p "MyHandler got body: " + context.request.body.not_nil!.gets_to_end
call_next(context)
end
end
server = HTTP::Server.new(42, [MyHandler.new]) do |context|
p "Server got body: " + context.request.body.not_nil!.gets_to_end
end
As expected, after MyHandler has read, server receives a empty body. How can copy the body without modifying original context?
Crystal supports streaming request bodies, which means that once you stream in the request, the IO is EOF and the second handler can't read any data.
A simple way to solve this would be to retrieve the entire content using body_string = context.request.body.try(&.gets_to_end), then set the request body to the returned string using context.request.body = body_string. This buffers the entire body to memory then sets the body to the buffer stored in memory. The downside to this approach is that an attacker can send an infinitely sized request body and eat all the memory on your server, causing a DOS attack. Another disadvantage is that if you're working with binary data, you would then need to convert the string into a slice using #to_slice to work with it.
One way to solve the DOS attack problem - if you have a maximum body size in mind - is to fail the request if the body is too large:
if body = context.request.body
body_io = IO::Memory.new
bytes_read = IO.copy(body, body_io, limit: 1_048_576) # 1GiB limit
body_io.rewind
if bytes_read == 1_048_576
# Fail request
end
# use body_io
body_io.rewind # Reset body_io to start
context.request.body = body_io
end
If you need to accept an infinitely sized body, and not buffer it to memory, you should create a custom IO implementation which wraps the existing body IO and runs the required transform inside IO#read(Bytes). This method is quite complex, and the previous method covers almost all situations, so I won't provide a code sample for this option.

Send BigText via WinHTTP 5.1 in Dynamics NAV 2009 SP1

I know how to send normal Text via WinHTTP 5.1 Automation and how to turn the reponse stream into an BigText object.
Now I want to send the content of a BigText via POST/PUT basicly this:
CREATE(bigText);
bigText.ADDTEXT('...');
...
CREATE(HTTP, TRUE, TRUE);
HTTP.OPEN('PUT', 'https://...', FALSE);
HTTP.SetCredentials('...', '...', 0);
HTTP.SEND(bigText);
The codeunit actually compiles and the automation object does send the request to the server, however with empty request body.
I tried to use OutStream but than the codeunit does not compile (Automation := OutStream).
I am using Dynamics NAV 2009 SP1, so no DotNet DataType available as well.
I got it to work by a stream juggle
// Variable Declaration:
// HTTP = Automation WinHTTP Services 5.1
// TempBlob = Record <TEMPORARY=YES> TempBlob
// blobOutStream = OutStream
// RequestBodyBigText = BigText
// ResponseBodyBigText = BigText
// RequestInStream = InStream
// ReponseInStream = InStream
// create WinHTTP client, force new instance, don't run locally
CREATE(HTTP, TRUE, FALSE);
HTTP.Open('PUT', '...', FALSE);
HTTP.SetCredentials('...', '....', 0);
// ...
// create InStream from BigText by the help of Temporary=YES TempBlob Record
TempBlob.INIT;
TempBlob."Blob".CREATEOUTSTREAM(blobOutStream);
// write the content of the reuquest body to the temp blob
RequestBodyBigText.WRITE(blobOutStream);
// important, calcfield the temp blob so that we can use the content
// in a new stream
TempBlob.CALCFIELDS("Blob");
TempBlob."Blob".CREATEINSTREAM(RequestInStream);
// send the stream
HTTP.Send(RequestInStream);
// timeout is in seconds
IF HTTP.WaitForResponse(30) THEN BEGIN
ResponseInStream := HTTP.ResponseStream;
CLEAR(ResponseBodyBigText);
ReponseBodyBigText.READ(ResponseInStream);
END;
// now we have a big text (ResponseBodyBigText) filled with the body of the response
If you encounter encoding problems you replace the ResponsBodyBigText.READ with a convertion function and a EOS loop. If you can't use DotNet Interop DataTypes (like me) you can use the ADOStream automation with charset set to UTF-8 or you use an own written COM Object (like I did)

Springs RestTemplate doesn't find the right MessageConverter when reciving data from the IMDB api

first I have to say, that I am pretty new to Springs RestTemplate.
I am trying to receive data from the imdb-api. (For example http://imdbapi.org/?title=Avatar&type=xml) Therefore I am using Springs RestTemplate.
But:
the webservice returns the data as application/octet-stream (even I declared that I want it as xml (when I browse the site with my browser I get the data as text/xml))
RestTemplate doesn't find my declared ByteArrayMessageConverter (to convert application/octet-stream)
I realy don't know where my mistakes are.
Here is the code to initialise the restTemplate:
public void onInit() {
_log.debug("Setting up the Spring Resttemplate");
_restTemplate = new RestTemplate();
List<HttpMessageConverter<?>> list = new ArrayList<HttpMessageConverter<?>>();
list.add(new SourceHttpMessageConverter<Source>());
list.add(new ByteArrayHttpMessageConverter());
_restTemplate.setMessageConverters(list);
_log.debug("Setting up the HTTP Headers for Restrequest");
List<MediaType> acceptableMediaTypes = new ArrayList<MediaType>();
_log.trace("allow {}", MediaType.APPLICATION_XML_VALUE);
acceptableMediaTypes.add(MediaType.APPLICATION_XML);
_log.trace("allow {}", MediaType.TEXT_HTML_VALUE);
acceptableMediaTypes.add(MediaType.TEXT_XML);
_log.trace("set accepted charset to uft-8");
List<Charset> acceptableCharsets = new ArrayList<Charset>();
acceptableCharsets.add(Charset.forName("utf-8"));
_httpHeaders = new HttpHeaders();
_httpHeaders.set("User-Agent", "something"); //only a user-agent, because the api returns a 403 if it is not set
_httpHeaders.setAcceptCharset(acceptableCharsets);
_httpHeaders.setAccept(acceptableMediaTypes);
}
Here is the code with the call:
_log.info("connect to Imdb-Webservice {}", _imbdWebserviceBaseUrl);
Map<String, Object> uriVariables = new HashMap<String, Object>();
uriVariables.put("title", pTitle);
ResponseEntity<Source> response = _restTemplate.exchange(_imbdWebserviceBaseUrl, HttpMethod.GET, new HttpEntity<String>(_httpHeaders), Source.class, uriVariables);
_imbdWebserviceBaseUrl is set to http://imdbapi.org/?title={title}&type=xml
Then I am getting this error message:
org.springframework.web.client.RestClientException: Could not extract response: no suitable HttpMessageConverter found for response type [interface javax.xml.transform.Source] and content type [application/octet-stream]
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:107)
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:687)
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:673)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:491)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:454)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:401)
at my.domain.projectname.integrationimpl.WebserviceHelper.getXml(WebserviceHelper.java:131)
Thanks for your help
the web service returns the data as application/octet-stream (even I declared that I want it as xml (when I browse the site with my browser I get the data as text/xml))
As far as I can see this rest service is not giving back the correct Content-Type (text/xml or similar). If your browser renders it correctly that's probably Chrome or Firefox, but IE will just show you html-ish kind of output.
RestTemplate doesn't find my declared ByteArrayMessageConverter (to convert application/octet-stream)
Well you are asking for a Source as far as I can see:
ResponseEntity<Source> response = _restTemplate.exchange(_imbdWebserviceBaseUrl, HttpMethod.GET, new HttpEntity<String>(_httpHeaders), Source.class, uriVariables);
The MessageConverters themselves have a method that determines if this converter is applicable, for ByteArrayHttpMessageConverter this is:
#Override
public boolean supports(Class<?> clazz) {
return byte[].class.equals(clazz);
}
Since you are asking for a Source.class it wont use this converter.

Qt QNetworkAccessManager or other method get html status code without getting page contenet?

i need to get web sites html status codes
today i just do simple get request to the domain , and then i get the status code as part of the response , but also the site index.html content .
pNetworkManager = new QNetworkAccessManager(this);
reply = pNetworkManager->get(request);
QVariant vStatusCodeV = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute);
QVariant redirectionTarget = reply->attribute(QNetworkRequest::RedirectionTargetAttribute);
data=reply->readAll();
this last function i like to avoid if it can be avoided ,
is there any way to get only the domain status code ?
Maybe you can send a HEAD request instead of a GET request?
This is not a Qt / client specific solution, but is the approach recommended by the HTTP protocol when you don't need the content, but just want to get the headers that a request would normally produce, for example in order to validate that the page exists.
I suppose this could be done with QNetworkAccessManager using the head() method
I agree with #shevron's answer, but if the site you're communicating with isn't "clever" enough to implement the HEAD request, you can still avoid the readAll() call.
QByteArray line = reply->readLine(); //< eg "HTTP/1.0 200 OK"
QList<QByteArray> chunks = line.split(' ');
QString statusCode = chunks[1];
That should avoid the memory overhead of readAll().

Abort user request with Node.js/formidable

I'm using formidable to receive a file upload with node.js. I send some fields together with a file in a multipart request.
As soon as certain fields arrived, I'm able to validate the authenticity of the request for instance, and I would like to abort the whole request if this is not correct to avoid waisting resources.
I have not found a right way to abort the incoming request. I tried to use req.connection.destroy(); as follow:
form
.on('field', function(field, value) {
fields[field] = value;
if (!fields['token'] || !fields['id'] || !fields['timestamp']) {
return;
}
if (!validateToken(fields['token'], fields['id'], fields['timestamp'])) {
res.writeHead(401, {'Content-Type' : 'text/plain' });
res.end('Unauthorized');
req.connection.destroy();
}
})
However, this triggers the following error:
events.js:45
throw arguments[1]; // Unhandled 'error' event
^
Error: Cannot resume() closed Socket.
at Socket.resume (net.js:764:11)
at IncomingMessage.resume (http.js:254:15)
at IncomingForm.resume (node_modules/formidable/lib/incoming_form.js:52:11)
at node_modules/formidable/lib/incoming_form.js:181:12
at node_modules/formidable/lib/file.js:51:5
at fs.js:1048:7
at wrapper (fs.js:295:17)
I also tried req.connection.end() but the file keeps uploading.
Any thoughts? Thanks in advance!
The problem is that formidable didn't understand that you want it to stop. Try this:
req.connection.destroy();
req.connection.resume = function(){};
Of course, this is a somewhat ugly workaround, I'd open an issue on github.