Handling large http response using boost::beast - c++

Question

The following code use to get http response message:
boost::beast::tcp_stream stream_;
boost::beast::flat_buffer buffer;
boost::beast::http::response<boost::beast::http::dynamic_body> res;
boost::beast::http::read(stream_, buffer, res);
However, In some cases, based on the preceding request, I can expect that the response message body will include large binary file.
Therefore, I’d like to read it directly to the filesystem and not through buffer variable to avoid excessive use of process memory. How can it be done ?
in Objective-c framework NSUrlSession there's an easy way to do it using NSURLSessionDownloadTask instead of NSURLSessionDataTask, so I wonder if it's also exist in boost.
Thanks !

Answer 1

In general, you can use the http::buffer_body to handle arbitrarily large request/response messages.
If you specifically want to read/write from a filesystem file, you can have the http::file_body instead.
Full Demo buffer_body
The documentation sample for buffer_body is here https://www.boost.org/doc/libs/1_77_0/libs/beast/doc/html/beast/using_http/parser_stream_operations/incremental_read.html.
Using it to write to std::cout: Live On Coliru
#include <boost/asio.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/beast.hpp>
#include <boost/beast/websocket.hpp>
#include <iostream>
namespace net = boost::asio;
namespace beast = boost::beast;
namespace http = beast::http;
using tcp = net::ip::tcp;
using socket_t = tcp::socket;
/* This function reads a message using a fixed size buffer to hold
portions of the body, and prints the body contents to a `std::ostream`.
*/
template<
bool isRequest,
class SyncReadStream,
class DynamicBuffer>
void
read_and_print_body(
std::ostream& os,
SyncReadStream& stream,
DynamicBuffer& buffer,
beast::error_code& ec)
{
http::parser<isRequest, http::buffer_body> p;
http::read_header(stream, buffer, p, ec);
if(ec)
return;
while(! p.is_done())
{
char buf[512];
p.get().body().data = buf;
p.get().body().size = sizeof(buf);
http::read(stream, buffer, p, ec);
if(ec == http::error::need_buffer)
ec = {};
if(ec)
return;
os.write(buf, sizeof(buf) - p.get().body().size);
}
}
int main() {
std::string host = "173.203.57.63"; // COLIRU 20210901
auto const port = "80";
net::io_context ioc;
tcp::resolver resolver{ioc};
socket_t s{ioc};
net::connect(s, resolver.resolve(host, port));
write(s, http::request<http::empty_body>{http::verb::get, "/", 11});
beast::error_code ec;
beast::flat_buffer buf;
read_and_print_body<false>(std::cout, s, buf, ec);
}
Full file_body example
This is much shorter, writing to body.html:
Live On Coliru
#include <boost/asio.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/beast.hpp>
#include <boost/beast/websocket.hpp>
#include <iostream>
namespace net = boost::asio;
namespace beast = boost::beast;
namespace http = beast::http;
using tcp = net::ip::tcp;
using socket_t = tcp::socket;
int main() {
std::string host = "173.203.57.63"; // COLIRU 20210901
auto const port = "80";
net::io_context ioc;
tcp::resolver resolver{ioc};
socket_t s{ioc};
net::connect(s, resolver.resolve(host, port));
write(s, http::request<http::empty_body>{http::verb::get, "/", 11});
beast::error_code ec;
beast::flat_buffer buf;
http::response<http::file_body> res;
res.body().open("body.html", beast::file_mode::write_new, ec);
if (!ec.failed())
{
read(s, buf, res, ec);
}
std::cout << "Wrote 'body.html' (" << ec.message() << ")\n";
std::cout << "Headers " << res.base() << "\n";
}
Prints
Wrote 'body.html' (Success)
Headers HTTP/1.1 200 OK
Content-Type: text/html;charset=utf-8
Content-Length: 8616
Server: WEBrick/1.4.2 (Ruby/2.5.1/2018-03-29) OpenSSL/1.0.2g
Date: Wed, 01 Sep 2021 19:52:20 GMT
Connection: Keep-Alive
With file body.html; wc body.html showing:
body.html: HTML document, ASCII text, with very long lines
185 644 8616 body.html
Beyond: streaming to child processes and streaming processing
I have an advanced example of that here: How to read data from Internet using muli-threading with connecting only once?.