Sending async https request using boost/beast - c++

Question

I had a synchronous method that send https request using http::write and than expect to read it's response using http::read.
However, in order to add timeout I had to move to async calls in my method. So I've tried to use http::async_read and http::async_write, but keep this overall flow synchronous so the method will return only once it has the https response.
here's my attempt :
class httpsClass {
std::optional<boost::beast::ssl_stream<boost::beast::tcp_stream>> ssl_stream_;
httpsClass(..) {
// notice that ssl_stream_ is initialized according to io_context_/ctx_
// that are class members that get set by c'tor args
ssl_stream_.emplace(io_context_, ctx_);
}
}
std::optional<boost::beast::http::response<boost::beast::http::dynamic_body>>
httpsClass::sendHttpsRequestAndGetResponse (
const boost::beast::http::request<boost::beast::http::string_body>
&request) {
try{
boost::asio::io_context ioc;
beast::flat_buffer buffer;
http::response<http::dynamic_body> res;
beast::get_lowest_layer(*ssl_stream_).expires_after(kTimeout);
boost::asio::spawn(ioc, [&, this](boost::asio::yield_context yield) {
auto sent = http::async_write(this->ssl_stream_.value(), request, yield);
auto received = http::async_read(this->ssl_stream_.value(), buffer, res, yield);
});
ioc.run();// this will finish only once the task above will be fully executed.
return res;
} catch (const std::exception &e) {
log("Error sending/receiving:{}", e.what());
return std::nullopt;
}
}
During trial, this method above reaches the task I assign for the internal io contexts (ioc). However, it gets stuck inside this task on the method async_write.
Anybody can help me figure out why it gets stuck? could it be related to the fact that ssl_stream_ is initialize with another io context object (io_context_) ?

Answer 1

Yes. The default executor for completion handlers on the ssl_stream_ is the outer io_context, which cannot make progress, because you're likely not running it.
My hint would be to:
avoid making the second io_context
also use the more typical future<Response> rather than optional<Response> (which loses the the error information)
avoid passing the io_context&. Instead pass executors, which you can more easily change to be a strand executor if so required.
Adding some code to make it self-contained:
class httpsClass {
ssl::context& ctx_;
std::string host_;
std::optional<beast::ssl_stream<beast::tcp_stream>> ssl_stream_;
beast::flat_buffer buffer_;
static constexpr auto kTimeout = 3s;
public:
httpsClass(net::any_io_executor ex, ssl::context& ctx, std::string host)
: ctx_(ctx)
, host_(host)
, ssl_stream_(std::in_place, ex, ctx_) {
auto ep = tcp::resolver(ex).resolve(host, "https");
ssl_stream_->next_layer().connect(ep);
ssl_stream_->handshake(ssl::stream_base::handshake_type::client);
log("Successfully connected to {} for {}",
ssl_stream_->next_layer().socket().remote_endpoint(), ep->host_name());
}
using Request = http::request<http::string_body>;
using Response = http::response<http::dynamic_body>;
std::future<Response> performRequest(Request const&);
};
Your implementation was pretty close, except for the unnecessary service:
std::future<httpsClass::Response>
httpsClass::performRequest(Request const& request) {
std::promise<Response> promise;
auto fut = promise.get_future();
auto coro = [this, r = request, p = std::move(promise)] //
(net::yield_context yield) mutable {
try {
auto& s = *ssl_stream_;
get_lowest_layer(s).expires_after(kTimeout);
r.prepare_payload();
r.set(http::field::host, host_);
auto sent = http::async_write(s, r, yield);
log("Sent: {}", sent);
http::response<http::dynamic_body> res;
auto received = http::async_read(s, buffer_, res, yield);
log("Received: {}", received);
p.set_value(std::move(res));
} catch (...) {
p.set_exception(std::current_exception());
}
};
spawn(ssl_stream_->get_executor(), std::move(coro));
return fut;
}
Now, it is important to have the io_service run()-ning for any asynchronous operations. With completely asynchronous code you wouldn't need threads, but as you are blocking on the response you will. The easiest way is to replace io_service with a thread_pool which does the run()-ning for you.
int main() {
net::thread_pool ioc;
ssl::context ctx(ssl::context::sslv23_client);
ctx.set_default_verify_paths();
for (auto query : {"/delay/2", "/delay/5"}) {
try {
httpsClass client(make_strand(ioc), ctx, "httpbin.org");
auto res = client.performRequest({http::verb::get, query, 11});
log("Request submitted... waiting for response");
log("Response: {}", res.get());
} catch (boost::system::system_error const& se) {
auto const& ec = se.code();
log("Error sending/receiving:{} at {}", ec.message(), ec.location());
} catch (std::exception const& e) {
log("Error sending/receiving:{}", e.what());
}
}
ioc.join();
}
As you can see this test will run two requests against https://httpbin.org/#/Dynamic_data/get_delay__delay_. The second will timeout because 5s exceeds the 3s expiration on the ssl_stream_.
Full Demo
Live On Coliru
#include <boost/asio.hpp>
#include <boost/asio/spawn.hpp>
#include <boost/beast.hpp>
#include <boost/beast/ssl.hpp>
#include <fmt/ostream.h>
#include <fmt/ranges.h>
#include <optional>
using namespace std::chrono_literals;
namespace net = boost::asio;
namespace beast = boost::beast;
namespace http = beast::http;
namespace ssl = net::ssl;
using net::ip::tcp;
////// LOG STUBS
template <> struct fmt::formatter<boost::source_location> : fmt::ostream_formatter {};
template <> struct fmt::formatter<tcp::endpoint> : fmt::ostream_formatter {};
template <bool isRequest, typename... Args>
struct fmt::formatter<http::message<isRequest, Args...>> : fmt::ostream_formatter {};
static inline void log(auto const& fmt, auto const&... args) {
fmt::print(fmt::runtime(fmt), args...);
fmt::print("\n");
std::fflush(stdout);
}
////// END LOG STUBS
class httpsClass {
ssl::context& ctx_;
std::string host_;
std::optional<beast::ssl_stream<beast::tcp_stream>> ssl_stream_;
beast::flat_buffer buffer_;
static constexpr auto kTimeout = 3s;
public:
httpsClass(net::any_io_executor ex, ssl::context& ctx, std::string host)
: ctx_(ctx)
, host_(host)
, ssl_stream_(std::in_place, ex, ctx_) {
auto ep = tcp::resolver(ex).resolve(host, "https");
ssl_stream_->next_layer().connect(ep);
ssl_stream_->handshake(ssl::stream_base::handshake_type::client);
log("Successfully connected to {} for {}",
ssl_stream_->next_layer().socket().remote_endpoint(), ep->host_name());
}
using Request = http::request<http::string_body>;
using Response = http::response<http::dynamic_body>;
std::future<Response> performRequest(Request const&);
};
std::future<httpsClass::Response>
httpsClass::performRequest(Request const& request) {
std::promise<Response> promise;
auto fut = promise.get_future();
auto coro = [this, r = request, p = std::move(promise)] //
(net::yield_context yield) mutable {
try {
auto& s = *ssl_stream_;
get_lowest_layer(s).expires_after(kTimeout);
r.prepare_payload();
r.set(http::field::host, host_);
auto sent = http::async_write(s, r, yield);
log("Sent: {}", sent);
http::response<http::dynamic_body> res;
auto received = http::async_read(s, buffer_, res, yield);
log("Received: {}", received);
p.set_value(std::move(res));
} catch (...) {
p.set_exception(std::current_exception());
}
};
spawn(ssl_stream_->get_executor(), std::move(coro));
return fut;
}
int main() {
net::thread_pool ioc;
ssl::context ctx(ssl::context::sslv23_client);
ctx.set_default_verify_paths();
for (auto query : {"/delay/2", "/delay/5"}) {
try {
httpsClass client(make_strand(ioc), ctx, "httpbin.org");
auto res = client.performRequest({http::verb::get, query, 11});
log("Request submitted... waiting for response");
log("Response: {}", res.get());
} catch (boost::system::system_error const& se) {
auto const& ec = se.code();
log("Error sending/receiving:{} at {}", ec.message(), ec.location());
} catch (std::exception const& e) {
log("Error sending/receiving:{}", e.what());
}
}
ioc.join();
}
Live on my system: