Memory Leak at async_handshake using boost beast and openssl - c++

Question

I'm using OpenSSL 1.1.1b and Boost 1.68 to create a simple server using https.
I followed the examples provided by boost beast and in particular the advance server flex.
The application seems to work properly. I can accept https session and also wss sessions.
The problem is when I exit from the application where the Visual Leak Detector finds 16 memory leaks that target at:
c:\openssl-1.1.1b\crypto\mem.c (233): abc.exe!CRYPTO_zalloc
c:\openssl-1.1.1b\crypto\err\err.c (716): abc.exe!ERR_get_state + 0x17 bytes
c:\openssl-1.1.1b\crypto\err\err.c (443): abc.exe!ERR_clear_error + 0x5 bytes
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\impl\engine.ipp (235): abc.exe!boost::asio::ssl::detail::engine::perform
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\impl\engine.ipp (137): abc.exe!boost::asio::ssl::detail::engine::handshake
I modified the pattern of the http session from the original boost beast code but it should perform exactly the same things.
I've tried to understand if the memory leaks increase based on the number of connections but it seems not. I don't understand how to get rid of this problem.
Following the code I used.
First a based http session class
class CApplicationServerBaseHttpSession
{
public:
std::shared_ptr<CApplicationServerSharedState> m_state = nullptr;
CApplicationServerHttpQueue m_queue;
// The parser is stored in an optional container so we can
// construct it from scratch it at the beginning of each new message.
boost::optional<boost::beast::http::request_parser<boost::beast::http::string_body>> parser_;
protected:
boost::asio::steady_timer m_timer;
boost::beast::flat_buffer buffer_;
boost::log::sources::severity_channel_logger<boost::log::trivial::severity_level> m_Logger{boost::log::keywords::channel = LOG_APPLICATION_SERVER_CHANNEL_ID};
boost::asio::strand<boost::asio::io_context::executor_type> m_strand;
public:
// Construct the session
CApplicationServerBaseHttpSession(
boost::asio::io_context& ioc,
boost::beast::flat_buffer buffer,
std::shared_ptr<CApplicationServerSharedState> const& state)
: m_state(state)
, m_strand(ioc.get_executor())
, m_timer(ioc,
(std::chrono::steady_clock::time_point::max)()
)
, m_queue(*this)
, buffer_(std::move(buffer))
{
}
void DoRead();
void OnRead(boost::system::error_code ec);
void OnWrite(boost::system::error_code ec, bool close);
virtual void WriteRequestStringBody(boost::beast::http::response<boost::beast::http::string_body> & msg) = 0;
virtual void WriteRequestFileBody(boost::beast::http::response<boost::beast::http::file_body> & msg) = 0;
protected:
virtual void ReadRequest() = 0;
virtual void DoEof() = 0;
virtual std::string GetRemoteAddress() = 0;
virtual void MakeWebSocketSession(boost::beast::http::request<boost::beast::http::string_body> req) = 0;
};
Here the implementation:
void CApplicationServerBaseHttpSession::DoRead()
{
// Set the timer
m_timer.expires_after(std::chrono::seconds(OCV_HTTP_SESSION_TIMER_EXPIRE_AFTER));
// Construct a new parser for each message
parser_.emplace();
// Apply a reasonable limit to the allowed size
// of the body in bytes to prevent abuse.
parser_->body_limit(HTTP_BODY_LIMIT);
this->ReadRequest();
}
void CApplicationServerBaseHttpSession::OnRead(boost::system::error_code ec)
{
// Happens when the timer closes the socket
if(ec == boost::asio::error::operation_aborted)
return;
// This means they closed the connection
if(ec == http::error::end_of_stream)
return this->DoEof();
if(ec == boost::asio::ssl::error::stream_truncated){
// "stream truncated" means that the other end closed the connection abruptly.
return warning(ec, "Http read", m_Logger);
}
if(ec)
return fail(ec, "Http read", m_Logger);
// See if it is a WebSocket Upgrade
if(websocket::is_upgrade(parser_->get())) {
// Get a websocket request handler to execute operation as authentication and authorization
// If these steps are allowed than the websocket session will be started
std::shared_ptr<CApplicationServerWsApiBase> endpointWs = m_state->GetEndpointWs(parser_->get().target().to_string());
if(endpointWs) {
int endpointErrorDefault = endpointWs->HandleRequest(parser_->get());
if(endpointErrorDefault > 0) { // Success Auth
// Make timer expire immediately, by setting expiry to time_point::min we can detect
// the upgrade to websocket in the timer handler
m_timer.expires_at((std::chrono::steady_clock::time_point::min)());
// Transfer the stream to a new WebSocket session
return MakeWebSocketSession(parser_->release());
} else {
// Authentication or Authorization failed
m_queue(endpointWs->GetResponseError(parser_->get(), endpointErrorDefault));
return;
}
} else {
// Wrong endpoint called: BadRequest
std::shared_ptr<CApplicationServerApiBase> endpoint = m_state->GetEndpoint(ApiURI::REQUEST_NOT_IMPLEMENTED);
if(endpoint) {
endpoint->HandleRequest(m_state->GetDocRoot(), parser_->release(), m_queue);
}
return;
}
}
BOOST_LOG_SEV(m_Logger, boost::log::trivial::trace) <<
"Request From: " <<
this->GetRemoteAddress() <<
" Request Target: " <<
parser_->get().target().to_string();
std::shared_ptr<CApplicationServerApiBase> endpoint = m_state->GetEndpoint(parser_->get().target().to_string());
if(endpoint) {
endpoint->HandleRequest(m_state->GetDocRoot(), parser_->release(), m_queue);
}
// If we aren't at the queue limit, try to pipeline another request
if(!m_queue.IsFull()) {
DoRead();
}
}
void CApplicationServerBaseHttpSession::OnWrite(boost::system::error_code ec, bool close)
{
// Happens when the timer closes the socket
if(ec == boost::asio::error::operation_aborted)
return;
if(ec)
return fail(ec, "write", m_Logger);
if(close) {
// This means we should close the connection, usually because
// the response indicated the "Connection: close" semantic.
return this->DoEof();
}
// Inform the queue that a write completed
if(m_queue.OnWrite()) {
// Read another request
DoRead();
}
}
The https session:
class COcvApplicationServerHttpSessionSSL
: public std::enable_shared_from_this<COcvApplicationServerHttpSessionSSL>
, public CApplicationServerBaseHttpSession
{
public:
public:
COcvApplicationServerHttpSessionSSL(boost::asio::ip::tcp::socket&& socket,boost::asio::ssl::context& ctx, boost::beast::flat_buffer&& buffer, std::shared_ptr<CApplicationServerSharedState> const& state);
~COcvApplicationServerHttpSessionSSL();
// Called by the base class
boost::beast::ssl_stream<boost::asio::ip::tcp::socket>& Stream();
boost::beast::ssl_stream<boost::asio::ip::tcp::socket> ReleaseStream();
void DoTimeout();
// Start the asynchronous operation
void Run();
void OnHandshake(boost::system::error_code ec, std::size_t bytes_used);
void OnShutdown(boost::system::error_code ec);
void OnTimer(boost::system::error_code ec);
private:
public:
boost::beast::ssl_stream<boost::asio::ip::tcp::socket> m_stream;
bool m_eof = false;
protected:
// Inherited via COcvApplicationServerBaseHttpSession
virtual void ReadRequest() override;
virtual void WriteRequestStringBody(boost::beast::http::response<boost::beast::http::string_body> & msg) override;
virtual void WriteRequestFileBody(boost::beast::http::response<boost::beast::http::file_body> & msg) override;
virtual void DoEof() override;
virtual std::string GetRemoteAddress() override;
virtual void MakeWebSocketSession(boost::beast::http::request<boost::beast::http::string_body> req) override;
};
and at the end the implementatition
COcvApplicationServerHttpSessionSSL::COcvApplicationServerHttpSessionSSL(tcp::socket&& socket, ssl::context & ctx, beast::flat_buffer&& buffer, std::shared_ptr<CApplicationServerSharedState> const & state)
: CApplicationServerBaseHttpSession(
socket.get_executor().context(),
std::move(buffer),
state)
, m_stream(std::move(socket), ctx)
{
}
COcvApplicationServerHttpSessionSSL::~COcvApplicationServerHttpSessionSSL()
{
}
beast::ssl_stream<tcp::socket> & COcvApplicationServerHttpSessionSSL::Stream()
{
return m_stream;
}
beast::ssl_stream<tcp::socket> COcvApplicationServerHttpSessionSSL::ReleaseStream()
{
return std::move(m_stream);
}
void COcvApplicationServerHttpSessionSSL::DoTimeout()
{
// If this is true it means we timed out performing the shutdown
if(m_eof)
return;
// Start the timer again
m_timer.expires_at(
(std::chrono::steady_clock::time_point::max)());
OnTimer({});
DoEof();
}
std::string COcvApplicationServerHttpSessionSSL::GetRemoteAddress()
{
return Stream().next_layer().remote_endpoint().address().to_string();
}
void COcvApplicationServerHttpSessionSSL::MakeWebSocketSession(boost::beast::http::request<boost::beast::http::string_body> req)
{
std::make_shared<CApplicationServerWebSocketSessionSSL>(
std::move(m_stream), m_state)->Run(std::move(req));
}
void COcvApplicationServerHttpSessionSSL::Run()
{
// Make sure we run on the strand
if(!m_strand.running_in_this_thread())
return boost::asio::post(
boost::asio::bind_executor(
m_strand,
std::bind(
&COcvApplicationServerHttpSessionSSL::Run,
shared_from_this())));
// Run the timer. The timer is operated
// continuously, this simplifies the code.
OnTimer({});
// Set the timer
m_timer.expires_after(std::chrono::seconds(OCV_HTTP_SESSION_TIMER_EXPIRE_AFTER));
// Perform the SSL handshake
// Note, this is the buffered version of the handshake.
m_stream.async_handshake(
ssl::stream_base::server,
buffer_.data(),
boost::asio::bind_executor(
m_strand,
std::bind(
&COcvApplicationServerHttpSessionSSL::OnHandshake,
shared_from_this(),
std::placeholders::_1,
std::placeholders::_2)));
}
void COcvApplicationServerHttpSessionSSL::OnHandshake(boost::system::error_code ec, std::size_t bytes_used)
{
// Happens when the handshake times out
if(ec == boost::asio::error::operation_aborted)
return;
if(ec)
return fail(ec, "handshake", m_Logger);
// Consume the portion of the buffer used by the handshake
buffer_.consume(bytes_used);
DoRead();
}
void COcvApplicationServerHttpSessionSSL::OnShutdown(boost::system::error_code ec)
{
// Happens when the shutdown times out
if(ec == boost::asio::error::operation_aborted || ec == boost::asio::ssl::error::stream_truncated)
return;
if(ec)
return fail(ec, "shutdown HTTPS", m_Logger);
// At this point the connection is closed gracefully
}
void COcvApplicationServerHttpSessionSSL::OnTimer(boost::system::error_code ec)
{
if(ec && ec != boost::asio::error::operation_aborted)
return fail(ec, "timer", m_Logger);
// Check if this has been upgraded to Websocket
if(m_timer.expires_at() == (std::chrono::steady_clock::time_point::min)())
return;
// Verify that the timer really expired since the deadline may have moved.
if(m_timer.expiry() <= std::chrono::steady_clock::now())
return DoTimeout();
// Wait on the timer
m_timer.async_wait(
boost::asio::bind_executor(
m_strand,
std::bind(
&COcvApplicationServerHttpSessionSSL::OnTimer,
shared_from_this(),
std::placeholders::_1)));
}
void COcvApplicationServerHttpSessionSSL::ReadRequest()
{
// Read a request
http::async_read(
Stream(),
buffer_,
*parser_,
boost::asio::bind_executor(
m_strand,
std::bind(
&CApplicationServerBaseHttpSession::OnRead,
shared_from_this(),
std::placeholders::_1)));
}
void COcvApplicationServerHttpSessionSSL::WriteRequestStringBody(boost::beast::http::response<boost::beast::http::string_body> & msg)
{
boost::beast::http::async_write(
Stream(),
msg,
boost::asio::bind_executor(
m_strand,
std::bind(
&CApplicationServerBaseHttpSession::OnWrite,
shared_from_this(),
std::placeholders::_1,
msg.need_eof()
)
)
);
}
void COcvApplicationServerHttpSessionSSL::WriteRequestFileBody(boost::beast::http::response<boost::beast::http::file_body> & msg)
{
boost::beast::http::async_write(
Stream(),
msg,
boost::asio::bind_executor(
m_strand,
std::bind(
&CApplicationServerBaseHttpSession::OnWrite,
shared_from_this(),
std::placeholders::_1,
msg.need_eof()
)
)
);
}
void COcvApplicationServerHttpSessionSSL::DoEof()
{
m_eof = true;
// Set the timer
m_timer.expires_after(std::chrono::seconds(OCV_HTTP_SESSION_TIMER_EXPIRE_DO_EOF));
// Perform the SSL shutdown
m_stream.async_shutdown(
boost::asio::bind_executor(
m_strand,
std::bind(
&COcvApplicationServerHttpSessionSSL::OnShutdown,
shared_from_this(),
std::placeholders::_1)));
}
The Visual Leak Detector gives me the following:
c:\openssl-1.1.1b\crypto\mem.c (233): abc.exe!CRYPTO_zalloc
c:\openssl-1.1.1b\crypto\err\err.c (716): abc.exe!ERR_get_state + 0x17 bytes
c:\openssl-1.1.1b\crypto\err\err.c (443): abc.exe!ERR_clear_error + 0x5 bytes
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\impl\engine.ipp (235): abc.exe!boost::asio::ssl::detail::engine::perform
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\impl\engine.ipp (137): abc.exe!boost::asio::ssl::detail::engine::handshake
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\buffered_handshake_op.hpp (70): abc.exe!boost::asio::ssl::detail::buffered_handshake_op<boost::asio::const_buffer>::process<boost::asio::const_buffer const * __ptr64> + 0x1F bytes
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\buffered_handshake_op.hpp (48): abc.exe!boost::asio::ssl::detail::buffered_handshake_op<boost::asio::const_buffer>::operator()
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\io.hpp (136): abc.exe!boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>,boost::asio::ssl::detail::buffered_handshake_op<boost::asio::const_buffer>,boost::asio::executor_binder<std::_Binder<std::_Unforced,void (__cdecl CabcApplicationServerH + 0x50 bytes
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\io.hpp (333): abc.exe!boost::asio::ssl::detail::async_io<boost::asio::basic_stream_socket<boost::asio::ip::tcp>,boost::asio::ssl::detail::buffered_handshake_op<boost::asio::const_buffer>,boost::asio::executor_binder<std::_Binder<std::_Unforced,void (__cdecl CabcApplicationServ + 0x87 bytes
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\stream.hpp (505): abc.exe!boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp> >::async_handshake<boost::asio::const_buffer,boost::asio::executor_binder<std::_Binder<std::_Unforced,void (__cdecl CabcApplicationServerHttpSessionSSL::*)(boost::system::erro + 0x5E bytes
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\beast\experimental\core\ssl_stream.hpp (485): abc.exe!boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp> >::async_handshake<boost::asio::const_buffer,boost::asio::executor_binder<std::_Binder<std::_Unforced,void (__cdecl CabcApplicationServerHttpSessionSSL::*)(boost::system::erro
c:\usr\work\abc_repo\util\capplicationserverhttpsession.cpp (343): abc.exe!CabcApplicationServerHttpSessionSSL::Run + 0x154 bytes
In some of the Leaks I also have:
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\impl\engine.ipp (290): abc.exe!boost::asio::ssl::detail::engine::do_accept
Of course seems related to the ssl handshake but I check the session shutdown and seems ok.
Thank you in advance.

Answer 1

Every thread that uses async_handshake() leaks memory. I added OPENSSL_thread_stop() at the end of my thread procedure and it solved the issue.
Took it from here: https://github.com/openssl/openssl/issues/3033#issuecomment-289838302