HTTP stress test tool using boost asio async_connect problem - c++

Question

Using the following boost::asio code I run a loop of 1M sequential http calls to a Docker node.js simple http service that generates random numbers, but after a few thousand calls I start getting async_connect errors. The node.js part is not producing any errors and I believe it works OK.
To avoid resolving the host in every call and trying to speed-up, I am caching the endpoint, which makes no difference, I have tested both ways.
Can anyone see what is wrong with my code below?
Are there any best practices for a stress-test tool using asio that I am missing?
//------------------------------------------------------------------------------
// https://www.boost.org/doc/libs/1_70_0/libs/beast/doc/html/beast/using_io/timeouts.html
HttpResponse HttpClientAsyncBase::_http(HttpRequest&& req)
{
using namespace boost::beast;
namespace net = boost::asio;
using tcp = net::ip::tcp;
HttpResponse res;
req.prepare_payload();
boost::beast::error_code ec = {};
const HOST_INFO host = resolve(req.host(), req.port, req.resolve);
net::io_context m_io;
boost::asio::spawn(m_io, [&](boost::asio::yield_context yield)
{
size_t retries = 0;
tcp_stream stream(m_io);
if (req.timeout_seconds == 0) get_lowest_layer(stream).expires_never();
else get_lowest_layer(stream).expires_after(std::chrono::seconds(req.timeout_seconds));
get_lowest_layer(stream).async_connect(host, yield[ec]);
if (ec) return;
http::async_write(stream, req, yield[ec]);
if (ec)
{
stream.close();
return;
}
flat_buffer buffer;
http::async_read(stream, buffer, res, yield[ec]);
stream.close();
});
m_io.run();
if (ec)
throw boost::system::system_error(ec);
return std::move(res);
}
I have tried both sync/async implementations of a boost http client and I get the exact same problem.
The error I get is "You were not connected because a duplicate name exists on the network. If joining a domain, go to System in Control Panel to change the computer name and try again. If joining a workgroup, choose another workgroup name [system:52]"

Answer 1

So, I decided to... just try. I made your code into self-contained example:
#include <boost/asio/spawn.hpp>
#include <boost/beast.hpp>
#include <fmt/ranges.h>
#include <iostream>
namespace http = boost::beast::http;
//------------------------------------------------------------------------------
// https://www.boost.org/doc/libs/1_70_0/libs/beast/doc/html/beast/using_io/timeouts.html
struct HttpRequest : http::request<http::string_body> { // SEHE: don't do this
using base_type = http::request<http::string_body>;
using base_type::base_type;
std::string host() const { return "127.0.0.1"; }
uint16_t port = 80;
bool resolve = true;
int timeout_seconds = 0;
};
using HttpResponse = http::response<http::vector_body<uint8_t> >; // Do this or aggregation instead
struct HttpClientAsyncBase {
HttpResponse _http(HttpRequest&& req);
using HOST_INFO = boost::asio::ip::tcp::endpoint;
static HOST_INFO resolve(std::string const& host, uint16_t port, bool resolve) {
namespace net = boost::asio;
using net::ip::tcp;
net::io_context ioc;
tcp::resolver r(ioc);
using flags = tcp::resolver::query::flags;
auto f = resolve ? flags::address_configured
: static_cast<flags>(flags::numeric_host | flags::numeric_host);
tcp::resolver::query q(tcp::v4(), host, std::to_string(port), f);
auto it = r.resolve(q);
assert(it.size());
return HOST_INFO{it->endpoint()};
}
};
HttpResponse HttpClientAsyncBase::_http(HttpRequest&& req) {
using namespace boost::beast;
namespace net = boost::asio;
using net::ip::tcp;
HttpResponse res;
req.prepare_payload();
boost::beast::error_code ec = {};
const HOST_INFO host = resolve(req.host(), req.port, req.resolve);
net::io_context m_io;
spawn(m_io, [&](net::yield_context yield) {
// size_t retries = 0;
tcp_stream stream(m_io);
if (req.timeout_seconds == 0)
get_lowest_layer(stream).expires_never();
else
get_lowest_layer(stream).expires_after(std::chrono::seconds(req.timeout_seconds));
get_lowest_layer(stream).async_connect(host, yield[ec]);
if (ec)
return;
http::async_write(stream, req, yield[ec]);
if (ec) {
stream.close();
return;
}
flat_buffer buffer;
http::async_read(stream, buffer, res, yield[ec]);
stream.close();
});
m_io.run();
if (ec)
throw boost::system::system_error(ec);
return res;
}
int main() {
for (int i = 0; i<100'000; ++i) {
HttpClientAsyncBase hcab;
HttpRequest r(http::verb::get, "/bytes/10", 11);
r.timeout_seconds = 0;
r.port = 80;
r.resolve = false;
auto res = hcab._http(std::move(r));
std::cout << res.base() << "\n";
fmt::print("Data: {::02x}\n", res.body());
}
}
(Side note, this is using docker run -p 80:80 kennethreitz/httpbin to run the server side)
While this is about 10x faster than running curl to do the equivalent requests in a bash loop, none of this is particularly stressing. There's nothing async about it, and it seems resource usage is mild and stable, e.g. memory profiled:
(for completeness I verified identical results with timeout_seconds = 1)
Since what you're doing is literally the opposite of async IO, I'd write it much simpler:
struct HttpClientAsyncBase {
net::io_context m_io;
HttpResponse _http(HttpRequest&& req);
static auto resolve(std::string const& host, uint16_t port, bool resolve);
};
HttpResponse HttpClientAsyncBase::_http(HttpRequest&& req) {
HttpResponse res;
req.requestObject.prepare_payload();
const auto host = resolve(req.host(), req.port, req.resolve);
beast::tcp_stream stream(m_io);
if (req.timeout_seconds == 0)
stream.expires_never();
else
stream.expires_after(std::chrono::seconds(req.timeout_seconds));
stream.connect(host);
write(stream, req.requestObject);
beast::flat_buffer buffer;
read(stream, buffer, res);
stream.close();
return res;
}
That's just simpler, runs faster and does the same, down to the exceptions.
But, you're probably trying to cause stress, perhaps you instead need to reuse some connections and multi-thread?
You can see a very complete example of just that here:
How do I make this HTTPS connection persistent in Beast?
It includes reconnecting dropped connections, connections to different hosts, varied requests etc.

Answer 2

Alan's comments gave me the right pointers and I soon found using netstat -a that it was a ports leakage problem with thousands of ports in TIME_WAIT state after running the code for some brief time.
The root cause was both on the client and the server:
In node.js server I had to make sure that responses close the connection by
adding
response.setHeader("connection", "close");
In boost::asio C++ code I replaced stream.close() with
stream.socket().shutdown(boost::asio::ip::tcp::socket::shutdown_both, ec);
That seems to make all the difference. Also I made sure to use
req.set(boost::beast::http::field::connection, "close");
in my requests.
I verfied with the tool running for over 5 hours with no problems at all, so I guess the problem is solved!

Answer 3

Implementing 'Abortive TCP/IP Close' with boost::asio to treat EADDRNOTAVAIL and TIME_WAIT for HTTP client stress test tool
I am revisting the issue to offer an alternative that actually worked much better. Reminding that the objective was to develop a stress test tool for hitting a server with 1M requests. Even though my previous solution worked on Windows, when I loaded the executable on Docker/Alpine it started crashing with SEGFAULT errors that I was unable to trace. The root cause seems to be related to boost::asio::spawn(m_io, [&](boost::asio::yield_context yield) but time pressured me to solve the HTTP problem.
I decided to use synch HTTP and treat EADDRNOTAVAIL and TIME_WAIT errors by following suggestions from Disable TIME_WAIT with boost sockets and TIME_WAIT with boost asio and template code from https://www.boost.org/doc/libs/1_80_0/libs/beast/example/http/client/sync/http_client_sync.cpp.
For anyone having EADDRNOTAVAIL and TIME_WAIT with boost::asio, the solution that worked for me and it is actually much faster than before on both Windows, Linux and Dockers is the following:
HttpResponse HttpClientSyncBase::_http(HttpRequest&& req)
{
namespace beast = boost::beast;
namespace http = beast::http;
namespace net = boost::asio;
using tcp = net::ip::tcp;
HttpResponse res;
req.prepare_payload();
const auto host = req.host();
const auto port = req.port;
const auto target = req.target();
const bool abortive_close = boost::iequals(req.header("Connection"), "close");
const bool download_large_file = boost::iequals(req.header("X-LARGE-FILE-HINT"), "YES");
beast::error_code ec;
net::io_context ioc;
// Resolve host:port for IPv4
tcp::resolver resolver(ioc);
const auto endpoints = resolver.resolve(boost::asio::ip::tcp::v4(), host, port);
// Create stream and set timeouts
beast::tcp_stream stream(ioc);
if (req.timeout_seconds == 0) boost::beast::get_lowest_layer(stream).expires_never();
else boost::beast::get_lowest_layer(stream).expires_after(std::chrono::seconds(req.timeout_seconds));
// Caution: we can get address_not_available[EADDRNOTAVAIL] due to TIME_WAIT port exhaustion
stream.connect(endpoints, ec);
if (ec == boost::system::errc::address_not_available)
throw beast::system_error{ ec };
// Write HTTP request
http::write(stream, req);
// Read HTTP response (or download large file >8MB)
beast::flat_buffer buffer;
if (download_large_file)
{
_HttpResponse tmp;
boost::beast::http::response_parser<boost::beast::http::string_body> parser{ std::move(tmp) };
parser.body_limit(boost::none);
boost::beast::http::read(stream, buffer, parser);
res = HttpResponse(std::move(parser.release()));
}
else
{
http::read(stream, buffer, res);
}
// Try to shut down socket gracefully
stream.socket().shutdown(tcp::socket::shutdown_both, ec);
if (abortive_close)
{
// Read until no more data are in socket buffers
// https://stackoverflow.com/questions/58983527/disable-time-wait-with-boost-sockets
try
{
http::response<http::dynamic_body> res;
beast::flat_buffer buffer;
http::read(stream, buffer, res);
}
catch (...)
{
// should get end of stream here, ignore it
}
// Perform "Abortive TCP/IP Close" to minimize TIME_WAIT port exhaustion
// https://stackoverflow.com/questions/35006324/time-wait-with-boost-asio
try
{
// enable linger with timeout 0 to force abortive close
boost::asio::socket_base::linger option(true, 0);
stream.socket().set_option(option);
stream.close();
}
catch (...)
{
}
}
else
{
try { stream.close(); } catch (...) {}
}
// Ignore not_connected and end_of_stream errors, handle the rest
if (ec && ec != beast::errc::not_connected && ec != beast::http::error::end_of_stream)
throw beast::system_error{ ec };
return std::move(res);
}
In the sample above I should add error handling in write but I guess anyone can do it. _HttpResponse is the following and is the base for HttpResponse.
using _HttpRequest = boost::beast::http::message<true, boost::beast::http::string_body, boost::beast::http::fields>;
using _HttpResponse = boost::beast::http::message<false, boost::beast::http::string_body, boost::beast::http::fields>;
using HttpHeaders = boost::beast::http::header<1, boost::beast::http::basic_fields<std::allocator<char>>>;
For what is worth, when I started the estimation for the job was 5-7 days. Using connetion=close in my previous solution it got down to 7-8 hours. Using Abortive TCP/IP Close I got down to 1.5 hours.
Funny thing is, the server, also boost::asio, could handle the stress while the original stress tool didn't. Finally both the server and its stress test tool work just fine! The code also demonstrates how to download a large file (over 8MB) which was another side-problem, as I needed to download the test results from the server.