How can I reuse http::response in beast - c++

I would like to periodically http GET some value from a website.
with beast, the code is like:
// Set up an HTTP GET request message
http::request<http::string_body> req{http::verb::get, target, version};
req.set(http::field::host, host);
req.set(http::field::user_agent, BOOST_BEAST_VERSION_STRING);
http::response<http::string_body> res;
while (true) {
// Send the HTTP request to the remote host
http::write(stream, req);
res = {};
// Receive the HTTP response
http::read(stream, buffer, res);
// Write the message to standard out
std::cout << res << std::endl;
}
But in the loop,
res = {}
brought one temporary object creation, one move/copy assignment and the destroy of the temporary object.
I am wondering if there is a better way to avoid these unnecessary costs。

Just remove the offending line and move the declaration of res inside the while loop. res will then be created and destroyed each time round the loop:
// Set up an HTTP GET request message
http::request<http::string_body> req{http::verb::get, target, version};
req.set(http::field::host, host);
req.set(http::field::user_agent, BOOST_BEAST_VERSION_STRING);
while (true) {
// Send the HTTP request to the remote host
http::write(stream, req);
// Receive the HTTP response
http::response<http::string_body> res;
http::read(stream, buffer, res);
// Write the message to standard out
std::cout << res << std::endl;
}

You can replace res = {}; with res.clear(); or drop it altogether.
The read free function instantiates a parser that takes ownership of all response resources (by move)¹. In its constructor, it unconditionally clears the message object anyways:
You can use a debugger to trace through these lines with a simple tester like this, which is exactly what I did.
¹ in the end the message is moved back into the response object passed, so it works out

Here's what I would do
http::read(stream, buffer, res);
// move the response object to the processing function
process(std::move(res)); // res will be as clean as newly constructed after being moved

Related

Should I clean up beast::flat_buffer when I see errors on on_read?

http_client_async_ssl
class session : public std::enable_shared_from_this<session>
{
...
beast::flat_buffer buffer_; // (Must persist between reads)
http::response<http::string_body> res_;
...
}
void on_write(beast::error_code ec, std::size_t bytes_transferred) {
if (ec)
{
fail(ec, "write");
return try_again();
}
// Receive the HTTP response
http::async_read(
stream_, buffer_, res_,
beast::bind_front_handler(&session::on_read, shared_from_this()));
}
void on_read(beast::error_code ec, std::size_t bytes_transferred) {
if (ec)
{
fail(ec, "read");
return try_again();
}
// Step 1: process response
//
const auto &body_data = res_.body().data();
user_parse_data(net::buffers_begin(body_data), net::buffers_end(body_data));
// Step 2: clean up buffer_
//
buffer_.consume(buffer_.size()); // clean up buffer_ after finishing reading it
// Step 3: continue to write
...
}
In the above implementation, I ONLY clean up the buffer_ when I finish parsing the data successfully.
Question> Should I clean up the buffer_ when I experience an error on the on_read too?
void on_read(beast::error_code ec, std::size_t bytes_transferred) {
if (ec)
{
// clean up buffer_
buffer_.consume(buffer_.size()); // Should we do the cleanup here too?
fail(ec, "read");
return try_again();
}
// Step 1: process response
//
const auto &body_data = res_.body().data();
user_parse_data(net::buffers_begin(body_data), net::buffers_end(body_data));
// Step 2: clean up buffer_
//
buffer_.consume(buffer_.size());
// Step 3: continue to write
...
}
// Should we do the cleanup here too?
That's asking the wrong question entirely.
One obvious question that comes first is "should we cleanup the read buffer at all".
And the more important question is: what do you do with the connection?
The buffer belongs to the connection, as it represents stream data.
The example you link always closes the connection. So the buffer is irrelevant after receiving the response - since the connection becomes irrelevant. Note that the linked example doesn't consume on the buffer either.
Should You Cleanup At All?
You should not cleanup after http::read!
The reason is that http::read already consumes any data that was parsed as part of the response message.
Even if you expect to read more messages from the same connection (e.g. HTTP pipelining), you need to start the next http::read with the same buffer since it might already contain (partial) data for the subsequent message.
What About Errors?
If you have an IO/parse error during HTTP transmissions, I expect in most circumstances the HTTP protocol specification will require you to shut down the connection.
There is no "try_again" in HTTP/1. Once you've lost the thread on stream contents, there is no way you can recover to a "known state".
Regardless, I'd always recommend shutting down failed HTTP sessions, because not doing so opens up to corrupted messages and security vulnerabilities.

Using co-routines in boost::beast websocket implementation - support parallel message handling

I'm using secure websocket boost::beast implementation and I'd like to be able to receive new message while current one is being handled. So I've chosen to try it using co-routines (method with yield)
this is my websocket object :
using Websocket = boost::beast::websocket::stream<
boost::beast::ssl_stream<boost::beast::tcp_stream>>;
std::optional<Websocket> ws_;
And this is how I call my websocket listener code
ws_.reset();
boost::asio::spawn(ioc_,
[=](const boost::asio::yield_context &yield) {
this->startAndServeWs(yield);
});
}
And this is how my websocket handler method works : notice this it's divided into 2 parts.
First, the initialization part.
Second, the websocket mainloop in which it ready to read new message.
So my Question is whether this code below is suitable to fetch new messages from server while handling the current message (in either sendPostFetchItems or sendPostDownloadNewVersion which can take a while since they trigger http post request and wait for server response). And if it doesn't, so can I assume that the new message will be queued, waiting for the next iteration when current message handle will be completed ?
My second question is about the catch statement, and if this is the proper way to retry the connection
void Comm::startAndServeWs(const boost::asio::yield_context &yield) {
try {
// webSocet init according to ssl context and io_context.
ws_.emplace(ioc_, ctx_);
boost::beast::get_lowest_layer(ws_.value())
.expires_after(std::chrono::seconds(30));
boost::asio::ip::tcp::resolver resolver(io_context_);
auto const results =
resolver.async_resolve(ip_.value(), port_.value(), yield);
auto ep = boost::beast::get_lowest_layer(ws_.value())
.async_connect(results, yield);
// Set SNI Hostname (many hosts need this to handshake successfully)
if (!SSL_set_tlsext_host_name(ws_.value().next_layer().native_handle(),
ip_.value().c_str())) {
throw("Failed to set SNI Hostname");
}
// Update the host_ string. This will provide the value of the
// Host HTTP header during the WebSocket handshake.
// Se e https://tools.ietf.org/html/rfc7230#section-5.4
auto address = ip_.value() + std::to_string(ep.port());
// Perform the SSL handshake
ws_.value().next_layer().handshake(boost::asio::ssl::stream_base::client);
// Turn off the timeout on the tcp_stream, because
// the websocket stream has its own timeout system.
boost::beast::get_lowest_layer(ws_.value()).expires_never();
// Set suggested timeout settings for the websocket
ws_.value().set_option(
boost::beast::websocket::stream_base::timeout::suggested(
boost::beast::role_type::client));
// Set a decorator to change the User-Agent of the handshake
ws_.value().set_option(boost::beast::websocket::stream_base::decorator(
[](boost::beast::websocket::request_type &req) {
req.set(boost::beast::http::field::user_agent, kWebsocketIdentifier);
}));
Log("trying to establish websocket in address {}", address);
ws_.value().async_handshake(address, "/ws", yield);
//////////////////// here's the websocket main loop.
for (;;) {
boost::beast::flat_buffer buffer;
// Read a message into our buffer
---> ws_.value().async_read(buffer, yield);
Log("websocket response buffer = {}",boost::beast::make_printable(buffer.data()));
try {
nlohmann::json response = nlohmann::json::parse(s);
if (response["command"] == "fetchItems") {
sendPostFetchItems();
} else if (response["command"] == "getLogs") {
sendPostDownloadNewVersion();
}...
}
} catch (std::exception &e) {
Log("websocket reconnect failed. reason = {}", e.what());
ws_.reset();
timer_websocket_.expires_at(timer_websocket_.expiry() +
boost::asio::chrono::seconds(10));
timer_websocket_.async_wait(
[this]([[maybe_unused]] const boost::system::error_code &error) {
boost::asio::spawn(io_context_,
[this](const boost::asio::yield_context &yield) {
this->startAndServeWs(yield);
});
});
}
}

301 Moved Permanently on GET request (some sites) C++

I just want to make a GET request to my telegram bot using C++ code but it's getting 301 Moved Permanently (but if I use web-browser it works fine).
Request to other sites work just fine, almost with no errors (google.com, ip2c.org ...).
Below I provide the code I'm using:
const std::string host = "api.telegram.org";
const std::string target = "/bot1781233486:AAENIwkZt0Lbv9zjks6l-4loBcWDzKyMhyU/sendMessage?chat_id=#ZiyodaEnglishPracticeTracker&text=Test+message+from+bot";
boost::asio::io_context ioc;
boost::asio::ip::tcp::resolver resolver(ioc);
boost::asio::ip::tcp::socket socket(ioc);
boost::asio::connect(socket, resolver.resolve(host, "80"));
http::request<http::string_body> req(http::verb::get, target, 11);
req.set(http::field::host, host);
req.set(http::field::user_agent, BOOST_BEAST_VERSION_STRING);
http::write(socket, req);
{
boost::beast::flat_buffer buffer;
http::response<http::dynamic_body> res;
http::read(socket, buffer, res);
std::cout << res << std::endl;
}
socket.shutdown(boost::asio::ip::tcp::socket::shutdown_both);
The bot's token is also shown in the code, it's just a temporary bot, so feel free to test it, but change chat id to see the result.
If you got an error try this one:
const std::string host = "ip2c.org";
const std::string target = "/192.199.20.22";
This one returns a valid result
I had to redirect it from HTTP to HTTPS, and the problem was solved by changing the port from 80 to 443

How to assign JSON body to Boost Library 1.70 http::request as POST request?

I am new to the Boost library. I try to create an Rest HTTP request using the Boost::http library.
My question is how can i simply assign the JSON payload to the http request.
the following code snippet shows my current try which connects successfully but the payload is not assigned.
http::request<http::string_body> req{ http::verb::post, LOGIN_PATH, 10 };
req.set(beast::http::field::content_type, "application/json");
req.body() = std::move(serviceUser);
// Send the HTTP request to the remote host
http::write(stream, req);
// This buffer is used for reading and must be persisted
beast::flat_buffer buffer;
// Declare a container to hold the response
http::response<http::dynamic_body> res;
// Receive the HTTP response
http::read(stream, buffer, res);
// Write the message to standard out
std::cout << res << std::endl;
return "OK";
The following snippet shows the JSON message.
std::stringstream strStream;
strStream << "{\"userName\" : lena, \"password\" : liebe }";
serviceUser = strStream.str();
Can you give me a simple example please. Importent to mention is that i use the Boost library version 1.70.

C++ proxy developed on top of the beast boost, cannot receive big responses from the host and forward them to the original (downstream) client

I have implemented a proxy with the boost beast library, where I have used the
async https server and client examples from boost organization. In my proxy I
was using the http::request<http::string_body> and
http::response<http::string_body> types of messages, as used in the examples.
This proxy works well, except that it cannot receive (download) big files and
streams.
So for that purpose, I decided to rework my transmitting mechanism by combining
two examples from the
https://www.boost.org/doc/libs/1_66_0/libs/beast/example/doc/http_examples.hpp.
The mentioned examples are the "Example: Incremental Read" and the "Example:
Send Child Process Output".
This example (that follows) partly works, but it has some issues.
Very often, when I have a series of requests to execute on one connection, I
failed to read the second or the third response (the header) to a successfully
written request, and therefore the connection gets broken, and the client
(browser) reconnects and tries to execute them in a different session. This
makes traffic very slow and annoying.
Although the proxy app is written as async one, this method is written in a
synced (blocking) manner and serves only to receive the response from the host
(upstream) in chunks and write the received data chunks directly as they are
received to the original client (downstream).
The question is what is it that I am doing wrong?
I believe an experienced beast boost user could pick up the problem from reading the example.
std::shared_ptr<beast::ssl_stream<beast::tcp_stream>> m_sslDownStream;
std::shared_ptr<beast::ssl_stream<beast::tcp_stream>> m_sslUpStream;
beast::flat_buffer m_upBuffer;
void Session::do_read_upstream_buffered_response()
{
beast::error_code ec;
size_t bytes_transferred = 0
beast::flat_buffer m_upBuffer;
http::response_parser<http::buffer_body> resPar;
resPar.body_limit(ULONG_MAX);
bytes_transferred = http::read_header(*m_sslUpStream.get(), m_upBuffer, resPar, ec);
if (ec)
{
return fail(ec, "read header");
}
http::response<http::buffer_body> bufRes;
bufRes.result(resPar.get().result_int());
bufRes.version(resPar.get().version());
int field_count = 0;
for (auto const& field : resPar.get())
{
bufRes.insert(field.name_string().to_string(), field.value().to_string());
}
// No data yet, but we set more = true to indicate
// that it might be coming later. Otherwise the
// serializer::is_done would return true right after
// sending the header.
bufRes.body().data = nullptr;
bufRes.body().more = true;
http::response_serializer<http::buffer_body, http::fields> resSer { bufRes };
bytes_transferred = http::write_header(*(m_sslDownStream.get()), resSer, ec);
if (ec)
{
LSPROXY_LOGD(LOG_MITM_PROXY, "Session[%d]::do_read_upstream_buffered_response(%s) Failed to write header to the original client (downstream) with error: %s", this, m_sessionStateStr, ec.message().c_str());
return fail(ec, "write header");
}
// Read the rest of the response body upstream and send it downstream
while (!resPar.is_done())
{
char buf[8192];
resPar.get().body().data = buf;
resPar.get().body().size = sizeof(buf);
bytes_transferred = http::read(*m_sslUpStream.get(), m_upBuffer, resPar, ec);
if (ec == http::error::need_buffer)
{
ec.message().c_str());
ec.assign(0, ec.category());
}
if (ec)
{
return fail(ec, "read body");
}
// Point to our buffer with the bytes that
// we received, and indicate that there may
// be some more data coming
bufRes.body().data = buf;
bufRes.body().size = sizeof(buf) - resPar.get().body().size;
bufRes.body().more = true;
bytes_transferred = http::write(*(m_sslDownStream.get()), resSer, ec);
// This error is returned by body_buffer during
// serialization when it is done sending the data
// provided and needs another buffer.
if (ec == http::error::need_buffer)
{
ec.message().c_str());
ec = {};
//continue;
}
if (ec)
{
return fail(ec, "write body");
}
} //while (!resPar.is_done())
// `nullptr` indicates there is no buffer
bufRes.body().data = nullptr;
// `false` means no more data is coming
bufRes.body().more = false;
// Send the response header to the original client (downstream).
bytes_transferred = http::write(*(m_sslDownStream.get()), resSer, ec);
// Read another request from the original client (downstream)
do_read_downstream();
}
Just for the sake of others having the same or similar problem, I'd like to post the resolution to my problem.
The answer was in the question all along.
To be more precise in the https://www.boost.org/doc/libs/1_66_0/libs/beast/example/doc/http_examples.hpp there is an Example: HTTP Relay wich is exaclty what I needed in the first place.
Theere are small differences in this example from what I have combined my self from the two other examples (mentioned in the original post).
The most important one, the Example: HTTP Relay doesn't use a buffered body response:
http::response<http::buffer_body> bufRes;
to construct the serializer with:
http::response_serializer<http::buffer_body, http::fields> resSer { bufRes };
it uses the receiving parser directly to construct the serializer with:
// Create a parser with a buffer body to read from the input.
parser<isRequest, buffer_body> p;
// Create a serializer from the message contained in the parser.
serializer<isRequest, buffer_body, fields> sr{p.get()};
With a little adaptation, the Example: HTTP Relay works just fine for my proxy all types of requests, small body requsts, but also with big files downloading and data streams as well.