I am creating a client application that connects to a server using a an ssl Websocket connection and an ssl Http (Keep-Alive) connection and I am using boost::beast package to do the same. So as to detect a dead connection i have implemented a simple ping-pong mechanism. These all work fine, but an issue comes up when handling the ping-pong failure. The issue is as follows:
For testing my code i connected to the remote server, sent few messages and then turned off my wifi. As expected after a certain period it detected that it did not receive any message from the server and it tried to do an async_shutdown for the http connection and an async_close for the websocket connection. First thing i noticed was that both these calls block their respective strands until the wifi is back up.
And after the wifi is up, the application tries to reset the stream before reconnect:
void HttpKeepAliveConnection::recreateSocket()
{
_receivedPongForLastPing = true;
_sslContext.reset(new boost::asio::ssl::context({boost::asio::ssl::context::sslv23_client}));
_stream.reset(new HttpStream(_ioContext, *_sslContext));
}
And reset ws variable for websocket:
void WebsocketConnection::recreateSocket()
{
_receivedPongForLastPing = true;
_sslContext.reset(new boost::asio::ssl::context({boost::asio::ssl::context::sslv23_client}));
_ws.reset(new WebSocket(_ioContext, *_sslContext));
}
Unfortunately it fails at either on_connect or on_ssl_handshake. Following are my logs:
156 AsioConnectionBase.cpp:53 (2018-08-06 15:34:38.458536) [0x00007ffff601e700] : Started connect sequence. Connection Name: HttpKeepAliveConn
157 AsioConnectionBase.cpp:122 (2018-08-06 15:34:38.459802) [0x00007ffff481b700] : Failed establishing connection to destination. Connection failed. Connection Name: HttpKeepAliveConn. Host: xxxxxxxxx. Port: 443. Error: Operation canceled
158 APIManager.cpp:175 (2018-08-06 15:34:38.459886) [0x00007ffff481b700] : Received error callback from connection. Restarting connection in a sec. Connection Name: HttpKeepAliveConn
159 AsioConnectionBase.cpp:53 (2018-08-06 15:34:39.460009) [0x00007ffff481b700] : Started connect sequence. Connection Name: HttpKeepAliveConn
160 HttpKeepAliveConnection.cpp:32 (2018-08-06 15:34:39.460515) [0x00007ffff481b700] : Failed ssl handshake. Connection failed.Connection Name: HttpKeepAliveConn. Host: xxxxxxxxx. Port: 443. Error: Bad file descriptor
161 APIManager.cpp:175 (2018-08-06 15:34:39.460674) [0x00007ffff481b700] : Received error callback from connection. Restarting connection in a sec. Connection Name: HttpKeepAliveConn
So I have 2 questions:
How do we close a connection if internet is down and a proper tcp close is not possible.
Before reconnecting what are the variables in boost::beast (or for that matter boost::asio as boost::beast is built on top of asio) that needs to be reset
Have been stuck trying to debug this for couple of hours. Any help is appreciated
EDIT
So I figured out where I went wrong. Both Alan Birtles and Vinnie Falco were right. The way to close a dead ssl connection after your ping timer has expired (and none of the handlers have returned yet) is
In your timer handler
_stream->lowest_layer().close();
For websocket
_ws->lowest_layer().close();
Wait for one of your handlers (typically read handler) to return with error (typically boost::asio::error::operation_aborted error). From there, queue the start of the next reconnect. (Do not queue the reconnect immediately after step 1, it will result in memory issues that I faced. I know this is asio 101, but is easy to forget)
For resetting socket, all that is required is for the stream to be reset
_stream.reset(new HttpStream(_ioContext, _sslContext));
For websocket
_ws.reset(new WebSocket(_ioContext, _sslContext));
I don't think asio::ssl::stream can be used again after being closed.
How do we close a connection if internet is down and a proper tcp close is not possible.
Simply allow the socket or stream object to be destroyed.
Related
I'm trying to get stats for my celery Que (rabbitmq). I'm using celery.app.control.Inspect().stats() API. I'm doing this on a web server, I can get the stats only one time. If I refresh the page I'm getting "[Errno 104] Connection reset by peer" Error. how can I deal with this.
/init.py
celtasks = Celery(app.name,"rabbit mq url")
/helpers.py
get_stats():
stats = celtasks.control.Inspect().stats()
return stats
whenever there is a request "get_stats" function is hit. It is only working for the first request after this, it says connection reset by peer error.
If I go by connection has been reset and try to create the connection again, I get error
updated /helpers.py
get_stats():
celtasks = Celery(app.name,"rabbit mq url")
stats = celtasks.control.Inspect().stats()
return stats
Rabbitmq logs
=WARNING REPORT==== 10-Jul-2017::14:11:54 ===
closing AMQP connection <0.29185.6> (10.246.170.70:48618 -> 10.24.83.115:5672):
connection_closed_abruptly
=WARNING REPORT==== 10-Jul-2017::14:11:54 ===
closing AMQP connection <0.29197.6> (10.246.170.70:48620 -> 10.24.83.115:5672):
connection_closed_abruptly
"rabbit#oser000300.log-20170625" 9054L, 361662C
AT most times , CONNECTION RESET BY PEER is because the server close the connection itself, however the client does not know . When client want to communicate to sever through this broke connection, it receive this ERROR. In your case , maybe the hang time (time interval between two stats()) is too long, and server think this connection is useless and close it .
Providing a MCVE is going to be hard, the scenario is the following:
a server written in c++ with boost asio offers some services
a client written in c++ with boost asio requests services
There are custom headers and most communication is done using multipart/form.
However, in the case where the server returns a 401 for an unauthorized access,
the client receives a broken pipe (system error 32).
AFAIK this happens when the server connection closes too early.
So, running into gdb, I can see that the problem is indeed the transition from the async_write which sends the request, to the async_read_until which reads the first line of the HTTP Header:
The connect routine sends the request from the client to the server:
boost::asio::async_write(*socket_.get(),
request_,
boost::bind(&asio_handler<http_socket>::write_request,
this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
And the write_request callback, checks if the request was sent OK, and then reads the first line (until the first newline):
template <class T>
void asio_handler<T>::write_request(const boost::system::error_code & err,
const std::size_t bytes)
{
if (!err) {
// read until first newline
boost::asio::async_read_until(*socket_,
buffer_,
"\r\n",
boost::bind(&asio_handler::read_status_line,
this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
else {
end(err);
}
}
The problem is that the end(err) is always called with a broken pipe (error code 32). Meaning, as far as I understand, that the server closed the connection. The server indeed closes the connection, but only after it has sent a message HTTP/1.1 401 Unauthorized.
using curl with the appropriate request, we do get the actual message/error before the server closes the connection
using our client written in C++/boost asio we only get the broken pipe and no data
only when the server leaves the connection open, do we get to the point of reading the error (401) but that defeats the purpose, since now the connection is left open.
I would really appreciate any hints or tips. I understand that without the code its hard to help, so I can add more source at any time.
EDIT:
If I do not check for errors between writing the request, and reading the server reply, then I do get the actual HTTP 401 error. However this seems counter-intuitive, and I am not sure why this happens or if it is supposed to happen.
The observed behavior is allowed per the HTTP specification.
A client or server may close the socket at anytime. The server can provide a response and close the connection before the client has finished transmitting the request. When writing the body, it is recommended that clients monitor the socket for an error or close notification. From the RFC 7230, HTTP/1.1: Message Syntax and Routing Section 6.5. Failures and Timeouts:
6.5. Failures and Timeouts
A client, server, or proxy MAY close the transport connection at any time. [...]
A client sending a message body SHOULD monitor the network connection for an error response while it is transmitting the request. If the client sees a response that indicates the server does not wish to receive the message body and is closing the connection, the client SHOULD immediately cease transmitting the body and close its side of the connection.
On a graceful connection closure, the server will send a response to the client before closing the underlying socket:
6.6. Tear-down
A server that sends a "close" connection option MUST initiate a close of the connection [...] after it sends the response containing "close". [...]
Given the above behaviors, there are three possible scenarios. The async_write() operation completes with:
success, indicating the request was written in full. The client may or may not have received the HTTP Response yet
an error, indicating the request was not written in full. If there is data available to be read on the socket, then it may contain the HTTP Response sent by the server before the connection terminated. The HTTP connection may have terminated gracefully
an error, indicating the request was not written in full. If there is no data available to be read on the socket, then the HTTP connection was not terminated gracefully
Consider either:
initiating the async_read() operation if the async_write() is successful or there is data available to be read
void write_request(
const boost::system::error_code & error,
const std::size_t bytes_transferred)
{
// The server may close the connection before the HTTP Request finished
// writing. In that case, the HTTP Response will be available on the
// socket. Only stop the call chain if an error occurred and no data is
// available.
if (error && !socket_->available())
{
return;
}
boost::asio::async_read_until(*socket_, buffer_, "\r\n", ...);
}
per the RFC recommendation, initiate the async_read() operation at the same time as the async_write(). If the server indicates the HTTP connection is closing, then the client would shutdown its send side of the socket. The additional state handling may not warrant the extra complexity
I've got this:
[info] asio async_read_at_least error: system:10058 (A request to send or receive data was disallowed because the socket had already been shut down in that direction with a previous shutdown call)
[error] handle_read_frame error: websocketpp.transport:2 (Underlying Transport Error)
after closing connection by server (not by browser client) in WebSocket++ lib. The server still works, but how can I fix this error?
SERVER CODE: http://pastebin.com/acbrjLvF
Fixed it by adding m_server.pause_reading(handler) before calling m_server.close on handler.
A boost-asio SSL/TLS TCP socket is implemented as an ssl::stream over a tcp::socket:
boost::asio::ssl::stream<boost::asio::ip::tcp::socket> ssl_socket;
In the TLS protocol, a cryptographically secure shutdown involves parties exchanging close_notify messages. Simply closing the lowest layer may make the session vulnerable to a truncation attack.
In boost asio ssl async_shutdown always finishes with an error? #Tanner Sansbury describes the SSL shutdown process in detail with a number of scenarios and proposes using an async_shutdown followed by an async_write to disconnect an SSL stream prior to closing the socket:
ssl_socket.async_shutdown(...);
const char buffer[] = "";
async_write(ssl_socket, buffer, [](...) { ssl_socket.close(); })
Performing an async_shutdown on an ssl::stream sends an SSL close_notify message and waits for a response from the other end. The purpose of writing to the stream after the async_shutdown is to be notified when async_shutdown has sent the close_notify so that the socket can be closed without waiting for the response. However, in the current (1.59) version of boost the call to async_write fails...
In How to gracefully shutdown a boost asio ssl client? #maxschlepzig proposes shutting down receiver of the underlying TCP socket:
ssl_socket.lowest_layer()::shutdown(tcp::socket::shutdown_receive);
This produces a short read error, and async_shutdown is called when it's detected in the error handler:
// const boost::system::error_code &ec
if (ec.category() == asio::error::get_ssl_category() &&
ec.value() == ERR_PACK(ERR_LIB_SSL, 0, SSL_R_SHORT_READ))
{
// -> not a real error:
do_ssl_async_shutdown();
}
Or cancelling the read/write operations on the socket and then calling SSL async shutdown, i.e.:
boost::system::error_code ec;
ssl_socket.cancel(ec);
ssl_socket.async_shutdown([](...) { ssl_socket.close(); };
I'm currently using this last method since it works with the current version of boost.
What is the correct/best way to securely disconnect a boost-asio SSL socket?
To securely disconnect, perform a shutdown operation and then close the underlying transport once shutdown has complete. Hence, the method you are currently using will perform a secure disconnect:
boost::system::error_code ec;
ssl_socket.cancel(ec);
ssl_socket.async_shutdown([](...) { ssl_socket.close(); };
Be aware that the current async_shutdown operation will be considered complete when either:
A close_notify has been received by the remote peer.
The remote peer closes the socket.
The operation has been cancelled.
Hence, if resources are bound to the lifetime of the socket or connection, then these resources will remain alive waiting for the remote peer to take action or until the operation is cancelled locally. However, waiting for a close_notify response is not required for a secure shutdown. If resources are bound to the connection, and locally the connection is considered dead upon sending a shutdown, then it may be worthwhile to not wait for the remote peer to take action:
ssl_socket.async_shutdown(...);
const char buffer[] = "";
async_write(ssl_socket, boost::asio::buffer(buffer),
[](...) { ssl_socket.close(); })
When a client sends a close_notify message, the client guarantees that the client will not send additional data across the secure connection. In essence, the async_write() is being used to detect when the client has sent a close_notify, and within the completion handler, will close the underlying transport, causing the async_shutdown() to complete with boost::asio::error::operation_aborted. As noted in the linked answer, the async_write() operation is expected to fail.
... as the write side of PartyA's SSL stream has closed, the async_write() operation will fail with an SSL error indicating the protocol has been shutdown.
if ((error.category() == boost::asio::error::get_ssl_category())
&& (SSL_R_PROTOCOL_IS_SHUTDOWN == ERR_GET_REASON(error.value())))
{
ssl_stream.lowest_layer().close();
}
The failed async_write() operation will then explicitly close the underlying transport, causing the async_shutdown() operation that is waiting for PartyB's close_notify to be cancelled.
I'm probably late to answer this but I want to report my experience.
This solution so far (using boost 1.78) did not produce any visible error on the client nor the server:
// sock type is boost::asio::ssl::stream<boost::asio::ip::tcp::socket>
sock->shutdown(ec);
sock->lowest_layer().shutdown(boost::asio::ip::tcp::socket::shutdown_both, ec);
sock->lowest_layer().cancel(ec);
sock->lowest_layer().close();
Sandbox server with: openssl s_server -cert server.crt -key server.key -4 -debug
With this solution the server gets this after the sock->shutdown(ec).
read from 0x55e5dff8c960 [0x55e5dff810f8] (19 bytes => 19 (0x13))
0000 - 44 bc 11 5b a9 b4 ee 51-48 e0 18 f7 99 a7 a8 a9 D..[...QH.......
0010 - 21 1a 60 !.`
DONE
shutting down SSL
CONNECTION CLOSED
Before I was using this code (used for both plain TCP and ssl socket)
sock->lowest_layer().shutdown(boost::asio::ip::tcp::socket::shutdown_both, ec);
sock->lowest_layer().cancel(ec);
sock->lowest_layer().close();
The old code, when leveraging ssl socket, produced this error on the server:
read from 0x55eb3d40b430 [0x55eb3d423513] (5 bytes => 0 (0x0))
ERROR
shutting down SSL
CONNECTION CLOSED
As mentioned before, to avoid this behavior a close_notify should be sent out by the client using ssl::stream::async_shutdown or ssl::stream::shutdown
The trick of async_write() could be useful in case you want to leverage the async_shutdown() function instead of the synchronous shutdown()
I am using TIdHTTPProxyServer and now I want to terminate connection when it is success to connect to the target HTTP server but receive no response for a long time(i.g. 3 mins)
Currently I find no related property or event about it. And even if the client terminate the connection before the proxy server receive the response from the HTTP server. OnException Event will not be fired until the proxy server receive the response. (That is, if the proxy server still receive no response from HTTP Server, I even do not know the client has already terminate the connection...)
Any help will be appreciated.
Thanks!
Willy
Indy uses infinite timeouts by default. To do what you are asking for, you need to set the ReadTimeout property of the outbound connection to the target server. You can access that connection via the TIdHTTPProxyServerContext.OutboundClient property. Use the OnHTTPBeforeCommand event, which is triggered just before the OutboundClient connects to the target server, eg:
#include "IdTCPClient.hpp"
void __fastcall TForm1::IdHTTPProxyServer1HTTPBeforeCommand(TIdHTTPProxyServerContext *AContext)
{
static_cast<TIdTCPClient*>(AContext->OutboundClient)->ReadTimeout = ...;
}