SSL_Read() returns SSL_ERROR_ZERO_RETURN but ERR_get_error() is 0 - c++

I am writing a non-blocking Websocket client and using OpenSSL for the TLS layer. I am able to connect to the remote server, complete the TLS handshake, send an Upgrade request, get an upgrade confirmed response, and get an actual websocket response afterwards before the TLS layer disconnects with a SSL_ERROR_ZERO_RETURN.
SSL_get_error(...) returns: 6 // SSL_ERROR_ZERO_RETURN
ERR_error_string(ERR_get_error(), nullptr) returns: error:00000000:lib(0):func(0):reason(0)
From my understanding, ERR_get_error() should pop off and return the first error on the error queue, and SSL_get_error() returns the last error of a SSL_* function. I do not understand why SSL_get_error() would return an error value but ERR_get_error() does not. According to this previous Stack Overflow Question, SSL_get_error() does NOT call ERR_get_error().
Following code gets called repeatedly (since it is a non-blocking socket) :
ERR_clear_error();
int ret = SSL_read(...);
if (ret > 0) {
// read bytes from socket
} else {
int err_code = SSL_get_error(ssl_session_, ret);
if (err_code == SSL_ERROR_ZERO_RETURN || err_code == SSL_ERROR_SYSCALL || err_code == SSL_ERROR_SSL) {
sprintf("Disconnected: %d %s", err_code, ERR_error_string(ERR_get_error(), nullptr));
// Disconnect Code
}
}
I have two questions:
Why am I not getting an error value for ERR_get_error()?
Why am I getting disconnected so quickly after establishing a TLS and Websocket session?
EDIT 1
I used wireshark to capture the packets between the client and the server. I confirmed that the TLS handshake, the websocket upgrade, and an initial server response are successful. I noticed after the initial server response, my client gets an Encrypted Alert 21 from the server which I believe is a fatal error and explains why the TLS session terminates immediately and my SSL error queue is empty (while it is probably a client side issue, I don't think it's the result of a recent action), and kind of explains the SSL_ERROR_ZERO_RETURN value I am getting after the SSL_Read.
I am not sure what the Encrypted Alert 21 entails. It might be the cert I am using (self signed). Need to investigate further.

Alright, the root cause of the issue has been determined but a lot of hoops were jumped through and time wasted to get there. I was able to decrypt the SSL traffic by grabbing the master key using the OpenSSL method, SSL_SESSION_get_master_key(), and the Client Hello's Random value with wireshark.
Relevant Code to output master key after SSL_Connect:
ERR_clear_error();
int ret = SSL_connect(ssl_ptr);
if (ret > 0) {
SSL_SESSION * ssl_session = SSL_get_session(ssl_ptr);
if(ssl_session != NULL) {
unsigned char master_key_buf[256];
size_t outlen = sizeof(master_key_buf);
size_t buf_size = SSL_SESSION_get_master_key(ssl_session, master_key_buf, outlen);
if(outlen > 0) {
char hex_encoded_master_buf[513];
// hex encode the master key
for(size_t i = 0; i < buf_size; ++i) {
sprintf(&hex_encoded_master_buf[2*i], "%02x", master_key_buf[i]);
}
hex_encoded_master_buf[(2*buf_size)] = '\0';
// log out the hex-encoded master key in master buf here
}
}
}
Using the NSS Key Log CLIENT_RANDOM format in wireshark to decrypt the captured SSL Traffic, I was able to examine the aforementioned Encrypted Alert 21 which ended up just being a WebSocket FIN and close_notify.
It turns out, the underlying reason was that during the handshake, my WSS Upgrade Request message did indeed containing the right headers, but I was actually sending a garbage payload with it. It was a case of setting the size using sizeof of the message buffer instead of strlen when sending the message. The server would have parsed the Upgrade message just fine and completed the handshake successfully, but the next time it checked its socket, it would read in garbage when it was expecting a WSS message. This caused the abrupt closure of the websocket connection.
In Summary, to answer my original two questions:
Why am I not getting an error value for ERR_get_error()?
The connection is being terminated on the server side, and the error queue would contain errors on the client side, which there are not any, at least on the SSL/TLS layer.
Why am I getting disconnected so quickly after establishing a TLS and Websocket session?
My Initial Upgrade request contained a valid Websocket Upgrade Request with Garbage Data following it. The Server parsed the Websocket Upgrade Request and confirmed the upgrade, and started sending back data. The next time the server checked its socket, it still had the garbage values that was sent with the original Websocket Upgrade Request. Since the server did not recognize it as a valid Websocket message, or anything else for that matter, it decided to terminate the connection with a close_notify.

Related

Cesanta Mongoose - problems when connecting to localhost

I'm having issues building an HTTP server using the Cesanta Mongoose web server library. The issue that I'm having occurs when I have an HTTP server built to listen on port 8080, and a client sending an HTTP request to localhost:8080. The problem is that the server processes the request fine and sends back a response, but the client only processes and prints the response after I kill the server process. Basically Mongoose works where you create connections which take an event handler function, ev_handler(). This event handler function is called whenever an
"event" occurs, such as the receiving of a request or a reply. On the server side, the event handler function is called fine when it receives a request from the client on 8080. However, the client-side event handler function is not called when the response sends the reply, but is called only after the server process is killed. I suspected that this may have something to do with the fact that the connection is on localhost, and I was right - this issue does not occur when the client sends requests to addresses other than localhost. The event handler function is called fine. Here is the ev_handler function on the client-side for reference:
static void ev_handler(struct mg_connection *c, int ev, void *p) {
if (ev == MG_EV_HTTP_REPLY) {
struct http_message *hm = (struct http_message *)p;
c->flags |= MG_F_CLOSE_IMMEDIATELY;
fwrite(hm->message.p, 1, (int)hm->message.len, stdout);
putchar('\n');
exit_flag = 1;
} else if (ev == MG_EV_CLOSE) {
exit_flag = 1;
};
}
Is this a common issue when trying to establish a connection on localhost with a server on the same computer?
The cause of such behavior is the fact that client connection does not fire an event until all data is read. How client knows the all data is read? There are 3 possibilities:
Server has sent Content-Length: XXX header and client has read XXX bytes of the message body, so it knows it received everything.
Server has sent Transfer-Encoding: chunked header, and sent all data chunks followed by an empty chunk. When client receives an empty chunk, it knows it received everything.
Server set neither Content-Lenth, nor Transfer-Encoding. Client does not know in this case what is the size of the body, and it keeps reading until server closes the connection.
What you see is (3). Solution: set Content-Length in your server code.

After Successfull TLS handshake the server closes with error SSL routines:SSL3_GET_RECORD:wrong version number

We are using openssl 1.0.2k for our TLS related functionalities.
In one of our deployment the client is able to complete the TLS handshakes using TLSv1.2 and was able to send application data towards server.After some requests the TLS connections closed from the server side with the below error
"error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number"
TLS handshake steps:
1. Client hello
2. Server Hello
3. Certificate,Certificate Request, Server hello done
4. Certificate,Client Key Exchange,Change Cipher spec,Encrypted handshake message
5. Change Cipher spec,Encrypted handshake message
6. Application data exchanges between client and server
7. Encrypted Alert(server to client)
8. Encrypted Alert( client to server
The error logs from server side says "error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number"
Can you please let us know the cause for this issue. If the ssl version is mismatching then the handshake phase should not succeed right?
But in our case handshake is successful and after some application data transfer our server is failing with this error.
If the ssl version is mismatching then the handshake phase should not succeed right?
No. Any TLS packet have header, and header has TLS version inside:
(
byte - record_type
byte[2] - version
byte[2] - length
) header
byte[length] - encrypted or raw data
Header is always in raw, it is never encrypted. Even if during handshake client sent TLS 1.2 version in all TLS packets, he can send another version after handshake is finished. Or someone in between can modify network traffic. In this case OpenSSL throws described error.
In my case, I was using OpenSSL for client functionality.
I was calling SSL_set_connect_state after SSL_connect. It should be called before.
SSL_set_connect_state (for client only) cleans up all the state!
snippet:
void SSL_set_connect_state(SSL *s)
{
s->server = 0;
s->shutdown = 0;
ossl_statem_clear(s);
s->handshake_func = s->method->ssl_connect;
clear_ciphers(s);
}
In my case:
1) Client <-> Server handshake succeeded.
2) SSL_write from client side (client sending message to server) lead to exact same error as mentioned in question (on server side)
I looked at pkt dump on server side.
read from 0x2651570 [0x2656c63] (5 bytes => 5 (0x5)) .
0000 - 16 03 01 01 e2 .....
ERROR
139688140752544:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong >version number:s3_pkt.c:337:
1) 5 Bytes read in the above snipped is the size of SSL record. Server received data, and it attempted reading SSL record.
2) 1'st byte of the record is the SSL record type In this case ===> x16 => '22'
This itself is wrong, as far as server is concerned, handshake was successful and it was expecting application data. Instead it received data with SSL record for handshake, hence it was throwing the error.
A correct snippet of application data is as follows: 'x17' ==> 23
read from 0x2664f80 [0x2656c63] (5 bytes => 5 (0x5)) .
0000 - 17 03 03 00 1c
Since SSL_set_connect_state was called after connecting, client state was lost and SSL_write will attempt handshake if handshake wasnt performed before (client thought so as its state was lost!)
More data on these SSL records can be found here:
https://www.ibm.com/support/knowledgecenter/SSB23S_1.1.0.12/gtps7/s5rcd.html

Send Recv from client to server socket by establish TCP Connection only once

I am working on a client/server solution in C++.
From the client, I am sending data to my server, and from this server I am sending to another server. I am able to configure port and IP address, and am able to send successfully.
But, the other server (which is not on my side) needs to establish only one TCP connection from my side, after that only sending and receiving needs to happen.
If I am connecting twice (say from two clients at the same time), it shows connection refused.
Part of the code is shown below:
while ((len = stream->receive(input, sizeof(input)-1)) > 0 )
{
input[len] = NULL;
//Code Addition by Srini starts here
//Client declaration
TCPConnector* connector_client = new TCPConnector();
printf("ip_client = %s\tport_client = %s\tport_client_int = %d\n", ip_client.c_str(), port_client.c_str(),atoi(port_client.c_str()));
TCPStream* stream_client = connector_client->connect(ip_client.c_str(), atoi(port_client.c_str()));
//Client declaration ends
if (stream_client)
{
//message = "Is there life on Mars?";
//stream_client->send(message.c_str(), message.size());
//printf("sent - %s\n", message.c_str());
stream_client->send(input, sizeof(input));
printf("sent - %s\n", input);
len = stream_client->receive(line, sizeof(line));
line[len] = NULL;
printf("received - %s\n", line);
delete stream_client;
}
//Code Additon by Srini ends here
stream->send(line, len);
printf("thread %lu, echoed '%s' back to the client\n",
(long unsigned int)self(), line);
}
The full thread code where receiving from client, sending to server, receiving from server, and sending to client is shown in the below link:
https://pastebin.com/UmPQJ70w
How can I change my design flow? Even in a basic diagram of client/server program. When the client calls connect(), then the server calls accept() every time, then sending/receiving happens. So, what can be done to modify the flow so that the client can connect only once?
Your intermediate server (which is acting as a proxy, so lets call it that) needs to maintain a single connection to the other server and delegate messaging with it in parallel to the messaging being done between your proxy and its clients.
I would suggest creating a separate thread whose sole task is to maintain that connection to the other server, and to send/receive messages with it.
When a client sends a message to your proxy, place the message in a thread-safe queue somewhere. Have the thread check the queue periodically and send any queued messages to the other server.
When the other server sends a message to your proxy, the thread can receive it and forward it to the appropriate client.

boost asio async_read header connection closes too early

Providing a MCVE is going to be hard, the scenario is the following:
a server written in c++ with boost asio offers some services
a client written in c++ with boost asio requests services
There are custom headers and most communication is done using multipart/form.
However, in the case where the server returns a 401 for an unauthorized access,
the client receives a broken pipe (system error 32).
AFAIK this happens when the server connection closes too early.
So, running into gdb, I can see that the problem is indeed the transition from the async_write which sends the request, to the async_read_until which reads the first line of the HTTP Header:
The connect routine sends the request from the client to the server:
boost::asio::async_write(*socket_.get(),
request_,
boost::bind(&asio_handler<http_socket>::write_request,
this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
And the write_request callback, checks if the request was sent OK, and then reads the first line (until the first newline):
template <class T>
void asio_handler<T>::write_request(const boost::system::error_code & err,
const std::size_t bytes)
{
if (!err) {
// read until first newline
boost::asio::async_read_until(*socket_,
buffer_,
"\r\n",
boost::bind(&asio_handler::read_status_line,
this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
else {
end(err);
}
}
The problem is that the end(err) is always called with a broken pipe (error code 32). Meaning, as far as I understand, that the server closed the connection. The server indeed closes the connection, but only after it has sent a message HTTP/1.1 401 Unauthorized.
using curl with the appropriate request, we do get the actual message/error before the server closes the connection
using our client written in C++/boost asio we only get the broken pipe and no data
only when the server leaves the connection open, do we get to the point of reading the error (401) but that defeats the purpose, since now the connection is left open.
I would really appreciate any hints or tips. I understand that without the code its hard to help, so I can add more source at any time.
EDIT:
If I do not check for errors between writing the request, and reading the server reply, then I do get the actual HTTP 401 error. However this seems counter-intuitive, and I am not sure why this happens or if it is supposed to happen.
The observed behavior is allowed per the HTTP specification.
A client or server may close the socket at anytime. The server can provide a response and close the connection before the client has finished transmitting the request. When writing the body, it is recommended that clients monitor the socket for an error or close notification. From the RFC 7230, HTTP/1.1: Message Syntax and Routing Section 6.5. Failures and Timeouts:
6.5. Failures and Timeouts
A client, server, or proxy MAY close the transport connection at any time. [...]
A client sending a message body SHOULD monitor the network connection for an error response while it is transmitting the request. If the client sees a response that indicates the server does not wish to receive the message body and is closing the connection, the client SHOULD immediately cease transmitting the body and close its side of the connection.
On a graceful connection closure, the server will send a response to the client before closing the underlying socket:
6.6. Tear-down
A server that sends a "close" connection option MUST initiate a close of the connection [...] after it sends the response containing "close". [...]
Given the above behaviors, there are three possible scenarios. The async_write() operation completes with:
success, indicating the request was written in full. The client may or may not have received the HTTP Response yet
an error, indicating the request was not written in full. If there is data available to be read on the socket, then it may contain the HTTP Response sent by the server before the connection terminated. The HTTP connection may have terminated gracefully
an error, indicating the request was not written in full. If there is no data available to be read on the socket, then the HTTP connection was not terminated gracefully
Consider either:
initiating the async_read() operation if the async_write() is successful or there is data available to be read
void write_request(
const boost::system::error_code & error,
const std::size_t bytes_transferred)
{
// The server may close the connection before the HTTP Request finished
// writing. In that case, the HTTP Response will be available on the
// socket. Only stop the call chain if an error occurred and no data is
// available.
if (error && !socket_->available())
{
return;
}
boost::asio::async_read_until(*socket_, buffer_, "\r\n", ...);
}
per the RFC recommendation, initiate the async_read() operation at the same time as the async_write(). If the server indicates the HTTP connection is closing, then the client would shutdown its send side of the socket. The additional state handling may not warrant the extra complexity

What is the proper way to securely disconnect an asio SSL socket?

A boost-asio SSL/TLS TCP socket is implemented as an ssl::stream over a tcp::socket:
boost::asio::ssl::stream<boost::asio::ip::tcp::socket> ssl_socket;
In the TLS protocol, a cryptographically secure shutdown involves parties exchanging close_notify messages. Simply closing the lowest layer may make the session vulnerable to a truncation attack.
In boost asio ssl async_shutdown always finishes with an error? #Tanner Sansbury describes the SSL shutdown process in detail with a number of scenarios and proposes using an async_shutdown followed by an async_write to disconnect an SSL stream prior to closing the socket:
ssl_socket.async_shutdown(...);
const char buffer[] = "";
async_write(ssl_socket, buffer, [](...) { ssl_socket.close(); })
Performing an async_shutdown on an ssl::stream sends an SSL close_notify message and waits for a response from the other end. The purpose of writing to the stream after the async_shutdown is to be notified when async_shutdown has sent the close_notify so that the socket can be closed without waiting for the response. However, in the current (1.59) version of boost the call to async_write fails...
In How to gracefully shutdown a boost asio ssl client? #maxschlepzig proposes shutting down receiver of the underlying TCP socket:
ssl_socket.lowest_layer()::shutdown(tcp::socket::shutdown_receive);
This produces a short read error, and async_shutdown is called when it's detected in the error handler:
// const boost::system::error_code &ec
if (ec.category() == asio::error::get_ssl_category() &&
ec.value() == ERR_PACK(ERR_LIB_SSL, 0, SSL_R_SHORT_READ))
{
// -> not a real error:
do_ssl_async_shutdown();
}
Or cancelling the read/write operations on the socket and then calling SSL async shutdown, i.e.:
boost::system::error_code ec;
ssl_socket.cancel(ec);
ssl_socket.async_shutdown([](...) { ssl_socket.close(); };
I'm currently using this last method since it works with the current version of boost.
What is the correct/best way to securely disconnect a boost-asio SSL socket?
To securely disconnect, perform a shutdown operation and then close the underlying transport once shutdown has complete. Hence, the method you are currently using will perform a secure disconnect:
boost::system::error_code ec;
ssl_socket.cancel(ec);
ssl_socket.async_shutdown([](...) { ssl_socket.close(); };
Be aware that the current async_shutdown operation will be considered complete when either:
A close_notify has been received by the remote peer.
The remote peer closes the socket.
The operation has been cancelled.
Hence, if resources are bound to the lifetime of the socket or connection, then these resources will remain alive waiting for the remote peer to take action or until the operation is cancelled locally. However, waiting for a close_notify response is not required for a secure shutdown. If resources are bound to the connection, and locally the connection is considered dead upon sending a shutdown, then it may be worthwhile to not wait for the remote peer to take action:
ssl_socket.async_shutdown(...);
const char buffer[] = "";
async_write(ssl_socket, boost::asio::buffer(buffer),
[](...) { ssl_socket.close(); })
When a client sends a close_notify message, the client guarantees that the client will not send additional data across the secure connection. In essence, the async_write() is being used to detect when the client has sent a close_notify, and within the completion handler, will close the underlying transport, causing the async_shutdown() to complete with boost::asio::error::operation_aborted. As noted in the linked answer, the async_write() operation is expected to fail.
... as the write side of PartyA's SSL stream has closed, the async_write() operation will fail with an SSL error indicating the protocol has been shutdown.
if ((error.category() == boost::asio::error::get_ssl_category())
&& (SSL_R_PROTOCOL_IS_SHUTDOWN == ERR_GET_REASON(error.value())))
{
ssl_stream.lowest_layer().close();
}
The failed async_write() operation will then explicitly close the underlying transport, causing the async_shutdown() operation that is waiting for PartyB's close_notify to be cancelled.
I'm probably late to answer this but I want to report my experience.
This solution so far (using boost 1.78) did not produce any visible error on the client nor the server:
// sock type is boost::asio::ssl::stream<boost::asio::ip::tcp::socket>
sock->shutdown(ec);
sock->lowest_layer().shutdown(boost::asio::ip::tcp::socket::shutdown_both, ec);
sock->lowest_layer().cancel(ec);
sock->lowest_layer().close();
Sandbox server with: openssl s_server -cert server.crt -key server.key -4 -debug
With this solution the server gets this after the sock->shutdown(ec).
read from 0x55e5dff8c960 [0x55e5dff810f8] (19 bytes => 19 (0x13))
0000 - 44 bc 11 5b a9 b4 ee 51-48 e0 18 f7 99 a7 a8 a9 D..[...QH.......
0010 - 21 1a 60 !.`
DONE
shutting down SSL
CONNECTION CLOSED
Before I was using this code (used for both plain TCP and ssl socket)
sock->lowest_layer().shutdown(boost::asio::ip::tcp::socket::shutdown_both, ec);
sock->lowest_layer().cancel(ec);
sock->lowest_layer().close();
The old code, when leveraging ssl socket, produced this error on the server:
read from 0x55eb3d40b430 [0x55eb3d423513] (5 bytes => 0 (0x0))
ERROR
shutting down SSL
CONNECTION CLOSED
As mentioned before, to avoid this behavior a close_notify should be sent out by the client using ssl::stream::async_shutdown or ssl::stream::shutdown
The trick of async_write() could be useful in case you want to leverage the async_shutdown() function instead of the synchronous shutdown()