SSL_connect results in SSL_ERROR_SYSCALL with WSAENOTCONN on Windows - c++

Question

I try to set SSL connection on Windows with OpenSSL. My steps follow:
Create TCP socket BIO.
Connect to server with TCP.
Add this BIO to SSL instance.
Upgrade connection to SSL.
However, when I try to call SSL_connect with BIO that for sure recently connected to TCP socket, I receive SSL_ERROR_SYSCALL with WSAENOTCONN on Windows.
My code follows.
this->TcpSocket = BIO_new(BIO_s_connect());
BIO_set_nbio(this->TcpSocket, 1);
BIO_set_conn_hostname(this->TcpSocket, hostname);
BIO_set_conn_port(this->TcpSocket, port);
int connectionResult;
while ((connectionResult = BIO_do_connect(this->TcpSocket)) <= 0 && BIO_should_retry(this->TcpSocket))
{
auto retryType = BIO_retry_type(this->TcpSocket);
if (retryType & BIO_FLAGS_READ != 0
|| retryType & BIO_FLAGS_WRITE != 0)
{
auto handle = BIO_get_fd(this->TcpSocket, NULL);
fd_set handles;
handles.fd_count = 1;
handles.fd_array[0] = handle;
timeval timeout;
timeout.tv_sec = seconds;
timeout.tv_usec = 0;
if (retryType & BIO_FLAGS_READ != 0)
select(handle + 1, &handles, NULL, NULL, &timeout);
else
select(handle + 1, NULL, &handles, NULL, &timeout);
}
else
Thread::Sleep(50);
}
this->SslContext = SSL_CTX_new(SSLv23_client_method());
SSL_CTX_set_verify(this->SslContext, SSL_VERIFY_NONE, NULL);
this->SslSocket = SSL_new(this->SslContext);
SSL_set_bio(this->SslSocket, this->TcpSocket, this->TcpSocket);
int sslConnectResult;
while ((sslConnectResult = SSL_connect(this->SslSocket)) == -1)
{
auto now = time(NULL);
int sslConnectErrorCode = SSL_get_error(this->SslSocket, sslConnectResult);
switch (sslConnectErrorCode)
{
case SSL_ERROR_WANT_READ:
case SSL_ERROR_WANT_WRITE:
if (now >= deadline)
throw SocketTimeoutException();
else
this->WaitForTcpSocket(deadline - now);
break;
case SSL_ERROR_SYSCALL:
{
auto err = GetLastError();
this->RaiseOpenSSLException();
}
break;
default:
this->RaiseOpenSSLException();
}
}
What is reason of the error? I understand that it means that client is disconnected from server. But I don't understand why. I have good Internet connection, and server also is stable, so it is unlikely that the reason in network connectivity.

Answer 1

Your TCP connect loop is not taking into account if BIO_do_connect() fails and BIO_should_retry() returns false. Your loop will stop in that condition and you will not have a connection, but you attempt to activate SSL anyway, which could cause the WSAENOTCONN error.
Try something more like this instead:
do
{
connectionResult = BIO_do_connect(this->TcpSocket);
if (connectionResult > 0)
break;
if (!BIO_should_retry(this->TcpSocket))
throw SocketException();
auto retryType = BIO_retry_type(this->TcpSocket);
if (retryType & (BIO_FLAGS_READ | BIO_FLAGS_WRITE))
{
auto handle = BIO_get_fd(this->TcpSocket, NULL);
fd_set handles;
FD_ZERO(&handles);
FD_SET(handle, &handles);
timeval timeout;
timeout.tv_sec = seconds;
timeout.tv_usec = 0;
if (retryType & BIO_FLAGS_READ)
selectResult = select(handle + 1, &handles, NULL, NULL, &timeout);
else
selectResult = select select(handle + 1, NULL, &handles, NULL, &timeout);
if (selectResult < 0)
throw SocketException();
if (selectResult == 0)
throw SocketTimeoutException();
}
else
Thread::Sleep(50);
}
while (true);