SSL_accept returns SSL_ERROR_SSL - c++

Question

I have a strange error working with openssl in c++ with non blocking sockets.
SSL_accept return SSL_ERROR_SSL.
error string is
"error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol"
as obtained from
ERR_error_string(ERR_get_error(), NULL);
ssldump:
New TCP connection #1: localhost(35677) <-> localhost(8084)
1 1 1443513828.4788 (0.0001) C>SV3.1(153) Handshake
ClientHello
Version 3.3
random[32]=
e9 fc d9 2b 25 20 77 9f 5f a4 b1 eb 5d 50 15 f2
38 7a a9 86 16 49 f8 a3 e9 00 7f 93 11 cd 7b b1
cipher suites
Unknown value 0xc02b
Unknown value 0xc02f
Unknown value 0xc00a
Unknown value 0xc009
Unknown value 0xc013
Unknown value 0xc014
Unknown value 0xc012
Unknown value 0xc007
Unknown value 0xc011
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
Unknown value 0x45
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
Unknown value 0x88
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
Unknown value 0x41
TLS_RSA_WITH_AES_256_CBC_SHA
Unknown value 0x84
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
compression methods
NULL
1 2 1443513828.4790 (0.0002) S>CV3.3(53) Handshake
ServerHello
Version 3.3
random[32]=
aa d6 10 d1 eb b5 de b1 ed f8 6b 2f 44 fa 9f 57
14 1d c0 27 a1 e9 24 3a 20 cb 09 fa 75 33 1c c7
session_id[0]=
cipherSuite TLS_RSA_WITH_AES_128_CBC_SHA
compressionMethod NULL
1 3 1443513828.4790 (0.0000) S>CV3.3(1003) Handshake
Certificate
certificate[993]=
30 82 03 dd 30 82 02 c5 a0 03 02 01 02 02 09 00
bc 6b da c1 c3 f0 3f 14 30 0d 06 09 2a 86 48 86
f7 0d 01 01 0b 05 00 30 81 84 31 0b 30 09 06 03
55 04 06 13 02 49 4e 31 0b 30 09 06 03 55 04 08
0c 02 49 4e 31 10 30 0e 06 03 55 04 07 0c 07 67
75 72 67 61 6f 6e 31 0e 30 0c 06 03 55 04 0a 0c
05 69 72 65 78 78 31 0c 30 0a 06 03 55 04 0b 0c
03 64 65 76 31 12 30 10 06 03 55 04 03 0c 09 6c
6f 63 61 6c 68 6f 73 74 31 24 30 22 06 09 2a 86
48 86 f7 0d 01 09 01 16 15 76 69 6b 61 73 73 6f
6f 64 31 31 40 67 6d 61 69 6c 2e 63 6f 6d 30 1e
17 0d 31 35 30 39 32 39 30 38 30 32 35 36 5a 17
0d 31 36 30 39 32 38 30 38 30 32 35 36 5a 30 81
84 31 0b 30 09 06 03 55 04 06 13 02 49 4e 31 0b
30 09 06 03 55 04 08 0c 02 49 4e 31 10 30 0e 06
03 55 04 07 0c 07 67 75 72 67 61 6f 6e 31 0e 30
0c 06 03 55 04 0a 0c 05 69 72 65 78 78 31 0c 30
0a 06 03 55 04 0b 0c 03 64 65 76 31 12 30 10 06
03 55 04 03 0c 09 6c 6f 63 61 6c 68 6f 73 74 31
24 30 22 06 09 2a 86 48 86 f7 0d 01 09 01 16 15
76 69 6b 61 73 73 6f 6f 64 31 31 40 67 6d 61 69
6c 2e 63 6f 6d 30 82 01 22 30 0d 06 09 2a 86 48
86 f7 0d 01 01 01 05 00 03 82 01 0f 00 30 82 01
0a 02 82 01 01 00 e2 9e cb 0c 9a c2 a4 0a ff 81
77 23 ad 74 c6 a0 12 9f f4 a4 86 94 bf 02 57 cc
ff 13 cf 4f 13 83 6e 70 8b 2d ee 92 93 94 44 d4
db 20 e1 67 65 3a 67 c1 d5 91 67 ab f9 98 aa 59
16 b4 38 99 92 66 c2 de e7 88 82 cd a0 10 95 89
78 f5 64 6a e5 b6 75 a2 e0 d0 76 c3 57 92 03 c7
d3 7d f3 c5 18 e5 bf 8d f5 71 17 4c dd 18 fa 11
8c d1 ed 8b 0f b6 5c fb 16 bf 5a 44 23 71 c8 83
83 5a 4d ca b0 8c 15 96 66 dc cd 9a 06 33 b3 eb
71 43 25 b1 0f df 1a c6 f1 7a 4c 4d 5e 34 12 b7
70 76 b6 f9 7e 40 6c 2e 70 df 26 cd f9 c9 0b 90
57 dd 0b c3 57 b8 55 ad 63 13 16 15 bf 8d 3b 48
9d 11 cf 15 55 a1 f2 ab ff 8f 43 48 09 e7 c9 b0
21 66 e7 e4 76 14 aa d4 a9 88 ef bf e1 db 0f b3
27 95 d0 c9 50 6f 36 c8 5e 47 3f c3 a8 59 9c a6
8a 75 57 0b db 83 c6 08 13 fa 9c 64 3c 5a 4e d5
ba 4f 23 39 67 77 02 03 01 00 01 a3 50 30 4e 30
1d 06 03 55 1d 0e 04 16 04 14 29 86 80 ff 9c 14
00 5e 2e 89 81 4a 1e 3c 32 82 ec 44 35 7b 30 1f
06 03 55 1d 23 04 18 30 16 80 14 29 86 80 ff 9c
14 00 5e 2e 89 81 4a 1e 3c 32 82 ec 44 35 7b 30
0c 06 03 55 1d 13 04 05 30 03 01 01 ff 30 0d 06
09 2a 86 48 86 f7 0d 01 01 0b 05 00 03 82 01 01
00 c9 f4 f7 0f bd a4 f2 81 e4 4e c5 4d 86 9d f3
30 fe df f6 b1 b5 a3 a6 ef 41 51 fc f3 1b c5 be
e1 1f 75 fe f1 ee 94 95 54 c7 07 13 d4 b6 89 2b
a3 86 e3 7b 12 6d 0a 51 66 31 74 d5 83 67 92 79
45 b3 03 1d 77 6b b0 71 a0 07 c0 06 e6 6b 81 74
01 8e 6d a6 c1 82 8b 34 b2 6c 52 b9 3e a6 c0 c5
ce f6 c8 b2 4b 6a 8c 43 4a 20 38 61 8a 09 1f 39
cd e6 4a 0b 6c 89 b8 88 74 d7 1e fa 86 dc 0a cd
d4 73 39 5f 53 4b 13 79 f8 cd 62 ee 38 42 61 56
26 31 44 ac 5b 39 24 2b 3f f9 21 b4 59 25 14 95
69 e6 33 41 df ef b5 ea 20 7d 38 a9 f4 c3 d1 f7
c5 37 4a c9 01 cd f4 3a e7 3c a7 16 33 0f 5d 46
19 bc 50 3d 33 05 63 ee 29 51 d1 79 c4 01 04 0f
be 27 65 57 ab e5 e8 d3 bf 4d 11 55 bc 52 e4 9f
d9 0c 87 be a9 62 9d be 65 21 a8 8a 46 63 d1 91
f9 2b e1 c6 5d 33 fb ae b6 9e a0 36 0b 37 2a fa
3b
1 4 1443513828.4790 (0.0000) S>CV3.3(4) Handshake
ServerHelloDone
1 5 1443513828.4800 (0.0009) C>SV3.3(262) Handshake
ClientKeyExchange
EncryptedPreMasterSecret[256]=
76 58 13 c2 dd ed fb 6e 7b 49 aa 0c 26 40 a1 1f
8b 87 b0 f6 a5 f8 24 fe 0c 7b a9 e6 b0 51 b1 92
2c 52 33 ba af 2a d2 52 b8 80 0f ea fd 24 b7 89
15 2f 0e 90 86 ef 9c 9d 52 02 ca 56 09 b5 81 5f
e1 05 a8 9a 44 32 7f 4f ec 58 33 4a 97 96 23 15
7d a8 1d 8c bc 5d 20 13 39 54 2f 61 27 be aa 6a
ec 49 be 82 cc 2b c7 3d a3 b0 cb 4f 26 cb 14 41
d4 4b 00 57 6f 52 2b 66 aa 23 97 ed e9 e7 9c d9
ed 81 28 0c 08 9f dd 9d e4 00 b5 ba aa a4 f6 3b
23 87 f6 da 64 c0 1e 33 87 bb ef bc c3 fe 28 64
d1 c8 ff ae f9 3f 5a fe 38 e6 ab 3e 34 2c b5 96
f7 a8 99 b6 8e b3 46 d3 c9 f8 f8 94 26 97 27 85
5f 3e ff 85 15 5f c8 1f 10 53 26 e4 88 32 4f 48
62 87 ac a8 9f 14 f1 e3 f4 c4 1a 71 49 e6 33 15
16 66 66 ec 68 ee f7 91 0c 4c b3 3a 49 88 e1 31
a5 fe 4c 31 35 3d dc 01 f6 be f2 e5 65 a0 bb 78
1 6 1443513828.4800 (0.0000) C>SV3.3(1) ChangeCipherSpec
1 7 1443513828.4800 (0.0000) C>SV3.3(64) Handshake
Unknown SSL content type 72
1 1443513828.4801 (0.0001) S>C TCP RST
I am not sure why the server resets the connection.
I am using a self signed certificate to test a websocket connection using scheme wss. Client is Mozilla Firefox and websocket server is what I am trying to implement.
Any pointers on why the server is resetting the connection?
error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
What possible reason could throw unknown protocol?
Thanks
Edited: Adding code. It is not all the code but gives a fair idea of what i did.
Here is how my code looks like:
/*
this the main method that accepts incoming connection on a listening socket.
*/
SecureServer::Accept()
{
int ret = 0;
while(!mFinished) {
ret = epoll_wait(millis);
if(ret == -1) {
continue;
}
else if(ret == 0) {
idle(0);
}
else if(ret > 0) {
//some events are triggered
for(int i=0; i<ret; i++) {
//getEvent method is in the base class.
//SecureServer is derived from this class.
//Base class is just an epoll wrapper
epoll_event* ev = getEvent(i);
//check for error
if( (ev->events & EPOLLERR) || (ev->events & EPOLLHUP) || !(ev->events & EPOLLIN)) {
ERROR(Log::eSecureServer, "Error occured on fd = %d", ev->data.fd);
removeEpollFd(ev->data.fd);
continue;
}
//check if its a new incoming connection on server socket
else if(ev->data.fd == mServerSocket) {
//accept the connection
struct sockaddr peerAddress;
socklen_t socklen = sizeof(peerAddress);
int newSocket = accept(mSocket, &peerAddress, &socklen);
if(newSocket == -1) {
ERROR(Log::eSecureServer, "Error in accept, listen address = %s, port = %d epoll_wait returned = %d", mHost.c_str(), mPort, ret);
continue;
}
rxUShort port = ntohs(*((rxUShort*) &(peerAddress.sa_data[0])));
char ipPort[32] = {0};
sprintf(ipPort, "%hhu.%hhu.%hhu.%hhu:%hu", peerAddress.sa_data[2],peerAddress.sa_data[3],peerAddress.sa_data[4],
peerAddress.sa_data[5],port);
INFO(Log::eSecureServer, "SecureTCPServer received incoming client connection from = %s", ipPort);
//add epoll fd
// This method also makes the socket NON BLOCKING
addEpollFd(newSocket);
}
//check the handshake from a previously accepted connection
else {
//process the handshake
processHandshake(ev->data.fd);
}
}
}
}
}
/*
This method determines the client and server protocol
*/
int Protocol::processHandshake(int fd)
{
int length = 0;
length = recv(fd, handshake, 1024, MSG_PEEK);
if(length == 0) {
return UNKNOWN_PROTOCOL;
}
else if((bcmp(handshake, "\x16", 1) == 0) ||
(bcmp(handshake, "\x80", 1) == 0)) {
//secure connection requested on this fd
SSL* pSSL = 0;
int ret = NewSSLSocket(pSSL, fd);
if(SSL_ERROR_NON == ret) {
///success
}
else {
if(SSL_ERROR_WANT_READ == ret || SSL_ERROR_WANT_WRITE == ret) {
NFO(Log::eProtocol, "Handshake error, in progress.");
return eProtocolHandshakeInProgress;
} else if(SSL_ERROR_SSL == ret) {
INFO(Log::eProtocol, "Handshake error, library error");
return eProtocolInvalidHeader;
} else if(SSL_ERROR_SYSCALL == ret) {
INFO(Log::eProtocol, "Handshake error, possibly syscall error");
return eProtocolInvalidHeader;
}
SSL_free(pSSL);
}
}
}
int Protocol::NewSSLSocket(SSL* pSSL, rxUInt fd)
{
pSSL = SSL_new(mSSLContext);
SSL_set_fd(pSSL, fd);
int ret = SSL_accept(pSSL);
switch(SSL_get_error(pSSL, ret)) {
case SSL_ERROR_WANT_READ:
case SSL_ERROR_WANT_WRITE:
{
INFO(Log::eNetwork, "SSL Handshake is in progress on fd = %d %s", fd, ERR_error_string(ERR_get_error(), NULL));
return SSL_ERROR_WANT_READ;
}
break;
case SSL_ERROR_SYSCALL:
{
INFO(Log::eNetwork, "SSL syscall error during handshake on fd = %d %s", fd, ERR_error_string(ERR_get_error(), NULL));
return SSL_ERROR_SYSCALL;
}
break;
case SSL_ERROR_SSL:
{
INFO(Log::eNetwork, "SSL library error during handshake on fd = %d %s", fd, ERR_error_string(ERR_get_error(), NULL));
return SSL_ERROR_SSL;
}
case SSL_ERROR_NONE:
{
INFO(Log::eNetwork, "SSL handshake on fd = %d complete", fd);
return SSL_ERROR_NONE;
}
break;
default:
{
INFO(Log::eNetwork, "SSL unknown error during handshake on fd = %d %s", fd, ERR_error_string(ERR_get_error(), NULL));
return -1;
}
break;
}
return -1;
}
//Protocol Constructor Init the SSL Context
Protocol::Protocol()
{
const SSL_METHOD* pSSLMethod = SSLv23_server_method();
char err[1024] = {0};
mSSLContext = SSL_CTX_new(pSSLMethod);
if(mSSLContext == NULL) {
ERROR(Log::eNetwork, "Failed to initialize SLS_CTX. TLS Unavaliable.");
ERROR(Log::eNetwork, ERR_error_string(ERR_get_error(), err));
} else
INFO(Log::eNetwork, "Initialized SSL Context.");
SSL_CTX_set_cipher_list(mSSLContext, "ALL");
//Load the SSL Certificates
rxUInt ret = 0;
ret = SSL_CTX_use_certificate_file(mSSLContext, mCACertificateFile.c_str(), SSL_FILETYPE_PEM);
if(ret <=0 ) {
ERROR(Log::eNetwork, "Failed to read ssl sertificate. We should exit.");
ERROR(Log::eNetwork, ERR_error_string(ERR_get_error(), err));
} else
INFO(Log::eNetwork, "SSL Certificate read complete. OK!");
//Load the private key file
ret = SSL_CTX_use_PrivateKey_file(mSSLContext, mPrivateKeyFile.c_str(), SSL_FILETYPE_PEM);
if(ret <=0 ) {
ERROR(Log::eNetwork, "Failed to read private key file. We should exit.");
ERROR(Log::eNetwork, ERR_error_string(ERR_get_error(), err));
} else
INFO(Log::eNetwork, "Private Key read complete. OK!");
if(!SSL_CTX_check_private_key(mSSLContext)) {
ERROR(Log::eNetwork, "Private key does not match the SSL Certificate. We should exit.");
ERROR(Log::eNetwork, ERR_error_string(ERR_get_error(), err));
} else
INFO(Log::eNetwork, "Private Key and SSL Certificate Matched. OK!");
}

Answer 1

1 7 1443513828.4800 (0.0000) C>SV3.3(64) Handshake
Unknown SSL content type 72
My guess is that you are sending some plain data on the connection, i.e. write to the plain socket instead to the SSL socket. This is just a guess because you did not provide any code but looking for such error one finds issues like this.