How can I validate hostnames with OpenSSL 1.1.0? - c++

Question

I'm trying to implement a sample hostname validation with OpenSSL.
The sample C/C++ code I have put together is:
// please note I'm connecting to https://openssl.org itself
// enable SNI
if(!SSL_set_tlsext_host_name(ssl, "www.openssl.org")) throw;
if(!SSL_connect(ssl)) throw;
// connection is fine, I can get the homepage via HTTP
X509 *cert = SSL_get_peer_certificate(ssl);
if(cert) {
if(!X509_VERIFY_PARAM_set1_host(SSL_get0_param(ssl), "google.com", 0)) throw;
SSL_set_verify(ssl, SSL_VERIFY_PEER, 0);
const long cert_res = SSL_get_verify_result(ssl);
if(cert_res == X509_V_OK) {
printf("Certificate verified!\n");
}
X509_free(cert);
}
As per code above I'm successfully connecting to the openssl.org domain; then I'm setting the name to verify as google.com to test failures, but the code still succeeds.
What am I doing wrong?
How can I implement a thorough verification of hostnames using OpenSSL APIs? I wouldn't want to re-implement (most likely with bugs/wrongly) what is already implemented in the library...
I'm using Ubuntu 16.04 and this libssl version: /lib/x86_64-linux-gnu/libssl.so.1.0.0.

Answer 1

As suggested by jww, one simply needs to set (at least)
if(!X509_VERIFY_PARAM_set1_host(SSL_get0_param(ssl), "google.com", 0)) throw;
before performing the connection itself:
if(!SSL_connect(ssl)) throw;
In this case, one case ensure OpenSSL to implement the check automatically at connection time by adding following code:
SSL_set_verify(ssl, SSL_VERIFY_PEER, 0);
before calling SSL_connect, or follow the same path as before and have return X509_V_ERR_HOSTNAME_MISMATCH by SSL_get_verify_result if one wants to handle things in more details:
// please note I'm connecting to https://openssl.org itself
// enable SNI
if(!SSL_set_tlsext_host_name(ssl, "www.openssl.org")) throw;
// enable name/domain verification
if(!X509_VERIFY_PARAM_set1_host(SSL_get0_param(ssl), "google.com", 0)) throw;
if(!SSL_connect(ssl)) throw;
// get certificate
X509 *cert = SSL_get_peer_certificate(ssl);
if(cert) {
const long cert_res = SSL_get_verify_result(ssl);
// in case of name/domain mismatch cert_res will
// be set as 62 --> X509_V_ERR_HOSTNAME_MISMATCH
if(cert_res != X509_V_OK) throw; // certificate has been tampered with
X509_free(cert);
} else throw; // we couldn't get certificate