Mutual authentication always succeeds with OpenSSL - c++

Question

I am using openssl and zmq to write a server and a client.
My client and server need mutual authentication.
but after I set SSL_CTX_set_verify(ssl_ctx,SSL_VERIFY_FAIL_IF_NO_PEER_CERT,NULL) on server, the handshake always successes whether the client send the certificate or not.
In addition, SSL_get_peer_certificate(tls->get_ssl_()) return null and SSL_get_verify_result(tls->get_ssl_()) return 0 which means X509_V_OK.
I am really confused and desperate now. Any suggestions or corrections?
This is part of my code:
OpenSSL_add_all_algorithms();
SSL_library_init();
SSL_load_error_strings();
ERR_load_BIO_strings();
const SSL_METHOD *meth;
SSL_CTX *ssl_ctx;
//**************************part of client************************
{
meth = SSLv23_client_method();
ssl_ctx = SSL_CTX_new(meth);
SSL_CTX_set_verify(ssl_ctx,SSL_VERIFY_PEER,NULL);
int rc1 = SSL_CTX_load_verify_locations(ssl_ctx, ".\\demoCA\\private\\server_chain.pem",".\\demoCA\\private\\");///
SSL_CTX_set_default_passwd_cb_userdata(ssl_ctx,"pw");
std::string cert_chain(".\\demoCA\\private\\client_chain.pem");
std::string cert(".\\demoCA\\private\\client_crt.pem");
std::string key(".\\demoCA\\private\\client_key.pem");
int code = SSL_CTX_use_certificate_chain_file(ssl_ctx,cert_chain.c_str());
if (code != 1)
{
std::cout<<"error1\n";
//throw TLSException("failed to read credentials.");
}
code = SSL_CTX_use_PrivateKey_file(ssl_ctx,key.c_str(),SSL_FILETYPE_PEM);
i f (code != 1)
{
std::cout<<"error2\n";
//throw TLSException("failed to read credentials.");
}
if(!SSL_CTX_check_private_key(ssl_ctx))
{
std::cout<<"key wrong";
system("pause");
exit(0);
}
}
//*****************part of server****************************
{
meth = SSLv23_server_method();
ssl_ctx = SSL_CTX_new(meth);
SSL_CTX_set_verify(ssl_ctx,SSL_VERIFY_FAIL_IF_NO_PEER_CERT,NULL)
SSL_CTX_set_client_CA_list(ssl_ctx,SSL_load_client_CA_file(".\\demoCA\\private\\client_chain.pem"));//
SSL_CTX_set_default_passwd_cb_userdata(ssl_ctx,"pw");
std::string cert_chain(".\\demoCA\\private\\server_chain.pem");
std::string cert(".\\demoCA\\private\\server_crt.pem");
std::string key(".\\demoCA\\private\\server_key.pem");
int rc = SSL_CTX_use_certificate_file(ssl_ctx,cert.c_str(),SSL_FILETYPE_PEM);
if (rc!=1)
{
//throw TLSException("failed to read credentials.");
std::cout<<"error1\n";
}
rc = SSL_CTX_use_PrivateKey_file(ssl_ctx,key.c_str(),SSL_FILETYPE_PEM);
if (rc!=1)
{
//throw TLSException("failed to read credentials.");
std::cout<<"error2\n";
}
int rcode = SSL_CTX_check_private_key(ssl_ctx);
if(rcode!=1)
{
std::cout<<"key wrong";
system("pause");
//exit(0);
}
}

Answer 1

From the documentation of SSL_CTX_set_verify:
SSL_VERIFY_FAIL_IF_NO_PEER_CERT
Server mode: if the client did not return a certificate, the TLS/SSL handshake is immediately terminated with a "handshake failure" alert. This flag must be used together with SSL_VERIFY_PEER.
You did not use it together with SSL_VERIFY_PEER as described in the documentation and thus it has no effect.