"unknown ca" with self-generated CA, certificates and client/server - c++

Question

I'm writing a custom client & server that I want to communicate securely over the public Internet, therefore I want to use OpenSSL and have both ends do peer verification to ensure that my client isn't mis-directed by a MITM, and likewise that an unauthorized client isn't able to connect to the server.
This is the error received from the server during the SSL_connect / SSL_accept phase:
15620:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:ssl\record\rec_layer_s3.c:1528:SSL alert number 48
I'm running under Windows 10, using OpenSSL 1.1.1. I'm using the following batch file to create them. I enter the ca private key passphrase by hand for obvious reasons.
openssl genrsa -out -des3 ca.key.pem 2048
openssl genrsa -out server.key.pem 2048
openssl genrsa -out client.key.pem 2048
openssl req -x509 -new -nodes -key ca.key.pem -sha256 -days 365 -out ca.cert.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=Foobar
openssl req -new -sha256 -key server.key.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=Foobar -out server.csr
openssl x509 -req -in server.csr -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out server.cert.pem -days 365 -sha256
openssl req -new -sha256 -key client.key.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=Foobar -out client.csr
openssl x509 -req -in client.csr -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out client.cert.pem -days 365 -sha256
The intent here is to create a self-signed CA, and then have that directly sign both the client and server keys.
ca.key.pem will be stored in a secure place: on an encrypted veracrypt volume.
Both client and server use the following call to enable peer verification:
SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT, nullptr);
I'm fairly certain this is a certificate issue because the errors go away if I remove that line.

Answer 1

Answering this myself so that it can help anyone else that might arrive here looking for solutions to this problem. The answer was found in another SO question, but is worth repeating here: The Common Name for the CA cannot be the same as the Common Name for the client and server certificates.
So changing the fourth line of the batch file to this:
openssl req -x509 -new -nodes -key ca.key.pem -sha256 -days 365 -out ca.cert.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=FoobarCA
fixed the problem.

Answer 2

$ openssl req -x509 -new ... -addext basicConstraints=critical,CA:TRUE
This essentially creates a certificate which has 2 basic contrains CA:TRUE extensions:
$ openssl x509 -in ca.cert.pem -text
X509v3 extensions:
...
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Basic Constraints: critical
CA:TRUE
Trying to use the CA to verify the server certificate will not work:
$ openssl verify -CAfile ca.cert.pem server.cert.pem
C = XX, ST = XX, L = XX, O = XX, CN = CA
error 24 at 1 depth lookup: invalid CA certificate
error server.cert.pem: verification failed
Given that this simple check does not work, the client will also not be able to validate the server certificate, resulting in an unknown ca alert:
...:tlsv1 alert unknown ca:...
When skipping the -addext it will create a self-signed certificate as documented, which already has CA:TRUE
$ openssl req -x509 -new ...
...
$ openssl x509 -in ca.cert.pem -text
X509v3 extensions:
...
X509v3 Basic Constraints: critical
CA:TRUE
And using this to verify the server certificate works:
$ openssl verify -CAfile ca.cert.pem server.cert.pem
server.cert.pem: OK
This certificate should also be successfully validated by your client, thus no longer resulting in unknown ca.