Failed SSL handshake in Kafka Producer - amazon-web-services

Question

I'm trying to set up a Kafka broker with SSL enabled communication.
I followed the guide as described in the Kafka documentation here under 7.2: https://kafka.apache.org/documentation/#security
When I'm trying to execute the command ./kafka-console-producer.sh --broker-list ec2-ip.eu-central-1.compute.amazonaws.com:9093 -topic test --producer.config ../config/producer.properties
I get:
ERROR [Producer clientId=console-producer] Connection to node -1 (ec2-ip.eu-central-1.compute.amazonaws.com/172.31.42.33:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2019-08-22 13:15:56,728] ERROR [Producer clientId=console-producer] Connection to node -1 (ec2-ip.eu-central-1.compute.amazonaws.com/172.31.42.33:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
Testing the SSl connection in general with
openssl s_client -debug -connect ec2-ip.eu-central-1.compute.amazonaws.com:9093 -tls1
results in
0020 - a0 41 45 81 42 b6 83 d3-2b 94 02 9f ac 42 73 42 .AE.B...+....BsB
---
Certificate chain
0 s:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown
i:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDuTCCAqGgAwIBAgIEPsshdTANBgkqhkiG9w0BAQsFADBsMRAwDgYDVQQGEwdV
bmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQHEwdVbmtub3duMRAwDgYD
VQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMRAwDgYDVQQDEwdVbmtub3du
MB4XDTE5MDgyMjEyNTIwOFoXDTIwMDgyMTEyNTIwOFowbDEQMA4GA1UEBhMHVW5r
bm93bjEQMA4GA1UECBMHVW5rbm93bjEQMA4GA1UEBxMHVW5rbm93bjEQMA4GA1UE
ChMHVW5rbm93bjEQMA4GA1UECxMHVW5rbm93bjEQMA4GA1UEAxMHVW5rbm93bjCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAI5EKxmSDLQvW/ePu9mB3qdf
E4cG1ok1kI48S6yL4gchx2egIC6Zjr5gBhuek36mhKV82H1nVYfDzJVyxoyaTZ5q
ux8xq+ooNN/chKHy0ExpCZ2HHvrQiTPQit//dyA5zc5yOyRcJo/m2EhYfRMuLfFM
kx1mEg2cs6Qa/hd6jKjCSvbcKDfrOhf4rLagHzI+2GRQfF6LXxYBE3Ti7u6fkqog
o67CTrNR6IER1b4wGIeHga8eXiVhOuP4JiWdpF69i5py0RVfzsesP0W8RB2qElte
+7EwShaFO72Xa7MPiDz1rE4D5pohQ09AWGb/B0YIcI7qh9jyInbj4eWhhlnd1lsC
AwEAAaNjMGEwQAYDVR0RBDkwN4I1ZWMyLTM1LTE1OC0yMzItMTc2LmV1LWNlbnRy
YWwtMS5jb21wdXRlLmFtYXpvbmF3cy5jb20wHQYDVR0OBBYEFNGugrNUmFiVKVyU
Uiepd0OUme1wMA0GCSqGSIb3DQEBCwUAA4IBAQAToYYUsvQ6MlOQ+HHWnfax5L0u
M/IrExjcrSmnxAevnTem5+Hnz+AmlRHSF1/FfCWhi2XMSHBHXU52uhIt+kDLNhFh
D17Je0FydthW8roBsLZ+XsuHjBi8MXo41YAA19+cjW60/se3+kWxOHH1rEfV9ky/
K9kWufQPl3U3fbUzh0rWwriSDaPUTjHg/IeROhVe8gt/QeBfTU2KUqnln6K58PfC
kE2c2zCTQwpsfwi5gqvrdqWZc/A5qriv1l39zKU1icfTZzdeejcjtbCdFqajOqar
lMXwyl1bPuPiOHas7K9J9MSREQYql89gbN5V9gO//IHmM97eLt4npFSpu8Cp
-----END CERTIFICATE-----
subject=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown
issuer=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown
---
No client certificate CA names sent
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1451 bytes and written 236 bytes
Verification error: self signed certificate
---
New, TLSv1.0, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1
Cipher : ECDHE-RSA-AES256-SHA
Session-ID: 5D5E9882C4E8DD80427108B85C7649F0EC1AA87B9DAA0EA1FCE8C7063C83A61B
Session-ID-ctx:
Master-Key: 861677EDA9E19E3D8926889A9B0DE299593C7FCD49DB8A55EBF4D222800169E16CDB74DCE0EC392A3B491268FCCF5F07
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1566480514
Timeout : 7200 (sec)
Verify return code: 18 (self signed certificate)
Extended master secret: yes
---
which looks ok to me.
Configuration of the Broker:
listeners=SSL://ec2-ip.eu-central-1.compute.amazonaws.com:9093,PLAINTEXT://ec2-ip.eu-central-1.compute.amazonaws.com:9092
group.initial.rebalance.delay.ms=0
advertised.listeners=PLAINTEXT://ec2-ip.eu-central-1.compute.amazonaws.com:9092,SSL://ec2-ip.eu-central-1.compute.amazonaws.com:9093
ssl.endpoint.identification.algorithm=SSL
ssl.keystore.location=/kafka/kafka_2.12-2.2.0/config/certs/server.keystore.jks
ssl.keystore.password=Welcome01
ssl.key.password=Welcome01
ssl.truststore.location=/kafka/kafka_2.12-2.2.0/config/certs/server.truststore.jks
ssl.truststore.password=Welcome01
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type=JKS
ssl.truststore.type=JKS
Configuration of the Producer:
bootstrap.servers=localhost:9092
security.protocol=SSL
ssl.truststore.location=/kafka/kafka_2.12-2.2.0/config/certs/client.truststore.jks
ssl.truststore.password=Welcome01
logging.level.org.apache.kafka: DEBUG
log4j.rootLogger=DEBUG
Do you see some misconfiguration? How can I proceed to investigate further to get down to the issue?
Thanks to mazaneiche I was able to get the error that leads to the failed handshake:
kafka-producer-network-thread | console-producer, fatal error: 46: General SSLEngine problem
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
%% Invalidated: [Session-3, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
kafka-producer-network-thread | console-producer, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown
Why does this error occur? How can it be fixed?

Answer 1

Apparently not the issue here but I have been struggling with the same error for a while and in my case the kafka-console-producer.sh was failing with the similar errors:
$ kafka-console-producer.sh --broker-list 192.168.1.173:9093 --topic kafka-security-topic --producer.config ~/ssl/client-ssl-auth.properties
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
checking with debug
$ export KAFKA_OPTS="-Djavax.net.debug=ssl"
kafka-producer-network-thread | console-producer, fatal error: 46: General SSLEngine problem
java.security.cert.CertificateException: No subject alternative names present
%% Invalidated: [Session-1, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
kafka-producer-network-thread | console-producer, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown
[2021-07-13 16:21:53,909] ERROR [Producer clientId=console-producer] Connection to node -1 (/192.168.1.173:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
In my case turned out the reason is using IP instead of hostname when specifying broker-list parameter.
That worked:
$ kafka-console-producer.sh --broker-list red03:9093 --topic kafka-security-topic --producer.config ~/ssl/client-ssl-auth.properties