Filebeat Kafka client failing SSL handshake with AWS MSK - amazon-web-services

Question

We are trying to send logs using Filebeat to AWS MSK (Provisioned) using kafka configuration available. We're using mTLS authentication with the setup of Root CA and Intermediate CA with Vault. The intermediate CA is available in AWS PCA which is assigned to AWS MSK cluster which in turn issues the certs to the brokers on AWS MSK.
We are able to do mTLS authentication using Kafka client with the Admin setup (Kafka client with required certificates), however filebeat kafka is failing to do SSL handshake. All the certs provided in the handshake are valid.
Filebeat docker image: docker.elastic.co/beats/filebeat:8.5.1
Our Filebeat config looks like
filebeat.yaml
---
filebeat.shutdown_timeout: 0
fields_under_root: false
logging.level: debug
.
.
.
output.kafka:
hosts: 'XXXXMSK_BOOTSTRAP_HOSTSXXXX'
ssl.enabled: true
ssl.verification_mode: 'certificate'
ssl.certificate: /path/to/obained-cert.crt'
ssl.key: /path/to/obained-key.pki.key'
ssl.authorities: [/path/to/root/int/ca/combined-file/msk_ca_chain.pem']
topic: 'XXXXKAFKA_TOPICXXXX'
codec.format:
string: '{"timestamp": "%{[#timestamp]}", "message": %{[message]}, "host": %{[host]}}'
close_inactive: 10m
required_acks: 1
partition.round_robin:
reachable_only: false
keep-alive: 30000ms
obained-cert.crt
-----BEGIN CERTIFICATE-----
MIIXXXXX
#Obtained Cert#
-----END CERTIFICATE-----
obained-key.pki.key
-----BEGIN RSA PRIVATE KEY-----
MIIXXXXX
#Obtained private key#
-----END RSA PRIVATE KEY-----
msk_ca_chain.pem
-----BEGIN CERTIFICATE-----
MIIXXXXX
#Intermediate CA Cert#
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIXXXXX
#Root CA Cert#
-----END CERTIFICATE-----
The error in Filebeat log is:
{"log.level":"error","#timestamp":"2023-01-06T10:59:48.701Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":337},"message":"Kafka (topic=XXXXKAFKA_TOPICXXXX): kafka: client has run out of available brokers to talk to (Is your cluster reachable?)","service.name":"filebeat","ecs.version":"1.6.0"}
The error on AWS Cloudwatch for the brokers is:
[2023-01-06 12:48:07,716] INFO [SocketServer listenerType=ZK_BROKER, nodeId=3] Failed authentication with /INTERNAL_IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2023-01-06 12:48:08,004] INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Failed authentication with /INTERNAL_IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2023-01-06 12:48:08,016] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
I've enabled debug logs on Filebeat, but I'm not seeing any information regarding why SSL handshake has failed.
Is there any way we could see any debug logs on Filebeat Kafka or AWS MSK Broker side to identify why SSL handshake is failing? Also, any pointers around possible problems in filebeat.yaml config are also appreciated.
Thanks in advance!!!

Answer 1

Sorry for answering my own question. I have resolved this issue now by appending intermediate CA cert to the certificate obtained by root CA and then supplying only root CA in authorities section.
The changes I made:
Appended intermediate CA cert to the file /path/to/obained-cert.crt for the parameter ssl.certificate
Provided only the root certificate instead of chain of certificates for the parameter ssl.authorities i.e. ['/path/to/root/ca/msk_root_ca.pem']
This has done the trick!!
So, if you've intermidiate CA in the PKI, always append that to the obtained cert in order to carry out SSL handshake.
I hope this helps others.