Why Kafka consumer freezes on Node failure while Producer stays unaffected? - google-cloud-platform

Question

I am new to Kafka and trying to create a Kafka cluster with 3 nodes for High Availability.
I have followed this guide and I have done the setup on Google Compute Instance (GCP VMs).
I tried creating topics with different --replication-factor.
Here is an example with replication-factor=3. I have tried the values 1 and 2 as well.
# With replication factor-3
bin/kafka-topics.sh --create \
--bootstrap-server xxx.xx.xx.xxx:9092,yy.yyy.yyy.yy:9092,zzz.zz.zz.zzz:9092 \
--replication-factor 3 --partitions 1 --topic sample-topic
This is what my consumer code looks like:
from kafka import KafkaConsumer
consumer = KafkaConsumer(
'sample-topic',
bootstrap_servers=[
'yyy.yy.yy.yyy'
],
client_id="sample-client-name",
auto_offset_reset="earliest",
group_id="sample-group-name")
for message in consumer:
print(message)
This is what my producer code looks like:
from time import sleep
from kafka import KafkaProducer
producer = KafkaProducer(
bootstrap_servers=[
'xxx.xx.xx.xxx', 'yyy.yy.yy.yyy', 'zzz.zz.zz.zzz'])
for i in range(1000):
message = str.encode("Message: " + str(i))
producer.send('sample-topic', value=message)
print("sent: {}".format(i))
sleep(5)
[xxx.xx.xx.xxx, yyy.yy.yy.yyy, zzz.zz.zz.zzz above are the IP addresses of the VMs]
Initially, the setup works pretty well. Now I start stopping the VMs (always keeping 2 VMs alive. Whenever I want to stop another VM I make sure at least two VMs are alive).
The producer code had the same bootstrap_servers as consumer code. I have tried keeping only 1 and 2 servers inside bootstrap_servers but it fails the same way for at least one of the VM going down( tried keeping all three VMs down one after another, making sure I'll have 2 VMs working).
Kafka consumers freeze because of stopping the VMs (only one VM down at a time, other two are brought up before stopping the targeted VM)
Is there any configuration I am missing? How to make the consumer stay intact like the producer?

Answer 1

If I read that linked post correctly, it doesn't mention that offsets.topic.replication.factor will have to be increased as well.
Otherwise, stopping the broker that holds the single replica will cause consumers fail to commit / lookup offsets