How to consume messages from Kafka using cppkafka - c++

Question

Ok, so I'm a new with all the Kafka stuff and cppkafka library in particular. I'm trying to write my convenience wrapper on top of cppkafka. The producer side is quite straightforward and looks like it does what it supposed to do
cppkafka::Producer producer(cppkafka::Configuration{{"metadata.broker.list", "localhost:9092"}});
producer.produce(cppkafka::MessageBuilder(topic[0])
.partition((message_counter++) % partitions)
.payload(buffer.str()));
producer.flush();
Error handling and retries, here and below, were removed for brevity.
This code produces messages that I can see using whatever kafka UI.
The consumer side is somewhat more complicated
cppkafka::Consumer consumer(cppkafka::Configuration{
{"metadata.broker.list", "localhost:9092"},
{"group.id", "MyGroup"},
{"client.id", "MyClient"},
// {"auto.offset.reset", "smallest"},
// {"auto.offset.reset", "earliest"},
// {"enable.auto.offset.store", "false"},
{"enable.partition.eof", "false"},
{"enable.auto.commit", "false"}});
cppkafka::TopicPartitionList assignment;
consumer.set_assignment_callback([&, this](cppkafka::TopicPartitionList& partitions_) {
assignment = partitions_;
LOG_INFO(log, "Partitions assigned");
});
consumer.set_rebalance_error_callback([this](cppkafka::Error error) {
LOG_ERROR(log, "Rebalancing error. Reason: {}", error.to_string());
});
consumer.set_revocation_callback([&, this](const cppkafka::TopicPartitionList&) {
assignment.clear();
LOG_INFO(log, "Partitions revoked");
});
consumer.subscribe({topic});
auto start = std::chrono::high_resolution_clock::now();
while (some_condition)
{
auto subscriptions = consumer.get_subscription();
if (subscriptions.empty())
{
consumer.subscribe(topic);
}
cppkafka::Message msg = consumer.poll(100ms);
if (!msg)
{
if (!assignment.empty())
{
auto committed_offset = consumer.get_offsets_committed(consumer.get_assignment());
consumer.assign(committed_offset);
}
continue;
}
try
{
std::string_view payload_view(reinterpret_cast<const char *>(msg.get_payload().get_data()), msg.get_payload().get_size());
consumer.commit(msg);
}
....
}
Looks like this code not picking old uncommitted messages, the poll always returns null message. I have no idea whats going on here and why it leaves old messages in the queue. As one can see in the beginning of the last code snippet I've tried various consumer settings to no avail. What I'm missing here? Maybe I have to configure something on the Kafka side? Like, some topic settings? Or the problem with the consumer configuration?
One important (I think) detail. Messages may be added before any consumer connects to the Kafka. Maybe it is important?