cluster computing – Kafka Consumer fails to rejoin its group in a single Kafka broker

I’ve been using Apache Kafka 3.3.2 for a while now as a single broker and not a cluster (I know, this started as a testing stuff) and now I bumped into some consumer error which I couldn’t solve by adjusting configs.

The consumer is a single member in its consumer group and the topic has 1 partition. At a certain point, it experiences a heartbeat expiration and doesn’t manage to rejoin the group, hence it is down (“transitioned to Dead”) without any exception thrown.

Kafka logs.log file contains the following logs:

[2024-09-17 13:19:48,500] INFO [GroupCoordinator 0]: Member rdkafka-5fc6100d-35b1-4475-9f6e-294983951b48 in group transactions-new has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2024-09-17 13:19:48,500] INFO [GroupCoordinator 0]: Preparing to rebalance group transactions-new in state PreparingRebalance with old generation 1 (__consumer_offsets-42) (reason: removing member rdkafka-5fc6100d-35b1-4475-9f6e-294983951b48 on heartbeat expiration) (kafka.coordinator.group.GroupCoordinator)
[2024-09-17 13:19:48,500] INFO [GroupCoordinator 0]: Group transactions-new with generation 2 is now empty (__consumer_offsets-42) (kafka.coordinator.group.GroupCoordinator)
[2024-09-17 13:23:03,208] INFO [GroupMetadataManager brokerId=0] Group transactions-new transitioned to Dead in generation 2 (kafka.coordinator.group.GroupMetadataManager)

I’ve tried to increase session.timeout.ms and max.poll.interval.ms configs and also to decrease the timeout param in the .poll() method, but it didn’t help. I’ve checked the CPU and memory load of the Ubuntu server that hosts the Kafka broker and saw nothing alerting, but I still think this happens because of my basic setup of a single broker in the cluster.

Would you scale it into a cluster? Or alternatively partition the topic, and add consumers?

Any help would be appreciated.