why does kafka broker store rebalance metadata for a consumer group in __consumer_offsets topic

While working on kafka streams we observed that Group Coordinator broker stores rebalance metadata in __consumer_offsets topic.


Since in our case we had 1200 StreamThreads, this metadata became huge (around 100 MB without compression and 25 MB after LZ4 compression), and since offsets.load.buffer.size parameter has default value of only 5MB, follower brokers of the corresponding __consumer_offsets topic partition were not able to read and this caused new GroupCoordinator to respond with error_code=16 (NotCoordinatorException) when there is failure of old GroupCoordinator.


Wondering why brokers store rebalance metadata for a consumer group in __consumer_offsets topic and is there a way to disable this ?

__consumer_offsets

__consumer_offsets

error_code=16 (NotCoordinatorException)

__consumer_offsets

1 Answer
1

The group metadata is store for fail-over reasons. If the GroupCoordinator dies and another broker takes over those groups, it needs this data to work properly. This is a consumer/broker feature and not specific to Kafka Streams.

For example, the new GroupCoordinator must know what member are in the group to apply corresponding timeouts (ie, session.timeout and max.poll.timeout).

session.timeout

max.poll.timeout

Thus, it's not possible to disable it, as it's essential for the cluster to work correctly. You change the corresponding config setting to give brokers large enough memory limits to handle the group metadata.

For plain KafkaConsumers group metadata does usually not grow as quickly and it's usually not a problem to handle large groups. However, KafkaStreams adds additional metadata compare to the client and it's metadata is particularly big. This is a know issue and I am sure it will be address in the future.

KafkaConsumers

KafkaStreams

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.