In this lesson, you will learn how to leverage incremental cooperative rebalancing to eliminate the "stop-the-world" downtime traditionally associated with Kafka consumer group membership changes.
The Problem: Stop-the-World Rebalancing
In older versions of Kafka, when a consumer joined or left a group, the entire group would perform an "Eager Rebalance." During this process, all consumers were forced to stop processing, revoke their current partition assignments, and wait for the coordinator to redistribute everything. For large consumer fleets, this leads to significant processing latency and potential state cache misses, as consumers are forced to dump their local state and restart from scratch.
The Solution: Cooperative Sticky Rebalancing
Cooperative Sticky Rebalancing changes this flow by allowing the consumer group to continue processing partitions that are not being reassigned. Instead of a global stop, only the affected partitions are paused.
- Cooperative: Consumers "agree" to participate in multiple rounds of rebalancing, allowing the group to reach a stable state incrementally.
- Sticky: The assignment strategy aims to minimize partition movement. If a consumer remains in the group, it keeps its existing partitions, reducing the need to rebuild local state (like RocksDB instances or memory caches).
Implementing the Strategy
To enable this, you must configure your consumers to use the CooperativeStickyAssignor. By default, many clients are still set to use the standard RangeAssignor or RoundRobinAssignor.
// Configure the consumer to use the CooperativeStickyAssignor
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
CooperativeStickyAssignor.class.getName());
When a new consumer joins, the cluster only moves the specific partitions necessary to achieve balance. Your existing consumers continue to process their assigned work without interruption, and the incoming consumer incrementally takes over its share. This maintains state continuity, ensuring that your applications remain highly available even during frequent code deployments or auto-scaling events.
By reducing the duration of partitions being "unassigned," you ensure that the throughput of your stream processing pipeline remains stable, preventing spikes in lag that typically follow a standard, disruptive rebalance.
Key Takeaway: By adopting Cooperative Sticky Rebalancing, you minimize cluster-wide processing pauses and maximize state retention by migrating only the bare minimum number of partitions during rebalancing events.