Rebalancing Protocol and Strategies
What you'll learn: How Kafka redistributes partitions among consumers when membership changes, and strategies that minimize disruption during these transitions.
What Triggers a Rebalance?
When consumers join or leave a consumer group, or when partition counts change, Kafka must reassign partitions to maintain balanced parallel consumption. Think of it like reassigning tables to waitstaff when someone arrives late or leaves early—you need to redistribute work fairly without dropping orders.
Common triggers include:
- A new consumer joins the group
- An existing consumer crashes or gracefully shuts down
- Consumer heartbeat timeout expires
- New partitions added to a topic
- Consumer subscription changes
Rebalancing Strategies
Eager Rebalancing (Stop-the-World)
The original protocol where all consumers stop processing, give up their partitions, and wait for a new assignment. It's simple but disruptive—like closing an entire restaurant to reassign tables.
- All consumers pause simultaneously
- Group coordinator computes new assignment
- Consumers resume with new partitions
- Downside: Complete processing halt during rebalance
Cooperative (Incremental) Rebalancing
A newer strategy where only affected partitions are reassigned while others continue processing. Most consumers keep working uninterrupted—like reassigning one table without stopping service at the others.
- Consumers retain unaffected partitions
- Only revoked partitions stop processing
- Partial assignment happens in stages
- Benefit: Minimal disruption, faster rebalancing
Minimizing Disruption
Use cooperative rebalancing when possible, tune session timeouts appropriately, commit offsets before shutdown, and monitor rebalance frequency as a health metric.
Key Takeaway: Rebalancing redistributes partitions when consumer group membership changes; cooperative strategies minimize disruption by only moving affected partitions rather than stopping all consumption.