Course contentsShow
← System Design
Lesson 717 of 1,91919. Event Streaming PlatformsFree lesson

Rebalancing Protocol and Strategies

Understanding partition rebalancing triggers, strategies (eager, cooperative), and minimizing disruption during reassignment.

Rebalancing Protocol and Strategies

What you'll learn: How Kafka redistributes partitions among consumers when membership changes, and strategies that minimize disruption during these transitions.

What Triggers a Rebalance?

When consumers join or leave a consumer group, or when partition counts change, Kafka must reassign partitions to maintain balanced parallel consumption. Think of it like reassigning tables to waitstaff when someone arrives late or leaves early—you need to redistribute work fairly without dropping orders.

Common triggers include:

  • A new consumer joins the group
  • An existing consumer crashes or gracefully shuts down
  • Consumer heartbeat timeout expires
  • New partitions added to a topic
  • Consumer subscription changes

Rebalancing Strategies

Eager Rebalancing (Stop-the-World)

The original protocol where all consumers stop processing, give up their partitions, and wait for a new assignment. It's simple but disruptive—like closing an entire restaurant to reassign tables.

  • All consumers pause simultaneously
  • Group coordinator computes new assignment
  • Consumers resume with new partitions
  • Downside: Complete processing halt during rebalance

Cooperative (Incremental) Rebalancing

A newer strategy where only affected partitions are reassigned while others continue processing. Most consumers keep working uninterrupted—like reassigning one table without stopping service at the others.

  • Consumers retain unaffected partitions
  • Only revoked partitions stop processing
  • Partial assignment happens in stages
  • Benefit: Minimal disruption, faster rebalancing

Minimizing Disruption

Use cooperative rebalancing when possible, tune session timeouts appropriately, commit offsets before shutdown, and monitor rebalance frequency as a health metric.

Key Takeaway: Rebalancing redistributes partitions when consumer group membership changes; cooperative strategies minimize disruption by only moving affected partitions rather than stopping all consumption.