Partition Assignment and Rebalancing
What you'll learn: How Kafka dynamically distributes partitions among consumers and adjusts when group membership changes.
The Assignment Problem
When you have multiple consumers in a consumer group reading from a topic with multiple partitions, Kafka must decide which consumer reads from which partition. This is partition assignment. Each partition is assigned to exactly one consumer within a group, but one consumer can handle multiple partitions.
Think of it like a restaurant kitchen where multiple cooks (consumers) prepare orders from different stations (partitions). The head chef needs to assign each station to a cook to avoid two cooks working on the same orders.
When Rebalancing Happens
A rebalance occurs when partition assignments must be recalculated:
- A new consumer joins the group
- An existing consumer leaves (crashes, network timeout, or graceful shutdown)
- New partitions are added to a topic
- A consumer takes too long to process messages (heartbeat timeout)
How Rebalancing Works
During a rebalance, Kafka pauses consumption temporarily and redistributes partitions across all available consumers. The goal is to balance the load—ideally, each consumer gets roughly the same number of partitions.
For example, with 6 partitions and 2 consumers, each gets 3 partitions. If a third consumer joins, assignments shift to 2-2-2.
The Cost of Rebalancing
Rebalancing briefly stops message processing for the entire group. Frequent rebalances hurt throughput. This is why consumer stability matters—avoid frequent joins/leaves.
Key Takeaway: Kafka automatically reassigns partitions to consumers when group membership changes through rebalancing, ensuring balanced load distribution but at the cost of temporary processing pauses.