Course contentsShow
← System Design
Lesson 709 of 1,91919. Event Streaming PlatformsFree lesson

Partition Assignment and Rebalancing

Understand how Kafka assigns partitions to consumers and triggers rebalancing when consumers join or leave the group.

Partition Assignment and Rebalancing

What you'll learn: How Kafka dynamically distributes partitions among consumers and adjusts when group membership changes.

The Assignment Problem

When you have multiple consumers in a consumer group reading from a topic with multiple partitions, Kafka must decide which consumer reads from which partition. This is partition assignment. Each partition is assigned to exactly one consumer within a group, but one consumer can handle multiple partitions.

Think of it like a restaurant kitchen where multiple cooks (consumers) prepare orders from different stations (partitions). The head chef needs to assign each station to a cook to avoid two cooks working on the same orders.

When Rebalancing Happens

A rebalance occurs when partition assignments must be recalculated:

  • A new consumer joins the group
  • An existing consumer leaves (crashes, network timeout, or graceful shutdown)
  • New partitions are added to a topic
  • A consumer takes too long to process messages (heartbeat timeout)

How Rebalancing Works

During a rebalance, Kafka pauses consumption temporarily and redistributes partitions across all available consumers. The goal is to balance the load—ideally, each consumer gets roughly the same number of partitions.

For example, with 6 partitions and 2 consumers, each gets 3 partitions. If a third consumer joins, assignments shift to 2-2-2.

The Cost of Rebalancing

Rebalancing briefly stops message processing for the entire group. Frequent rebalances hurt throughput. This is why consumer stability matters—avoid frequent joins/leaves.

Key Takeaway: Kafka automatically reassigns partitions to consumers when group membership changes through rebalancing, ensuring balanced load distribution but at the cost of temporary processing pauses.