Course contentsShow
← System Design
Lesson 716 of 1,91919. Event Streaming PlatformsFree lesson

Consumer Groups and Partition Assignment

How Kafka distributes partitions among consumers in a group for parallel processing and automatic rebalancing.

Consumer Groups and Partition Assignment

You'll learn: How Kafka distributes partitions among consumers in a group to enable parallel processing and automatic rebalancing.

The Core Idea

In Kafka, a consumer group is a logical grouping of consumers that coordinate to consume messages from a topic. Each partition is assigned to exactly one consumer within the group at any given time, but a consumer can handle multiple partitions. This design enables horizontal scalability—add more consumers to process partitions in parallel.

Think of it like a restaurant kitchen: if you have 6 dishes (partitions) to prepare and 3 chefs (consumers) in your team (consumer group), each chef gets assigned 2 dishes. If a chef leaves, the remaining chefs automatically divide the workload. If a new chef joins, everyone gets fewer dishes to manage.

How Assignment Works

When consumers in a group subscribe to a topic, Kafka's group coordinator distributes partitions among them. The assignment strategy determines which consumer gets which partitions:

  • Range: Assigns contiguous partition ranges per topic
  • Round-robin: Distributes partitions evenly across consumers
  • Sticky: Minimizes partition movement during rebalancing

If you have 6 partitions and 2 consumers, each consumer might get 3 partitions. Add a third consumer, and they'll rebalance to 2 partitions each.

Rebalancing

When consumers join or leave the group, or when partitions change, Kafka triggers a rebalance—redistributing partition assignments. During rebalancing, consumption briefly pauses as the group reorganizes.

Key Takeaway: Consumer groups enable parallel processing by assigning each partition to one consumer within the group, automatically rebalancing when membership changes to maintain even distribution.