Consumer Groups and Partition Assignment
You'll learn: How Kafka distributes partitions among consumers in a group to enable parallel processing and automatic rebalancing.
The Core Idea
In Kafka, a consumer group is a logical grouping of consumers that coordinate to consume messages from a topic. Each partition is assigned to exactly one consumer within the group at any given time, but a consumer can handle multiple partitions. This design enables horizontal scalability—add more consumers to process partitions in parallel.
Think of it like a restaurant kitchen: if you have 6 dishes (partitions) to prepare and 3 chefs (consumers) in your team (consumer group), each chef gets assigned 2 dishes. If a chef leaves, the remaining chefs automatically divide the workload. If a new chef joins, everyone gets fewer dishes to manage.
How Assignment Works
When consumers in a group subscribe to a topic, Kafka's group coordinator distributes partitions among them. The assignment strategy determines which consumer gets which partitions:
- Range: Assigns contiguous partition ranges per topic
- Round-robin: Distributes partitions evenly across consumers
- Sticky: Minimizes partition movement during rebalancing
If you have 6 partitions and 2 consumers, each consumer might get 3 partitions. Add a third consumer, and they'll rebalance to 2 partitions each.
Rebalancing
When consumers join or leave the group, or when partitions change, Kafka triggers a rebalance—redistributing partition assignments. During rebalancing, consumption briefly pauses as the group reorganizes.
Key Takeaway: Consumer groups enable parallel processing by assigning each partition to one consumer within the group, automatically rebalancing when membership changes to maintain even distribution.