In this lesson, you will learn how Kafka distributes responsibilities between leader and follower replicas to ensure both high performance and data durability.
When we create a topic, Kafka distributes its partitions across various brokers. To prevent data loss if a broker fails, Kafka replicates each partition. Within this set of replicas, exactly one broker is designated as the Leader, while the others act as Followers.
The Role of the Leader
The leader is the "active" broker for a partition. Every single read and write request from producers and consumers must go through the leader. The leader maintains the ground truth for that partition, managing the sequence of offsets and ensuring that all data is correctly ordered. By funneling all traffic to the leader, Kafka ensures strict consistency, preventing the "split-brain" scenarios that arise when multiple nodes try to accept writes simultaneously.
The Role of Followers
Followers have a purely passive role in terms of request handling; they do not serve data to clients. Instead, they constantly pull data from the leader to stay in sync. A follower is considered "In-Sync" (ISR) if it is successfully keeping up with the leader’s latest offsets.
Consider this structural view of a partition with a replication factor of 3:
Partition 0:
- Broker 101 (Leader): Handles all reads/writes.
- Broker 102 (Follower): Pulls data from 101.
- Broker 103 (Follower): Pulls data from 101.
If Broker 101 crashes, Kafka’s controller automatically promotes either Broker 102 or 103 to be the new leader. Because the followers have been continuously replicating the log, they have a near-identical copy of the data, allowing the partition to resume operations with minimal downtime.
By offloading the replication work to background processes, the leader can focus entirely on serving incoming producer writes and consumer reads, keeping the system responsive even while maintaining fault tolerance.
Key Takeaway: The leader serves all read/write traffic to ensure data consistency, while followers replicate the partition log in the background to guarantee high availability in the event of a broker failure.