In this lesson, you will learn how Kafka ensures that your data remains safe and accessible even if an individual broker fails by replicating partition logs across the cluster.
The Role of Replication
While partitioning (Lesson 2) allows us to scale throughput, it creates a risk: if the broker holding a specific partition crashes, that data becomes unavailable. To solve this, Kafka implements a Replication Mechanism. For every partition, Kafka designates one broker as the Leader and one or more as Followers.
- The Leader: Handles all read and write requests for the partition. It ensures that every new message is written to its local log.
- Followers: They act as passive observers. They continuously pull data from the Leader’s log to keep their own local copies synchronized.
In-Sync Replicas (ISR)
A critical concept here is the ISR (In-Sync Replica). An ISR is a follower that is "caught up" with the leader—meaning it has successfully fetched all the latest messages. If a follower lags behind the leader due to network issues or high load, it may be dropped from the ISR set.
Why does this matter? When a producer sends a message with acks=all (Lesson 8), the leader waits until all brokers in the ISR have acknowledged the write before confirming success to the producer. This ensures that even if the leader fails, the data is guaranteed to exist on at least one other broker.
Visualizing the Cluster
If you have a replication factor of 3, your partition state looks like this:
Broker 1: Partition 0 [LEADER]
Broker 2: Partition 0 [FOLLOWER] (ISR)
Broker 3: Partition 0 [FOLLOWER] (ISR)
If Broker 1 crashes, the cluster uses its coordination mechanism (Lesson 5) to trigger a leader election. One of the followers from the ISR set is promoted to become the new Leader. Because the followers were kept up-to-date, the transition is seamless, and no data is lost during the switch.
Key Takeaway: Kafka achieves fault tolerance by maintaining multiple In-Sync Replicas of every partition, ensuring that a cluster can survive broker failures without data loss or downtime.