BiteSizedChunks.comLearn one small thing at a time.
Course contentsShow
← Data Engineering
Lesson 530 of 8258. Real-time Data Streaming with KafkaFree lesson

Defining In-Sync Replicas and ISR Sets

Identify the criteria for a replica to be considered in-sync and how the ISR set maintains the cluster's consistency guarantees.

In this lesson, you will learn how Kafka determines which replicas are reliable enough to participate in data consensus and maintain cluster consistency.

The Criteria for Synchronization

In Kafka’s replication model, the Leader handles all reads and writes, while Followers simply copy data. However, not every Follower is "in-sync" at all times. For a replica to be considered an In-Sync Replica (ISR), it must meet two critical criteria:

  1. Liveness: The replica must send regular "heartbeat" requests to the Leader. If it fails to communicate within a specific window (defined by replica.lag.time.max.ms), the broker assumes the follower has crashed or is too slow.
  2. Catch-up Progress: The replica must not fall too far behind the Leader’s log. If the number of missing messages exceeds a threshold, it is removed from the ISR set to ensure that a partition’s progress isn't stalled by a lagging follower.

Maintaining Consistency via the ISR Set

The ISR set is a dynamic list of replicas that the Leader "trusts." When a producer sends a message with acks=all, the Leader only considers the write successful once all brokers currently in the ISR set have acknowledged the data.

Consider a partition with three replicas: [Broker 1 (Leader), Broker 2, Broker 3]. If Broker 3 experiences a network hiccup and stops fetching data, the Leader will remove it from the ISR set. The cluster now operates as [1, 2]. Because the ISR set is smaller, the Leader can continue to commit messages as long as Broker 2 keeps up. If Broker 1 fails, Kafka will only promote a replica from the current ISR set to be the new Leader. This guarantees that no committed data is lost, as any replica in the ISR set is guaranteed to have the full sequence of acknowledged messages.

// Visualization of the ISR lifecycle
ISR Set: [Broker 1, Broker 2, Broker 3]
1. Broker 3 hangs -> Removed from ISR
ISR Set: [Broker 1, Broker 2]
2. Broker 3 recovers -> Re-joins ISR once caught up

Key Takeaway: The ISR set acts as a safety buffer that balances data durability with availability, ensuring only fully caught-up replicas are eligible to take over as Leaders or satisfy acknowledgment requirements.