In this lesson, you will learn how to fine-tune replica synchronization thresholds to maintain cluster stability while preventing slow followers from dragging down your throughput.
The Mechanism of the ISR
In Kafka, the leader replica handles all reads and writes, while follower replicas pull data from the leader to stay synchronized. A follower is considered an "In-Sync Replica" (ISR) as long as it is caught up with the leader. If a follower falls too far behind, the leader deems it "out-of-sync" and removes it from the ISR list. This prevents the leader from waiting indefinitely for a slow follower to acknowledge messages, which would otherwise stall the entire partition’s production pipeline.
Managing the Thresholds
To control this behavior, we focus on two critical broker configurations:
replica.lag.time.max.ms: This defines the maximum time a follower can go without sending a fetch request to the leader before it is kicked out of the ISR. If your network is jittery, you might increase this to avoid constant "churn" where followers are repeatedly dropped and re-added.replica.lag.max.messages: (Note: This is deprecated in newer versions in favor of time-based tracking, but remains relevant for understanding the concept). This historically set the maximum number of messages a follower could lag behind the leader before being demoted.
Impact on Cluster Stability
If you set your lag thresholds too aggressively (e.g., a very short time window), you create a "flapping" scenario where followers are constantly ejected and rejoined to the ISR. This triggers metadata updates across the entire cluster, increasing CPU and Zookeeper/KRaft load. Conversely, if you set them too loosely, a sluggish follower will stay in the ISR longer than it should, which can force your producer to wait longer for acknowledgments (if acks=all is enabled) or allow data loss if the leader crashes and the partition promotes an outdated, slow follower.
# Example configuration in server.properties
# Allow followers 30 seconds to catch up before removal
replica.lag.time.max.ms=30000
By balancing these settings, you ensure that only healthy, responsive followers participate in the ISR, keeping your data pipeline fluid and resilient.
Key Takeaway: Configuring replica lag thresholds is a balancing act between protecting the cluster from slow nodes and preventing unnecessary, resource-heavy rebalancing cycles.