In this lesson, you will learn how to bring a failed Kafka broker back online and ensure it safely synchronizes missing data without compromising the cluster's integrity.
When a broker crashes, it stops serving requests and misses new writes. During this time, the cluster relies on the remaining replicas in the In-Sync Replica (ISR) set to handle traffic. Once the failed broker comes back online, it is essentially "stale." It must rejoin the cluster and catch up on the logs it missed while it was offline.
The Recovery Process
- Restart the Broker: Start the Kafka process on the host machine. The broker will initialize its logs and connect to the Zookeeper/KRaft controller.
- Joining the Cluster: The broker registers itself as active. However, it cannot immediately serve as a leader for partitions. It must first synchronize with the current leader to become an ISR again.
- Synchronization: The broker fetches the missing log segments from the current partition leader. During this period, the broker is technically a follower, but it is not yet "in-sync."
- Returning to ISR: Once the replica’s log end offset is sufficiently close to the leader’s, the controller updates the metadata to include this broker back in the ISR.
You can monitor the recovery progress by checking the UnderReplicatedPartitions metric via JMX:
# Example command to check for under-replicated partitions
kafka-log-dirs --bootstrap-server localhost:9092 --describe --broker-list 1
If a broker stays in a "recovering" state for too long, it may indicate that the replica lag is too high to bridge, often caused by heavy producer load or slow disk I/O. It is crucial to ensure your min.insync.replicas configuration is tuned to protect the cluster during this recovery phase; if too many brokers fail simultaneously, the system will stop accepting writes to avoid data loss. By managing these parameters, you ensure that as a broker rejoins, it gradually becomes a reliable member of the cluster once again.
Key Takeaway: Broker recovery is an automated synchronization process where a returning follower catches up on missing log segments from the leader until it regains its status as an In-Sync Replica.