In this lesson, you will learn how to use preferred replica election to restore balance to your Kafka cluster by migrating partition leadership back to the original, designated brokers.
The Concept of the "Preferred" Replica
When you create a Kafka topic, the partition assignment logic automatically designates the first replica in the list as the "preferred leader." Ideally, this replica should be the one serving all read and write requests for that partition.
However, throughout the lifecycle of a cluster, brokers may fail or undergo maintenance. When a broker goes offline, Kafka triggers an automatic leader election, moving leadership to an In-Sync Replica (ISR) on a different broker. Even after the original broker comes back online, it often remains a follower, while the new, "unpreferred" broker continues to handle the load. This leads to an uneven distribution of resource usage across your cluster.
Restoring Balance
To rectify this, we use Preferred Replica Election. This process instructs the Kafka Controller to revert leadership back to the "preferred" replica defined during the partition creation. By doing this, you ensure that the cluster returns to your initial architectural plan, preventing certain nodes from becoming "hot spots" while others remain idle.
In practice, you trigger this via the command-line utility provided by Kafka:
kafka-preferred-replica-election --bootstrap-server localhost:9092
When this command runs, the controller identifies all partitions that are not currently led by their preferred replica and initiates a leader election to shift control back to them.
Why It Matters
If you don't perform this maintenance, your cluster’s resource utilization will drift further from your initial design. By migrating leadership back to the preferred replicas, you maintain predictable performance and ensure that your hardware utilization stays balanced as originally intended when the cluster was provisioned. Note that while this is a low-impact operation, it does involve a brief moment where the leadership changes, which is why it is best performed during periods of low traffic.
Key Takeaway: Preferred replica election is a maintenance operation used to restore cluster balance by forcing partition leadership back to the original broker assigned at topic creation, ensuring even workload distribution.