BiteSizedChunks.comLearn one small thing at a time.
Course contentsShow
← Data Engineering
Lesson 534 of 8258. Real-time Data Streaming with KafkaFree lesson

Handling Unclean Leader Elections

Evaluate the trade-offs of allowing out-of-sync replicas to become leaders when no in-sync replicas are available during a disaster.

In this lesson, you will learn how to evaluate the critical trade-offs between system availability and data consistency when an "unclean" leader election occurs in your Kafka cluster.

The Dilemma of Unclean Elections

Normally, when a partition leader fails, the Kafka controller triggers an election to promote one of the In-Sync Replicas (ISR) to become the new leader. Because ISRs are perfectly up-to-date, this transition is seamless and ensures no data is lost.

However, in a catastrophic disaster, it is possible for all members of the ISR set to go offline simultaneously. At this point, your partition has no "in-sync" candidates. You are left with two choices:

  1. Maintain Consistency: Keep the partition offline until an original ISR member recovers. This guarantees no data loss, but results in downtime.
  2. Prioritize Availability: Allow an "out-of-sync" replica (a follower that missed some data) to become the leader. This restores availability immediately, but results in the permanent loss of the data that the new leader never received.

Configuring the Trade-off

You control this behavior using the unclean.leader.election.enable setting at the topic level.

  • unclean.leader.election.enable = false (Default): The partition stays offline if no ISRs are available. This is preferred for financial or transactional data where correctness is non-negotiable.
  • unclean.leader.election.enable = true: The system will elect an out-of-sync replica. This is often preferred for high-volume telemetry or sensor data where system uptime is more valuable than maintaining every single data point.
# Example: Enabling unclean election for a high-availability topic
kafka-configs --bootstrap-server localhost:9092 --alter \
  --entity-type topics --entity-name sensor-data \
  --add-config unclean.leader.election.enable=true

When you allow an unclean election, the new leader starts from whatever offset it last successfully replicated. Any writes acknowledged by the original leader but not yet copied to this follower vanish from the perspective of the consumers, potentially leading to discrepancies in your downstream analytics.

Key Takeaway: An unclean leader election forces a trade-off between system uptime and data durability; choose false for strict consistency and true only when service availability outweighs the cost of data loss.