Course contentsShow
← System Design
Lesson 705 of 1,91919. Event Streaming PlatformsFree lesson

Replication and Fault Tolerance

Understand how Kafka replicates partition data across brokers to provide durability and availability during failures.

Replication and Fault Tolerance

What You'll Learn

You'll understand how Kafka ensures your data survives broker failures by replicating partition data across multiple machines.

The Problem: Machines Fail

In any distributed system, individual machines (brokers) will eventually fail due to hardware issues, network problems, or software crashes. Without protection, losing a broker means losing all the partition data it held—and your streaming platform goes down.

How Kafka Replicates Data

When you create a topic in Kafka, you specify a replication factor (commonly 3). This means each partition gets copied to multiple brokers:

  • Leader replica: One broker holds the primary copy and handles all reads and writes for that partition
  • Follower replicas: Other brokers maintain synchronized copies by continuously pulling updates from the leader

Think of it like having backup copies of an important document stored in different filing cabinets. If one cabinet burns down, you still have the others.

Fault Tolerance in Action

When a broker hosting a leader fails, Kafka automatically promotes one of the in-sync follower replicas to become the new leader. Producers and consumers seamlessly switch to the new leader—often within seconds.

The replication factor determines how many failures you can tolerate. With a replication factor of 3, you can lose up to 2 brokers and still maintain availability (though you'd want to restore capacity quickly).

In-Sync Replicas (ISRs)

Not all followers are equal. Kafka tracks which replicas are "in-sync"—meaning they've successfully replicated recent data. Only in-sync replicas can become leaders, ensuring you don't lose committed messages during failover.

Key Takeaway: Kafka's replication mechanism creates multiple synchronized copies of each partition across brokers, enabling automatic failover and ensuring your event streams remain available even when individual machines fail.