Multi-Datacenter Replication
What You'll Learn
You'll learn how to replicate Kafka data across geographically distributed datacenters using MirrorMaker 2 for disaster recovery, data locality, and global distribution.
The Challenge of Geographic Distribution
When your Kafka deployment spans multiple regions or datacenters, you need a way to replicate event streams across these boundaries. Unlike the built-in replication we covered earlier (which keeps copies within a single cluster), multi-datacenter replication moves data between entirely separate Kafka clusters.
Why Replicate Across Datacenters?
Think of it like having backup offices in different cities:
- Disaster Recovery: If one datacenter fails, another has a complete copy of your data
- Geographic Distribution: Serve users from nearby datacenters for lower latency
- Data Locality: Comply with regulations requiring data processing in specific regions
- Data Aggregation: Collect streams from edge locations into a central analytics cluster
MirrorMaker 2: Cross-Cluster Replication
MirrorMaker 2 (MM2) is Kafka's built-in tool for replicating topics between clusters. It's essentially a special Kafka Connect application that:
- Consumes from topics in the source cluster
- Produces those events to the destination cluster
- Preserves partition counts and key-based ordering
- Handles offset translation so consumers can fail over seamlessly
- Replicates consumer group offsets and ACLs
Unlike message brokers that push data around, MM2 follows Kafka's pull model—the destination cluster actively pulls data from the source.
Replication Patterns
- Active-Passive: One primary cluster, replicas for failover only
- Active-Active: Multiple clusters all accept writes (requires careful design)
- Hub-and-Spoke: Regional clusters replicate to a central aggregation cluster
Key Takeaway: MirrorMaker 2 enables multi-datacenter Kafka deployments by replicating entire topics between independent clusters, supporting disaster recovery strategies and global event distribution patterns.