Brokers and Cluster Architecture
You'll learn: How Kafka brokers work together as a cluster to distribute data and maintain system reliability.
What Is a Kafka Broker?
A broker is a single Kafka server that stores data and serves client requests. Think of it like one librarian in a library system. But Kafka runs multiple brokers together as a cluster — like having multiple librarians managing different sections of the library simultaneously.
Each broker:
- Stores partitions (the actual data segments from topics)
- Handles producer writes to its assigned partitions
- Serves consumer reads from its partitions
- Communicates with other brokers to maintain cluster state
How Clusters Distribute Work
When you create a topic with multiple partitions, Kafka distributes those partitions across different brokers. This means:
- No single broker holds all the data
- Load is spread across the cluster
- If one broker fails, others continue serving their partitions
For example, if you have 6 partitions and 3 brokers, each broker typically manages 2 partitions (plus replicas for fault tolerance).
Coordination: ZooKeeper vs KRaft
Brokers need to agree on critical metadata: which broker leads each partition, which brokers are alive, and what the cluster topology looks like.
Historically, ZooKeeper (an external coordination service) managed this metadata — tracking broker membership, partition leadership, and configuration.
KRaft (Kafka Raft) is the newer approach that eliminates ZooKeeper by implementing consensus directly within Kafka itself, using a Raft-based protocol among controller brokers.
The Controller
One broker acts as the cluster controller, elected through consensus. It orchestrates partition leadership assignments and responds to broker failures by reassigning leadership to healthy replicas.
Key Takeaway: Kafka brokers form clusters that distribute partitions across multiple servers, coordinating through ZooKeeper or KRaft to manage metadata, leadership, and cluster state for high availability and scalability.