Course contentsShow
← System Design
Lesson 700 of 1,91919. Event Streaming PlatformsFree lesson

Kafka Overview and Core Components

Understand Kafka's role as a distributed event streaming platform and identify its core components: brokers, topics, partitions.

Kafka Overview and Core Components

What You'll Learn

You'll understand Apache Kafka's role as a distributed event streaming platform and identify its three fundamental building blocks: brokers, topics, and partitions.

What Is Kafka?

Apache Kafka is a distributed event streaming platform designed to handle massive volumes of real-time events. Think of it as a durable, scalable commit log that sits at the heart of your data infrastructure, allowing multiple applications to publish and subscribe to streams of events reliably.

Unlike traditional message brokers that delete messages after consumption, Kafka retains events for a configurable period, enabling replay and multiple independent consumers to process the same event stream.

Core Components

Brokers

A broker is a Kafka server that stores data and serves client requests. Kafka runs as a cluster of one or more brokers, providing fault tolerance and horizontal scalability. Each broker handles reads, writes, and replication for the partitions it hosts.

Analogy: Think of brokers as the physical warehouses in a distribution network—each one stores inventory and handles shipments.

Topics

A topic is a logical channel or category for organizing events. Producers write events to topics; consumers read from topics. Topics are the primary abstraction for organizing your event streams by subject matter (e.g., "user-clicks," "payment-events").

Analogy: Topics are like newspaper sections (Sports, Business)—each groups related content.

Partitions

Each topic is divided into partitions—ordered, immutable sequences of events. Partitions enable parallelism: different partitions can live on different brokers and be consumed independently. Events within a single partition maintain strict ordering.

Analogy: If a topic is a highway, partitions are the individual lanes—traffic flows independently in each lane, enabling parallel throughput.

Key Takeaway: Kafka's architecture centers on brokers (servers), topics (logical channels), and partitions (ordered substreams), working together to deliver scalable, durable event streaming with replay capabilities.