Course contentsShow
← System Design
Lesson 752 of 1,91920. Data Pipelines and ETLFree lesson

Kappa Architecture Overview

Learn how Kappa architecture simplifies Lambda by using only stream processing with replayable event logs.

Kappa Architecture Overview

You'll learn: How Kappa architecture simplifies data pipelines by eliminating the batch layer and using only stream processing.

What Is Kappa Architecture?

Kappa architecture is a streamlined alternative to Lambda architecture that ditches the complexity of maintaining two separate processing paths (batch and speed layers). Instead, everything is treated as a stream, processed by a single stream processing engine backed by a replayable event log.

The key insight: if your event log (like Kafka) can store data indefinitely and replay it from any point in time, you don't need a separate batch processing system. Need to reprocess historical data? Just replay the stream from the beginning.

How It Works

Kappa architecture has two main components:

  1. Stream Processing Layer: All data flows through a stream processing framework that handles both real-time and historical data
  2. Replayable Event Log: An immutable, persistent log (like Kafka) that stores all events and allows replay from any offset

When you need to fix a bug or deploy a new algorithm:

  • Deploy the new version alongside the old one
  • Replay the event stream from the start (or any checkpoint)
  • Once caught up, switch traffic to the new version
  • Decommission the old version

The Core Simplification

Think of it like having only one recipe book instead of two. Lambda architecture maintains separate "batch recipes" and "streaming recipes" for the same logic. Kappa says: "Why not write one streaming recipe that works for both fresh and historical data?"

This eliminates code duplication, reduces operational complexity, and makes the system easier to reason about.

Key Takeaway: Kappa architecture simplifies Lambda by using a single stream processing path with replayable logs, eliminating the need for separate batch and speed layers at the cost of requiring powerful stream processing infrastructure.