Course contentsShow
← System Design
Lesson 749 of 1,91920. Data Pipelines and ETLFree lesson

Lambda Architecture: Speed Layer

Explore how the speed layer provides low-latency real-time views by processing only recent data incrementally.

Lambda Architecture: Speed Layer

You'll learn how the speed layer delivers real-time insights by processing only the freshest data with low latency.

What Is the Speed Layer?

While the batch layer processes all historical data thoroughly (but slowly), the speed layer handles only the most recent data—streaming it incrementally to provide near-instant views. Think of it as an express checkout lane at a grocery store: it doesn't handle everyone, but it gets the latest customers through fast.

The speed layer compensates for the batch layer's high latency. Since batch jobs might run hourly or daily, you'd otherwise have stale data until the next batch completes. The speed layer bridges this gap by processing incoming events immediately, updating real-time views within seconds or milliseconds.

How It Works

  1. Ingests streaming data from message queues or event streams (Kafka, Kinesis)
  2. Processes incrementally using stream processing frameworks (Storm, Flink, Spark Streaming)
  3. Updates real-time views in fast-access stores (Redis, Cassandra, in-memory databases)
  4. Stores only recent results—old data is discarded once the batch layer catches up

The Tradeoff

The speed layer prioritizes speed over accuracy. It may use approximate algorithms or skip complex joins to maintain low latency. This is acceptable because the batch layer will eventually compute precise results.

For example: counting website visits in real-time might use probabilistic counters (HyperLogLog) for speed, while the batch layer provides exact counts later.

Integration Point

Queries merge results from both layers: final_result = batch_view + speed_view. When fresh batch results arrive, corresponding speed layer data is purged since it's now redundant.

Key Takeaway:
The speed layer trades precision for velocity, processing only recent events to deliver low-latency views that complement the batch layer's comprehensive but delayed results.