Course contentsShow
← System Design
Lesson 748 of 1,91920. Data Pipelines and ETLFree lesson

Lambda Architecture: Batch Layer

Learn how the batch layer precomputes batch views from immutable, append-only master datasets for accuracy.

Lambda Architecture: Batch Layer

What you'll learn: How the batch layer in Lambda Architecture precomputes accurate views from immutable master datasets.


The Foundation of Truth

In Lambda Architecture, the batch layer is your source of absolute truth. Think of it like an accounting department that closes the books at month-end—it takes all the raw data (the immutable master dataset) and systematically computes comprehensive, accurate views, even if it takes hours or days.

The batch layer operates on an append-only master dataset that stores every event that ever happened. Nothing is deleted or modified—events are only added. This immutability guarantees you can always recompute views from scratch if needed.

How It Works

The batch layer runs periodic batch jobs (using frameworks like MapReduce, Spark, or others you've studied) that:

  1. Read the entire master dataset (or relevant portions)
  2. Compute batch views (aggregations, reports, analytics)
  3. Replace old views with newly computed ones

For example, if you're tracking product sales, the batch layer might run nightly jobs that:

  • Calculate total sales per product for all time
  • Compute customer lifetime values
  • Generate inventory reports

The results are precomputed views stored in a serving layer database, ready for fast queries. While these views might be hours old, they're guaranteed to be accurate and complete.

The Accuracy-Latency Tradeoff

The batch layer sacrifices real-time freshness for correctness and completeness. It handles late-arriving data naturally—when you recompute tomorrow, everything is included. This is the "eventually accurate" guarantee that makes Lambda Architecture resilient to data quality issues.

Key Takeaway: The batch layer precomputes accurate, comprehensive views from an immutable master dataset using periodic batch jobs, trading latency for guaranteed correctness.