Course contentsShow
← System Design
Lesson 772 of 1,91920. Data Pipelines and ETLFree lesson

Apache Beam Programming Model

Understand Beam's unified batch and stream API with runners for Spark, Flink, and Dataflow.

Apache Beam Programming Model

You'll learn how Apache Beam provides a single API for both batch and stream processing that runs on multiple execution engines.

The Unified Pipeline Abstraction

Apache Beam (short for Batch + Stream) solves a fundamental problem: writing data pipelines twice—once for batch processing and once for streaming. Instead, Beam provides one programming model that works for both.

Think of Beam as writing a recipe that can be cooked in different kitchens (execution engines). You write your data transformation logic once, and Beam translates it to run on Apache Spark, Apache Flink, Google Cloud Dataflow, or other runners.

Core Programming Concepts

Beam pipelines work with two key abstractions:

  • PCollections: Immutable datasets (bounded for batch, unbounded for streams)
  • PTransforms: Operations that transform data (like Map, Filter, GroupByKey)

You define transformations declaratively, and the runner optimizes and executes them. Windowing, triggers, and watermarks handle time-based operations the same way whether you're processing a static file or a live Kafka stream.

# Conceptual Beam pipeline (same code for batch or stream)
input | ReadFromSource()
      | Filter(valid_records)
      | Window.into(FixedWindows(60))
      | GroupByKey()
      | WriteToSink()

Runner Portability

The same pipeline code can execute on:

  • Dataflow: Google's managed service
  • Spark: Using your existing cluster
  • Flink: For low-latency stream processing

You choose the runner at runtime based on scale, cost, and latency requirements—without rewriting logic.

Key Takeaway: Apache Beam lets you write data pipelines once using a unified batch/stream API, then execute them on different runners (Spark, Flink, Dataflow) without code changes, maximizing portability and minimizing duplication.