Apache Beam Programming Model
You'll learn how Apache Beam provides a single API for both batch and stream processing that runs on multiple execution engines.
The Unified Pipeline Abstraction
Apache Beam (short for Batch + Stream) solves a fundamental problem: writing data pipelines twice—once for batch processing and once for streaming. Instead, Beam provides one programming model that works for both.
Think of Beam as writing a recipe that can be cooked in different kitchens (execution engines). You write your data transformation logic once, and Beam translates it to run on Apache Spark, Apache Flink, Google Cloud Dataflow, or other runners.
Core Programming Concepts
Beam pipelines work with two key abstractions:
- PCollections: Immutable datasets (bounded for batch, unbounded for streams)
- PTransforms: Operations that transform data (like
Map,Filter,GroupByKey)
You define transformations declaratively, and the runner optimizes and executes them. Windowing, triggers, and watermarks handle time-based operations the same way whether you're processing a static file or a live Kafka stream.
# Conceptual Beam pipeline (same code for batch or stream)
input | ReadFromSource()
| Filter(valid_records)
| Window.into(FixedWindows(60))
| GroupByKey()
| WriteToSink()
Runner Portability
The same pipeline code can execute on:
- Dataflow: Google's managed service
- Spark: Using your existing cluster
- Flink: For low-latency stream processing
You choose the runner at runtime based on scale, cost, and latency requirements—without rewriting logic.
Key Takeaway: Apache Beam lets you write data pipelines once using a unified batch/stream API, then execute them on different runners (Spark, Flink, Dataflow) without code changes, maximizing portability and minimizing duplication.