Transformation Patterns: Map, Filter, Aggregate
You'll learn: The three fundamental transformation patterns that form the building blocks of nearly every data pipeline transformation step.
The Core Idea
When building data pipelines, you'll apply transformations repeatedly. Rather than writing custom logic every time, three universal patterns handle the vast majority of transformations: Map (transform each record), Filter (keep or discard records), and Aggregate (combine multiple records into summaries).
Think of these as verbs in the language of data transformation:
- Map = "change every row" (e.g., convert temperatures from Celsius to Fahrenheit)
- Filter = "keep only rows that match criteria" (e.g., only customers from California)
- Aggregate = "summarize groups of rows" (e.g., total sales by region)
Why These Patterns Matter
These patterns are composable—you chain them together to build complex transformations. A typical pipeline stage might:
- Filter out incomplete records
- Map to standardize field formats (lowercase emails, trim whitespace)
- Aggregate to calculate daily totals
- Map again to enrich with metadata
Because these patterns are so universal, every major data processing framework (Spark, Dask, SQL, pandas) implements them, though sometimes under different names. Understanding the conceptual pattern helps you translate between tools.
Pattern Recognition in Pipeline Design
When designing transformations, ask yourself:
- Am I changing each record independently? → Map
- Am I deciding which records to keep? → Filter
- Am I combining multiple records into fewer? → Aggregate
This framing helps you write cleaner, more maintainable pipelines with explicit transformation intent.
Key Takeaway: Map, Filter, and Aggregate are the three foundational transformation patterns that can be composed to handle nearly any data reshaping, cleaning, or enrichment task in your pipelines.