Batch Processing Architectures
What you'll learn: You'll understand the three primary architectural patterns for transforming data in batch pipelines: ETL, ELT, and medallion architectures.
Core Architectural Patterns
When designing batch processing systems, your choice of architecture determines where, when, and how data transformation happens as it flows from source to destination.
ETL (Extract-Transform-Load)
The classic pattern: extract data from sources, transform it in a staging area or processing cluster, then load the cleaned, enriched data into the target warehouse. Think of it like preprocessing ingredients before serving—you clean vegetables, season meat, and prepare everything before plating.
ETL makes sense when:
- Target systems have limited processing power
- You need strict data quality gates before storage
- Network bandwidth to the destination is expensive
ELT (Extract-Load-Transform)
The modern cloud approach: extract and immediately load raw data into a powerful data warehouse (like Snowflake or BigQuery), then transform using the warehouse's compute. It's like buying pre-washed salad—you leverage the supplier's capabilities instead of doing it yourself.
ELT excels when:
- Cloud warehouses provide elastic, cost-effective compute
- You want full historical raw data preserved
- Schema flexibility and iterative transformation are priorities
Medallion Architecture
A layered refinement pattern, often called bronze-silver-gold:
- Bronze: Raw ingested data (minimal transformation)
- Silver: Cleaned, validated, deduplicated data
- Gold: Business-level aggregates and features
This combines ETL and ELT principles into progressive quality zones, making data governance and lineage clearer.
Choosing Your Architecture
Your streaming experience prepared you for thinking about data flow. Batch architectures differ mainly in where compute happens and data quality boundaries—decisions that impact cost, flexibility, and time-to-insight.
Key Takeaway: ETL transforms before loading, ELT transforms after loading using warehouse compute, and medallion architecture creates progressive quality layers—each pattern optimizes for different storage costs, compute resources, and data governance needs.