Pipeline Components: Sources, Transformations, Sinks
What you'll learn: How every data pipeline is built from three essential building blocks that define the journey data takes from origin to destination.
The Three Core Components
Think of a data pipeline like a water system: water flows from a source (reservoir), passes through filters and treatment plants (transformations), and arrives at a destination (your faucet). Data pipelines work exactly the same way.
Every data pipeline, whether simple or complex, consists of these three fundamental components:
1. Sources
Sources are where your data originates. They're the entry points of your pipeline. Sources can be:
- Databases (MySQL, PostgreSQL)
- APIs delivering real-time data
- Files (CSV, JSON, Parquet)
- Streaming platforms (Kafka)
- Third-party services (Salesforce, Google Analytics)
The source's job is to make data available for extraction. In ETL patterns, this is where the "Extract" happens.
2. Transformations
Transformations are the processing steps that clean, reshape, enrich, or aggregate your data. This is where the real work happens:
- Cleaning null values
- Joining multiple data sources
- Calculating derived metrics
- Filtering unwanted records
- Standardizing formats
Transformations convert raw data into something useful for your business. This is the "Transform" in ETL or ELT.
3. Sinks
Sinks are the final destinations where transformed data lands. They're the outputs of your pipeline:
- Data warehouses (Snowflake, BigQuery)
- Databases for applications
- Files for archival
- Dashboards and visualization tools
- Machine learning model inputs
The sink's job is to store or consume the processed data. This is the "Load" phase.
Key Takeaway: Every data pipeline consists of three components—sources (where data comes from), transformations (how data is processed), and sinks (where data goes)—creating a complete flow from raw input to usable output.