Course contentsShow
← Data Science
Lesson 1823 of 2,14543. Data Pipelines and OrchestrationFree lesson

Pipeline Components: Sources, Transformations, Sinks

Identify the three core components of any data pipeline and their roles in the data flow.

Pipeline Components: Sources, Transformations, Sinks

What you'll learn: How every data pipeline is built from three essential building blocks that define the journey data takes from origin to destination.

The Three Core Components

Think of a data pipeline like a water system: water flows from a source (reservoir), passes through filters and treatment plants (transformations), and arrives at a destination (your faucet). Data pipelines work exactly the same way.

Every data pipeline, whether simple or complex, consists of these three fundamental components:

1. Sources

Sources are where your data originates. They're the entry points of your pipeline. Sources can be:

  • Databases (MySQL, PostgreSQL)
  • APIs delivering real-time data
  • Files (CSV, JSON, Parquet)
  • Streaming platforms (Kafka)
  • Third-party services (Salesforce, Google Analytics)

The source's job is to make data available for extraction. In ETL patterns, this is where the "Extract" happens.

2. Transformations

Transformations are the processing steps that clean, reshape, enrich, or aggregate your data. This is where the real work happens:

  • Cleaning null values
  • Joining multiple data sources
  • Calculating derived metrics
  • Filtering unwanted records
  • Standardizing formats

Transformations convert raw data into something useful for your business. This is the "Transform" in ETL or ELT.

3. Sinks

Sinks are the final destinations where transformed data lands. They're the outputs of your pipeline:

  • Data warehouses (Snowflake, BigQuery)
  • Databases for applications
  • Files for archival
  • Dashboards and visualization tools
  • Machine learning model inputs

The sink's job is to store or consume the processed data. This is the "Load" phase.

Key Takeaway: Every data pipeline consists of three components—sources (where data comes from), transformations (how data is processed), and sinks (where data goes)—creating a complete flow from raw input to usable output.