Implementing Source Definitions for Data Lineage

In this lesson, you will learn how to bridge the gap between your raw data and your dbt transformation models by using sources to create a visual and logical map of your data’s journey.

The Role of Sources in Lineage

In dbt, lineage is the visual representation of how data flows from your source systems into your final analytical models. If you reference a raw table directly in your model using a hardcoded string like FROM raw_database.public.users, dbt doesn't "know" where that data originated. This creates a "black hole" in your DAG (Directed Acyclic Graph) where the lineage simply stops.

By defining a source in a YAML file, you explicitly tell dbt: "This table is an external dependency." Once defined, dbt connects your transformation models to these sources, allowing the lineage graph to show the full path from raw data to end-user reports.

Defining Your First Source

To define a source, you create a YAML file (usually in your models/ directory) and group your tables under a source block. Here is a simple example:

version: 2

sources:
  - name: raw_crm
    database: analytics_db
    schema: raw_salesforce
    tables:
      - name: customers
      - name: orders

Once this file exists, you stop hardcoding table names. Instead, you use the {{ source() }} function in your models:

-- models/stg_customers.sql
SELECT 
    id, 
    email 
FROM {{ source('raw_crm', 'customers') }}

Why This Matters

When you use {{ source('raw_crm', 'customers') }}, dbt automatically draws an arrow from raw_crm.customers to your stg_customers model in the lineage graph. This does three critical things:

1. Visibility: You can see exactly which models depend on which raw tables.
2. Impact Analysis: If a source column changes, you can instantly see every downstream model that might break.
3. Documentation: Your source tables now appear in your project documentation, making it easier for team members to identify the origin of their data.

Key Takeaway: By using source definitions, you turn hidden database dependencies into explicit, documented, and fully trackable components of your project's lineage.