In this lesson, you will learn how to create YAML source files to document your raw upstream tables, ensuring your data lineage remains clear and traceable as it flows into dbt.
Why Define Sources?
In a dbt project, your "raw" data—the tables that exist in your warehouse before dbt performs any transformations—are called Sources. While you could hardcode table names directly into your SQL models, that makes it nearly impossible to track where your data originated or to manage changes when a source schema evolves.
By defining sources in a YAML file, you create a "Source of Truth" for your upstream data. This allows dbt to understand the relationships between your raw tables and your transformed models, enabling features like freshness checks and full-project lineage graphs.
Creating a sources.yml File
To define a source, you create a file inside your models/ directory (typically named src_crm.yml or similar) that maps to your database schema. Here is a simple example:
version: 2
sources:
- name: raw_data
database: analytics_db
schema: sales_raw
tables:
- name: customers
- name: orders
Once defined, you don’t reference the raw table using its full database name. Instead, you use the {{ source() }} Jinja function in your dbt models:
-- models/stg_orders.sql
select
order_id,
customer_id
from {{ source('raw_data', 'orders') }}
The Benefits of Traceability
When you use the source() function, dbt automatically links the model to that source in its internal graph. If you run dbt docs generate, you can visualize the entire journey of your data—from the raw tables in your warehouse all the way to your final business intelligence dashboards. This visibility is critical for debugging; if a downstream report breaks, you can instantly see if the upstream orders table failed to load or changed its schema.
Key Takeaway: Defining source schemas in YAML centralizes your raw data definitions, enabling end-to-end lineage tracking and simplifying how you reference upstream data throughout your pipeline.