In this lesson, you will learn how to transform your technical dbt project into a user-friendly, interactive documentation site that maps your entire data lineage.
One of the greatest challenges in data engineering is "black box" syndrome—where downstream users don't understand how a report was calculated. Since you are already modularizing your SQL with ref functions, dbt keeps track of the relationships between your tables automatically. We can leverage this metadata to generate a visual graph and documentation site that explains the journey of your data from raw sources to final business models.
Generating the Site
To build your documentation site, you use the following commands in your terminal:
# Generate the documentation JSON file from your project
dbt docs generate
# Host the documentation site locally
dbt docs serve
When you run dbt docs generate, the tool parses your project's schema.yml files (where you describe columns and business logic) and your compiled SQL files. It stitches these together into an interactive website.
What the Lineage Map Shows
Once the site is served (usually at localhost:8080), you will see a DAG (Directed Acyclic Graph). This is a visual map showing how each model flows into the next.
- Nodes: Represent your tables/models.
- Edges (Arrows): Represent the dependencies you created using the
{{ ref('model_name') }}function.
If a stakeholder asks, "Where does this metric come from?", you can simply point them to this graph. They can click on any node to read the descriptions you’ve written in your .yml files, view the underlying SQL code, and see exactly which upstream tables feed that specific calculation. By documenting your data lineage here, you bridge the gap between complex SQL transformations and the people who need to trust the data for decision-making.
Key Takeaway: Generating a documentation site allows you to provide stakeholders with an interactive, self-service visual map of your data lineage, ensuring total transparency in your transformation logic.