Integrating dbt with Airflow Orchestration

In this lesson, you will learn how to orchestrate your dbt transformations as a formal step within a larger Airflow DAG that includes upstream data ingestion.

The Role of Orchestration

While dbt handles the "T" (Transform) in ELT, it doesn't handle the "E" (Extract) or the "L" (Load). In a production environment, you need a way to ensure that your data is fully loaded into your warehouse before dbt begins processing it. Airflow acts as the "conductor" for your entire data pipeline, ensuring these dependencies are respected.

Using the BashOperator or DbtRunOperator

The most common way to integrate dbt into Airflow is by using the BashOperator to trigger the dbt CLI or by utilizing specialized operators like the DbtRunOperator (often provided by the airflow-dbt or cosmos packages).

Imagine you have a task that pulls raw sales data from an API into a staging table. You only want to run your dbt models once that load is successful. Your DAG structure would look like this:

from airflow import DAG
from airflow.operators.bash import BashOperator

with DAG('sales_pipeline', schedule_interval='@daily') as dag:
    # 1. Upstream Data Ingestion
    ingest_data = BashOperator(
        task_id='ingest_raw_data',
        bash_command='python scripts/extract_load_sales.py'
    )

    # 2. dbt Transformation
    run_dbt = BashOperator(
        task_id='dbt_run',
        bash_command='dbt run --select models/sales'
    )

    # Define the dependency
    ingest_data >> run_dbt

Why this matters

By placing dbt run inside an Airflow DAG, you gain two critical features:

1. Dependency Management: Airflow ensures dbt won't run if the ingestion fails, preventing "dirty" data from entering your refined models.
2. Retries and Alerting: If your dbt model fails due to a temporary network issue or database lock, Airflow can automatically retry the task and notify your team via Slack or email.

This integration transforms your dbt project from a standalone script into a resilient, production-grade data pipeline that handles the entire lifecycle of your data.

Key Takeaway: Airflow provides the workflow control necessary to ensure your dbt transformations only run once your upstream data ingestion is complete and successful.