Enforcing Data Freshness and Source Latency

In this lesson, you will learn how to configure dbt source freshness checks to monitor the timeliness of your incoming data and proactively alert stakeholders to pipeline delays.

Why Freshness Matters

In batch ELT pipelines, your transformation models are only as good as the raw data they process. If your ingestion tool (like Fivetran or Airbyte) fails or slows down, your dbt models might finish running successfully, but they will be calculating insights based on stale data. Enforcing "Source Freshness" allows you to define a "Contract of Timeliness" for your data sources.

Configuring Freshness

To monitor your data, you define freshness requirements directly in your sources.yml file. This tells dbt exactly how "old" your data is allowed to be before it flags a warning or an error.

For example, if you expect your orders table to receive new records at least every 24 hours, you configure it like this:

sources:
  - name: raw_data
    tables:
      - name: orders
        freshness:
          warn_after: {count: 24, period: hour}
          error_after: {count: 48, period: hour}
        loaded_at_field: updated_at

• loaded_at_field: This is the column in your raw table (usually a timestamp) that dbt uses to calculate the latency.
• warn_after: If the most recent timestamp in the table is older than this threshold, dbt will report a warning.
• error_after: If the data is older than this, dbt will report an error, which is useful for triggering alerts in your orchestration layer.

Executing the Check

Once configured, you don't run freshness with a standard dbt run. Instead, you use the dbt source freshness command. This command inspects the loaded_at_field, compares it against the current time, and outputs the result.

When you integrate this into your CI/CD or scheduling tool, you can ensure that if the data is stale, the downstream pipeline execution is halted or a notification is sent to the engineering team. This prevents users from making business decisions based on yesterday’s stagnant data.

Key Takeaway: By defining freshness thresholds in your source configuration, you treat data latency as a quality metric, ensuring stakeholders are notified immediately when ingestion pipelines fall behind schedule.