In this lesson, you will learn how to leverage full-refresh execution modes to ensure your data models are rebuilt from scratch when schema drift or corruption threatens your pipeline's integrity.
Why Full Refresh Matters
In previous lessons, we discussed incremental models, which optimize performance by only processing new records. However, incremental logic assumes your source structure stays constant. If you rename a column, change a data type, or need to correct a historical bug that affected months of data, simply appending new rows won't fix the existing state. A full-refresh is the "nuclear option" that drops the existing table and recreates it using the current model logic.
Controlling the Refresh Behavior
In dbt, you trigger a full refresh during execution to force the database to ignore the current state and perform a clean rebuild. You do this by adding the --full-refresh flag to your command:
dbt run --select my_model_name --full-refresh
When you run this, dbt drops the target table and runs the CREATE TABLE AS statement anew. This ensures that any logic changes—such as a new transformation rule or a corrected calculation—are applied to the entire dataset, maintaining absolute consistency across your historical and current data.
When to Use It
- Development Cycles: During initial development, you often change your SQL logic frequently. Running a full refresh ensures your model reflects the most recent version of your code.
- Emergency Recovery: If a source system update breaks downstream data (e.g., changing a date format), incremental updates will continue to fail or populate incorrect data. A full refresh allows you to reprocess the entire history once the fix is applied.
- Data Integrity Audits: When you suspect underlying drift in your warehouse state, a full refresh serves as a baseline reset to guarantee that the final model is an accurate representation of your source transformations.
By mastering this control, you treat your data models as reproducible artifacts rather than static entities that become brittle over time.
Key Takeaway: The --full-refresh flag is your primary tool for resetting model state, ensuring that structural changes and historical corrections are accurately propagated across your entire dataset.