Feature Materialization and Backfilling
What You'll Learn
You'll learn how to pre-compute features for historical data and keep online feature stores updated with fresh values—essential for maintaining consistency between training and serving.
The Core Problem
When you've defined features in your feature store (lesson 2885), you need to actually compute those features and store them. This process has two distinct flavors:
Materialization is the ongoing process of computing feature values from raw data and writing them to your feature store—both offline (for training) and online (for serving). Think of it like a factory continuously producing products and stocking shelves.
Backfilling is computing features for historical data, typically when you:
- Add a new feature definition after your system is already running
- Need to retrain a model with updated features
- Fix a bug in feature computation logic
Why This Matters
Without materialization, your feature definitions are just recipes—you need the actual cooked meals ready to serve. The offline store needs historical features for model training with point-in-time correctness (lesson 2886), while the online store needs fresh features for real-time predictions.
The Workflow
- Define: Register feature definitions in your feature store
- Backfill: Compute features for all historical data (e.g., past 2 years)
- Materialize: Schedule regular jobs to compute new features as data arrives
- Sync: Push computed features to both offline and online stores
The tricky part? Ensuring backfilled historical features use the exact same logic as ongoing materialization—otherwise, training-serving skew creeps back in.
Key Takeaway: Feature materialization keeps your feature stores stocked with fresh, ready-to-use values, while backfilling ensures historical completeness—both are critical for maintaining the consistency promise of feature stores.