Handling Irregular Sampling and Missing Data in Foundation Models
You'll learn: How modern foundation models for time series manage non-uniform timestamps and data gaps during inference without breaking their predictions.
The Challenge of Real-World Time Series
Unlike the clean, evenly-spaced data we often train with, real-world time series are messy. Medical sensors might record every few minutes or hours depending on patient state. Financial data might be missing during weekends. IoT devices fail intermittently. Foundation models need to handle these irregularities gracefully.
How Foundation Models Adapt
Modern foundation models like TimeGPT, Lag-Llama, and Chronos use several strategies:
1. Temporal Embeddings
Instead of assuming fixed intervals, these models encode the actual time difference between observations. Think of it like GPS coordinates rather than grid squares—you track exact positions, not just "next step."
2. Masking Mechanisms
Models use attention masks to ignore missing values during computation. The self-attention layers (from Transformers) simply don't compute relationships with missing timesteps, like skipping empty seats when calculating average room temperature.
3. Time-Aware Tokenization
Chronos tokenizes time series with explicit time metadata. Each token carries both its value and timestamp, so the model knows "this reading came 3 hours after the last one, not 1 hour."
4. Interpolation During Encoding
Some models pre-fill gaps using simple strategies (forward-fill, mean imputation) before feeding data to the foundation model, leveraging the model's robustness to minor imputation errors.
5. Learned Positional Encodings
Unlike fixed positional encodings, these models learn to weight observations based on their temporal distance, automatically adjusting for irregular spacing.
Key Takeaway: Foundation models handle irregular sampling by explicitly encoding temporal distances, using attention masks for missing data, and learning flexible positional representations—treating time as a continuous coordinate system rather than fixed steps.