Transformers for Time Series Forecasting
What you'll learn: How to adapt transformer architectures—originally designed for language—to forecast time series by adding temporal positional encodings and causal masking.
Why Transformers Need Adaptation for Time Series
Transformers were built for text, where word order matters but time isn't continuous. In time series, we have temporal dynamics with continuous timestamps and causality constraints: the future can't influence the past.
Two critical adaptations make transformers time-series-ready:
Temporal Positional Encodings
Standard transformers use sinusoidal positional encodings based on sequence position (1st, 2nd, 3rd token). For time series, we enhance this with temporal information:
- Absolute timestamps: Hour of day, day of week, month
- Relative time distances: The gap between observations matters (1 minute vs 1 week)
- Learnable temporal embeddings: Let the model discover temporal patterns
Think of it like a calendar: knowing something happened on "Tuesday at 3pm" (absolute) and "2 days after the last event" (relative) gives richer context than just "observation #5."
Causal Masking
The self-attention mechanism naturally lets each position attend to all others—dangerous for forecasting! We apply causal masking (also called "look-ahead masking") to ensure position t can only attend to positions ≤ t.
This prevents "cheating" during training: your model can't use tomorrow's values to predict today. It's like covering the right side of a page while reading left-to-right—forcing the model to respect time's arrow.
Together, these adaptations preserve the transformer's parallel processing power while respecting temporal structure and causality.
Key Takeaway: Transformers become time-series forecasters by adding temporal positional encodings (capturing when events occur) and causal masking (ensuring predictions only use past information).