Course contentsShow
← Machine Learning and Deep Learning
Lesson 2417 of 3,53852. Time Series and ForecastingFree lesson

Transformers for Time Series Forecasting

Adapt transformer architectures for time series by incorporating temporal positional encodings and causal masking.

Transformers for Time Series Forecasting

What you'll learn: How to adapt transformer architectures—originally designed for language—to forecast time series by adding temporal positional encodings and causal masking.

Why Transformers Need Adaptation for Time Series

Transformers were built for text, where word order matters but time isn't continuous. In time series, we have temporal dynamics with continuous timestamps and causality constraints: the future can't influence the past.

Two critical adaptations make transformers time-series-ready:

Temporal Positional Encodings

Standard transformers use sinusoidal positional encodings based on sequence position (1st, 2nd, 3rd token). For time series, we enhance this with temporal information:

  • Absolute timestamps: Hour of day, day of week, month
  • Relative time distances: The gap between observations matters (1 minute vs 1 week)
  • Learnable temporal embeddings: Let the model discover temporal patterns

Think of it like a calendar: knowing something happened on "Tuesday at 3pm" (absolute) and "2 days after the last event" (relative) gives richer context than just "observation #5."

Causal Masking

The self-attention mechanism naturally lets each position attend to all others—dangerous for forecasting! We apply causal masking (also called "look-ahead masking") to ensure position t can only attend to positions ≤ t.

This prevents "cheating" during training: your model can't use tomorrow's values to predict today. It's like covering the right side of a page while reading left-to-right—forcing the model to respect time's arrow.

Together, these adaptations preserve the transformer's parallel processing power while respecting temporal structure and causality.

Key Takeaway: Transformers become time-series forecasters by adding temporal positional encodings (capturing when events occur) and causal masking (ensuring predictions only use past information).