Course contentsShow
← Machine Learning and Deep Learning
Lesson 2426 of 3,53852. Time Series and ForecastingFree lesson

Lag-Llama: Language Model Architecture for Time Series

Adapting decoder-only transformer architectures from LLMs to univariate and multivariate forecasting tasks.

Lag-Llama: Language Model Architecture for Time Series

What you'll learn: How decoder-only transformer architectures from large language models can be adapted to forecast univariate and multivariate time series.

The Core Idea

Lag-Llama takes the decoder-only transformer architecture popularized by LLMs like GPT and repurposes it for time series forecasting. Just as language models predict the next word given previous words, Lag-Llama predicts the next time step given previous time steps. The key innovation is treating time series as "sequences of numbers" rather than "sequences of tokens," allowing the same architectural principles to apply.

Why Decoder-Only for Time Series?

In language modeling, decoder-only transformers use causal self-attention — they can only look backward, never forward. This naturally fits forecasting, where you predict the future using only past observations. Unlike encoder-decoder models (like Temporal Fusion Transformers), Lag-Llama uses a single stack of decoder blocks with autoregressive generation.

Architectural Adaptations

From tokens to lag features: Instead of word embeddings, Lag-Llama uses lag features (previous time steps) as input. Think of [t-7, t-6, ..., t-1] as your "context window."

Univariate to multivariate: For multiple time series, Lag-Llama can process them as separate channels or interleave them, similar to how multimodal LLMs handle different input types.

Probabilistic outputs: Rather than outputting raw predictions, Lag-Llama can generate distribution parameters (like DeepAR), enabling uncertainty quantification through sampling.

The architecture leverages pretraining on massive time series datasets, then fine-tunes or zero-shot forecasts on new series — borrowing the transfer learning paradigm from NLP.

Key Takeaway: Lag-Llama demonstrates that decoder-only transformer architectures from LLMs translate effectively to time series by treating historical observations as sequential context and forecasting as autoregressive generation with causal attention.