Course contentsShow
← Machine Learning and Deep Learning
Lesson 1217 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

T5 Architecture and Design Choices

Explore T5's encoder-decoder structure, relative positional embeddings, and architectural modifications from vanilla Transformers.

T5 Architecture and Design Choices

What you'll learn: How T5 modifies the vanilla Transformer to create a unified text-to-text architecture with clever design improvements.

The Core Design Philosophy

T5 (Text-to-Text Transfer Transformer) uses an encoder-decoder architecture, unlike GPT's decoder-only design. Think of it like a translator: the encoder reads and understands the input completely, then the decoder generates the output word-by-word. This two-part structure makes T5 flexible for both understanding tasks (like classification) and generation tasks (like translation).

Key Architectural Modifications

Encoder-Decoder Structure

T5 keeps both halves of the original Transformer. The encoder uses bidirectional self-attention (like BERT) to understand context from all directions. The decoder uses causal attention (like GPT) plus cross-attention to the encoder's output, allowing it to "look back" at the input while generating.

Relative Positional Embeddings

Instead of absolute position embeddings that say "this is token #5," T5 uses relative positions that capture "this token is 2 positions after that one." This helps the model generalize better to sequences of different lengths—like learning patterns rather than memorizing fixed positions.

Layer Normalization Placement

T5 places layer normalization before attention and feedforward layers (pre-norm), not after. This subtle change improves training stability, especially as models scale larger.

Simplified Architecture

T5 removes some components like separate segment embeddings and dropout variations, opting for a cleaner, more uniform design that's easier to scale and experiment with.

Key Takeaway: T5's encoder-decoder design with relative positional embeddings and pre-norm layer placement creates a flexible, scalable architecture optimized for the text-to-text framework.