Course contentsShow
← Machine Learning and Deep Learning
Lesson 1606 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Causal Self-Attention Masking

Learn how causal masking prevents tokens from attending to future positions, enabling autoregressive generation.

Causal Self-Attention Masking

What you'll learn: How causal masking enables language models to generate text one token at a time without "peeking" into the future.

The Core Problem

In decoder-only architectures like GPT, we need the model to predict the next word based only on previous words. But standard self-attention lets every token see all other tokens in the sequence—including future ones. This creates a data leak during training: the model could "cheat" by looking ahead at what it's supposed to predict.

How Causal Masking Works

Causal masking (also called autoregressive masking) prevents tokens from attending to future positions by setting attention weights to negative infinity (or zero after softmax) for any position that comes later in the sequence.

Think of it like reading a book with blinders: at word 5, you can only see words 1-5, never words 6 onward. Each position has a triangular "visibility window" that only looks backward (and at itself).

The Mask Matrix

During attention computation, we apply a mask matrix that looks like this:

[1, 0, 0, 0]  ← Token 1 sees only itself
[1, 1, 0, 0]  ← Token 2 sees tokens 1-2
[1, 1, 1, 0]  ← Token 3 sees tokens 1-3
[1, 1, 1, 1]  ← Token 4 sees tokens 1-4

This lower-triangular pattern ensures each token can only attend to itself and previous tokens, enabling true autoregressive generation where each new token depends only on the past.

Key Takeaway: Causal self-attention masking enforces the autoregressive property by blocking future token visibility, allowing language models to generate sequences one token at a time while being trained in parallel on entire sequences.