Course contentsShow
← Machine Learning and Deep Learning
Lesson 1060 of 3,53824. The Transformer ArchitectureFree lesson

Causal (Masked) Self-Attention for Autoregressive Models

Implement masking to prevent positions from attending to future tokens in generation tasks.

Causal (Masked) Self-Attention for Autoregressive Models

What You'll Learn

You'll learn how to prevent a model from "cheating" by looking ahead at future tokens during text generation.

The Problem with Plain Self-Attention

When generating text one word at a time (autoregressive generation), we have a critical issue: regular self-attention lets each position look at all other positions in the sequence, including future ones. During training, this means the model could peek at the answer it's supposed to predict!

The Solution: Masking Future Positions

Causal masking (also called "look-ahead masking") ensures each position can only attend to itself and previous positions—never future ones. Think of it like taking a test where you can only see questions 1-3 when answering question 3, not questions 4-5.

We implement this by adding a mask to our attention scores before the softmax operation. The mask sets future positions to negative infinity (-inf), so after softmax they become zero—effectively blocking attention to those positions.

import torch
import torch.nn.functional as F

seq_len = 5
# Create lower triangular mask (1s for allowed positions, 0s for masked)
mask = torch.tril(torch.ones(seq_len, seq_len))

# Convert 0s to -inf for masking
mask = mask.masked_fill(mask == 0, float('-inf'))
mask = mask.masked_fill(mask == 1, 0.0)

print("Causal mask:\n", mask)
# When added to attention scores before softmax,
# -inf positions become 0 after softmax

This triangular mask structure ensures token at position i only sees tokens 0 through i, preserving the autoregressive property during both training and generation.

Key Takeaway: Causal masking prevents positions from attending to future tokens by setting attention scores to negative infinity before softmax, ensuring autoregressive models can only use past context during generation.