Implementing Single-Head Attention in PyTorch
What you'll learn: How to code the complete attention mechanism from scratch using PyTorch matrix operations.
The Core Idea
Now that you understand the conceptual pieces of attention—queries, keys, values, dot products, scaling, softmax, and weighted sums—it's time to bring them all together in actual code. You'll implement single-head attention as a series of matrix operations that process entire batches of sequences simultaneously.
Building the Complete Mechanism
Here's the full attention implementation step-by-step:
import torch
import torch.nn.functional as F
def single_head_attention(Q, K, V, mask=None):
"""
Q: queries (batch_size, seq_len, d_k)
K: keys (batch_size, seq_len, d_k)
V: values (batch_size, seq_len, d_v)
"""
d_k = Q.size(-1)
# Step 1: Compute attention scores (QK^T)
scores = torch.matmul(Q, K.transpose(-2, -1))
# Step 2: Scale by sqrt(d_k)
scores = scores / torch.sqrt(torch.tensor(d_k, dtype=torch.float32))
# Step 3: Apply mask if provided (for padding or future tokens)
if mask is not None:
scores = scores.masked_fill(mask == 0, -1e9)
# Step 4: Apply softmax to get attention weights
attention_weights = F.softmax(scores, dim=-1)
# Step 5: Multiply weights by values
output = torch.matmul(attention_weights, V)
return output, attention_weights
This function chains all the operations you've learned: computing dot products between queries and keys, scaling, normalizing with softmax, and producing the final weighted sum of values.
Key Takeaway: Single-head attention is implemented as a sequence of matrix operations—score computation, scaling, softmax normalization, and weighted value summation—that can process entire batches efficiently in parallel.