Course contentsShow
← Machine Learning and Deep Learning
Lesson 1058 of 3,53824. The Transformer ArchitectureFree lesson

Implementing Single-Head Attention in PyTorch

Code the complete attention mechanism from scratch using matrix operations.

Implementing Single-Head Attention in PyTorch

What you'll learn: How to code the complete attention mechanism from scratch using PyTorch matrix operations.

The Core Idea

Now that you understand the conceptual pieces of attention—queries, keys, values, dot products, scaling, softmax, and weighted sums—it's time to bring them all together in actual code. You'll implement single-head attention as a series of matrix operations that process entire batches of sequences simultaneously.

Building the Complete Mechanism

Here's the full attention implementation step-by-step:

import torch
import torch.nn.functional as F

def single_head_attention(Q, K, V, mask=None):
    """
    Q: queries (batch_size, seq_len, d_k)
    K: keys (batch_size, seq_len, d_k)
    V: values (batch_size, seq_len, d_v)
    """
    d_k = Q.size(-1)
    
    # Step 1: Compute attention scores (QK^T)
    scores = torch.matmul(Q, K.transpose(-2, -1))
    
    # Step 2: Scale by sqrt(d_k)
    scores = scores / torch.sqrt(torch.tensor(d_k, dtype=torch.float32))
    
    # Step 3: Apply mask if provided (for padding or future tokens)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    
    # Step 4: Apply softmax to get attention weights
    attention_weights = F.softmax(scores, dim=-1)
    
    # Step 5: Multiply weights by values
    output = torch.matmul(attention_weights, V)
    
    return output, attention_weights

This function chains all the operations you've learned: computing dot products between queries and keys, scaling, normalizing with softmax, and producing the final weighted sum of values.

Key Takeaway: Single-head attention is implemented as a sequence of matrix operations—score computation, scaling, softmax normalization, and weighted value summation—that can process entire batches efficiently in parallel.