Broadcasting Rules
What You'll Learn
You'll learn how NumPy automatically stretches arrays of different shapes so they can work together in vectorized operations—a fundamental trick that makes ML code fast and concise.
The Core Idea
Broadcasting is NumPy's clever system for handling operations between arrays of different shapes without making wasteful copies. Instead of forcing you to manually expand arrays to matching sizes, NumPy follows specific rules to "virtually" stretch smaller arrays to match larger ones.
Think of it like a photocopy machine that automatically tiles a small pattern to fill a larger page—but without actually using extra paper.
The Broadcasting Rules
NumPy checks array shapes from right to left and applies these rules:
- If arrays have different numbers of dimensions, pad the smaller shape with ones on the left until both have the same length
- For each dimension, the sizes must either:
- Be equal, OR
- One of them must be 1 (which gets "stretched")
- If any dimension fails these conditions, broadcasting fails
Example Walkthrough
# Shape (3, 4) array + shape (4,) array
A.shape = (3, 4)
b.shape = (4,)
# Step 1: Pad b's shape on left: (4,) becomes (1, 4)
# Step 2: Compare dimensions right-to-left:
# - Last dim: 4 == 4 ✓
# - Second-to-last: 3 vs 1 → 1 stretches to 3 ✓
# Result shape: (3, 4)
This is why you can add a bias vector to every row of a weight matrix without loops—broadcasting handles the repetition automatically during computation.
Key Takeaway: Broadcasting lets NumPy perform operations on arrays with compatible (not identical) shapes by automatically expanding dimensions of size 1, enabling fast, memory-efficient vectorized ML operations without explicit loops or array copying.