The Reparameterization Trick
What you'll learn: How to make random sampling differentiable so we can train VAEs with backpropagation.
The Problem
In the previous lesson, you learned that VAE encoders output probability distributions (mean μ and standard deviation σ) instead of fixed vectors. To get a latent vector z, we need to sample from that distribution. But here's the catch: sampling is a random operation, and you can't backpropagate through randomness. Gradients need a deterministic path!
The Trick
The reparameterization trick cleverly separates the randomness from the learnable parameters:
z = μ + σ ⊙ ε where ε ~ N(0, I)
Instead of sampling directly from N(μ, σ), we:
- Sample ε (epsilon) from a standard normal distribution N(0,1) — this is pure noise, independent of our network
- Scale it by σ and shift by μ — these ARE learnable parameters from our encoder
Why It Works
Think of it like adjusting a radio dial. The static (ε) is random and unchangeable, but the tuning knobs (μ and σ) control what signal you get. Gradients can flow through μ and σ because they're just regular operations (addition and multiplication), while the randomness is isolated in ε.
Now backpropagation works! The gradient bypasses the random sampling and flows directly through μ and σ, allowing your encoder to learn the right distribution parameters.
Key Takeaway: The reparameterization trick makes sampling differentiable by moving randomness outside the computational graph: sample standard noise ε, then deterministically transform it with learnable μ and σ.