The Reparameterization Trick Implementation
What you'll learn: How to write code that samples from learned latent distributions while keeping the computation graph differentiable for backpropagation.
The Core Challenge
You've learned why the reparameterization trick matters—it lets gradients flow through stochastic sampling. Now let's see how to implement it. The key is to separate randomness (which can't have gradients) from learned parameters (which must).
The Implementation Pattern
Instead of sampling z directly from N(μ, σ²), you:
- Sample epsilon (ε) from a standard normal
N(0, 1)— this is random but parameter-free - Transform it using your learned
μandσ:z = μ + σ * ε
This way, gradients flow through μ and σ, bypassing the random sampling step entirely.
Code Structure
def reparameterize(mu, log_var):
"""
mu: learned mean, shape [batch_size, latent_dim]
log_var: learned log variance, shape [batch_size, latent_dim]
"""
# Compute standard deviation from log variance
std = torch.exp(0.5 * log_var)
# Sample epsilon from standard normal
eps = torch.randn_like(std)
# Apply reparameterization: z = μ + σ * ε
z = mu + std * eps
return z
Why log_var instead of variance directly? It ensures numerical stability and guarantees positive variance after exp().
Why randn_like(std)? It creates random noise with the exact same shape as your parameters, making the math work per-dimension.
Analogy
Think of it like mixing paint: instead of randomly grabbing a pre-mixed color (non-differentiable), you start with pure white (ε) and adjust it using learnable amounts of pigment (μ, σ).
Key Takeaway: The reparameterization trick implementation splits sampling into a fixed random component (eps) and a learnable transformation (mu + std * eps), enabling gradients to update the encoder's parameters during training.