Course contentsShow
← Machine Learning and Deep Learning
Lesson 1460 of 3,53832. Generative Models: Autoencoders and VAEsFree lesson

The Reparameterization Trick Implementation

Writing the epsilon-based sampling procedure that enables backpropagation through stochastic latent variables.

The Reparameterization Trick Implementation

What you'll learn: How to write code that samples from learned latent distributions while keeping the computation graph differentiable for backpropagation.

The Core Challenge

You've learned why the reparameterization trick matters—it lets gradients flow through stochastic sampling. Now let's see how to implement it. The key is to separate randomness (which can't have gradients) from learned parameters (which must).

The Implementation Pattern

Instead of sampling z directly from N(μ, σ²), you:

  1. Sample epsilon (ε) from a standard normal N(0, 1) — this is random but parameter-free
  2. Transform it using your learned μ and σ: z = μ + σ * ε

This way, gradients flow through μ and σ, bypassing the random sampling step entirely.

Code Structure

def reparameterize(mu, log_var):
    """
    mu: learned mean, shape [batch_size, latent_dim]
    log_var: learned log variance, shape [batch_size, latent_dim]
    """
    # Compute standard deviation from log variance
    std = torch.exp(0.5 * log_var)
    
    # Sample epsilon from standard normal
    eps = torch.randn_like(std)
    
    # Apply reparameterization: z = μ + σ * ε
    z = mu + std * eps
    
    return z

Why log_var instead of variance directly? It ensures numerical stability and guarantees positive variance after exp().

Why randn_like(std)? It creates random noise with the exact same shape as your parameters, making the math work per-dimension.

Analogy

Think of it like mixing paint: instead of randomly grabbing a pre-mixed color (non-differentiable), you start with pure white (ε) and adjust it using learnable amounts of pigment (μ, σ).

Key Takeaway: The reparameterization trick implementation splits sampling into a fixed random component (eps) and a learnable transformation (mu + std * eps), enabling gradients to update the encoder's parameters during training.