Course contentsShow
← Machine Learning and Deep Learning
Lesson 2656 of 3,53857. Model Compression: QuantizationFree lesson

Binarization Training Techniques

How to train networks with binary weights using straight-through estimators and gradient approximations.

Binarization Training Techniques

What you'll learn: How to actually train networks with binary weights using gradient approximations, since binary operations aren't differentiable.

The Core Challenge

When you binarize weights to {-1, +1}, you face a fundamental problem: the sign function used for binarization has zero gradient almost everywhere. During backpropagation, gradients would vanish completely, making learning impossible. You need clever techniques to work around this non-differentiability.

The Straight-Through Estimator (STE)

The most common solution is the straight-through estimator. During the forward pass, you use binary weights:

w_binary = sign(w_real)

But during the backward pass, you pretend the sign function is the identity function and pass gradients straight through unchanged:

∂Loss/∂w_real ≈ ∂Loss/∂w_binary

Think of it like a "gradient lie" — you compute outputs with binary values but update the underlying real-valued parameters as if no binarization happened.

Gradient Approximation Strategies

Beyond the basic STE, several approximations exist:

  • Clipped identity: Gradient is 1 when |w| < 1, else 0
  • Polynomial approximations: Use smooth functions that approximate the sign function
  • Stochastic binarization: Sample from probability distributions during training

Training Flow

You maintain real-valued latent weights throughout training. Each forward pass:

  1. Binarize the latent weights
  2. Compute loss with binary values
  3. Backpropagate using STE
  4. Update the real-valued latents with standard optimizers

The latent weights act as a "memory" that accumulates small gradient updates over time.

Key Takeaway: Binarization training uses straight-through estimators to approximate gradients for the non-differentiable sign function, maintaining real-valued latent weights that get binarized only during forward passes.