Binarization Training Techniques
What you'll learn: How to actually train networks with binary weights using gradient approximations, since binary operations aren't differentiable.
The Core Challenge
When you binarize weights to {-1, +1}, you face a fundamental problem: the sign function used for binarization has zero gradient almost everywhere. During backpropagation, gradients would vanish completely, making learning impossible. You need clever techniques to work around this non-differentiability.
The Straight-Through Estimator (STE)
The most common solution is the straight-through estimator. During the forward pass, you use binary weights:
w_binary = sign(w_real)
But during the backward pass, you pretend the sign function is the identity function and pass gradients straight through unchanged:
∂Loss/∂w_real ≈ ∂Loss/∂w_binary
Think of it like a "gradient lie" — you compute outputs with binary values but update the underlying real-valued parameters as if no binarization happened.
Gradient Approximation Strategies
Beyond the basic STE, several approximations exist:
- Clipped identity: Gradient is 1 when |w| < 1, else 0
- Polynomial approximations: Use smooth functions that approximate the sign function
- Stochastic binarization: Sample from probability distributions during training
Training Flow
You maintain real-valued latent weights throughout training. Each forward pass:
- Binarize the latent weights
- Compute loss with binary values
- Backpropagate using STE
- Update the real-valued latents with standard optimizers
The latent weights act as a "memory" that accumulates small gradient updates over time.
Key Takeaway: Binarization training uses straight-through estimators to approximate gradients for the non-differentiable sign function, maintaining real-valued latent weights that get binarized only during forward passes.