Binary and Ternary Neural Networks
What you'll learn: How extreme 1-bit and 2-bit weight quantization can compress models to their theoretical limits.
The Ultimate Compression Frontier
After exploring various quantization schemes (INT8, mixed-precision, QAT), we now reach the absolute extreme: representing weights with just 1 or 2 bits. Binary Neural Networks (BNNs) use weights constrained to {-1, +1}, while Ternary Neural Networks (TNNs) add zero: {-1, 0, +1}.
Binary Networks: 1-Bit Representation
In a BNN, each weight is either +1 or -1. The sign function defines the mapping:
w_binary = sign(w_full-precision)
Where sign(w) = +1 if w ≥ 0, else -1.
This achieves 32× compression compared to FP32 (1 bit vs 32 bits per weight). Matrix multiplications become simple XNOR and bit-counting operations—extremely fast on specialized hardware.
Ternary Networks: 2-Bit Representation
TNNs add a third state (zero), allowing the network to "ignore" certain connections:
w_ternary ∈ {-Δ, 0, +Δ}
The threshold function determines which weights become zero (typically those near zero in magnitude), while Δ (the scaling factor) is learned or computed to minimize quantization error. Common formulation:
- If |w| < threshold: w_ternary = 0
- Else: w_ternary = sign(w) × Δ
This provides 2 bits per weight with slightly better accuracy than pure binary.
The Mathematical Challenge
Both BNNs and TNNs face the gradient problem: the sign function has zero gradient almost everywhere. Solutions use the Straight-Through Estimator (which you've seen in QAT) to approximate gradients during backpropagation, allowing training despite the discrete nature.
Key Takeaway: Binary and ternary networks achieve extreme compression (1-2 bits per weight) by constraining weights to {-1,+1} or {-1,0,+1}, trading significant accuracy loss for massive efficiency gains in specialized applications.