Course contentsShow
← Machine Learning and Deep Learning
Lesson 2655 of 3,53857. Model Compression: QuantizationFree lesson

Binary and Ternary Neural Networks

Understanding 1-bit and 2-bit weight representations and their mathematical formulations for extreme compression.

Binary and Ternary Neural Networks

What you'll learn: How extreme 1-bit and 2-bit weight quantization can compress models to their theoretical limits.

The Ultimate Compression Frontier

After exploring various quantization schemes (INT8, mixed-precision, QAT), we now reach the absolute extreme: representing weights with just 1 or 2 bits. Binary Neural Networks (BNNs) use weights constrained to {-1, +1}, while Ternary Neural Networks (TNNs) add zero: {-1, 0, +1}.

Binary Networks: 1-Bit Representation

In a BNN, each weight is either +1 or -1. The sign function defines the mapping:

w_binary = sign(w_full-precision)

Where sign(w) = +1 if w ≥ 0, else -1.

This achieves 32× compression compared to FP32 (1 bit vs 32 bits per weight). Matrix multiplications become simple XNOR and bit-counting operations—extremely fast on specialized hardware.

Ternary Networks: 2-Bit Representation

TNNs add a third state (zero), allowing the network to "ignore" certain connections:

w_ternary ∈ {-Δ, 0, +Δ}

The threshold function determines which weights become zero (typically those near zero in magnitude), while Δ (the scaling factor) is learned or computed to minimize quantization error. Common formulation:

  • If |w| < threshold: w_ternary = 0
  • Else: w_ternary = sign(w) × Δ

This provides 2 bits per weight with slightly better accuracy than pure binary.

The Mathematical Challenge

Both BNNs and TNNs face the gradient problem: the sign function has zero gradient almost everywhere. Solutions use the Straight-Through Estimator (which you've seen in QAT) to approximate gradients during backpropagation, allowing training despite the discrete nature.

Key Takeaway: Binary and ternary networks achieve extreme compression (1-2 bits per weight) by constraining weights to {-1,+1} or {-1,0,+1}, trading significant accuracy loss for massive efficiency gains in specialized applications.