Gradient Clipping and Noise Calibration
You'll learn how to bound per-example gradient norms and calibrate noise to achieve your privacy budget in DP-SGD.
Why Clip Gradients?
In Differentially Private Stochastic Gradient Descent (DP-SGD), we add noise to gradients to protect individual training examples. But here's the problem: if one person's gradient is enormous and another's is tiny, the same amount of noise protects them differently. The giant gradient "sticks out" through the noise, leaking information.
Per-example gradient clipping solves this by capping each individual example's gradient norm at a threshold C before aggregating. Think of it like standardizing answer sheets: everyone's response is "normalized" to the same maximum length before being shuffled together, making individuals indistinguishable.
Computing the Right Noise Scale
Once gradients are clipped, you add Gaussian noise scaled to your privacy budget. The noise standard deviation σ depends on three things:
- Clipping norm C: Higher clipping = more sensitivity = more noise needed
- Privacy budget (ε, δ): Tighter privacy = more noise
- Batch size: Larger batches = averaging dilutes noise = need less per gradient
The noise scale formula (from advanced composition):
σ = C · √(2 ln(1.25/δ)) / ε
This ensures adding noise to the clipped, aggregated gradient satisfies (ε, δ)-differential privacy per batch. Over training, composition theorems track total privacy loss across all batches.
The Balancing Act
Choose C too small and you destroy useful gradient information; too large and noise becomes prohibitively high. Typical values: C = 1.0 to C = 10.0, tuned empirically.
Key Takeaway: Per-example gradient clipping bounds sensitivity uniformly, enabling calibrated Gaussian noise that guarantees differential privacy without individual gradients overwhelming the protection.