Differentially Private Stochastic Gradient Descent
What you'll learn: How DP-SGD modifies standard gradient descent to train models while formally guaranteeing privacy for training data.
The Core Problem
Standard SGD can memorize training examples, making models vulnerable to privacy attacks you've seen earlier (membership inference, data extraction). Even individual gradients can leak information about specific training samples. DP-SGD solves this by making each gradient update differentially private.
How DP-SGD Works
The algorithm modifies standard SGD with two key operations per training step:
1. Gradient Clipping
Before computing the average gradient, DP-SGD clips each individual example's gradient to have bounded norm (typically L2 norm ≤ C). Think of this as preventing any single training example from having outsized influence on the update.
If a gradient has norm 10 but your clip threshold C = 1, you scale it down to norm 1. This bounds the global sensitivity of the gradient computation—the maximum change one example can cause.
2. Noise Addition
After clipping and averaging, DP-SGD adds calibrated Gaussian noise (using the Gaussian Mechanism you learned earlier) to the gradient. The noise scale depends on:
- The clipping threshold C
- Your target privacy budget (ε, δ)
- The number of training steps (via composition theorems)
The Privacy-Utility Trade-off
The noise protects privacy but reduces gradient signal quality. Smaller clipping thresholds and larger noise harm model accuracy. Careful tuning of C, noise scale, and batch size is essential for practical DP-SGD.
Key Takeaway: DP-SGD makes model training differentially private by clipping per-example gradients to bound sensitivity, then adding calibrated Gaussian noise to each update, providing formal privacy guarantees at the cost of some model utility.