Course contentsShow
← Machine Learning and Deep Learning
Lesson 3346 of 3,53873. Privacy in Machine LearningFree lesson

Differentially Private Stochastic Gradient Descent

DP-SGD algorithm: clipping gradients and adding calibrated noise during training for private learning.

Differentially Private Stochastic Gradient Descent

What you'll learn: How DP-SGD modifies standard gradient descent to train models while formally guaranteeing privacy for training data.

The Core Problem

Standard SGD can memorize training examples, making models vulnerable to privacy attacks you've seen earlier (membership inference, data extraction). Even individual gradients can leak information about specific training samples. DP-SGD solves this by making each gradient update differentially private.

How DP-SGD Works

The algorithm modifies standard SGD with two key operations per training step:

1. Gradient Clipping

Before computing the average gradient, DP-SGD clips each individual example's gradient to have bounded norm (typically L2 norm ≤ C). Think of this as preventing any single training example from having outsized influence on the update.

If a gradient has norm 10 but your clip threshold C = 1, you scale it down to norm 1. This bounds the global sensitivity of the gradient computation—the maximum change one example can cause.

2. Noise Addition

After clipping and averaging, DP-SGD adds calibrated Gaussian noise (using the Gaussian Mechanism you learned earlier) to the gradient. The noise scale depends on:

  • The clipping threshold C
  • Your target privacy budget (ε, δ)
  • The number of training steps (via composition theorems)

The Privacy-Utility Trade-off

The noise protects privacy but reduces gradient signal quality. Smaller clipping thresholds and larger noise harm model accuracy. Careful tuning of C, noise scale, and batch size is essential for practical DP-SGD.

Key Takeaway: DP-SGD makes model training differentially private by clipping per-example gradients to bound sensitivity, then adding calibrated Gaussian noise to each update, providing formal privacy guarantees at the cost of some model utility.