Implementing DP-SGD with PyTorch and Opacus
You'll learn how to integrate the Opacus library into PyTorch training loops to enable differentially private model training and track your privacy budget.
What is Opacus?
Opacus is Meta's library that makes implementing DP-SGD (Differentially Private Stochastic Gradient Descent) in PyTorch straightforward. Instead of manually clipping gradients and adding calibrated noise—as you've learned in theory—Opacus automates this entire process while tracking your privacy budget (ε and δ).
The Privacy Engine
The core component is the PrivacyEngine, which wraps your optimizer and modifies the training loop to enforce differential privacy:
from opacus import PrivacyEngine
# Standard PyTorch setup
model = MyModel()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
data_loader = DataLoader(dataset, batch_size=32)
# Attach privacy engine
privacy_engine = PrivacyEngine()
model, optimizer, data_loader = privacy_engine.make_private(
module=model,
optimizer=optimizer,
data_loader=data_loader,
noise_multiplier=1.1, # Controls noise scale
max_grad_norm=1.0, # Gradient clipping bound
)
The PrivacyEngine handles gradient clipping per sample, adds Gaussian noise calibrated to your parameters, and leverages privacy amplification by sampling through the modified data loader.
Tracking Privacy Budget
During training, query the accumulated privacy cost:
epsilon = privacy_engine.get_epsilon(delta=1e-5)
print(f"Current ε = {epsilon:.2f} at δ = 1e-5")
This uses advanced composition to give you a tight bound on your total privacy loss across all training iterations.
Key Takeaway: Opacus simplifies DP-SGD implementation by automating gradient clipping and noise addition through the PrivacyEngine, while providing real-time privacy budget tracking to ensure your model meets specified (ε, δ) guarantees.