Measuring Privacy Leakage Empirically
You'll learn: How to quantify privacy risks in machine learning systems using concrete metrics and experimental methods.
Now that you understand various privacy attacks (membership inference, model inversion, data extraction), you need practical ways to measure how much privacy is actually leaking from your models. Think of it like stress-testing a dam—you need gauges and measurements, not just theories about potential failure.
Core Metrics for Privacy Leakage
Attack Success Rate is your primary metric. If a membership inference attack correctly identifies training members 95% of the time versus 50% (random guessing), that's a 45-point privacy leakage. You run the attack multiple times with different samples and report accuracy, precision, and recall.
Reconstruction Error quantifies how well an attacker can recreate sensitive data. For model inversion attacks, measure pixel-wise distance (MSE, SSIM) or perceptual similarity between reconstructed and original images. Lower error = higher privacy risk.
Canary Tests embed known "canary" data points—synthetic records with specific patterns—into your training set. After training, test whether the model has memorized these exact sequences. If your language model reproduces a planted 128-bit random string, you've confirmed memorization risk. It's like hiding a tracer in water to see where it flows.
Experimental Setup
Design controlled experiments: train shadow models on data with known membership, run attacks, and measure success rates across different hyperparameters, model architectures, and dataset sizes. Compare against baselines (random guessing, majority class prediction). Report confidence intervals—statistical rigor matters.
Key Takeaway: Privacy leakage becomes actionable when you measure it empirically through attack success rates, reconstruction quality metrics, and canary insertion tests, enabling you to make informed risk assessments and design decisions.