What Privacy Means in Machine Learning Contexts
What you'll learn: You'll understand the three key dimensions of privacy that matter when building and deploying ML systems.
The Core Idea
Privacy in machine learning isn't just about keeping data secret—it's about protecting information at multiple stages of the ML pipeline. Think of an ML model like a bank vault: you need to protect the money going in (training data), the vault's internal mechanism (model behavior), and the transactions people make (inference queries).
Three Privacy Surfaces
1. Training Data Privacy
Your training dataset may contain sensitive personal information—medical records, financial transactions, user behavior. Privacy means preventing anyone from extracting or reconstructing individual training examples from your trained model. If someone can look at your model and figure out "John Doe was in the training set," privacy is violated.
2. Model Behavior Privacy
The model's internal weights, architecture decisions, and learned patterns are intellectual property and strategic assets. Protecting model behavior means preventing unauthorized parties from replicating your model through techniques like model extraction attacks or reverse engineering.
3. Inference Query Privacy
When users make predictions with your model, their queries themselves can be sensitive. If someone asks "Will I get cancer?" or "What's my credit score?", you need to protect both the query and the response from eavesdropping or logging by untrusted parties.
Why It Matters
Unlike traditional software where you explicitly program logic, ML models learn from data—which means they can inadvertently memorize and leak it. A seemingly innocent model might reveal whether specific individuals participated in training, or expose sensitive patterns about groups.
Key Takeaway: Privacy in ML requires protecting three distinct attack surfaces: the training data from being reverse-engineered, the model itself from theft, and user queries from exposure—each requiring different technical safeguards.