Course contentsShow
← Machine Learning and Deep Learning
Lesson 3334 of 3,53873. Privacy in Machine LearningFree lesson

Privacy Risks in Model APIs and Inference Services

Analyze privacy threats when exposing models via APIs: query-based attacks, input-output correlations, and side channels.

Privacy Risks in Model APIs and Inference Services

What you'll learn: How exposing models through APIs creates exploitable privacy vulnerabilities via query-based attacks, input-output correlations, and side channels.

The API Privacy Problem

When you deploy a model as a service (like GPT-4's API or a hosted image classifier), you're giving adversaries a special tool: unlimited querying power. Unlike stealing a snapshot of your training data, attackers can craft strategic inputs and observe outputs, gradually extracting sensitive information through repeated interactions.

Think of it like interviewing someone versus reading their diary. With enough clever questions and patience, you can extract secrets they never intended to share.

Three Major Attack Vectors

Query-Based Attacks

Attackers submit carefully crafted inputs thousands of times. For membership inference (lesson 3328), they probe specific samples repeatedly to determine if they were in training data. For model inversion (3329), they iteratively refine queries to reconstruct training examples. The API's unlimited access becomes a gradient descent oracle for privacy breaches.

Input-Output Correlations

Observing patterns across many queries reveals hidden information. If your medical diagnosis API consistently flags rare conditions for certain demographic combinations, attackers can infer sensitive training data patterns. Output confidence scores, especially, leak disproportionate information about whether specific examples were memorized during training.

Side Channels

Beyond the model's logical outputs, APIs leak metadata: response times, error messages, GPU memory usage spikes, or query rate limits. An attacker might detect overfitting (lesson 3333) by timing how quickly the model responds to training data versus novel inputs—memorized examples often process faster.

Real-World Implications

Commercial APIs must balance utility with privacy. Rate limiting helps but doesn't eliminate risks. Differential privacy (future lessons) and output randomization add defenses, but fundamentally: every query is a potential privacy leak.

Key Takeaway: Model APIs transform one-shot privacy attacks into iterative optimization problems for adversaries—query access enables systematic extraction of training data and model internals through strategic probing, output analysis, and side-channel observation.