Course contentsShow
← Machine Learning and Deep Learning
Lesson 3339 of 3,53873. Privacy in Machine LearningFree lesson

The Failure Probability Parameter (δ)

Understanding delta as the probability of privacy breach; (ε,δ)-differential privacy definition.

The Failure Probability Parameter (δ)

You'll learn how δ captures the small probability that differential privacy guarantees might fail, completing the (ε,δ)-differential privacy definition.

Beyond Pure ε-Differential Privacy

In the previous lesson, you learned that ε controls how much neighboring datasets' outputs can differ. However, pure ε-differential privacy can be too strict for practical machine learning algorithms. This is where δ (delta) comes in as a relaxation parameter.

What δ Represents

Think of δ as a "get out of jail free" card that you can use with tiny probability. It's the probability of catastrophic privacy failure — a small chance that the strong privacy guarantees don't hold at all.

More formally, δ represents the probability that the mechanism might completely violate the ε-privacy bound. Instead of guaranteeing that outputs are indistinguishable up to factor e^ε for all possible outputs, we now say this holds for most outputs, except with probability at most δ.

(ε,δ)-Differential Privacy Definition

A mechanism M satisfies (ε,δ)-differential privacy if for all neighboring datasets D and D' (differing by one record) and all possible output sets S:

P[M(D) ∈ S] ≤ e^ε · P[M(D') ∈ S] + δ

The "+δ" term allows rare exceptions. Typically, δ is set to be cryptographically small (e.g., 10^-5 or smaller than 1/n where n is dataset size), making privacy breaches vanishingly unlikely while enabling more practical algorithms.

Why This Matters

Many useful ML algorithms, like those using Gaussian noise instead of Laplace noise, require this relaxation. The δ parameter lets us balance stronger practical utility with an acceptably tiny risk of privacy failure.

Key Takeaway: δ is the probability that differential privacy guarantees completely fail; (ε,δ)-DP requires privacy bounds to hold except with probability at most δ, enabling more practical privacy-preserving algorithms.