Course contentsShow
← Machine Learning and Deep Learning
Lesson 324 of 3,5387. Instance-Based and Probabilistic ClassifiersFree lesson

Choosing K: The Bias-Variance Tradeoff

How small K creates flexible decision boundaries with high variance while large K smooths predictions but increases bias.

Choosing K: The Bias-Variance Tradeoff

What you'll learn: How the choice of K in K-Nearest Neighbors directly controls the balance between overfitting and underfitting through the bias-variance tradeoff.

The Core Tension

In K-Nearest Neighbors, the value of K determines how many neighbors vote on a prediction. This single number dramatically affects your model's behavior in opposite directions.

Small K (like K=1 or K=3):

  • Creates highly flexible decision boundaries that twist and turn to follow every training point closely
  • Results in high variance: your model is sensitive to noise and outliers in the training data
  • Low bias: the model makes few assumptions and can capture complex patterns
  • Risk: overfitting — memorizing training data rather than learning general patterns

Large K (like K=50 or K=100):

  • Creates smooth, simple decision boundaries that ignore local variations
  • Results in low variance: predictions are stable and consistent
  • High bias: the model makes strong assumptions by averaging over many points
  • Risk: underfitting — missing important patterns in the data

The Goldilocks Problem

Think of small K as a nervous student who changes their answer based on sitting next to just one classmate. Large K is like polling the entire auditorium — you get a consensus, but lose nuance.

The sweet spot depends on your data's complexity, noise level, and sample size. Typically, you'll use cross-validation to test different K values and choose the one with the best validation performance.

Key Takeaway: Small K creates flexible, high-variance models prone to overfitting; large K creates smooth, high-bias models prone to underfitting. Finding the right K means balancing these competing forces.