Course contentsShow
← Computer Science
Lesson 2865 of 2,87263. Classical ML AlgorithmsFree lesson

Choosing the Value of K

Explore how k affects model complexity, bias-variance tradeoff, and decision boundaries.

Choosing the Value of K

What you'll learn: How the choice of k in KNN fundamentally shapes your model's behavior, complexity, and predictive performance.

The Core Idea

In k-Nearest Neighbors, the value of k determines how many neighbors vote on a prediction. This single hyperparameter controls the entire character of your model—and there's no universal "right" answer.

How K Affects Model Complexity

Small k (e.g., k=1):

  • Creates highly complex, flexible decision boundaries that wiggle around individual training points
  • Model memorizes training data, capturing noise as if it were signal
  • High variance: predictions change dramatically with different training sets
  • Low bias: can fit very complex patterns

Large k (e.g., k=100):

  • Produces smooth, simple decision boundaries
  • Model averages over many neighbors, ignoring local details
  • Low variance: predictions are stable across training sets
  • High bias: may miss important patterns and underfit

The Bias-Variance Tradeoff in Action

Think of k like a camera's focus. k=1 is hyperfocused on every pixel (overfitting); k=total_samples is blurry and sees only the average (underfitting). You need the sweet spot in between.

As k increases:

  • Bias increases (model becomes too simple)
  • Variance decreases (predictions become more stable)
  • Decision boundaries smooth out from jagged to nearly linear

Choosing K in Practice

Typical approaches:

  • Use cross-validation to test multiple k values (e.g., k=1, 3, 5, 7, 11, 15)
  • Choose k with the best validation performance
  • Odd values for binary classification avoid ties
  • Start with k=√n as a rule of thumb

Key Takeaway: Choosing k is your primary tool for controlling KNN's bias-variance tradeoff—small k gives flexible boundaries that risk overfitting, while large k produces smooth boundaries that risk underfitting. Always validate your choice empirically.