Course contentsShow
← Computer Science
Lesson 2863 of 2,87263. Classical ML AlgorithmsFree lesson

KNN: Distance-Based Classification and Regression

Understand how KNN predicts labels by finding the k nearest training examples in feature space.

KNN: Distance-Based Classification and Regression

What You'll Learn

You'll learn how K-Nearest Neighbors (KNN) makes predictions by finding the most similar training examples in feature space.

The Core Idea

KNN is an instance-based learning algorithm that doesn't build an explicit model during training. Instead, it stores all training examples and makes predictions by examining the k closest neighbors to a new data point.

Here's how it works:

  1. Given a new data point, compute its distance to every training example in feature space
  2. Identify the k nearest neighbors (the k training points with smallest distances)
  3. For classification: assign the most common label among those k neighbors (majority vote)
  4. For regression: predict the average value of those k neighbors

Think of it like asking your k closest friends for advice: if you're deciding whether to watch a movie (classification), you'd pick based on what most of them recommend. If you're estimating how much you'll like it (regression), you'd average their ratings.

Distance Matters

The "nearness" is typically measured using Euclidean distance in feature space, though other distance metrics (Manhattan, Minkowski) can be used. This makes feature scaling critical—features with larger ranges would otherwise dominate the distance calculation.

The choice of k is crucial: small k values (like k=1) are sensitive to noise and lead to complex decision boundaries, while large k values smooth predictions but may ignore local patterns (the bias-variance tradeoff applies here).

Key Takeaway: KNN predicts by computing distances to all training examples and using the k nearest neighbors' labels or values—no training phase required, just direct lookup at prediction time.