The K-Nearest Neighbors Algorithm
What you'll learn: How K-Nearest Neighbors makes predictions by finding the closest training examples and letting them vote on the answer.
Core Idea
K-Nearest Neighbors (KNN) is wonderfully simple: to classify a new data point, find the K training examples that are closest to it, then take a majority vote of their labels. If 7 out of your 10 nearest neighbors are "cats," you predict "cat."
Think of it like asking your K nearest neighbors in a theater what movie they liked. If most of them loved it, you'll probably love it too.
Step-by-Step Mechanics
1. Measure Distance to All Training Points
When a new example arrives, calculate its distance to every point in your training set. Common distance metrics include Euclidean distance (straight-line) or Manhattan distance (grid-based).
2. Find the K Nearest Neighbors
Sort all training points by distance and select the K closest ones. The choice of K is a hyperparameter you decide (e.g., K=5 means you consider the 5 nearest neighbors).
3. Majority Voting for Classification
Look at the labels of those K neighbors. Count how many belong to each class. The class with the most votes wins and becomes your prediction.
Example: For a new patient's medical record, find K=7 nearest patients. If 5 were diagnosed "healthy" and 2 were "sick," predict "healthy."
Why It Works
KNN assumes that similar inputs have similar outputs. Points close together in feature space likely share the same label—a natural assumption for many real-world problems.
Key Takeaway: K-Nearest Neighbors classifies by measuring distances to all training points, selecting the K nearest, and using majority voting among their labels—no complex math required, just proximity and democracy.