Drag the target point to see how the KNN algorithm classifies it based on proximity.
Overview
Overview
The K-Nearest Neighbors (KNN) algorithm is one of the simplest and most intuitive classification algorithms in machine learning. Its core idea is based on the saying, "You are known by the company you keep." To classify a new, unknown data point, KNN looks at the 'K' closest data points from the training set and makes a prediction based on a majority vote. This lab lets you explore this process interactively.
Parameters
3
5
Visualization
Drag the target circle to classify.
Real-time Analysis
Result: Based on the 5 nearest neighbors, the target is classified as:
Predicted Class
--
Neighbor Votes
Inside K-Nearest Neighbors (KNN)
The K-Nearest Neighbors (KNN) algorithm is one of the simplest and most intuitive classification algorithms in machine learning. Its core idea is based on the saying, "You are known by the company you keep." To classify a new, unknown data point, KNN looks at the 'K' closest data points from the training set and makes a prediction based on a majority vote. This lab lets you explore this process interactively.
How KNN Works: A Simple Democracy
Imagine you have a new student (the target point) and you want to predict which class they belong to. KNN doesn't learn a "model" in the traditional sense. Instead, it follows a simple, lazy procedure at prediction time.
1. Find the Neighbors
First, the algorithm calculates the distance (usually Euclidean distance) from the new target point to every single point in the training data. It then identifies the 'K' points that are closest to the target—these are its "nearest neighbors." In the visualization, a circle is drawn around the target to show the neighborhood, and lines connect to the K neighbors within it.
2. Hold a Vote
Next, the algorithm looks at the classes of these K neighbors and holds a vote. Each neighbor gets one vote for its class. The class with the most votes wins, and that becomes the prediction for the new target point. The "Neighbor Votes" panel on the right shows this process in real-time.
Nothing is learned until you ask
KNN is unusual in being a model with no training step worth the name. fit() stores the data. That is it. All the work happens when a prediction is requested:
Measure the distance from the new point to every stored point.
Take the k nearest.
For classification, take the majority vote of their labels. For regression, take their average.
This is called lazy learning, and it has consequences you feel immediately. Training is instant. Prediction is slow, because every prediction scans the entire dataset. Memory use is the size of your data, forever, because the data is the model. And there is no equation to inspect afterwards — you cannot show anyone "the model", only the neighbours it consulted.
It also means KNN adapts instantly to new data. Add a row and it is available to the next prediction with no retraining. For a small, slowly-changing dataset that is a genuine advantage.
What k actually controls
k is the only important setting, and it slides the model along the bias-variance line.
k = 1 — every point takes the label of its single nearest neighbour. Training accuracy is 100% by construction, because each point is its own nearest neighbour. The decision boundary is a jagged outline that wraps around every individual point, including every mislabelled one. Maximum variance.
k = 5 to 20 — the usual working range. Boundaries smooth out and single odd points stop mattering.
k = n — every prediction is the majority class of the whole dataset. The model has become a constant. Maximum bias.
Two practical rules. Use an odd k for two-class problems so a vote cannot tie. And choose k by cross-validation rather than by folklore — the square root of n is a common starting suggestion, not an answer.
Distance weighting is the refinement worth knowing: instead of every neighbour getting one vote, weight each vote by 1/distance so closer neighbours count for more. It softens the choice of k considerably, because a distant neighbour scraping into the top k barely affects the outcome.
Scaling is not optional here
KNN is distance, and distance is unitless arithmetic on whatever numbers you hand it. Consider predicting from age (18–80) and salary (20,000–200,000).
Two people: A is 25 with a salary of 30,000; B is 60 with a salary of 30,500. The age difference is 35, the salary difference is 500. Squared, that is 1,225 against 250,000. The distance calculation is 99.5% salary. Age has effectively been deleted from the model.
Standardise both columns — subtract the mean, divide by the standard deviation — and the two features get comparable say. This is the number one cause of KNN "not working", and it is a one-line fix.
The same logic applies to the curse of dimensionality, which hits KNN harder than most models. In high dimensions, distances between all pairs of points converge towards each other: the nearest neighbour is barely nearer than the furthest. With 100 mostly-irrelevant features, "nearest" becomes meaningless and KNN degrades to guessing. Feature selection or PCA before KNN is often the difference between a working model and a coin flip.
The model that does nothing until you ask
k-NN has no training step worth the name -- it stores the data and does all its work at prediction time. That single fact explains every one of its trade-offs.
example_01.pyscikit-learn
import numpy as np
import time
from sklearn.datasets import make_classification
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_classification(n_samples=4000, n_features=10, n_informative=5,
flip_y=0.10, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
knn = KNeighborsClassifier(5)
t0 = time.time(); knn.fit(Xtr, ytr); fit_s = time.time() - t0
t0 = time.time(); knn.predict(Xte); pred_s = time.time() - t0
print("fit took %.4f s, predicting %d rows took %.4f s" % (fit_s, len(Xte), pred_s))
print("that ratio is backwards from every other model, and it is the point:")
print("'fitting' is just storing the array.")
print()
print("one prediction, by hand:")
row = Xte[0]
d = np.linalg.norm(Xtr - row, axis=1)
near = np.argsort(d)[:5]
print(" the five nearest training rows:")
for i in near:
print(" distance %.4f, label %d" % (d[i], ytr[i]))
print(" majority vote -> %d" % np.bincount(ytr[near]).argmax())
print(" sklearn says -> %d" % knn.predict([row])[0])
print()
print("k is a smoothing knob:")
print("%6s %10s %10s" % ("k", "train", "test"))
for k in (1, 3, 5, 15, 50, 200):
m = KNeighborsClassifier(k).fit(Xtr, ytr)
print("%6d %10.4f %10.4f" % (k, m.score(Xtr, ytr), m.score(Xte, yte)))
print(" k=1 scores 1.0000 on training data by definition -- the nearest")
print(" neighbour of a training point is itself. it means nothing.")
print()
print("scaling is not optional here, it is the algorithm:")
Xs = X.copy(); Xs[:, 0] *= 500
print(" one column x500, raw : %.4f"
% cross_val_score(KNeighborsClassifier(5), Xs, y, cv=5).mean())
print(" same data, scaled : %.4f"
% cross_val_score(make_pipeline(StandardScaler(), KNeighborsClassifier(5)),
Xs, y, cv=5).mean())
print()
print("and it degrades as dimensions grow, because distances stop separating:")
for dim in (2, 5, 20, 100, 400):
Xd, yd = make_classification(n_samples=1200, n_features=dim,
n_informative=2, n_redundant=0,
flip_y=0.05, random_state=0)
s = cross_val_score(make_pipeline(StandardScaler(), KNeighborsClassifier(5)),
Xd, yd, cv=5).mean()
print(" %3d features (still only 2 informative): %.4f" % (dim, s))
print()
print("the informative signal never changed. the noise dimensions drowned it,")
print("because every extra dimension adds to the distance whether it means")
print("anything or not. that is the curse of dimensionality, and k-NN feels it")
print("harder than almost anything else.")
Output
Guided tour
The most crucial parameter in KNN is 'K'. Let's see how it affects the outcome.
The Effect of a Small K (K=1): Set the "Neighbors (K)" slider to 1. Now, drag the target point around the canvas. Notice that the prediction is extremely volatile; it changes to the class of the single closest point. This creates a very complex and jagged decision boundary, which is a classic sign of high variance and overfitting. The model is too sensitive to individual data points, including noise.
The Effect of a Large K (K=20): Now, set the "Neighbors (K)" slider to its maximum value, 20. Drag the target point again. You'll see that the prediction is much more stable and changes less frequently. The decision boundary becomes much smoother. However, if K is too large, the model might become too generalized and ignore local patterns, leading to high bias and underfitting. For example, a small cluster of one class might be "outvoted" by the more dominant class in the wider neighborhood.
Finding a "Good" K: Set K to a moderate value, like 5 or 7. Drag the target point to a boundary region between two classes. With K=5, if three neighbors are Class A and two are Class B, the target will be classified as A. Now, slowly increase K. You might find a point where the prediction flips because the expanding neighborhood now includes more points from Class B. This demonstrates the trade-off: a good K balances the influence of local patterns with overall stability.
The "Curse of Dimensionality": While this 2D visualization is simple, imagine if we had 100 features (dimensions). In high-dimensional space, the concept of "distance" becomes less meaningful. All points tend to be far away from each other, making it difficult to find truly "close" neighbors. This is a major challenge for KNN in practice.
Where that leaves you
Lazy Learning: KNN is a "lazy" algorithm because it doesn't do any work during the training phase. It simply stores the entire training dataset. All the computation happens at prediction time.
Non-Parametric: It makes no assumptions about the underlying data distribution. This allows it to learn complex, non-linear decision boundaries.
Computationally Expensive at Prediction: Because it has to compare the new point to every single training point, making predictions can be slow for large datasets.
Importance of Feature Scaling: Since KNN relies on distance, features with larger scales can dominate the distance calculation. It's crucial to scale your data (e.g., using normalization or standardization) before applying KNN.
Making it fast enough to use
A brute-force prediction compares against every stored point: with 1 million rows and 50 features, that is 50 million multiply-adds per prediction. Fine for a notebook, fatal for an API with a 50ms budget.
The standard remedies:
KD-trees partition space along feature axes so whole regions can be skipped. Excellent up to roughly 20 dimensions, then they stop helping.
Ball trees partition into nested hyperspheres and cope somewhat better with more dimensions.
Approximate nearest neighbours (HNSW, IVF, product quantisation) give up the guarantee of finding the exact nearest neighbours in exchange for enormous speed-ups. Every vector database is built on these, and for search and recommendation the approximation is invisible.
Reduce first. PCA down to 20–50 components makes both the distance computation and the tree structures dramatically more effective.
Scikit-learn's algorithm="auto" picks between brute force, KD-tree and ball tree for you based on the data's shape, which is usually the right call.
Where KNN is genuinely the right tool
Recommendation systems. "Users like you also bought" is literally a nearest-neighbour query over user vectors.
Missing value imputation.KNNImputer fills a gap with the average of the most similar complete rows, which respects relationships between columns in a way that filling with the column mean does not.
Anomaly detection. A point whose k-th nearest neighbour is unusually far away is, by definition, somewhere the data is thin.
Small tabular problems with irregular boundaries. When you have a few thousand rows, a handful of well-scaled features and a genuinely non-linear boundary, KNN is quick to get working and hard to beat by much.
A baseline. It takes three lines and gives you a number your fancier model has to beat.
Where it is the wrong tool: very large datasets with tight latency budgets, high-dimensional raw data, heavily imbalanced classes (the majority class simply outvotes the minority in every neighbourhood), and any situation where you must explain the decision as a rule rather than as "these five similar cases".
Questions people ask
How do I choose k? Cross-validate over a range and plot the score. You will usually see a rise, a plateau, and a slow decline; pick from the plateau rather than the single best point, which is often noise.
Which distance should I use? Euclidean for continuous, well-scaled features; Manhattan when features are sparse or high-dimensional; cosine when direction matters more than magnitude, as with text; Hamming for binary or categorical data.
Does KNN work with categorical features? Only after encoding, and even then the distances one-hot encoding produces are crude. Consider Gower distance for mixed data, or a model that handles categories natively.
Why is my accuracy perfect on training data and poor on test data? You are almost certainly using k = 1. Every training point is its own neighbour, so training accuracy is a meaningless 100%. Increase k.
How does it handle imbalanced classes? Badly, by default. Distance weighting helps a little; resampling or a class-weighted vote helps more.
Is KNN affected by irrelevant features? Severely. Every useless column adds noise to every distance. Feature selection matters more for KNN than for tree-based models, which can simply ignore a column.
Recap in one screen
KNN stores the data and answers each query by consulting the k closest stored points.
No training cost, high prediction cost, and the dataset is the model.
Small k means jagged, unstable boundaries; large k means smooth, eventually constant ones.
Scale your features, or the widest-ranging column becomes the only column.
It degrades badly in high dimensions — reduce first.
Ideal as a baseline, an imputer, and the engine behind "similar items".
Check yourself
0 of 3
Answer without scrolling back up.
You set K = 1 and the decision boundary becomes jagged and unstable. That is a symptom of:
With K = 1 every single point, including mislabelled ones, gets its own territory. High variance and a jagged boundary are the classic signature of overfitting.
Why does KNN need its features scaled?
Distance sums squared differences. A salary in the tens of thousands swamps an age in the tens, so the model quietly becomes 'nearest by salary' no matter what you intended.
KNN is called a lazy learner because:
Training is just storing the dataset. All the cost lands on prediction, when it must measure the new point against every stored one - which is why KNN is expensive to serve at scale.
Cheat sheet
K-Nearest Neighbors
The K-Nearest Neighbors (KNN) algorithm is one of the simplest and most intuitive classification algorithms in machine learning. Its core idea is based on the saying, "You are known by the company you keep." To classify a new, unknown data point, KNN looks at the 'K' closest data points from the training set and makes a prediction based on a majority vote. This lab lets you explore this process interactively.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.