A robust supervised learning model for classification. Add points to see the margin maximize.
Overview
The Core Idea of SVM
A Support Vector Machine (SVM) is a powerful supervised learning algorithm used for classification and regression. For classification, its primary goal is to find the optimal hyperplane that best separates data points of different classes in a high-dimensional space. The "best" hyperplane is the one that has the largest possible margin—the distance between the hyperplane and the nearest data point from either class.
Input Mode
Hyperplane Boundary
Status: Ready
Tip: Add points from both classes to calculate the decision boundary.
Model Metrics
Support Vectors0
Bias Parameter (b)0.000
"Support Vectors" are the points that 'support' the boundary. Only these points influence the final model position.
Legend
Decision Plane
Soft Margin
Active SV
Support Vector Machines: A Visual Guide
Explore the core concepts of SVMs by interacting with the model directly.
Maximizing the Margin
Why is a large margin so important? A larger margin implies a more confident and robust classification model. It means the decision boundary is as far as possible from the data points of both classes, making it less sensitive to small variations in the data and more likely to generalize well to new, unseen data.
The hyperplane is defined by the equation w·x - b = 0, where w is a weight vector and b is the bias. The two parallel hyperplanes that define the margin are w·x - b = 1 and w·x - b = -1.
What are Support Vectors?
The data points that lie exactly on the margin boundaries are called Support Vectors. These are the most critical data points in the dataset because they alone "support" or define the position and orientation of the optimal hyperplane.
If you were to move a support vector, the optimal hyperplane would also move.
If you were to remove a non-support vector, the hyperplane would not change at all.
In the visualization, these points are highlighted with a green circle. Notice how few points are actually needed to define the entire boundary!
The widest street between two classes
Many straight lines can separate two groups of points. A support vector machine asks a sharper question: which line leaves the most room?
Picture the boundary not as a line but as a road laid between the two classes. Push the kerbs outwards until they touch the nearest points on each side. The width of that road is the margin, and the SVM chooses the orientation that makes it as wide as possible.
The reason this is a good idea is generalisation. A boundary squeezed up against the training points will misclassify a new point that falls slightly on the wrong side. A boundary with a wide buffer on both sides tolerates that wobble.
The points that touch the kerbs are the support vectors, and they are the only points that matter. Delete every other training point and refit — you get exactly the same boundary. That is an unusual and useful property: the model is defined by a handful of borderline cases rather than by the bulk of the data.
Soft margins, and the C setting
Real data overlaps, so a road with no cars parked in it usually does not exist. Soft-margin SVMs allow violations — points inside the margin, or on the wrong side entirely — and charge for them.
C sets the price:
Small C (0.01–1) — violations are cheap. The model prefers a wide margin and tolerates mistakes. Smoother boundary, more bias, less variance.
Large C (100–10,000) — violations are expensive. The model contorts the boundary to classify every training point correctly. Narrow margin, low bias, high variance.
C is therefore the regularisation dial in disguise, and it is the first thing to tune. On noisy data, small C almost always wins, because insisting on classifying every training point correctly means fitting the noise.
The kernel trick, in one picture
Some data cannot be separated by any straight line. The classic example is a small circle of one class surrounded by a ring of the other — no line works, in any orientation.
Now lift each point into a third dimension, with height equal to its distance from the centre. The inner circle rises into a hill, the outer ring stays low, and a flat plane slices cleanly between them. Projected back down to two dimensions, that plane appears as a circle.
That is the whole idea. Map the data into a higher-dimensional space where a flat boundary exists, and the flat boundary becomes a curved one back home.
The trick part is that you never actually compute the mapping. The SVM's maths only ever needs dot products between pairs of points, and a kernel function computes what that dot product would be in the higher space, directly from the original coordinates. You get the benefit of an enormous — sometimes infinite-dimensional — feature space at the cost of one extra function call.
Kernel
Shape it draws
Use when
Linear
A straight boundary
Many features, text, very large datasets
RBF (Gaussian)
Smooth, closed, curved regions
The default for non-linear numeric data
Polynomial
Curved with a chosen degree
Interactions of a known order
Sigmoid
Similar to a small neural net
Rarely the best choice today
For the RBF kernel, gamma controls how far each training point's influence reaches. Low gamma means broad influence and smooth boundaries; high gamma means each point only influences its immediate neighbourhood, which produces islands around individual points and overfits enthusiastically. Tuning an RBF SVM means searching over C and gamma together — a small grid over powers of ten is the standard approach.
Only a few rows matter
An SVM's boundary is decided by the handful of points nearest to it. Delete every other row and refit -- the boundary does not move.
example_01.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification, make_circles
from sklearn.svm import SVC, LinearSVC
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_classification(n_samples=800, n_features=2, n_informative=2,
n_redundant=0, n_clusters_per_class=1,
class_sep=1.2, random_state=0)
svm = SVC(kernel="linear", C=1.0).fit(X, y)
print("trained on %d rows; %d of them are support vectors."
% (len(X), len(svm.support_)))
print()
# refit using ONLY the support vectors
svm2 = SVC(kernel="linear", C=1.0).fit(X[svm.support_], y[svm.support_])
print("refit on those %d rows alone:" % len(svm.support_))
print(" original coefficients", np.round(svm.coef_[0], 6), "b %.6f" % svm.intercept_[0])
print(" refit coefficients ", np.round(svm2.coef_[0], 6), "b %.6f" % svm2.intercept_[0])
print(" identical predictions on all %d rows: %s"
% (len(X), (svm.predict(X) == svm2.predict(X)).all()))
print()
print("the other %d rows contributed nothing. no other common model has this"
% (len(X) - len(svm.support_)))
print("property -- move a far-away point in logistic regression and the fit shifts.")
print()
print("C is how much you are willing to pay to avoid a mistake:")
print("%8s %10s %14s %12s" % ("C", "support", "margin width", "train acc"))
for C in (0.01, 0.1, 1.0, 100.0):
m = SVC(kernel="linear", C=C).fit(X, y)
width = 2 / np.linalg.norm(m.coef_[0])
print("%8.2f %10d %14.4f %12.4f"
% (C, len(m.support_), width, m.score(X, y)))
print(" small C: a wide margin, more points inside it, more tolerance.")
print(" large C: a narrow margin that bends to fit awkward points -- overfitting.")
print()
Xc, yc = make_circles(n_samples=800, noise=0.22, factor=0.5, random_state=0)
print("the kernel trick, on data no line can split (one ring inside another):")
for name, m in (("linear kernel", SVC(kernel="linear")),
("polynomial d=3", SVC(kernel="poly", degree=3)),
("rbf kernel", SVC(kernel="rbf"))):
print(" %-16s cv accuracy %.4f"
% (name, cross_val_score(m, Xc, yc, cv=5).mean()))
print()
print("the rbf kernel never builds the higher-dimensional space it implies --")
print("it only computes distances in it. that is what makes it affordable.")
print()
print("gamma controls how far one point's influence reaches:")
print("%10s %10s %10s %10s" % ("gamma", "train", "cv", "gap"))
for g in (0.01, 0.1, 1.0, 10.0, 100.0, 1000.0):
tr = SVC(kernel="rbf", gamma=g).fit(Xc, yc).score(Xc, yc)
cv = cross_val_score(SVC(kernel="rbf", gamma=g), Xc, yc, cv=5).mean()
print("%10.2f %10.4f %10.4f %10.4f" % (g, tr, cv, tr - cv))
print()
print("at gamma 0.01 the influence of each point is so wide that the two rings")
print("blur into one -- it is barely better than guessing, both ways.")
print("as gamma climbs, each point gets its own small island: training accuracy")
print("keeps rising while the honest score turns and falls. the gap column is")
print("the overfitting, and it is doing exactly what a gap should do.")
Output
Guided experiments
Use the canvas above to build your intuition:
Start Fresh: Click "Reset Canvas". Add a few "Indigo" points on the left and a few "Amber" points on the right. Observe how the white decision boundary and the dashed margin lines appear, separating the two classes.
Identify Support Vectors: Add a point very close to the boundary. Notice it gets a green circle, becoming a support vector. The "Support Vectors" count in the metrics panel will increase.
Test Insensitivity: Now, add a new point far away from the boundary, deep within its class territory. Observe that the boundary does not move. This point is not a support vector and has no influence on the model.
Create a Challenge: Place an "Indigo" point in the middle of the "Amber" cluster. This is a misclassified point. The SVM will try its best to find a boundary, but it demonstrates the concept of a "soft margin," where some misclassifications are tolerated to find a boundary that generalizes better than one that perfectly separates the training data (which could be overfit).
Move a Support Vector: Reset the canvas and create a simple, clean separation. Now, add a new point that is closer to the boundary than the current support vectors. See how the margin shrinks and the hyperplane adjusts its position. This proves that only the support vectors dictate the outcome.
The Kernel Trick
What if the data isn't linearly separable? This is where SVMs truly shine. The kernel trick is a powerful technique that allows SVMs to create non-linear decision boundaries. It works by mapping the data into a higher-dimensional space where it becomes linearly separable.
Imagine data points arranged in a circle, with one class inside and another outside. A straight line can't separate them. But if we add a third dimension (e.g., z = x² + y²), we can lift the points into a 3D space where a simple plane can separate them. The kernel trick achieves this without the computational cost of actually transforming the data, by modifying the dot product calculation.
Why Use SVM?
Effective in High-Dimensional Spaces: Works well even when the number of dimensions exceeds the number of samples.
Memory Efficient: It only uses a subset of training points (the support vectors) in the decision function.
Versatile: Different kernel functions can be specified for the decision function, making it adaptable to various data types.
Practical use, and the scaling requirement
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import GridSearchCV
pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf"))
grid = GridSearchCV(pipe, {
"svc__C": [0.1, 1, 10, 100],
"svc__gamma": ["scale", 0.01, 0.1, 1],
}, cv=5)
grid.fit(X_train, y_train)
The StandardScaler is not optional. SVMs are built on distances, so an unscaled feature with a large range dominates the kernel and the margin becomes a statement about that one column. This is the most common reason an SVM "does not work".
Cost is the other practical constraint. Training scales roughly between the square and the cube of the number of rows, which makes kernel SVMs impractical much beyond about 100,000 examples. For larger data use LinearSVC, which scales far better, or switch to a linear model trained by stochastic gradient descent.
For probabilities, note that an SVM natively outputs a signed distance from the boundary, not a probability. probability=True fits a calibration model on top, at the cost of an internal cross-validation, and the resulting probabilities can disagree with the raw decision function near the boundary.
Where SVMs are still the right answer
Small to medium datasets with clear margins. A few thousand rows and well-separated classes is where SVMs shine and often beat everything else.
High-dimensional data. Text classification with thousands of features and fewer documents than features is a classic linear-SVM win; the margin idea copes with dimensions that make other models nervous.
Bioinformatics. Gene expression datasets — many features, few samples — are the textbook use case.
Novelty detection. One-class SVMs learn a boundary around "normal" and flag whatever falls outside.
Where they lose: very large datasets (too slow), problems needing calibrated probabilities (not native), heavily imbalanced classes (use class_weight="balanced"), and situations demanding interpretability — an RBF boundary is not a story anyone can tell.
Questions people ask
Which kernel should I start with? RBF, with scaled features and a small grid over C and gamma. If the linear kernel performs equally well, keep it — it is faster and more interpretable.
What does gamma="scale" do? It sets gamma from the number of features and the data's variance, which is a sensible default and usually better than the older "auto".
Can SVMs do regression? Yes — support vector regression fits a tube of width ε around the data and penalises only points outside it, which makes it robust to small errors.
Why is my SVM so slow? Almost certainly too many rows for a kernel SVM. Subsample, switch to LinearSVC, or use an approximate kernel map such as Nystroem followed by a linear model.
How does it handle more than two classes? By training many binary classifiers — one-vs-one in scikit-learn's SVC, which is why training time grows quickly with the number of classes.
Do outliers hurt? Less than you might expect, because only points near the boundary matter. An outlier far inside its own class is ignored entirely; an outlier sitting among the other class is a support vector and does damage — lower C to reduce it.
Recap in one screen
The SVM picks the boundary with the widest margin between the classes.
Only the closest points — the support vectors — define it.
C prices margin violations: small C for noisy data, large C to insist on perfect training accuracy.
Kernels give curved boundaries by computing dot products in a higher space without ever visiting it.
Scale the features, tune C and gamma together, and switch to a linear model past ~100,000 rows.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The Core Idea of SVM”?
A Support Vector Machine (SVM) is a powerful supervised learning algorithm used for classification and regression. For classification, its primary goal is to find the optimal hyperplane that best separates data points of different classes in a high-dimensional space. The "best" hyperplane is the one that has the largest possible margin —the distance between the hyperplane and the nearest data point from either class.
What does this module say about “Maximizing the Margin”?
Why is a large margin so important? A larger margin implies a more confident and robust classification model. It means the decision boundary is as far as possible from the data points of both classes, making it less sensitive to small variations in the data and more likely to generalize well to new, unseen data.
What does this module say about “What are Support Vectors”?
The data points that lie exactly on the margin boundaries are called Support Vectors . These are the most critical data points in the dataset because they alone "support" or define the position and orientation of the optimal hyperplane.
Cheat sheet
Support Vector Machines
A Support Vector Machine (SVM) is a powerful supervised learning algorithm used for classification and regression. For classification, its primary goal is to find the optimal hyperplane that best separates data points of different classes in a high-dimensional space. The "best" hyperplane is the one that has the largest possible margin—the distance between the hyperplane and the nearest data point from either class.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.