How the boundary is chosen
For a linear binary classifier, the score is w·x + b, and the boundary is where that score is zero. Here x is a feature vector, w sets the direction of the boundary, and b shifts it. With a conventional scaling of w, the margin edges are w·x + b = 1 and w·x + b = −1; their separation is 2 / ||w||.
Real classes often overlap. A soft-margin SVM allows points to fall inside the margin or be misclassified, then penalizes those violations. In scikit-learn, C controls the trade-off: a larger value penalizes violations more strongly, while a smaller value allows more violations in exchange for stronger regularisation. Tune it on validation data rather than assuming that perfect training separation is best.
Which points set the boundary?
Support vectors are the training examples with nonzero influence on the fitted boundary. In a hard-margin solution they sit on the margin; in a soft-margin solution they can sit on or inside it, including on the wrong side. Moving a support vector can change the fit. A non-support vector does not directly constrain the current solution, though changing the training set or preprocessing can affect a refit.
In the visualiser, choose Indigo Class or Amber Class and add points near the boundary. Click Reset Canvas to start again. Points close to the margin become the useful cases to inspect; a point deep inside its class usually contributes no new constraint.
Kernels allow curved boundaries
A straight boundary cannot separate a centre group from a surrounding ring. A kernel lets an SVM compute similarities as if the points had been mapped into a different feature space, without explicitly building all the new coordinates. A linear boundary there can appear nonlinear in the original features.
| Kernel | Typical use | Trade-off |
|---|
| Linear | Many features or a roughly linear relationship | Fast and easier to inspect; may miss nonlinear structure |
| RBF | Nonlinear numeric data | Flexible; tune C and gamma together |
| Polynomial | A known interaction pattern may matter | Degree and scale need care |
For RBF, gamma sets how local each training point’s influence is. A high value can create a jagged boundary around individual observations; a low value makes the boundary smoother and may underfit. A grid over a modest range of C and gamma, evaluated with cross-validation, is a practical starting point.
Fit it without leaking or letting units dominate
SVMs are not scale-invariant. If income is measured in thousands and age in years, distances and the fitted margin can be dominated by the larger-valued feature. Scaling is usually important, especially for RBF kernels. Fit the scaler inside each training fold so validation rows do not influence its statistics:
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
model = make_pipeline(StandardScaler(), SVC(kernel="rbf"))
search = GridSearchCV(model, {
"svc__C": [0.1, 1, 10],
"svc__gamma": ["scale", 0.01, 0.1],
}, cv=5)
search.fit(X_train, y_train)
Kernel SVMs can be costly as the number of training rows grows. The scikit-learn SVC documentation notes that fit time scales at least quadratically with sample count and can become impractical beyond tens of thousands of samples. For large datasets, compare a linear model such as LinearSVC or SGDClassifier.
What SVM scores do and do not tell you
An SVM’s decision function is a signed score, not a calibrated probability. If a task needs probabilities, calibrate on held-out or out-of-fold predictions and evaluate calibration separately. SVMs can work well with small or medium datasets and carefully chosen features, but they are not automatically the best choice for every problem. Compare against a simple baseline using the same validation design.
The original support-vector networks paper describes the margin approach for classification. The scikit-learn SVM guide documents kernels, scaling, and computational trade-offs. Continue with feature scaling and cross-validation.