IndusAI Docs
indusai.co

Support vector machines

Lesson 52 of 62, about 6 min

A support vector machine (SVM) draws the boundary between two classes so that the gap on either side of it is as wide as possible. That simple objective, plus a trick for handling curved boundaries, made SVMs the strongest general purpose classifier for a decade, and they remain excellent for small to medium datasets with many features.

The widest street

Many lines can separate two well separated classes. Logistic regression picks one based on probabilities. An SVM picks the one that maximises the margin: the distance from the boundary to the nearest point of either class. Those nearest points are the support vectors. Move any other point and the boundary does not change; move a support vector and it does.

The boundary is w⋅x+b=0w \cdot x + b = 0. The margin width turns out to be 2/∥w∥2 / \|w\|, so maximising the margin means minimising ∥w∥\|w\|, subject to every point being on the correct side:

min⁡w,b12∥w∥2subject toyi(w⋅xi+b)≥1 for all i\min_{w, b} \tfrac{1}{2}\|w\|^2 \quad \text{subject to} \quad y_i (w \cdot x_i + b) \ge 1 \ \text{for all } i

where the labels yiy_i are +1+1 or −1-1.

Soft margins

Real data overlaps. The soft margin version allows points inside the margin or on the wrong side, but charges a penalty for each. The parameter CC sets the price:

min⁡w,b12∥w∥2+C∑iξi\min_{w, b} \tfrac{1}{2}\|w\|^2 + C \sum_i \xi_i
  • Large CC: violations are expensive, the model bends to fit every point, narrow margin, risk of overfitting.
  • Small CC: violations are cheap, wide margin, some training points misclassified, better generalisation.

Linear SVM in scikit-learn

python
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

svm = make_pipeline(StandardScaler(), SVC(kernel="linear", C=1.0)).fit(X_train, y_train)
print("test accuracy:", round(svm.score(X_test, y_test), 3))
print("support vectors:", svm[-1].n_support_)

Scaling is essential: the margin is measured in feature units.

The kernel trick

Some classes cannot be split by a straight line in their original features, but can be after mapping the features into a higher dimensional space. For example, points inside a circle versus outside become linearly separable if you add the feature x12+x22x_1^2 + x_2^2.

The kernel trick lets the SVM work in that higher space without ever computing the new features. It only needs the similarity between pairs of points, given by a kernel function:

  • Linear: K(a,b)=a⋅bK(a, b) = a \cdot b
  • Polynomial: K(a,b)=(γ a⋅b+r)dK(a, b) = (\gamma\, a \cdot b + r)^d
  • RBF (Gaussian): K(a,b)=exp⁡(−γ∥a−b∥2)K(a, b) = \exp(-\gamma \|a - b\|^2)

RBF is the default and the usual choice. γ\gamma controls how far the influence of a single training point reaches: small γ\gamma gives smooth boundaries, large γ\gamma gives wiggly ones that hug the training points.

python
import numpy as np
from sklearn.svm import SVC

# a ring of class 1 around a core of class 0: not linearly separable
rng = np.random.default_rng(0)
angles = rng.uniform(0, 2 * np.pi, 200)
r = np.where(np.arange(200) < 100, rng.uniform(0, 1, 200), rng.uniform(2, 3, 200))
X = np.column_stack([r * np.cos(angles), r * np.sin(angles)])
y = (np.arange(200) >= 100).astype(int)

for kernel in ("linear", "rbf"):
    clf = SVC(kernel=kernel).fit(X, y)
    print(f"{kernel:6s} training accuracy: {clf.score(X, y):.3f}")

Tuning C and gamma

They interact, so search over a grid with cross-validation.

python
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline

X, y = load_breast_cancer(return_X_y=True)
pipe = make_pipeline(StandardScaler(), SVC())
grid = {"svc__C": [0.1, 1, 10], "svc__gamma": ["scale", 0.01, 0.001]}
search = GridSearchCV(pipe, grid, cv=5).fit(X, y)
print(search.best_params_, round(search.best_score_, 3))

Probabilities and multi-class

An SVM outputs a signed distance from the boundary, not a probability. SVC(probability=True) fits an extra calibration step if you need predict_proba. For more than two classes scikit-learn trains one SVM per pair of classes and votes.

SVMs for regression

SVR fits a tube of width ϵ\epsilon around the data and only penalises points outside it. It is useful when you want a robust fit that ignores small noise.

Strengths and weaknesses

  • Very effective in high dimensional spaces, including when features outnumber samples (text classification).
  • Memory efficient: only the support vectors matter after training.
  • Training scales badly with the number of rows (roughly quadratic), so beyond about 100,000 samples use LinearSVC or a different family.
  • Needs scaling and tuning; not naturally probabilistic.

Practice

  1. Train the linear SVM above with C=0.01 and C=100 and compare the number of support vectors and the test accuracy.
  2. On the ring dataset, try gamma=0.1 and gamma=50 with the RBF kernel and compare accuracy on a fresh sample of points.
  3. Use LinearSVC on the breast cancer data and compare speed and accuracy with SVC(kernel="linear").