Overfitting and regularisation

A model that does brilliantly on its training data and badly on new data has overfit. A model that does badly on both has underfit. Steering between the two is most of the craft of machine learning, and regularisation is the main tool for it.

Underfitting and overfitting

• Underfitting: the model is too simple to capture the pattern. A straight line through curved data. Training error is high, test error is high.
• Overfitting: the model is so flexible that it fits the noise in the training data as well as the signal. A degree 15 polynomial through 20 points. Training error is tiny, test error is large.
• Good fit: training error is low and test error is close to it.

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(1)
X = np.sort(rng.uniform(0, 3, 20)).reshape(-1, 1)
y = np.sin(2 * X.ravel()) + rng.normal(0, 0.25, 20)
X_test = np.linspace(0, 3, 100).reshape(-1, 1)
y_test = np.sin(2 * X_test.ravel())

for degree in (1, 3, 15):
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(X, y)
    train_err = mean_squared_error(y, model.predict(X))
    test_err = mean_squared_error(y_test, model.predict(X_test))
    print(f"degree {degree:2d}: train MSE={train_err:.3f}  test MSE={test_err:.3f}")

Degree 1 underfits, degree 15 overfits, degree 3 is about right.

Bias and variance

These two words name the two failure modes.

• Bias is error from wrong assumptions: the model cannot represent the truth no matter how much data it sees. High bias means underfitting.
• Variance is error from sensitivity to the particular training sample: retrain on a different sample and you get a very different model. High variance means overfitting.

Expected test error decomposes as $\text{bias}^2 + \text{variance} + \text{irreducible noise}$. Simple models have high bias and low variance; flexible models have low bias and high variance. Increasing model complexity trades one for the other, and the sweet spot is where their sum is smallest.

Diagnosing with learning curves

Plot training and validation error against the number of training examples.

• Both high and close together: high bias. More data will not help; use a more flexible model or better features.
• Training low, validation high, gap not closing: high variance. More data helps, and so does regularisation.

import numpy as np
from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
sizes, train_scores, val_scores = learning_curve(
    DecisionTreeClassifier(random_state=0), X, y, cv=5, train_sizes=[0.2, 0.5, 1.0], random_state=0)
for n, tr, va in zip(sizes, train_scores.mean(axis=1), val_scores.mean(axis=1)):
    print(f"n={n:3d}  train={tr:.3f}  validation={va:.3f}")

Regularisation

Regularisation adds a penalty on large weights to the cost function. The model must now justify every unit of weight with a real reduction in error, which stops it from contorting itself around noise.

L2 (ridge)

Shrinks all weights towards zero smoothly. Weights become small but rarely exactly zero. The standard choice.

L1 (lasso)

Drives some weights to exactly zero, which removes those features from the model. Useful when you suspect many features are irrelevant and want automatic selection.

Elastic net

A mix of the two.

$\lambda$ (called alpha in scikit-learn's regressors and 1/C in its classifiers) controls the strength. Larger means more shrinkage, more bias, less variance. Features must be scaled first, or the penalty punishes features with small units unfairly.

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(0)
X = rng.normal(size=(60, 20))
y = 3 * X[:, 0] - 2 * X[:, 1] + rng.normal(0, 1, 60)     # only 2 of 20 features matter

for name, model in [("ols", LinearRegression()), ("ridge", Ridge(alpha=5)), ("lasso", Lasso(alpha=0.2))]:
    pipe = make_pipeline(StandardScaler(), model)
    score = cross_val_score(pipe, X, y, cv=5, scoring="r2").mean()
    pipe.fit(X, y)
    nonzero = (np.abs(pipe[-1].coef_) > 1e-6).sum()
    print(f"{name:6s} cv R2={score:.3f}  nonzero weights={nonzero}")

Lasso finds that only two features matter.

Other ways to fight overfitting

• More data. The most reliable fix for high variance.
• Simpler model. Fewer features, smaller tree depth, fewer polynomial terms.
• Early stopping. For iterative training, stop when validation error starts rising.
• Ensembles. Averaging many high variance models lowers variance (random forests).
• Dropout and weight decay in neural networks.
• Cross-validation to choose all of the above honestly, which is the next lesson.

Practice

1. Extend the polynomial example to degrees 1 through 12 and find the degree with the lowest test error.
2. Fit Ridge with alpha in [0.01, 0.1, 1, 10, 100] on the 20 feature data and print the cross-validated R squared for each.
3. Take the breast cancer learning curve and replace the tree with logistic regression. Which one shows the bigger gap between training and validation?