Cross-validation and tuning

A single train/test split gives you one noisy estimate of performance. Cross-validation gives you several and averages them, so you can compare models and settings with confidence and still keep the test set untouched for the final report.

k-fold cross-validation

1. Shuffle the training data and cut it into $k$ equal folds (5 or 10 is usual).
2. For each fold: train on the other $k - 1$ folds, evaluate on this one.
3. Average the $k$ scores. The spread between them tells you how stable the estimate is.

Every example is used for validation exactly once and for training $k - 1$ times.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scores = cross_val_score(model, X, y, cv=5)
print(scores.round(3))
print(f"mean={scores.mean():.3f}  std={scores.std():.3f}")

Because the pipeline includes the scaler, scaling is refit inside each fold. Preprocessing outside the pipeline would leak.

Stratified folds

For classification, cross_val_score uses StratifiedKFold by default, so each fold has the same class proportions. For regression it uses plain KFold. You can pass either explicitly:

from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier

X, y = load_iris(return_X_y=True)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
print(cross_val_score(DecisionTreeClassifier(random_state=0), X, y, cv=cv).mean().round(3))

Other splitters

• KFold: plain folds, use for regression.
• RepeatedStratifiedKFold: run k-fold several times with different shuffles for a smoother estimate.
• LeaveOneOut: $k$ equals the number of rows. Very expensive; only for tiny datasets.
• GroupKFold: keep rows from the same group (patient, customer) in the same fold, so the model is tested on unseen groups.
• TimeSeriesSplit: folds move forward in time.

Several metrics at once

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_validate
from sklearn.ensemble import RandomForestClassifier

X, y = load_breast_cancer(return_X_y=True)
result = cross_validate(RandomForestClassifier(n_estimators=100, random_state=0), X, y, cv=5,
                        scoring=["accuracy", "f1", "roc_auc"])
for key in ("test_accuracy", "test_f1", "test_roc_auc"):
    print(f"{key:14s} {result[key].mean():.3f}")

Hyperparameters

Parameters are what the model learns (weights). Hyperparameters are what you set before training: tree depth, C, k, learning rate, number of trees. They cannot be learned from the training set directly, because the choice that fits training data best is always "maximum flexibility". They are chosen by cross-validation.

Grid search

Try every combination of a small set of values and keep the best by cross-validated score.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

X, y = load_breast_cancer(return_X_y=True)
grid = {"n_estimators": [50, 150], "max_depth": [3, 6, None], "min_samples_leaf": [1, 4]}
search = GridSearchCV(RandomForestClassifier(random_state=0), grid, cv=5, n_jobs=-1).fit(X, y)
print(search.best_params_)
print(round(search.best_score_, 4))

search.best_estimator_ is a model already refit on all the data with the best settings.

Random search

When there are many hyperparameters, sampling random combinations finds good settings faster than a full grid, because most hyperparameters matter little and random sampling explores the important ones more finely.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import GradientBoostingClassifier
from scipy.stats import randint, uniform

X, y = load_breast_cancer(return_X_y=True)
dist = {"n_estimators": randint(50, 300), "learning_rate": uniform(0.01, 0.3), "max_depth": randint(2, 6)}
search = RandomizedSearchCV(GradientBoostingClassifier(random_state=0), dist, n_iter=8, cv=3, random_state=0).fit(X, y)
print(search.best_params_)
print(round(search.best_score_, 4))

Beyond these, libraries like Optuna do Bayesian optimisation, which learns from earlier trials where to look next.

The full protocol

1. Split off a test set and lock it away.
2. On the remaining data, use cross-validation to compare models and tune hyperparameters.
3. Refit the winner on all the non test data.
4. Evaluate once on the test set. That is the number you report.

If you tune on the test set, your reported score is optimistic and you will not find out until production.

Nested cross-validation

When you want an unbiased estimate of a tuned model and the data is too small to spare a test set, run the grid search inside each outer fold. It is expensive but honest.

Practice

1. Compare k-NN, logistic regression and a random forest on the iris data with 10-fold cross-validation. Print mean and standard deviation for each.
2. Grid search C in [0.01, 0.1, 1, 10, 100] for a scaled logistic regression on the breast cancer data.
3. Explain in two sentences why grid searching on the test set is a mistake.