Train, validation and test sets

The only thing that matters about a model is how it performs on data it has never seen. To measure that, you hide some data from it during training. This lesson covers how to split data, and the two mistakes that silently ruin evaluations.

Why a test set

If you evaluate a model on the same rows it trained on, a model that simply memorised the answers scores perfectly. That tells you nothing about tomorrow's data. A held out test set is a stand in for the future.

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
print("train accuracy:", tree.score(X_train, y_train))
print("test accuracy: ", round(tree.score(X_test, y_test), 3))

Train accuracy is 1.0 because a deep tree can memorise. Test accuracy is the honest number.

train_test_split

• test_size is the fraction (or count) held out. 20 to 30 percent is typical; less if you have millions of rows.
• random_state fixes the shuffle so you get the same split every run. Always set it while developing.
• shuffle=True is the default. Turn it off only for time series, where the test set must be the later data.
• stratify=y keeps the class proportions the same in both parts. Use it for classification, especially with imbalanced classes.

from sklearn.model_selection import train_test_split
import numpy as np

y = np.array([0] * 90 + [1] * 10)
X = np.arange(100).reshape(-1, 1)

_, _, _, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
print("without stratify:", y_test.sum(), "positives in test")
_, _, _, y_test = train_test_split(X, y, test_size=0.2, random_state=1, stratify=y)
print("with stratify:   ", y_test.sum(), "positives in test")

Three way split

If you use the test set to choose between models or settings, it stops being unseen: you are tuning to it. The fix is a third set.

• Training set: fit the model.
• Validation set: compare models and tune hyperparameters.
• Test set: touched once, at the very end, to report the final number.

from sklearn.model_selection import train_test_split
import numpy as np

X = np.arange(1000).reshape(-1, 1)
y = np.arange(1000)

X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, random_state=0)
print(len(X_train), len(X_val), len(X_test))

Cross-validation, covered later, makes the validation step more reliable when data is limited.

Data leakage

Leakage is when information from the test set, or from the future, sneaks into training. The model looks brilliant in evaluation and falls apart in production. Common sources:

• Fitting a scaler or imputer on the full dataset before splitting.
• Features that are only known after the outcome (a "refund issued" column when predicting fraud).
• Duplicate rows that land in both train and test.
• For time series, training on later data and testing on earlier data.

Rule: split first, then do everything else inside the training set.

Time series splits

For data ordered in time, use the past to predict the future.

from sklearn.model_selection import TimeSeriesSplit
import numpy as np

X = np.arange(10).reshape(-1, 1)
for train_idx, test_idx in TimeSeriesSplit(n_splits=3).split(X):
    print("train", train_idx, "test", test_idx)

Practice

1. Split the iris dataset 70/30 with stratification and confirm each class appears in the test set in the same proportion as in the full data.
2. List two features that would be leakage in a model predicting whether a student passes an exam.
3. Create a train/validation/test split of 60/20/20 and print the sizes.