Random forests and ensembles

One decision tree overfits. A hundred decision trees, each trained a little differently, and averaged together, do not. That is the idea behind ensembles, and random forests and gradient boosting are the two ensemble methods that win most tabular data problems in practice.

Why averaging helps

Every model makes errors. If the errors of many models are different from each other (uncorrelated), averaging their predictions cancels much of the noise while keeping the signal. The trick is making the models different without making them bad.

Bagging

Bootstrap aggregating: train each model on a random sample of the training rows drawn with replacement (so some rows repeat and about a third are left out), then average the predictions (or take a majority vote).

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import BaggingClassifier

X, y = load_breast_cancer(return_X_y=True)
single = DecisionTreeClassifier(random_state=0)
bagged = BaggingClassifier(DecisionTreeClassifier(), n_estimators=50, random_state=0)
print("single tree:", round(cross_val_score(single, X, y, cv=5).mean(), 3))
print("bagged 50:  ", round(cross_val_score(bagged, X, y, cv=5).mean(), 3))

Random forest

A random forest is bagging of decision trees with one extra source of randomness: at each split, the tree only considers a random subset of the features (by default $\sqrt{n}$ for classification). This stops every tree from grabbing the same strong feature at the top and makes the trees more different, which makes the average better.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

X, y = load_breast_cancer(return_X_y=True)
rf = RandomForestClassifier(n_estimators=200, random_state=0)
print("random forest:", round(cross_val_score(rf, X, y, cv=5).mean(), 3))

Important settings:

• n_estimators: number of trees. More is better up to a point, then it just costs time. 100 to 500 is typical.
• max_features: features considered per split.
• max_depth, min_samples_leaf: as for single trees, though forests tolerate deep trees well.
• n_jobs=-1: use every CPU core.

Out-of-bag score

Each tree never saw about a third of the rows. Predicting those rows with only the trees that skipped them gives a free validation estimate.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier

X, y = load_breast_cancer(return_X_y=True)
rf = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0).fit(X, y)
print("out-of-bag accuracy:", round(rf.oob_score_, 3))

Feature importance

Averaged across all trees, and more reliable than from a single tree.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier

data = load_breast_cancer()
rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(data.data, data.target)
ranked = sorted(zip(data.feature_names, rf.feature_importances_), key=lambda t: -t[1])
for name, imp in ranked[:5]:
    print(f"{name:25s} {imp:.3f}")

Boosting

Bagging trains trees in parallel and averages. Boosting trains them one after another, each new tree focusing on the examples the previous ones got wrong. The final prediction is a weighted sum.

Gradient boosting fits each new tree to the residual errors of the ensemble so far, effectively doing gradient descent in the space of functions. It uses many shallow trees and a small learning rate.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import GradientBoostingClassifier

X, y = load_breast_cancer(return_X_y=True)
gb = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, max_depth=3, random_state=0)
print("gradient boosting:", round(cross_val_score(gb, X, y, cv=5).mean(), 3))

Key settings: n_estimators and learning_rate trade off (more trees with a smaller rate is usually better), max_depth is small (2 to 5), and subsample < 1 adds randomness.

Outside scikit-learn, XGBoost, LightGBM and CatBoost are highly optimised gradient boosting libraries and are the usual choice for competitions and production tabular models. HistGradientBoostingClassifier in scikit-learn is a fast built in alternative.

Random forest or gradient boosting

• Random forest: harder to get wrong, parallel, good default, a little less accurate at the top end.
• Gradient boosting: usually more accurate when tuned, sequential, easier to overfit, more knobs.

Start with a random forest to get a baseline in minutes, then try boosting if you need more.

Regression versions

RandomForestRegressor and GradientBoostingRegressor work the same way and predict the mean of the leaves.

Practice

1. On the breast cancer data, plot (or print) cross-validated accuracy for a random forest with 1, 5, 20, 100 and 300 trees.
2. Compare a random forest and gradient boosting on the iris data with 5-fold cross-validation.
3. Train a RandomForestRegressor on the California housing dataset (sklearn.datasets.fetch_california_housing, needs internet) or on synthetic data from make_regression, and print the top three features.