Regression metrics

A regression model predicts numbers, so its errors are numbers too. The metrics in this lesson summarise those errors in different ways, and each answers a slightly different question. Know all four and choose the one that matches what you care about.

Setup

import numpy as np
y_true = np.array([200, 150, 300, 250, 100], dtype=float)
y_pred = np.array([210, 140, 280, 270, 130], dtype=float)
errors = y_pred - y_true
print("errors:", errors)

Every metric below is built from this errors array.

Mean absolute error (MAE)

The average size of the error, in the same units as the target. Easy to explain: "on average we are off by 18 rupees". Every error counts in proportion to its size.

import numpy as np
from sklearn.metrics import mean_absolute_error
y_true = np.array([200, 150, 300, 250, 100.0]); y_pred = np.array([210, 140, 280, 270, 130.0])
print(np.abs(y_pred - y_true).mean())
print(mean_absolute_error(y_true, y_pred))

Mean squared error (MSE)

Squaring punishes large errors far more than small ones: an error of 30 costs 900, an error of 10 costs 100. This is the quantity most models minimise during training. Its units are squared, which makes it hard to read directly.

import numpy as np
from sklearn.metrics import mean_squared_error
y_true = np.array([200, 150, 300, 250, 100.0]); y_pred = np.array([210, 140, 280, 270, 130.0])
print(((y_pred - y_true) ** 2).mean())
print(mean_squared_error(y_true, y_pred))

Root mean squared error (RMSE)

Back in the target's units, but still weighted towards large errors. RMSE is always at least as large as MAE; a big gap between them tells you a few predictions are badly off.

import numpy as np
from sklearn.metrics import root_mean_squared_error
y_true = np.array([200, 150, 300, 250, 100.0]); y_pred = np.array([210, 140, 280, 270, 130.0])
print(root_mean_squared_error(y_true, y_pred).round(3))

R squared

The fraction of the target's variance that the model explains, compared with simply predicting the mean $\bar{y}$ every time. 1.0 is perfect; 0 means no better than the mean; negative means worse than the mean. It is unitless, so it is handy for comparing across problems, but it says nothing about the size of the errors in practical terms.

import numpy as np
from sklearn.metrics import r2_score
y_true = np.array([200, 150, 300, 250, 100.0]); y_pred = np.array([210, 140, 280, 270, 130.0])
ss_res = ((y_pred - y_true) ** 2).sum()
ss_tot = ((y_true.mean() - y_true) ** 2).sum()
print(round(1 - ss_res / ss_tot, 4))
print(round(r2_score(y_true, y_pred), 4))

Mean absolute percentage error (MAPE)

Error as a percentage of the true value. Intuitive for business reporting, but it explodes when true values are near zero and it penalises over prediction more than under prediction.

import numpy as np
from sklearn.metrics import mean_absolute_percentage_error
y_true = np.array([200, 150, 300, 250, 100.0]); y_pred = np.array([210, 140, 280, 270, 130.0])
print(f"{mean_absolute_percentage_error(y_true, y_pred):.1%}")

Which one to use

Report at least one absolute metric (MAE or RMSE) alongside R squared: a model can have R squared of 0.9 and still be useless if the remaining error is bigger than the decisions it informs.

Always compare with a baseline

import numpy as np
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error

rng = np.random.default_rng(0)
X = rng.uniform(0, 10, (200, 1))
y = 3 * X.ravel() + 5 + rng.normal(0, 2, 200)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

for name, model in [("mean baseline", DummyRegressor()), ("linear", LinearRegression())]:
    model.fit(X_train, y_train)
    print(f"{name:14s} MAE = {mean_absolute_error(y_test, model.predict(X_test)):.2f}")

Residual plots

Plot errors against predictions. A healthy model shows a shapeless cloud around zero. A curve means the model is missing a non linear pattern; a funnel means the errors grow with the prediction and a log transform of the target may help.

Practice

1. Compute MAE, RMSE and R squared by hand for y_true = [3, 5, 2, 7] and y_pred = [2.5, 5, 4, 8], then verify with scikit-learn.
2. Construct two prediction arrays with the same MAE but very different RMSE.
3. Fit a linear model to a clearly curved dataset (for example $y = x^2$) and look at the pattern in the residuals.