Classification metrics

Accuracy is the first number everyone looks at and the one most likely to mislead. A classifier that predicts "no fraud" for every transaction is 99.9 percent accurate. This lesson gives you the metrics that see through that.

The confusion matrix

Everything starts here. For a binary problem, each prediction lands in one of four cells:

import numpy as np
from sklearn.metrics import confusion_matrix

y_true = np.array([1, 0, 1, 1, 0, 0, 1, 0, 0, 1])
y_pred = np.array([1, 0, 0, 1, 0, 1, 1, 0, 0, 0])
print(confusion_matrix(y_true, y_pred))

scikit-learn prints rows as actual, columns as predicted, with the negative class first: [[TN, FP], [FN, TP]].

Accuracy

The fraction of predictions that were right. Fine when classes are balanced and both kinds of mistake cost the same. Misleading otherwise.

Precision

Of everything the model flagged as positive, how much really was? High precision means few false alarms. Matters when acting on a positive is expensive: sending a fraud team, blocking an account, recommending surgery.

Recall

Of everything that really was positive, how much did the model catch? High recall means few misses. Matters when missing a positive is expensive: cancer screening, detecting a security breach.

Precision and recall pull against each other. Flag more things and recall rises while precision falls.

F1 score

The harmonic mean of the two. It is high only when both are high, so it is a good single number for imbalanced problems. If one matters more, the $F_\beta$ score weights recall $\beta$ times as much as precision.

import numpy as np
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

y_true = np.array([1, 0, 1, 1, 0, 0, 1, 0, 0, 1])
y_pred = np.array([1, 0, 0, 1, 0, 1, 1, 0, 0, 0])
print("accuracy: ", accuracy_score(y_true, y_pred))
print("precision:", precision_score(y_true, y_pred))
print("recall:   ", recall_score(y_true, y_pred))
print("f1:       ", round(f1_score(y_true, y_pred), 3))

The classification report

One call gives all of the above per class.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, stratify=y)
clf = LogisticRegression(max_iter=5000).fit(X_train, y_train)
print(classification_report(y_test, clf.predict(X_test), target_names=["malignant", "benign"]))

support is the number of true examples of each class. macro avg treats classes equally; weighted avg weights by support.

Why accuracy fails on imbalanced data

import numpy as np
from sklearn.metrics import accuracy_score, recall_score

y_true = np.array([0] * 990 + [1] * 10)
always_zero = np.zeros(1000, dtype=int)
print("accuracy:", accuracy_score(y_true, always_zero))
print("recall:  ", recall_score(y_true, always_zero))

Perfect looking accuracy, zero usefulness.

Thresholds and the ROC curve

Most classifiers output a probability and the label is "probability above 0.5". Change the threshold and every metric above changes. The ROC curve plots the true positive rate (recall) against the false positive rate for every possible threshold. The area under it (AUC) summarises the ranking quality of the model regardless of threshold: 1.0 is perfect, 0.5 is random guessing.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, precision_recall_curve

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, stratify=y)
clf = LogisticRegression(max_iter=5000).fit(X_train, y_train)
probs = clf.predict_proba(X_test)[:, 1]
print("ROC AUC:", round(roc_auc_score(y_test, probs), 4))

precision, recall, thresholds = precision_recall_curve(y_test, probs)
for t in (0.2, 0.5, 0.8):
    i = (thresholds >= t).argmax()
    print(f"threshold {t}: precision={precision[i]:.3f} recall={recall[i]:.3f}")

For heavily imbalanced data the precision-recall curve and its area (average precision) are more informative than ROC, because ROC's false positive rate barely moves when negatives are plentiful.

Log loss

When the probabilities themselves matter (you will rank customers by risk, or feed the output into another system), measure how good the probabilities are with log loss. Lower is better. It punishes confident mistakes.

Multi-class

All of the above extends to several classes. The confusion matrix becomes $k \times k$; precision, recall and F1 are computed per class and averaged (average="macro" or "weighted").

Choosing a metric

Practice

1. From the confusion matrix [[50, 10], [5, 35]] compute accuracy, precision, recall and F1 by hand.
2. Train any classifier on the breast cancer data and find the threshold that gives recall of at least 0.98. What precision do you get?
3. Build a deliberately imbalanced dataset with make_classification(weights=[0.95, 0.05]) and compare accuracy, F1 and ROC AUC of a logistic regression with those of a dummy classifier.