Logistic regression

Despite the name, logistic regression is a classification algorithm. It takes the linear model from the previous lessons and squashes its output into a probability between 0 and 1, then predicts the class with the higher probability.

From a line to a probability

A linear model produces any number from minus infinity to infinity. To turn that into a probability we pass it through the sigmoid function:

where $z = w_1 x_1 + \dots + w_n x_n + b$. The sigmoid maps large negative $z$ to nearly 0, large positive $z$ to nearly 1, and $z = 0$ to exactly 0.5.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

for z in (-5, -2, 0, 2, 5):
    print(f"z={z:+d}  sigmoid={sigmoid(z):.3f}")

The prediction is "class 1 if $\sigma(z) \ge 0.5$, else class 0". Because $\sigma(z) = 0.5$ exactly when $z = 0$, the decision boundary is the line (or plane) where $w \cdot x + b = 0$. It is still a linear boundary; the sigmoid only converts distance from that boundary into confidence.

The cost function: log loss

Mean squared error does not work well with a sigmoid: the cost surface becomes bumpy. Instead we use log loss (binary cross-entropy):

Read it for one example: if the true label is 1, the cost is $-\log(\hat{p})$, which is near 0 when $\hat{p}$ is near 1 and huge when $\hat{p}$ is near 0. If the label is 0, the roles flip. Confident wrong answers are punished heavily. This cost is convex, so gradient descent finds the single minimum.

Conveniently, the gradient has the same form as for linear regression:

From scratch

import numpy as np

# hours studied -> passed
x = np.array([0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5], dtype=float)
y = np.array([0, 0, 0, 0, 1, 0, 1, 1, 1, 1], dtype=float)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

w, b, alpha = 0.0, 0.0, 0.5
for step in range(2000):
    p = sigmoid(w * x + b)
    w -= alpha * ((p - y) * x).mean()
    b -= alpha * (p - y).mean()

loss = -(y * np.log(p) + (1 - y) * np.log(1 - p)).mean()
print(f"w={w:.2f} b={b:.2f} loss={loss:.3f}")
print("boundary at x =", round(-b / w, 2), "hours")

With scikit-learn

import numpy as np
from sklearn.linear_model import LogisticRegression

X = np.array([[0.5], [1], [1.5], [2], [2.5], [3], [3.5], [4], [4.5], [5]])
y = np.array([0, 0, 0, 0, 1, 0, 1, 1, 1, 1])

clf = LogisticRegression().fit(X, y)
print("predict:      ", clf.predict([[1], [2.7], [4]]))
print("probabilities:", clf.predict_proba([[1], [2.7], [4]]).round(3))

predict_proba returns a row per example with the probability of each class. That second column is what you threshold.

Moving the threshold

0.5 is only a default. If missing a positive is costly (a disease screen), lower the threshold to catch more; if false alarms are costly, raise it. The classification metrics lesson covers how to choose.

import numpy as np
from sklearn.linear_model import LogisticRegression

X = np.array([[0.5], [1], [1.5], [2], [2.5], [3], [3.5], [4], [4.5], [5]])
y = np.array([0, 0, 0, 0, 1, 0, 1, 1, 1, 1])
clf = LogisticRegression().fit(X, y)

p = clf.predict_proba([[2.2]])[0, 1]
print("p =", round(p, 3))
print("threshold 0.5:", int(p >= 0.5), " threshold 0.3:", int(p >= 0.3))

More than two classes

For three or more classes scikit-learn trains one logistic model per class and picks the highest probability (or uses the softmax function, the multi-class generalisation of the sigmoid). You do not need to change your code.

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
clf = LogisticRegression(max_iter=500).fit(X_train, y_train)
print("test accuracy:", round(clf.score(X_test, y_test), 3))

Reading the weights

A positive weight means the feature pushes towards class 1; the size of the weight (on scaled features) shows how strongly. $e^{w}$ is the odds ratio: how many times the odds of class 1 multiply for a one unit increase in that feature. This interpretability is why logistic regression remains the standard in credit scoring and medicine.

Regularisation

scikit-learn applies L2 regularisation by default, controlled by C (smaller C means stronger regularisation). Scale your features first so the penalty treats them equally. The overfitting lesson explains why.

Practice

1. Train on the hours-studied data and print the probability of passing for 1, 2, 3, 4 and 5 hours in a table.
2. Add a second feature (hours slept) with made up values and check whether accuracy improves.
3. Compute log loss for your predictions by hand and compare with sklearn.metrics.log_loss.