Neural networks and backpropagation

A neural network is layers of the units from the previous lesson, each layer feeding the next. Training it means adjusting every weight in every layer to lower a loss, and backpropagation is the algorithm that works out how much each weight is to blame. This lesson builds and trains a small network from scratch in NumPy, then does the same in scikit-learn.

Architecture

• Input layer: one node per feature.
• Hidden layers: each node computes $f(w \cdot a_{\text{prev}} + b)$ over every node of the previous layer. "Deep" learning means several hidden layers.
• Output layer: one node for regression or binary classification (sigmoid), or one per class with softmax.

A layer with $n$ inputs and $h$ units has a weight matrix $W$ of shape $(n, h)$ and a bias vector of length $h$. For a whole batch of examples $A_{\text{prev}}$ (shape $m \times n$):

Computing the output from the input this way is the forward pass.

The loss

For binary classification use log loss on the sigmoid output, exactly as in logistic regression. For regression, mean squared error. For multi-class, cross-entropy on the softmax output.

Backpropagation

We need $\partial L / \partial W$ for every layer. The chain rule says the gradient of the loss with respect to a layer's weights is the gradient with respect to that layer's output, times how the output depends on the weights. Working backwards from the loss:

1. At the output, with sigmoid and log loss, the gradient of the loss with respect to $Z$ is simply $A - y$.
2. For each earlier layer, the gradient with respect to its output is the next layer's $Z$ gradient multiplied by the next layer's weights (transposed), and the gradient with respect to its own $Z$ is that times the derivative of its activation.
3. The weight gradient at any layer is $A_{\text{prev}}^T \cdot \partial L / \partial Z$ divided by the batch size; the bias gradient is the column sum.

Then apply gradient descent to every $W$ and $b$.

A two layer network from scratch

Solving XOR, which one perceptron could not.

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)

rng = np.random.default_rng(1)
W1 = rng.normal(0, 1, (2, 4)); b1 = np.zeros((1, 4))    # hidden layer, 4 units
W2 = rng.normal(0, 1, (4, 1)); b2 = np.zeros((1, 1))    # output layer

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

lr = 0.5
for epoch in range(5000):
    # forward pass
    Z1 = X @ W1 + b1
    A1 = np.tanh(Z1)
    Z2 = A1 @ W2 + b2
    A2 = sigmoid(Z2)

    # loss
    loss = -(y * np.log(A2) + (1 - y) * np.log(1 - A2)).mean()

    # backward pass
    dZ2 = A2 - y
    dW2 = A1.T @ dZ2 / len(X)
    db2 = dZ2.mean(axis=0, keepdims=True)
    dA1 = dZ2 @ W2.T
    dZ1 = dA1 * (1 - A1 ** 2)          # derivative of tanh
    dW1 = X.T @ dZ1 / len(X)
    db1 = dZ1.mean(axis=0, keepdims=True)

    # gradient descent
    W2 -= lr * dW2; b2 -= lr * db2
    W1 -= lr * dW1; b1 -= lr * db1

    if epoch % 1000 == 0:
        print(f"epoch {epoch:4d} loss {loss:.4f}")

print("predictions:", A2.round(2).ravel())

Eighteen weights and a few lines of calculus, and the network learns a boundary no single line could draw.

With scikit-learn

MLPClassifier and MLPRegressor implement multi layer perceptrons with Adam optimisation, mini-batches and early stopping.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, stratify=y)

mlp = make_pipeline(StandardScaler(),
                    MLPClassifier(hidden_layer_sizes=(32, 16), max_iter=500, random_state=0))
mlp.fit(X_train, y_train)
print("test accuracy:", round(mlp.score(X_test, y_test), 3))
print("layers trained:", mlp[-1].n_iter_, "iterations")

Scaling is not optional for neural networks; unscaled inputs make gradients explode or vanish.

Training details that matter

• Initialisation. Weights start small and random (never all zero, or every unit learns the same thing).
• Learning rate. The most important hyperparameter. Too high diverges, too low crawls. Adam adapts it per weight and is the usual default.
• Batch size. 32 to 256. Smaller is noisier but often generalises better.
• Epochs and early stopping. Watch validation loss; stop when it rises.
• Regularisation. L2 weight decay (alpha in scikit-learn), dropout (randomly silencing units during training), and data augmentation.
• Vanishing gradients. Deep sigmoid stacks multiply many small derivatives together. ReLU, careful initialisation, batch normalisation and residual connections solve this in modern architectures.

Beyond the multi layer perceptron

The same forward and backward machinery powers every deep learning architecture:

• Convolutional networks share weights across positions in an image.
• Recurrent networks share weights across time steps in a sequence.
• Transformers use attention to let every position look at every other, and are the basis of modern language models.

PyTorch and TensorFlow compute the backward pass automatically (autograd), so in practice you only write the forward pass.

When to use a neural network

For tabular data of ordinary size, gradient boosting usually wins and is far easier to tune. Neural networks earn their cost with images, audio, text, very large datasets, or when you need to learn representations rather than hand craft features.

Practice

1. Change the hidden layer in the from scratch network to 2 units, then to 8. Does it still learn XOR? How fast?
2. Replace tanh with ReLU in the hidden layer (and its derivative, which is 1 where $Z > 0$ and 0 elsewhere).
3. Try hidden_layer_sizes=(8,), (64, 64) and (128, 64, 32) on the breast cancer data and compare test accuracy and training iterations.