The perceptron

Lesson 60 of 62, about 6 min

A neural network is built from a single simple unit repeated many times. That unit, the perceptron, is nothing more than a weighted sum followed by a decision. Understand it and the rest of deep learning is a matter of stacking.

The unit

A perceptron takes inputs $x_1, \dots, x_n$ , multiplies each by a weight, adds a bias, and passes the total through an activation function $f$ :

z = w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b = w \cdot x + b

a = f(z)

If $f$ is a step function (1 when $z \ge 0$ , else 0), this is the original 1958 perceptron. Notice that $w \cdot x + b$ is exactly the linear model from the regression lessons: a single neuron is logistic regression with a different activation.

The perceptron learning rule

For each training example, predict, and if wrong, nudge the weights towards the right answer:

w \leftarrow w + \eta\,(y - \hat{y})\, x \qquad b \leftarrow b + \eta\,(y - \hat{y})

with $\eta$ the learning rate. If the classes are linearly separable, this is guaranteed to converge.

python

import numpy as np

# learn the AND function
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 0, 0, 1])

w = np.zeros(2)
b = 0.0
eta = 0.1
for epoch in range(10):
    errors = 0
    for xi, yi in zip(X, y):
        y_hat = int(w @ xi + b >= 0)
        update = eta * (yi - y_hat)
        w += update * xi
        b += update
        errors += int(update != 0)
    if errors == 0:
        print("converged at epoch", epoch)
        break

print("weights", w, "bias", b)
print("predictions", [int(w @ xi + b >= 0) for xi in X])

What one unit cannot do

A single perceptron draws one straight line. The XOR function (true when exactly one input is true) has no such line: the two positive cases sit on opposite corners. This limitation stalled neural network research for years. The fix is to stack units into layers, which is the next lesson.

python

import numpy as np
from sklearn.linear_model import Perceptron

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
for name, y in (("AND", [0, 0, 0, 1]), ("OR", [0, 1, 1, 1]), ("XOR", [0, 1, 1, 0])):
    clf = Perceptron(max_iter=100, random_state=0).fit(X, y)
    print(f"{name}: predictions {clf.predict(X)}  accuracy {clf.score(X, y):.2f}")

Activation functions

Stacking layers only helps if the activation is non linear; a stack of linear functions is still linear. The common choices:

Name	Formula	Range	Used for
Sigmoid	$\frac{1}{1 + e^{-z}}$	(0, 1)	Output of binary classifiers
Tanh	$\frac{e^z - e^{-z}}{e^z + e^{-z}}$	(-1, 1)	Older hidden layers, some RNNs
ReLU	$\max(0, z)$	[0, inf)	Hidden layers, the default
Leaky ReLU	$\max(0.01z, z)$		Hidden layers when ReLU units die
Softmax	$\frac{e^{z_k}}{\sum_j e^{z_j}}$	probabilities summing to 1	Output of multi-class classifiers

python

import numpy as np

z = np.array([-3, -1, 0, 1, 3], dtype=float)
sigmoid = 1 / (1 + np.exp(-z))
tanh = np.tanh(z)
relu = np.maximum(0, z)
softmax = np.exp(z) / np.exp(z).sum()
print("sigmoid", sigmoid.round(3))
print("tanh   ", tanh.round(3))
print("relu   ", relu)
print("softmax", softmax.round(3), "sum =", softmax.sum().round(3))

ReLU is the default in hidden layers because it is cheap and its gradient is either 0 or 1, which avoids the vanishing gradients that sigmoid and tanh suffer in deep stacks.

Why gradients matter

The step function has zero gradient everywhere, so gradient descent cannot train it. Replacing it with a smooth activation lets us define a loss and differentiate it with respect to every weight, and that is what makes training networks with many layers possible.

A neuron as a feature detector

Think of each unit as asking one weighted question about its inputs ("how much does this look like pattern $w$ ?") and answering with a strength. A layer asks many such questions at once. The next layer asks questions about those answers. Deep networks build up from simple to abstract features this way.

Practice

Change the from scratch perceptron to learn OR, then NAND. Print the final weights and interpret them.
Try to make it learn XOR and watch the error count never reach zero.
Plot (or tabulate) sigmoid, tanh and ReLU for $z$ from -5 to 5 and note where each saturates.