Naive Bayes

Lesson 53 of 62, about 5 min

Naive Bayes classifies by asking "given these features, which class is most probable?" and answering with Bayes theorem. It is fast, needs little data, and is the classic algorithm for text problems such as spam filtering and sentiment analysis.

Bayes theorem

P(\text{class} \mid \text{features}) = \frac{P(\text{features} \mid \text{class})\, P(\text{class})}{P(\text{features})}

$P(\text{class})$ is the prior: how common the class is before looking at any features.
$P(\text{features} \mid \text{class})$ is the likelihood: how likely these feature values are for examples of that class.
$P(\text{class} \mid \text{features})$ is the posterior: what we want.
The denominator is the same for every class, so it can be ignored when comparing them.

Prediction: compute the numerator for each class and pick the largest.

The naive assumption

The likelihood of many features together is hard to estimate. Naive Bayes assumes the features are independent given the class, so the joint likelihood is just the product of individual likelihoods:

P(x_1, x_2, \dots, x_n \mid c) = \prod_{j=1}^{n} P(x_j \mid c)

This is almost never literally true (the words "machine" and "learning" are clearly not independent), yet the classifier still works well, because it only needs to rank classes correctly, not to get the probabilities exactly right.

A worked example

Spam filter with two words. From 100 emails: 30 spam, 70 not spam. The word "offer" appears in 24 of the spam and 7 of the not spam; "meeting" in 3 of the spam and 42 of the not spam. A new email contains "offer" but not "meeting".

python

p_spam, p_ham = 0.30, 0.70
p_offer_spam, p_offer_ham = 24 / 30, 7 / 70
p_meet_spam, p_meet_ham = 3 / 30, 42 / 70

spam_score = p_spam * p_offer_spam * (1 - p_meet_spam)
ham_score = p_ham * p_offer_ham * (1 - p_meet_ham)
total = spam_score + ham_score
print(f"P(spam | offer, no meeting) = {spam_score / total:.3f}")

Variants

MultinomialNB: features are counts (word counts in a document). The standard for text.
BernoulliNB: features are binary (word present or not).
GaussianNB: features are continuous and assumed to follow a normal distribution within each class. For general numeric data.
ComplementNB: a variant of multinomial that works better with imbalanced classes.

GaussianNB on numeric data

python

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
nb = GaussianNB().fit(X_train, y_train)
print("test accuracy:", round(nb.score(X_test, y_test), 3))
print("class priors:", nb.class_prior_.round(3))

Text classification

Turn text into word counts with CountVectorizer, then fit MultinomialNB.

python

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

texts = [
    "win a free prize now", "limited offer click here", "free money guaranteed",
    "meeting at 10 tomorrow", "please review the attached report", "lunch with the team",
    "claim your free gift", "project update and next steps",
]
labels = ["spam", "spam", "spam", "ham", "ham", "ham", "spam", "ham"]

clf = make_pipeline(CountVectorizer(), MultinomialNB()).fit(texts, labels)
for msg in ["free offer inside", "report for the meeting", "team prize"]:
    print(f"{msg!r:28s} -> {clf.predict([msg])[0]}")

TfidfVectorizer is a common upgrade over raw counts: it downweights words that appear in every document.

Laplace smoothing

If a word never appeared in spam during training, its likelihood is 0 and the product wipes out the whole class. Smoothing adds a small count (the alpha parameter, default 1) to every word so nothing is exactly zero.

P(w \mid c) = \frac{\text{count}(w, c) + \alpha}{\text{count}(c) + \alpha \cdot |V|}

Strengths and weaknesses

Extremely fast to train and predict; works with tiny datasets and with hundreds of thousands of features.
A strong baseline for text. Always try it before anything more complex.
Probabilities are poorly calibrated (too confident) because of the independence assumption.
Cannot learn interactions between features; correlated features get double counted.

Practice

Recompute the worked example for an email that contains both "offer" and "meeting".
Add three more training emails to the text example, including one that mixes spam and ham words, and see how the predictions change.
Swap CountVectorizer for TfidfVectorizer and compare.