The maths
For a document d made of words w1, w2, …, wn, Bayes' theorem gives:
P(class | d) ∝ P(class) × ∏ P(wi | class)
P(class) is the prior: the fraction of training documents that belong to that class. P(wi | class) is the likelihood: how often the word wi appears in documents of that class. You compute both from counts, multiply them together, and pick the class with the highest score.
That is the entire algorithm. The visualisation above lets you type a sentence and watch each word contribute evidence for or against each class.
Why log-probabilities
Multiplying many small probabilities together produces numbers so small that floating-point arithmetic rounds them to zero. A document with 200 words can push the product below 10−300, which underflows in 64-bit floating point.
The fix is to work in log space. Since log is monotonic, the class that maximises the product also maximises the sum of logs:
log P(class | d) = log P(class) + ∑ log P(wi | class)
Addition is cheaper than multiplication and never underflows. This is what every practical implementation does, and what the visualisation above shows.
Laplace smoothing
A word that never appeared in positive training documents gets P(word | positive) = 0. One zero anywhere makes the entire product zero, so the model ignores all the other evidence. A single unseen word vetoes the entire classification.
Laplace smoothing (add-1 smoothing) fixes this by adding a pseudocount of 1 to every word count:
P(w | class) = (count(w, class) + 1) / (total words in class + V)
V is the vocabulary size. The denominator grows by V to keep the probabilities summing to 1. The effect is that no word ever has zero probability. The cost is that rare words get a slight boost relative to their true frequency, but this is almost always worth it.
You can generalise to add-α smoothing by using a fractional pseudocount. α = 1 is the most common choice.
Where it works and where it fails
Naive Bayes works well for:
- Spam filtering. The bag-of-words model is a natural fit because spam is mostly about which words appear, not their exact arrangement.
- Sentiment analysis. Words like "excellent", "terrible" and "disappointing" carry strong class signal on their own.
- Topic classification. Documents about sport contain sport words; documents about politics contain politics words.
- Medical text triage. Routing clinical notes by specialty based on keyword frequency.
It struggles when:
- Word order matters. "The movie was not good" and "The movie was good, not bad" have overlapping word sets but opposite meanings. Naive Bayes cannot distinguish them.
- Words interact. "Hot dog" means something different from "hot" and "dog" separately.
- Classes are defined by absence. Some categories are defined less by which words appear and more by which words do not. A bag-of-words model has no natural way to represent that.
For tasks where order matters, n-grams partially help by treating bigrams as features. For richer representations, embeddings capture meaning that word counts miss, and text encoding explains why the conversion from text to numbers is needed at all.
A Naive Bayes classifier from scratch
import math
from collections import Counter, defaultdict
# ---- tiny training set ----
train = [
("I love this movie", "positive"),
("great film and wonderful acting", "positive"),
("terrible waste of time", "negative"),
("awful movie I hated it", "negative"),
]
# ---- build counts ----
class_counts = Counter()
word_counts = defaultdict(Counter)
vocab = set()
for text, label in train:
words = text.lower().split()
class_counts[label] += 1
for w in words:
word_counts[label][w] += 1
vocab.add(w)
V = len(vocab)
total = sum(class_counts.values())
# ---- classify a new document ----
def classify(text):
words = text.lower().split()
scores = {}
for label in class_counts:
log_p = math.log(class_counts[label] / total)
n = sum(word_counts[label].values())
for w in words:
log_p += math.log((word_counts[label][w] + 1) / (n + V))
scores[label] = log_p
return max(scores, key=scores.get), scores
pred, scores = classify("I love great acting")
print(f"Prediction: {pred}")
for label, s in scores.items():
print(f" {label}: {s:.4f}")
The code uses Laplace smoothing (the +1 in the numerator) and works entirely in log space. It trains in one pass and classifies in time proportional to the document length times the number of classes.
Questions people ask about Text Classification with Naive Bayes
Is Naive Bayes actually "naive"? Yes. The conditional independence assumption is almost never true for real language. But the classifier needs correct rankings, not correct probabilities, and the independence assumption rarely changes which class scores highest.
When should I use Naive Bayes over logistic regression? When you have very little training data, many features, or need a fast baseline. Naive Bayes has fewer parameters to estimate, so it handles sparse high-dimensional data well. With enough data, logistic regression usually wins on accuracy.
Does Naive Bayes work with TF-IDF features? The multinomial variant uses raw counts. If you want TF-IDF, the complement Naive Bayes variant or a simple logistic regression is a better match. Scikit-learn provides both.
What is the difference between multinomial and Bernoulli Naive Bayes? Multinomial uses word counts. Bernoulli uses binary presence or absence. For short texts like tweets, Bernoulli sometimes works better because repeated words add little information.
Can Naive Bayes handle more than two classes? Yes. The formula generalises to any number of classes. You compute P(class) and the word likelihoods for each class independently, then pick the class with the highest score.
Why does Naive Bayes still appear in production systems? Because it trains in milliseconds, classifies in microseconds, and gives a strong baseline. Many production spam filters and content classifiers still use it, sometimes as the first stage of a pipeline that escalates uncertain cases to a heavier model.