Uncover the probability logic behind weather-based predictions.
Overview
Overview
The Naive Bayes Classifier is a simple yet powerful algorithm for predictive modeling. It's based on Bayes' Theorem and is particularly useful for text classification, like spam filtering. Its core idea is to calculate the probability of a certain outcome (e.g., "Should I play tennis?") based on the evidence provided by a set of features (e.g., the weather outlook, temperature, and wind).
Inference Engine
Calculation Story
Ready
Adjust inputs and click 'Next Step' to start the mathematical breakdown.
Knowledge Base
Results
Probability Play = Yes0%
Probability Play = No0%
Prediction: YES
How Naive Bayes Works: A Step-by-Step Guide
Follow the "Calculation Story" as you click through the steps to see the logic unfold.
The "Naive" Assumption: A Key Simplification
The "naive" part of the name comes from a key assumption the algorithm makes: it assumes that all features are independent of each other. In our example, it assumes that the 'Outlook' has no effect on the 'Temperature' or 'Wind'. While this is often not true in the real world (a sunny outlook usually implies hotter temperatures), this simplification makes the calculations much easier and faster. Despite this "naive" assumption, the classifier works surprisingly well in many real-world scenarios.
Bayes' theorem without the fear
Everything here rests on one line, and the line is less intimidating written as a sentence.
"How likely is this class, given what I have seen?" equals "how typical is this evidence for that class" times "how common is that class in the first place".
The second term is the one people forget, and it is the one that stops the classic mistake. A test for a disease that affects 1 in 10,000 people, with a 1% false-positive rate, produces about 100 false positives for every true one — not because the test is bad but because the prior is tiny. Naive Bayes never forgets the prior, which is one reason it behaves sensibly on rare classes.
To classify, you compute that product for every class and pick the biggest. The denominator that would turn the product into a proper probability is the same for all classes, so it can be ignored while you are only ranking them.
What "naive" is doing
The hard part of the true calculation is P(evidence | class) when the evidence is several features at once. To know P("free" and "money" and "click" | spam) properly, you would need to know how those words co-occur, and with a vocabulary of 20,000 words that is more combinations than there are atoms available to count them with.
The naive assumption cuts through this: assume every feature is independent of every other, given the class. Then the joint probability is just the product:
Each of those is a single count divided by another count. The whole model becomes a table of frequencies, computable in one pass over the data.
The assumption is, of course, false. "New" and "York" are wildly dependent. And yet the classifier works, because for ranking classes you do not need the probabilities to be right — you only need the right class to come out on top. The estimated probabilities are typically far too extreme (0.9999 and similar), while the decisions they lead to are sound. Trust the argmax, distrust the number, and calibrate if you need the number.
A spam calculation you can follow
Training data: 100 emails, 30 spam and 70 not. Among the spam, "free" appears in 24; among the rest, in 7.
Spam wins. Normalising, 0.24 / (0.24 + 0.07) = 77% probability of spam.
Now suppose the email also contains "meeting", a word that appeared in zero spam emails during training. P(meeting | spam) = 0, and the entire spam score is multiplied by zero — one unseen word has vetoed the whole calculation.
The fix is Laplace smoothing: add 1 to every count (and the vocabulary size to every denominator), so no probability is ever exactly zero. With a vocabulary of 1,000 words, P(meeting | spam) becomes 1/(30 + 1000) = 0.00097 instead of 0. Small, but survivable. Every practical implementation does this by default; scikit-learn calls the parameter alpha.
The other implementation detail: multiplying thousands of small probabilities underflows to zero in floating point, so real implementations add logarithms instead of multiplying probabilities. Same ranking, no underflow.
The Calculation Process: A Walkthrough
The interactive panel walks you through the exact steps the algorithm takes to make a prediction. Let's use the default input: Outlook=Sunny, Temp=Cool, Wind=Strong.
Step 1: Calculate Prior Probabilities
First, the algorithm looks at the historical data (the "Knowledge Base") to determine the overall probability of each outcome. This is the "prior" belief before considering any new evidence.
P(Play=Yes): Out of 14 total days, 'Play' was 'Yes' on 9 of them. So, P(Yes) = 9/14.
P(Play=No): 'Play' was 'No' on 5 of them. So, P(No) = 5/14.
Step 2: Calculate Likelihoods for Each Feature
Next, for each class ('Yes' and 'No'), the algorithm calculates the probability of our specific input features occurring. This is the "likelihood".
Likelihood for 'Yes':
P(Outlook=Sunny | Play=Yes): Among the 9 'Yes' days, 2 were 'Sunny'. (Prob = 2/9)
P(Temp=Cool | Play=Yes): Among the 9 'Yes' days, 3 were 'Cool'. (Prob = 3/9)
P(Wind=Strong | Play=Yes): Among the 9 'Yes' days, 3 were 'Strong'. (Prob = 3/9)
Likelihood for 'No':
P(Outlook=Sunny | Play=No): Among the 5 'No' days, 3 were 'Sunny'. (Prob = 3/5)
P(Temp=Cool | Play=No): Among the 5 'No' days, 1 was 'Cool'. (Prob = 1/5)
P(Wind=Strong | Play=No): Among the 5 'No' days, 3 were 'Strong'. (Prob = 3/5)
Note: The visualization uses Laplace Smoothing (adding 1 to the numerator and the number of unique feature values to the denominator) to avoid zero-probability issues, so the exact fractions will look slightly different in the "Calculation Story".
Step 3: Combine and Calculate Scores
Now, we combine everything. For each outcome, we multiply its prior probability by all of its likelihoods.
The scores are not final probabilities. To get a clean 0-100% result, we normalize them by dividing each score by the sum of both scores. The class with the higher final probability is our prediction.
Which flavour to use
Variant
Feature type
Typical use
Multinomial
Counts
Word counts or TF-IDF for text classification
Bernoulli
Binary
Word present / absent; short texts
Gaussian
Continuous
Numeric features, assumed normal within each class
Complement
Counts
Text with imbalanced classes; often beats multinomial
Gaussian Naive Bayes is the one to be careful with: it assumes each feature follows a bell curve within each class. On a heavily skewed feature such as income, that assumption is badly wrong, and a log transform beforehand usually helps more than switching models.
Where it still earns its place
Naive Bayes is old, and it is still deployed, for reasons that have not been undermined by anything newer:
Speed. Training is one pass of counting. On millions of documents it finishes while other models are still loading.
Small data. With a few hundred labelled examples it often outperforms models with more parameters to fit, because it has so few of its own.
Streaming. Counts update incrementally, so the model can learn continuously with no retraining step.
Baselines. If your transformer cannot beat Naive Bayes on your text classification task, something is wrong with the pipeline, not with the transformer.
High dimensions. A vocabulary of 100,000 features is not a problem; each contributes one number per class.
It is a poor choice when feature interactions carry the signal — the naive assumption discards them by construction — and when you need well-calibrated probabilities rather than a ranking.
An assumption that is wrong and works anyway
Naive Bayes assumes every feature is independent, which is almost never true. Here is the assumption being violated and the classifier still doing its job.
example_01.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.naive_bayes import GaussianNB, MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
X, y = make_classification(n_samples=2000, n_features=8, n_informative=5,
n_redundant=0, flip_y=0.08, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
nb = GaussianNB().fit(Xtr, ytr)
print("what the model actually stores -- a mean and a variance per class,")
print("per feature. that is all of it:")
print(" class priors ", np.round(nb.class_prior_, 4))
print(" means, class 0 ", np.round(nb.theta_[0][:4], 4), "...")
print(" means, class 1 ", np.round(nb.theta_[1][:4], 4), "...")
print(" variances, class 0", np.round(nb.var_[0][:4], 4), "...")
print()
row = Xte[0]
print("one prediction, rebuilt by hand:")
logp = []
for c in (0, 1):
ll = -0.5 * np.sum(np.log(2 * np.pi * nb.var_[c])
+ (row - nb.theta_[c]) ** 2 / nb.var_[c])
logp.append(np.log(nb.class_prior_[c]) + ll)
print(" log P(x | class %d) + log prior = %.6f" % (c, logp[-1]))
logp = np.array(logp)
post = np.exp(logp - logp.max()); post /= post.sum()
print(" normalised : %s" % np.round(post, 6))
print(" sklearn predict_proba : %s" % np.round(nb.predict_proba([row])[0], 6))
print()
print("the 'naive' step is that sum: it multiplies the per-feature")
print("probabilities together, which is only valid if they are independent.")
print()
print("so let us break the assumption. find the single most useful column,")
print("then paste copies of it alongside the originals:")
best = int(np.argmax([cross_val_score(GaussianNB(), X[:, [c]], y, cv=5).mean()
for c in range(X.shape[1])]))
print(" column %d is the strongest on its own." % best)
print()
print("%14s %12s %12s" % ("copies added", "naive bayes", "logistic reg"))
for k in (0, 1, 3, 10, 30):
Xd = np.column_stack([X] + [X[:, best]] * k) if k else X
print("%14d %12.4f %12.4f"
% (k, cross_val_score(GaussianNB(), Xd, y, cv=5).mean(),
cross_val_score(LogisticRegression(max_iter=3000), Xd, y, cv=5).mean()))
print()
print("no new information was added at any step -- every copy is the same")
print("column. naive bayes multiplies its evidence in once per copy, so that")
print("one feature drowns out the other seven. logistic regression splits the")
print("weight across the copies and lands in the same place it started.")
print()
print("its probabilities are the part you should not trust:")
p = nb.predict_proba(Xte)[:, 1]
print(" predictions above 0.99 : %d of %d" % ((p > 0.99).sum(), len(p)))
print(" of those, actually 1 : %.4f" % yte[p > 0.99].mean())
print(" it claimed better than 99%% and was right on %.1f%% of them."
% (100 * yte[p > 0.99].mean()))
print()
print(" now the same measurement after adding 30 copies of one column:")
Xd = np.column_stack([X] + [X[:, best]] * 30)
nb2 = GaussianNB().fit(Xd[:len(Xtr)], y[:len(Xtr)])
p2 = nb2.predict_proba(Xd[len(Xtr):])[:, 1]
y2 = y[len(Xtr):]
print(" predictions above 0.99 : %d of %d" % ((p2 > 0.99).sum(), len(p2)))
print(" of those, actually 1 : %.4f" % y2[p2 > 0.99].mean())
print()
print(" multiplying many small numbers together drives the result toward 0")
print(" or 1 much faster than the evidence justifies, and correlated")
print(" features make it worse. use the ranking, not the number.")
print()
print("it earns its place when you have many features and few rows: it needs")
print("only a mean and a variance per feature per class, so it will train on")
print("data that leaves almost anything else underdetermined. that, and it is")
print("fast enough to be the baseline you compare everything against.")
Output
Questions people ask
Why does it work if the independence assumption is false? Because classification only needs the correct class to score highest. Correlated features distort the magnitude of the scores in the same direction for all classes, which often leaves the ordering intact.
What does alpha do? It is the smoothing constant. alpha=1 is Laplace smoothing; smaller values trust the observed counts more; alpha=0 reintroduces the zero-probability veto. Tune it — on small vocabularies it matters more than people expect.
Can it handle continuous and categorical features together? Not in one standard estimator. Either bin the continuous features into categories, or train separate models and combine their log-probabilities.
Why are the predicted probabilities so extreme? Multiplying many correlated likelihoods double-counts evidence, pushing outputs towards 0 or 1. Apply isotonic or sigmoid calibration if you need trustworthy probabilities.
Is it affected by imbalanced classes? The prior handles imbalance sensibly, but for text specifically, Complement Naive Bayes was designed to correct a bias multinomial NB shows on skewed class distributions.
Recap in one screen
Score each class by "how common is this class" times "how typical is this evidence for it", and pick the winner.
The naive assumption — features independent given the class — turns an impossible count into a product of easy ones.
Smoothing is mandatory, or a single unseen feature zeroes out a class.
Fast, tiny, excellent on text and small datasets, and a benchmark every heavier model should have to beat.
Trust its decisions more than its probabilities.
A last word on why it survives
Naive Bayes is roughly as old as the field, is based on an assumption everyone agrees is false, and is still in production in spam filters, document routers and medical triage tools. That combination is worth understanding, because it teaches something general about modelling.
The lesson is that a model does not have to be right to be useful — it has to be wrong in a way that does not affect the decision being made. The independence assumption distorts the magnitude of the probabilities, and the decision only depends on their order. So the error goes almost entirely into a quantity nobody uses.
Compare that with a model whose errors land directly on the decision boundary, and the difference in practical value is enormous even if their stated assumptions look equally shaky.
The second reason it survives is that its cost profile is unusual. Training is one pass of counting, memory is a table of frequencies, and prediction is a handful of additions in log space. On a device with no GPU, in a stream with no batch window, or in a system that must update continuously, that profile beats a more accurate model that cannot be deployed.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Overview”?
The Naive Bayes Classifier is a simple yet powerful algorithm for predictive modeling. It's based on Bayes' Theorem and is particularly useful for text classification, like spam filtering. Its core idea is to calculate the probability of a certain outcome (e.g., "Should I play tennis?") based on the evidence provided by a set of features (e.g., the weather outlook, temperature, and wind).
What does this module say about “The "Naive" Assumption: A Key Simplification”?
The "naive" part of the name comes from a key assumption the algorithm makes: it assumes that all features are independent of each other . In our example, it assumes that the 'Outlook' has no effect on the 'Temperature' or 'Wind'. While this is often not true in the real world (a sunny outlook usually implies hotter temperatures), this simplification makes the calculations much easier and faster.
What does this module say about “Bayes' theorem without the fear”?
Everything here rests on one line, and the line is less intimidating written as a sentence.
Cheat sheet
Naive Bayes Classifier
The Naive Bayes Classifier is a simple yet powerful algorithm for predictive modeling. It's based on Bayes' Theorem and is particularly useful for text classification, like spam filtering. Its core idea is to calculate the probability of a certain outcome (e.g., "Should I play tennis?") based on the evidence provided by a set of features (e.g., the weather outlook, temperature, and wind).
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.