Conditional Probability

The formula is just a recount

P(A | B) = P(A and B) / P(B)

Look at what this does with the grid above. The denominator was 400 — the whole sample space. Conditioning on B replaces it with however many outcomes are in B. The numerator counts the outcomes that are in both, because those are the only ones in the shrunken world where A is also true.

Tick Condition on B and the picture does exactly this: everything outside B is discarded, and the readout underneath shows the fraction as a plain count of dots. The formula is bookkeeping for that operation, not a separate idea.

Independence, defined properly

A and B are independent when knowing B tells you nothing about A:

P(A | B) = P(A)

Shrinking the world to B leaves A's share of it unchanged. Equivalently, and more symmetrically, P(A and B) = P(A) × P(B) — which is the multiplication rule everyone learns, now with the condition attached that makes it valid.

Multiplying probabilities when the events are not independent is one of the most common errors in applied statistics, and it always understates joint risk when the events tend to occur together.

Probability, once you know something

Conditional probability is the probability of one event given that another has happened.

P(A|B) = P(A and B) / P(B)

Read it as: restrict attention to the cases where B happened, and ask what fraction of those also have A. The denominator is the new, smaller universe.

A worked example on 100 people:

• P(owns a car) = 55/100 = 0.55
• P(owns a car | lives rurally) = 35/40 = 0.875
• P(owns a car | lives in a city) = 20/60 = 0.33

Knowing where someone lives changes the answer dramatically, which is the entire point: conditioning is what turns a population statistic into a prediction about an individual. Every supervised model is an estimate of P(target | features).

Independence, stated properly

Two events are independent when conditioning changes nothing:

P(A|B) = P(A)   equivalently   P(A and B) = P(A) × P(B)

In the table above, car ownership and location are clearly not independent: 0.875 against 0.55. Two coin flips are independent; two words in the same sentence are not.

The distinction matters because independence is what lets probabilities multiply. Assume it wrongly and you double-count evidence, producing models that are far too confident — which is precisely the known behaviour of Naive Bayes on correlated features.

Mutually exclusive is not the same as independent. If two events cannot both happen, then knowing one occurred tells you the other definitely did not — which is maximal dependence, not independence.

Chaining conditions

The definition rearranges into the multiplication rule, which extends to any number of events:

P(A and B) = P(B) × P(A|B)

P(A, B, C) = P(A) × P(B|A) × P(C|A, B)

This chain rule is the backbone of sequence modelling. A language model assigns a probability to a sentence by multiplying the probability of each word given everything before it:

P(w₁, w₂, …, wₙ) = ∏ P(wᵢ | w₁ … wᵢ₋₁)

That is exactly what "predict the next token" means, and it is why a model trained only to guess the next word can assign a probability to an entire document.

Because these products underflow, implementations sum log-probabilities instead — which is where perplexity, the exponentiated average negative log-probability, comes from.

Both directions, measured

P(A given B) and P(B given A) are different numbers. Here is a population where you can count both.

import numpy as np

rng = np.random.default_rng(3)
n = 400_000

# 30% of people own a car; owners commute by car 80% of the time, others 5%
owns = rng.random(n) < 0.30
p_drive = np.where(owns, 0.80, 0.05)
drives = rng.random(n) < p_drive

print("P(owns)          %.4f" % owns.mean())
print("P(drives)        %.4f" % drives.mean())
print("P(owns and dr.)  %.4f" % (owns & drives).mean())
print()
print("P(drives | owns) %.4f   <- measured   0.80 by construction"
      % drives[owns].mean())
print("P(owns | drives) %.4f   <- the other direction, a different number"
      % owns[drives].mean())
print()
print("division, not magic:")
print("  P(A and B) / P(B) = %.4f / %.4f = %.4f"
      % ((owns & drives).mean(), drives.mean(),
         (owns & drives).mean() / drives.mean()))

Try it yourself

1. Watch the world shrink. Tick Condition on B. The dots outside B fade out and P(A given B) is the fraction of the survivors that are blue. Untick it and the denominator goes back to all 400.
2. Find independence. Set the Preset to Independent. P(A given B) and P(A) now read the same number — knowing B happened has told you nothing at all about A.
3. Break it deliberately. Drag the Separation slider away from that setting. The two numbers come apart immediately, and Independent flips to no. Independence is a knife-edge condition, not a typical one.
4. Make them incompatible. Set the Preset to Mutually exclusive. P(A and B) is zero, so P(A given B) is zero: if B happened, A certainly did not. Note that this is the opposite of independent — B is now maximally informative.
5. See a certainty. Set the Preset to A inside B. Every outcome in A is also in B, so P(B given A) is exactly 1 while P(A given B) is much smaller. The two conditionals are nothing like each other.
6. Meet the base-rate trap. Set the Preset to Rare A, common B and compare P(A given B) with P(B given A). One is small, the other is large, from the same two events. Reversing them is the error behind almost every misread medical test.

P(A|B) is not P(B|A)

This is worth stating in its own section because it causes more real-world damage than any other confusion in probability.

"Given the patient has the disease, the test is positive 99% of the time" is a statement about P(positive | disease). What a patient wants to know is P(disease | positive). Those are different numbers, and when the disease is rare they are wildly different — the second can be under 10% while the first is 99%.

The Rare A preset above shows the same asymmetry with nothing but dots. Converting between the two directions requires knowing how common the disease is to begin with, and the machinery for doing it is Bayes' theorem, which is the next module and is built directly on this one.

Where it shows up in machine learning

• Every classifier is estimating P(class | features). That is a conditional probability and nothing else.
• Naive Bayes is named for the assumption that features are conditionally independent given the class — which is usually false, and works anyway.
• Precision and recall are the two directions of the same pair. Precision is P(actually positive | predicted positive); recall is P(predicted positive | actually positive). Reporting one when you meant the other is exactly this mistake.
• Language models predict P(next token | everything so far) — a conditional probability with a very large condition.

Where this goes wrong

• Swapping the two sides. Covered above, and worth checking every single time you read a conditional claim in the wild.
• Multiplying without independence. P(A and B) = P(A)P(B) only holds when the events are independent. Applied to correlated failures it dramatically understates the chance of both happening.
• Confusing exclusive with independent. They are close to opposites. Mutually exclusive events are maximally dependent: one occurring guarantees the other did not.
• Conditioning on something with probability zero. The formula divides by P(B), so it is undefined when B cannot happen. Shrink the world to nothing and there is nothing left to count.

Worth remembering

Conditioning on B does not change the outcomes, it discards the ones where B did not happen, so P(A | B) is simply the count of outcomes in both events divided by the count in B. Independence is the special case where that shrinking leaves A's share unchanged, which is exactly when P(A and B) = P(A)P(B) is valid and not before. The two conditionals P(A|B) and P(B|A) are different numbers and can differ enormously when one event is rare — converting between them is what Bayes' theorem is for.

Confounding, and why conditioning matters for causation

Conditional probabilities are also how you check whether an association is real or an artefact.

The classic illustration: ice cream sales and drowning incidents rise together. P(drowning | high ice cream sales) is genuinely higher than P(drowning). But condition on temperature as well, and within any given temperature band the association disappears. Temperature was driving both.

That structure — a third variable causing both — is a confounder, and it is why "controlling for" variables is the central move in observational statistics.

Simpson's paradox is the extreme case: a trend that holds in every subgroup can reverse when the groups are pooled. A treatment can be better for mild cases and better for severe cases, and appear worse overall, simply because it was given more often to severe cases. The lesson is not that statistics lie; it is that the conditional and unconditional questions are genuinely different, and you must decide which one you are asking.

Questions people ask

Does P(A|B) equal P(B|A)? Almost never. P(rain | clouds) is high; P(clouds | rain) is essentially 1. Confusing the two is the most common probability error there is.

What if P(B) is zero? The conditional probability is undefined — you cannot condition on something impossible.

How is this related to Bayes' theorem? Bayes' theorem is a rearrangement of this definition, and it is what lets you convert P(B|A) into P(A|B).

Is machine learning just conditional probability? Supervised learning largely is: estimate P(target | features) from examples. Classifiers estimate it directly; regressors estimate the conditional mean.

What does "conditionally independent" mean? Independent once you know a third variable. Two symptoms may be correlated overall but independent given the disease — which is exactly the assumption Naive Bayes makes.

Why do dependent features hurt Naive Bayes? Because it multiplies their probabilities as if each were fresh evidence, so correlated features count the same signal several times and the output becomes overconfident.

Recap in one screen

• P(A|B) restricts attention to the cases where B happened: P(A and B) / P(B).
• Conditioning is what turns a population rate into an individual prediction — supervised learning is P(target | features).
• Independence means conditioning changes nothing, and it is what allows probabilities to multiply.
• The chain rule factorises a joint probability into conditionals, which is how language models score sentences.
• Confounders and Simpson's paradox show that conditional and unconditional answers can differ, and even reverse.