The p-value is a shaded area under an assumption. Move the observation and watch the area, and the assumption, stay exactly where they were.
Overview
The machinery
The null hypothesis is a exact, boring claim: no effect, no difference, nothing happening. It's exact on purpose, because a exact claim implies a spread for the test statistic.
That spread is the curve. It's what the statistic would look like across repeated samples if the null were true. each use of a p-value inherits that if-based.
The p-value is the shaded tail: the chance of a statistic at least as extreme as the one seen, under the null.
Drag the seen statistic and watch the area shrink as it moves outward. A large statistic is unusual under the null. So Its tail is small — and a small tail is what "unlikely, if the null were true" means.
Hypothesis Testing and p-values
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
The curve is the distribution of the test statistic assuming the null hypothesis is true. Everything else follows from that.
The p-value is the shaded area: the probability of a result at least this extreme, if the null held.
It is not the probability the null is true, and it is not the probability your result was a fluke.
0.05 is a convention Fisher suggested as a rule of thumb, not a property of nature.
Hypothesis Testing and p-values
The most used and most misread number in statistics, defined just enough to stop misreading it.
One test, worked by hand
The modelling below finds a p-value by brute force. It's worth doing one the plain way first, on numbers small enough to check, so the quantity stops being what the library returned.
Five readings per group:
group
values
mean
control
12, 14, 11, 13, 15
13
variant
15, 17, 14, 16, 18
16
The deviations from the mean are the same five numbers in both groups — −1, 1, −2, 0, 2. So each group's sum of squared deviations is 10, and each sample variance is 10 / 4 = 2.5. Cutting by n − 1 rather than n is the correction covered in expectation and variance.
The difference between the means is 3. What decides whether 3 is a lot is the standard error of that difference:
SE = √(2.5/5 + 2.5/5) = √1 = 1
t = 3 / 1 = 3, on 5 + 5 − 2 = 8 degrees of freedom, which gives a two-tailed p = 0.0171.
Read that last number as the meaning demands: *if the two groups were really the same*, a gap of 3 or more in either direction would turn up in about one sample in 59. It says nothing about how likely it's that they're the same, and nothing about whether a gap of 3 matters to anyone.
Notice which quantity did the work. The difference of 3 was never enough on its own — it had to be measured against the SE. The SE fell as the spread fell and as the sample grew. Quadruple the sample and the SE halves, because it cuts by √n rather than n. That single square root is why the fourth hundred readings buy so much less than the first hundred.
What it is not
Three misreadings, all common enough to have names.
Not the chance that the null is true. The p-value is found *given* the null. It can't then tell you how likely the assumption was. Getting from P(data | null) to P(null | data) needs Bayes' theorem and a prior, and the two quantities can differ by orders of magnitude.
Not the chance the result was chance. Same error, differently worded.
Not a measure of effect size. With enough data, a difference of no hands-on consequence makes a tiny p-value. The p-value answers "is there evidence of *any* effect", never "is the effect large enough to matter". This is why trust intervals are increasingly preferred: they carry the size and the uncertainty in one object.
Failing to reject is not accepting
Set the seen statistic near zero. The p-value is large, and the conclusion is that the data is steady with the null.
It's not that the null is true. A large p-value is made both by a real lack of effect and by a study too small to detect one. Distinguishing those needs power, and it's why "no major difference" in an underpowered study means almost nothing.
One tail or two
Switch the option. The two-tailed p-value is just double the one-tailed one for the same statistic. The same area is being counted on both sides.
The choice must be made before seeing the data. Deciding afterwards that you only cared about one direction halves your p-value for free, and that is a recognised form of cheating rather than a modelling choice.
Why one in twenty, exactly
The claim that a 0.05 threshold lets one test in twenty through isn't a rough guess. It's exact, and the reason is a fact about p-values that is rarely stated: when the null is true, the p-value is uniformly spread between 0 and 1. each value is equally likely. A p of 0.03 is just as probable as a p of 0.87.
Once that is granted, all else follows arithmetically. The chance that a uniform value falls below α is α. So a 0.05 cutoff admits 5% of true nulls by build. Not as a regrettable side effect, but as the meaning of what the cutoff is doing. Twenty thousand tests with no effect in them at all:
threshold
fraction of nulls below it
uniform predicts
p < 0.01
0.0111
0.01
p < 0.05
0.0520
0.05
p < 0.10
0.1003
0.10
p < 0.25
0.2492
0.25
p < 0.50
0.4953
0.50
p < 0.90
0.8968
0.90
The deciles come out 0.10, 0.20, 0.30, 0.40, 0.50, 0.61, 0.71, 0.81, 0.90 — the identity line, which is what a flat spread looks like from the inside.
This is also the mechanism behind p-hacking, and it makes the size of the problem computable rather than vague. Running k free tests on nothing means drawing k uniforms and keeping the smallest, and the smallest of k uniforms has an mean of 1/(k + 1):
tests run
average smallest p
chance at least one clears 0.05
5
0.167
1 − 0.95⁵ = 22.6%
20
0.048
1 − 0.95²⁰ = 64.2%
100
0.010
1 − 0.95¹⁰⁰ = 99.4%
Twenty metrics on an test that changed nothing, and the odds are roughly two in three that one of them looks major. At a hundred it's a near certainty. No one has to be dishonest for this to happen — it's what the math does on its own. This Is why the correction has to be decided before the tests are run.
Why 0.05 is under attack
The threshold is a rule. Fisher suggested it as a rough guide and by name didn't intend it as a choice rule.
The problems are structural.
p-hacking. Test enough hypotheses and a thing crosses 0.05 by chance. At 0.05, one test in twenty does so with no effect present at all.
Publication bias. Major results get published, so the literature over-shows them, and the effect sizes it reports are inflated.
The cliff. p = 0.049 and p = 0.051 are the same as evidence and are treated as opposites.
The responses in circulation: report exact p-values rather than thresholds, report effect sizes and intervals alongside, pre-register the analysis, and correct for multiple checks. Some fields have moved the threshold to 0.005; others argue for abandoning the dichotomy fully.
Where it goes wrong with Hypothesis Testing and p-values
Reading p as the chance the null is true. The central error.
Treating "not major" as "no effect." Check the power.
Choosing the tail after seeing the data.
Running many tests without correction. Bonferroni is crude and better than nothing. False find rate control is often the better tool.
Questions people ask about Hypothesis Testing and p-values
What is a p-value? The probability of seeing data at least as extreme as yours if the null hypothesis were true. It is not the probability that the null is true.
What does it mean to fail to reject the null? That the data did not give enough evidence against it. It is not proof that the null is correct, since the test may simply lack power.
Why is 0.05 the usual threshold? It is a convention that dates back to early statistical practice, not a law of nature. Choose the threshold to match the cost of a wrong decision.
What is the difference between one-tailed and two-tailed tests? A one-tailed test looks for a difference in one stated direction. A two-tailed test looks for any difference and is the safer default.
What is p-hacking? Running many tests or analyses and reporting the ones that cross the line. With 20 tests at 0.05 you expect one false positive by chance.
Does a small p-value mean a large effect? No. With enough data a trivial effect gets a tiny p-value. Always report the effect size and an interval as well.
Where p-values come from, and what breaks them
A p-value found by modelling rather than looked up in a table, then the same test run two hundred times on data with no effect in it at all.
example_01.pyNumPy
import numpy as np
import math
rng = np.random.default_rng(0)
control = rng.normal(0.1000, 0.02, 300)
variant = rng.normal(0.1045, 0.02, 300)
observed = variant.mean() - control.mean()
print("A/B test: control %.4f, variant %.4f, difference %+.4f"
% (control.mean(), variant.mean(), observed))
print()
# the null hypothesis says the labels do not matter. so shuffle them.
pool = np.concatenate([control, variant])
null_diffs = np.empty(20_000)
for i in range(20_000):
rng.shuffle(pool)
null_diffs[i] = pool[300:].mean() - pool[:300].mean()
p = (np.abs(null_diffs) >= abs(observed)).mean()
print("shuffling the labels 20,000 times, so no real effect can survive:")
print(" the shuffled differences spread out to sd %.5f" % null_diffs.std())
print(" %.2f%% of them were at least as big as the one we measured" % (100 * p))
print(" that fraction is the p-value: %.4f" % p)
print()
print("it is NOT the chance the null is true; it is the chance of data this")
print("extreme IF the null were true.")
def p_value(a, b):
d = b.mean() - a.mean()
se = np.sqrt(a.var(ddof=1) / len(a) + b.var(ddof=1) / len(b))
return 2 * (1 - 0.5 * (1 + math.erf(abs(d / se) / np.sqrt(2))))
print("now the failure mode. 1000 tests, no effect in any of them:")
ps = np.array([p_value(rng.normal(0.1, 0.02, 300), rng.normal(0.1, 0.02, 300))
for _ in range(1000)])
hits = (ps < 0.05).sum()
print(" %d of 1000 came out under 0.05, from pure noise." % hits)
print(" expected: 1000 * 0.05 = 50.0")
print()
print("smallest from that batch:", " ".join("%.4f" % v for v in np.sort(ps)[:5]))
print()
print("a 5%% threshold lets one in twenty through by chance. tighten it")
print(" Bonferroni for 1000 tests: %.7f" % (0.05 / 1000))
print(" tests still under it: %d" % (ps < 0.05 / 1000).sum())
Output
Check yourself
0 of 3
Answer without scrolling back up.
A p-value is the probability of:
It is computed assuming the null, so it cannot tell you how likely that assumption was. Going the other way needs Bayes' theorem and a prior.
A study reports p = 0.4. What can you conclude?
A large p-value is produced both by a real absence of effect and by a study too small to detect one. Distinguishing them requires power.
Why must the choice of one or two tails be made before seeing the data?
The two-tailed p-value is exactly double the one-tailed one for the same statistic, so the decision has to precede the observation.
Cheat sheet
Hypothesis Testing and p-values
Run a hypothesis test step by step. See what the p-value actually measures and why a small one does not mean the effect is large.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.