Two overlapping distributions and one threshold between them. Every choice you make trades one error for the other.
Overview
Two curves, one line
The grey curve is the test statistic when the null is true. The orange curve is the same statistic when the effect is real. They share, and the share is the entire problem.
The dashed line is the threshold: above it you reject the null, below it you do not.
Type I error — the grey area to the right of the line. The null was true and you rejected it. A false plus.
Type II error — the orange area to the left. The effect was real and you missed it. A false minus.
Power is what is left of the orange curve: 1 - beta, the chance of catching a real effect.
Type I and Type II Errors
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
Type I is a false positive: rejecting a null that was true. Its rate is α, and you choose it.
Type II is a false negative: failing to reject a null that was false. Its rate is β, and you mostly do not.
Power is 1 − β: the chance of detecting an effect that is really there.
Moving the threshold trades one for the other. Only more data, or a larger true effect, improves both.
Type I and Type II Errors
The two ways a test can be wrong, why you can't cut both, and what power really buys.
The trade
Change alpha from 0.05 to 0.01. The threshold moves right, the grey tail shrinks — and the orange tail grows. Beta rises, power falls.
Move it the other way and the reverse happens.
You can't reduce both by moving the line. That isn't a limitation of the method. It's what "the spreads share" means. Any threshold is a statement about which error you'd rather make.
What actually helps
Two things, and only two.
More data. Drag n upward. Both curves narrow, because their spread is the standard error and that falls like sqrt(n). The share shrinks, and both error rates can fall at once. This is the only lever that raises both without given any of it.
A larger true effect. Drag the effect control. The curves separate. You do not often control this, but it's why detecting a large effect needs so much less data than detecting a small one.
Choosing alpha honestly
The rule is 0.05, and treating it as fixed is the mistake. The right value depends on which error costs more.
A screening test for a treatable disease. A false minus means missing a case. A false plus means an unneeded follow-up. Loosen alpha, gain power.
A criminal conviction. The system by name prefers false negatives, which is what "beyond reasonable doubt" codes.
Particle physics. The five-sigma standard is alpha near 3 in 10 million, because the field has been burned by fluctuations and there's enough data to afford it.
A/B testing a button colour. A wrong call is cheap. Standard thresholds are generous, and the real risk is running many tests and picking winners.
Power, before the experiment
Power analysis is the math no one enjoys and everybody should do: given a target power — often 80% — an alpha, and the smallest effect worth detecting, how large a sample is needed?
Set the effect to 0.2 and watch what n has to be before power reaches 80%. The needed sample grows roughly with the inverse square of the effect size, which is why detecting small effects is so costly and why so much published work is underpowered.
An underpowered study is worse than no study. It often fails to find a real effect — and when it does find one, the estimate is inflated, because only an unusually large sample fluctuation could have crossed the threshold at that sample size. That is the winner's curse, and it's a substantial part of why published effects shrink on replication.
The question alpha and power cannot answer alone
Alpha is the chance of a false plus given that the null is true, and power is the chance of a true plus given that it's false. Neither of them answers the question anyone really has after a major result: how likely is it that *this* finding is real?
That question needs one more number, and it's the one no one reports — the proportion of the hypotheses being tested that were true to begin with. Work it through on a thousand tests, all done right, alpha at 0.05 and power at a fair 80%, in a field where one hypothesis in ten is true:
true (100)
false (900)
significant
80
45
not significant
20
855
There are 125 major results and 45 of them are false. Thirty-six per cent of the findings are wrong, with no p-hacking, no bad faith and no methodological error — only 5% of many false hypotheses being larger than 80% of a small number of true ones.
All follows from the ratio of those two counts:
true hypotheses
alpha
power
share of significant results that are real
10%
0.05
0.80
64.0%
10%
0.05
0.35
43.8%
1%
0.05
0.80
13.9%
10%
0.005
0.80
94.7%
50%
0.05
0.80
94.1%
Read the third row. In exploratory work where one hypothesis in a hundred is true — screening compounds, scanning the genome, trying features — about six of each seven major results are false, and the study was run just right. Read the second: halving the power doesn't halve the errors, it raises the false share from 36% to 56%. This Is the concrete cost of the underpowered study given above.
Then read the last two rows, because they're the levers. Tightening alpha tenfold takes the false share from 36% to 5%. This Is the actual arg behind moving the threshold to 0.005. Testing better hypotheses does the same thing — a field where half the ideas are right barely has the problem. Neither lever is inside the statistics. They're choices about how hard you're willing to make it to publish, and about what you bother to test.
Where it goes wrong with Type I and Type II Errors
Reading a non-major result as no effect. Report the power, or the interval.
Running the power analysis afterwards. Post-hoc power found from the seen effect is a restatement of the p-value and carries no new info.
Fixing alpha at 0.05 even so of costs. It's a rule.
Peeking at results and stopping when major. This inflates the true Type I rate far above the nominal one. Sequential testing methods exist and correct for it.
Questions people ask about Type I and Type II Errors
What is a Type I error? Rejecting a true null hypothesis, a false positive. Its probability is the significance level alpha.
What is a Type II error? Failing to reject a false null hypothesis, a false negative. Its probability is called beta.
What is statistical power? Power is 1 minus beta: the chance of detecting a real effect of a given size. Aim for 80 percent or more before running an experiment.
How do I reduce both errors at once? Increase the sample size. With a fixed sample, lowering alpha raises beta, so you have to trade one against the other.
Which error is worse? It depends on the cost. A false alarm in a drug trial and a missed disease in screening carry very different consequences, so pick alpha with that in mind.
How do I compute the sample size I need? Fix alpha, the desired power and the smallest effect you care about, then use a power calculation. Libraries such as statsmodels provide it.
Both errors, counted
Lower your threshold and you catch more real effects while raising more false alarms. Here both rates are measured across a sweep. The trade is visible rather than given.
example_01.pyNumPy
import numpy as np
import math
rng = np.random.default_rng(0)
N = 200
TRIALS = 4000
def p_value(effect):
a = rng.normal(0.0, 1.0, N)
b = rng.normal(effect, 1.0, N)
d = b.mean() - a.mean()
se = np.sqrt(a.var(ddof=1) / N + b.var(ddof=1) / N)
z = abs(d / se)
return 2 * (1 - 0.5 * (1 + math.erf(z / np.sqrt(2))))
null_ps = np.array([p_value(0.00) for _ in range(TRIALS)]) # no effect exists
real_ps = np.array([p_value(0.25) for _ in range(TRIALS)]) # a real effect
print("%10s %18s %18s %10s" % ("threshold", "type I (false alarm)",
"type II (missed)", "power"))
for alpha in (0.20, 0.10, 0.05, 0.01, 0.001):
type1 = (null_ps < alpha).mean()
type2 = (real_ps >= alpha).mean()
print("%10.3f %17.3f %17.3f %9.3f" % (alpha, type1, type2, 1 - type2))
print()
print("the type I rate tracks the threshold you chose -- that is what alpha is.")
print("tightening it from 0.05 to 0.001 cuts false alarms by 50x and misses")
print("far more real effects. there is no setting that fixes both.")
print()
print("the other lever is sample size:")
for n in (50, 200, 800, 3200):
N = n
ps = np.array([p_value(0.25) for _ in range(1500)])
print(" n=%5d power at alpha=0.05: %.3f" % (n, (ps < 0.05).mean()))
print()
print("more data lowers the miss rate without touching the false alarm rate.")
print("that is the only way to improve both at once.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
You tighten alpha from 0.05 to 0.01. What happens to power?
The threshold moves right, shrinking the false-positive tail and growing the false-negative one. Moving the line always trades one error for the other.
What improves both error rates at once?
Their spread is the standard error, which falls like root n, so the overlap shrinks. A larger true effect also works, but you rarely control that.
Why are effects from underpowered studies often inflated?
The winner's curse - and a substantial part of why published effects shrink on replication.
Cheat sheet
Type I and Type II Errors
Adjust significance level and sample size and watch Type I and Type II error rates trade off. See what statistical power actually measures.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.