What it is not
Three misreadings, all common enough to have names.
Not the probability that the null is true. The p-value is computed *assuming* the null. It cannot then tell you how likely the assumption was. Getting from P(data | null) to P(null | data) needs [Bayes' theorem](bayes_theorem.html) and a prior, and the two quantities can differ by orders of magnitude.
Not the probability the result was chance. Same error, differently worded.
Not a measure of effect size. With enough data, a difference of no practical consequence produces a tiny p-value. The p-value answers "is there evidence of *any* effect", never "is the effect large enough to matter". This is why [confidence intervals](confidence_intervals.html) are increasingly preferred: they carry the size and the uncertainty in one object.
Failing to reject is not accepting
Set the observed statistic near zero. The p-value is large, and the conclusion is that the data is consistent with the null.
It is not that the null is true. A large p-value is produced both by a real absence of effect and by a study too small to detect one. Distinguishing those requires [power](type_i_and_type_ii_errors.html), and it is why "no significant difference" in an underpowered study means almost nothing.
One tail or two
Switch the alternative. The two-tailed p-value is exactly double the one-tailed one for the same statistic, because the same area is being counted on both sides.
The choice must be made before seeing the data. Deciding afterwards that you only cared about one direction halves your p-value for free, and that is a recognised form of cheating rather than a modelling choice.
Why 0.05 is under attack
The threshold is a convention. Fisher suggested it as a rough guide and explicitly did not intend it as a decision rule.
The problems are structural.
p-hacking. Test enough hypotheses and something crosses 0.05 by chance. At 0.05, one test in twenty does so with no effect present at all.
Publication bias. Significant results get published, so the literature over-represents them, and the effect sizes it reports are inflated.
The cliff. p = 0.049 and p = 0.051 are indistinguishable as evidence and are treated as opposites.
The responses in circulation: report exact p-values rather than thresholds, report effect sizes and intervals alongside, pre-register the analysis, and correct for multiple comparisons. Some fields have moved the threshold to 0.005; others argue for abandoning the dichotomy entirely.
Where it goes wrong
Reading p as the probability the null is true. The central error.
Treating "not significant" as "no effect." Check the power.
Choosing the tail after seeing the data.
Running many tests without correction. Bonferroni is crude and better than nothing; false discovery rate control is usually the better tool.