P-Value

Operational definition

The p-value is the probability of observing a result at least as extreme as the one actually found in a sample, calculated on the assumption that the null hypothesis is true. It is not a statement about how likely the null hypothesis itself is, and it does not measure the size or practical importance of an effect. A p-value only answers one narrow question: given a specified null hypothesis (typically ‘no effect’ or ‘no difference’), how surprising is data this extreme, or more extreme, if that null hypothesis were actually correct?

Formally, for a test statistic computed from sample data, the p-value is P(observing a result this extreme or more extreme | H₀ is true) — a conditional probability calculated under the null hypothesis (H₀), not a probability attached to any hypothesis after the fact. Reversing that conditional — treating the p-value as P(H₀ is true | the data) — is one of the most persistent errors in applied statistics and is explicitly flagged as a misinterpretation by the American Statistical Association (see below).

The significance-threshold convention

Researchers commonly compare the p-value against a pre-specified significance threshold, conventionally α = 0.05, and describe a result as ‘statistically significant’ when the p-value falls below it. The 0.05 convention traces to the statistician Ronald Fisher’s early-twentieth-century work and became a durable norm across many fields, but the threshold itself is an arbitrary convention, not a mathematical law — some fields (e.g. particle physics, genome-wide association studies) use far stricter thresholds precisely because a single fixed cutoff performs badly when thousands of comparisons are tested at once. A p-value just above or below 0.05 (e.g. 0.049 vs 0.051) does not represent a meaningful difference in the underlying evidence, even though the significant/non-significant label treats it as a hard line.

Worked examples

Example 1 — correct interpretation. A trial compares a new intervention to a control and reports p = 0.03 for the primary outcome. The correct reading: if the intervention truly had no effect (the null hypothesis), a difference this large or larger would occur only about 3% of the time by chance alone, across many repeated identical studies. It does not mean there is a 97% probability the intervention works, and it says nothing about the size of the benefit — a statistically significant result can still be clinically trivial.

Example 2 — a large p-value. A study reports p = 0.42 for a comparison of two groups. This means data this extreme, or more extreme, would be expected roughly 42% of the time even if there were truly no difference between the groups — unsurprising data under the null hypothesis. This is routinely, and incorrectly, summarized as ‘proving no difference exists’; a high p-value indicates the data are consistent with the null hypothesis, not that the null hypothesis has been confirmed. A non-significant result is frequently just as consistent with an underpowered study as with a genuinely null effect — the p-value alone cannot distinguish the two, which is why reporting an confidence interval alongside it matters.

Counter-example: what a p-value does not tell you

A p-value of 0.001 is sometimes described in the literature as showing the effect is ‘highly important’ or ‘large.’ This is a category error: statistical significance describes how unlikely the observed data pattern would be under the null hypothesis, not how big or practically meaningful the effect is. With a large enough sample size, even a trivially small, practically meaningless difference will reliably produce a very small p-value. Reporting an effect size alongside the p-value, and specifying which statistical test produced it, is necessary to interpret a result responsibly — the p-value on its own is an incomplete summary of the evidence.

The ASA statement and the debate over p-value misuse

Concern that p-values were being systematically misinterpreted and misused across the scientific literature led the American Statistical Association (ASA) to take the unusual step, in March 2016, of publishing an official statement on the topic: ‘The ASA Statement on p-Values: Context, Process, and Purpose,’ by Ronald L. Wasserstein and Nicole A. Lazar, in The American Statistician (vol. 70, issue 2, pp. 129–133). It sets out six principles, including that p-values do not measure the probability that the studied hypothesis is true or the probability the data were produced by random chance alone, and that a p-value, or statistical significance, does not measure the size of an effect or the importance of a result.

The ASA statement was followed by continued methodological debate, including a 2019 special issue of The American Statistician in which the ASA’s leadership discussed moving beyond the phrase ‘statistically significant’ altogether, and a 2021 ASA President’s Task Force statement on statistical significance and replicability. The recurring concern across this literature is practices such as data dredging (p-hacking) — running or reporting only the subset of analyses that happen to cross the significance threshold — which inflates the apparent rate of ‘significant’ findings and contributes to non-reproducible results. None of this literature argues that the p-value is meaningless; the consensus position is narrower: a p-value is one piece of evidence, correctly interpreted under the null-hypothesis-conditional definition above, and should be reported alongside effect sizes, confidence intervals, and enough methodological detail (pre-registration, full reporting of tests conducted) for a reader to judge the result in context.

Related terms

• Confidence interval — the companion statistic that expresses the range of plausible effect sizes, addressing what a bare significant/non-significant p-value label leaves out.
• Chi-square test — one of the named statistical tests that produces a p-value, used for categorical data.
• Statistical test — the general procedure (of which p-value calculation is step three) underlying every named hypothesis test.
• Frequentist vs. Bayesian statistics — the p-value is a frequentist concept; Bayesian methods answer a related but distinct question using posterior probabilities instead.