What Is a P Value? Definition, Misinterpretations, and How to Read One

A p value is the probability, calculated under the assumption that the null hypothesis is true, of observing a result at least as extreme as the one actually obtained — or more extreme — if that experiment were repeated indefinitely. It is a single number, conventionally reported between 0 and 1, that summarizes how compatible an observed dataset is with a specified null hypothesis. That is the entire claim a p value makes. It says nothing directly about whether the null hypothesis is true, how large an effect is, or whether a finding will replicate — three claims p values are routinely, and incorrectly, asked to support.

A p value belongs to inferential statistics, answering a question about a hypothesized population; it is a different kind of claim than the summary numbers described in CASRAI’s guide to descriptive statistics, which describe only the sample actually collected. Neither replaces the other — a full results section normally reports both.

The correct definition, stated precisely

Formally: for a test statistic computed from sample data, the p value is P(data this extreme or more extreme | H₀ is true) — a conditional probability calculated under the null hypothesis, not a probability attached to any hypothesis after seeing the data. Every word in that definition is load-bearing:

• “At least as extreme” — the p value is a tail probability. It accounts for every outcome as unusual or more unusual than the one observed, not just the exact observed value.
• “Assuming the null hypothesis is true” — the calculation is entirely conditional on H₀. The p value never asks “how likely is the null hypothesis,” only “how surprising would this data be if the null hypothesis were correct.”
• A property of the data given the hypothesis, not of the hypothesis given the data — reversing that conditional (treating p as the probability the null hypothesis is true) is one of the most persistent and consequential errors in applied statistics, sometimes called the “inverse probability” or “prosecutor’s” fallacy. The two quantities are related through Bayes’ theorem, but they are not the same number, and calculating one does not tell you the other without additional information (a prior probability) that a p value does not supply.

A p value is always computed relative to a specific statistical test and a specific null hypothesis; the same dataset can produce different p values under different tests or different null hypotheses, which is one reason the test and hypothesis used need to be reported alongside the number, not just the number itself. A t-test comparing two means, a chi-square test of independence between two categorical variables, and a correlation coefficient‘s own significance test are three of the most common sources of a p value in the research literature, and each rests on its own null hypothesis and assumptions that are worth checking before interpreting the p value it produces.

What a p value is not

Because the correct definition is conditional and narrow, several intuitive-sounding restatements of it are wrong. These four are the errors that show up most often in manuscripts, journal clubs, and press coverage of research findings, and they are worth stating explicitly rather than leaving implicit.

It is not the probability that the null hypothesis is true

P(data | H₀) is not P(H₀ | data). A p value of 0.03 does not mean there is a 3% chance the null hypothesis is correct, and 1 minus the p value is not the probability that the alternative hypothesis is correct. Computing the probability that a hypothesis is true, given the data, requires a prior probability for that hypothesis — information a p value’s calculation does not use and cannot supply. This is the single most common p value misinterpretation in the literature and the one the American Statistical Association’s 2016 statement addresses first and most directly.

It is not “the probability the results occurred by chance”

This everyday phrasing sounds close to correct but elides the conditioning: a p value is the probability of data this extreme given that only chance (the null hypothesis) is operating, not the probability that chance alone, as opposed to a real effect, produced the observed data. Once the data exist, they were produced by whatever actually happened — a real effect, pure sampling variation, measurement error, or some mixture — and the p value cannot partition that mixture after the fact.

It is not the probability that the finding will replicate

A small p value describes how unlikely the current dataset would be under the null hypothesis; it does not describe how likely an independent future study is to find a similar result. Replication probability depends on the true effect size, the study’s statistical power, and sources of variability the original p value cannot capture. Studies with very small p values have failed to replicate, and studies with p values just above 0.05 have replicated reliably — p value size and replicability are correlated in aggregate but are not interchangeable at the level of a single study. See CASRAI’s coverage of the replication crisis for how this gap between statistical significance and reproducibility plays out across fields.

1 minus p is not the probability the alternative hypothesis is true

This follows from the same conditional-probability error as the first misreading, but is common enough to name separately: subtracting a p value from 1 produces another number derived entirely from data-given-null-hypothesis reasoning. It does not become a data-given-alternative-hypothesis, or hypothesis-given-data, quantity through subtraction. There is no valid arithmetic operation on a p value alone that converts it into a probability statement about a hypothesis.

The 0.05 threshold: convention, not law

Researchers commonly compare a p value against a pre-specified significance threshold — conventionally α = 0.05 — and label a result “statistically significant” when the p value falls below it. That threshold traces to Ronald Fisher’s early-twentieth-century work, where 0.05 was offered as one reasonable convenience among several, not a derived mathematical constant. It became a durable cross-field norm largely through repetition and institutional habit rather than any principled argument that 0.05 is the correct cutoff for scientific claims in general.

Two consequences follow directly from the threshold being arbitrary rather than fundamental:

• The cutoff is not universal. Fields that routinely test many hypotheses at once, such as particle physics (which requires “5-sigma,” roughly p < 3 × 10⁻⁷, before claiming a discovery) or genome-wide association studies (which commonly use thresholds near p < 5 × 10⁻⁸), use far stricter cutoffs precisely because a fixed 0.05 threshold performs badly once thousands of comparisons are run — see multiple comparisons, below.
• Dichotomizing at any threshold discards information. A p value of 0.049 and a p value of 0.0001 are both “significant” under a 0.05 rule, and a p value of 0.051 is not, even though 0.049 and 0.051 represent essentially identical evidence and 0.0001 represents much stronger evidence than either. Collapsing a continuous quantity into a binary significant/not-significant label loses exactly the information — the actual strength of evidence — a reader most needs. This is why current methodological guidance, including the ASA’s own follow-up work, increasingly favors reporting the exact p value and treating “statistical significance” as a description on a continuum rather than a pass/fail gate. CASRAI’s companion guide on how to report p values covers the mechanics of exact-value versus threshold reporting in APA style.

P values and sample size

A p value is sensitive to sample size in a way that is easy to state but often forgotten in practice: for a fixed true effect, larger samples produce smaller p values, because a larger sample estimates the effect more precisely and the same true difference produces a larger, less noise-dominated test statistic. The practical consequences run in both directions:

• Large samples make trivial effects statistically significant. With enough observations, almost any nonzero difference — including one too small to matter to anyone — will eventually cross a p < 0.05 threshold. A statistically significant result from a very large sample can still be practically meaningless.
• Small samples can miss real effects. An underpowered study can fail to reach significance even when a genuine, practically important effect exists, simply because the sample was too small to estimate it precisely. A non-significant p value from a small study is compatible with both “no effect” and “an effect the study wasn’t equipped to detect” — the p value alone cannot distinguish the two.

Because of this, a p value can never be interpreted responsibly in isolation from the study’s sample size and statistical power. Planning the sample size in advance, via power analysis and sample size calculation, and reporting an effect size and confidence interval alongside any p value are the two standard remedies — see “Alternatives and companions” below.

The ASA Statement on p-values (2016)

Concern that p values were being systematically misinterpreted and misused across the published literature led the American Statistical Association to take the unusual step of issuing an official statement on the topic: Ronald L. Wasserstein and Nicole A. Lazar, “The ASA’s Statement on p-Values: Context, Process, and Purpose,” The American Statistician, volume 70, issue 2, pages 129–133 (2016). The statement sets out six principles for the use and interpretation of p values:

1. P values can indicate how incompatible the data are with a specified statistical model.
2. P values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p value passes a specific threshold.
4. Proper inference requires full reporting and transparency — disclosing how many analyses were conducted and how results were selected for reporting.
5. A p value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a p value does not provide a good measure of evidence regarding a model or hypothesis.

The statement was explicit that the answer is not to abandon p values, but to stop asking them to answer questions they cannot answer. It was followed by continued methodological debate, including a 2019 special issue of The American Statistician in which ASA leadership discussed moving beyond the phrase “statistically significant” altogether, and a 2021 ASA President’s Task Force statement on statistical significance and replicability. CASRAI’s comparison of statistical significance vs. clinical significance covers a closely related distinction the ASA statement’s fifth principle is directly about: a significant p value and a meaningful effect are not the same claim.

P-hacking, HARKing, and the garden of forking paths

The gap between the 0.05 threshold and career or publication incentives has produced a well-documented set of practices, sometimes grouped as questionable research practices, that inflate the apparent rate of “significant” findings without inflating the amount of genuine evidence behind them:

• P-hacking (data dredging) — trying multiple analyses, subgroups, outcome measures, or exclusion criteria on the same dataset and reporting only the combination that crosses the significance threshold, without disclosing the others that were tried. Because roughly 1 in 20 tests will cross p < 0.05 purely by chance even when the null hypothesis is true, testing enough variations on the same data makes finding at least one “significant” result almost guaranteed — it just isn’t evidence of anything real.
• HARKing (Hypothesizing After the Results are Known) — presenting a hypothesis discovered by exploring the data as though it had been specified in advance. This converts an exploratory, hypothesis-generating finding into what reads like a confirmatory test, without the p value’s underlying logic (which assumes the hypothesis was fixed before the data were seen) actually holding.
• The garden of forking paths — a related but subtler problem, described by statisticians Andrew Gelman and Eric Loken: a single, honestly reported analysis can still be a “hack” in effect if the specific analytical choices (which covariates to include, how to code a variable, which subgroup to examine) were made after looking at the data, even without deliberately trying multiple versions. The researcher only ran one analysis, but it was one of many plausible analyses that could have been run, and the p value’s validity depends on the analysis path having been fixed independently of the data.

CASRAI’s direct comparison of p-hacking vs. HARKing covers how these two practices differ mechanically and how each is detected. All three problems are a major contributor to the replication crisis and share the same structural fix: fixing the hypothesis, outcome measures, and analysis plan before seeing the data, and disclosing that plan publicly.

Preregistration and registered reports

Because p values are only interpretable when the hypothesis and analysis plan were genuinely fixed in advance, the field’s main structural remedy is to make that advance commitment verifiable rather than simply asserted after the fact:

• Preregistration is the practice of publicly time-stamping a study’s hypotheses, design, and planned analysis before data collection begins, closing off the ability to silently change the analysis plan after seeing the results. CASRAI covers the mechanics in guides on preregistration of a study protocol and OSF preregistration.
• Registered reports go a step further: the introduction and methods are peer-reviewed and provisionally accepted for publication before data collection, so publication no longer depends on which p value the results happen to produce. See CASRAI’s guide on how to write a registered report.

Neither eliminates the underlying statistical limits of a p value, but both remove the incentive and the opportunity to p-hack, HARK, or quietly wander the garden of forking paths.

Multiple comparisons

Every additional hypothesis test run against the same data raises the chance that at least one of them crosses p < 0.05 purely by chance, even if every null hypothesis being tested is actually true. Run 20 independent tests at α = 0.05 and the probability of at least one false positive climbs to roughly 64%, not 5%. Two standard correction approaches address this directly:

• Bonferroni correction divides the significance threshold by the number of comparisons (α / n), which controls the family-wise error rate — the probability of any false positive across the whole set of tests — but becomes very conservative, and can miss real effects, as the number of comparisons grows large.
• Benjamini-Hochberg false discovery rate (FDR) correction instead controls the expected proportion of false positives among the tests called significant, which is typically far less conservative than Bonferroni and is the standard approach in fields that routinely run very large numbers of comparisons at once, such as genomics.

Either way, the number of comparisons actually attempted — not just the ones reported — has to be disclosed for a reader to judge whether correction was needed and applied correctly; this is the same full-reporting principle the ASA statement’s fourth principle addresses.

Alternatives and companions to the p value

Current methodological guidance treats a bare p value as an incomplete summary of evidence and expects it to appear alongside at least one of the following, not in isolation:

• Confidence intervals report a range of plausible values for the effect itself, showing both its estimated size and the precision of that estimate — information a significant/non-significant label discards entirely.
• Effect size reports how large a difference or relationship actually is, independent of sample size, directly addressing what the “p values and sample size” section above shows a p value cannot tell you on its own.
• Likelihood ratios compare how well two competing hypotheses (not just the null) explain the observed data, without the significance-threshold framing.
• Bayes factors quantify the relative evidence for one hypothesis over another in a way that, unlike a p value, can be combined with a prior probability to produce an actual posterior probability that a hypothesis is true — the very quantity a p value is most often mistaken for.
• Equivalence testing (e.g. the two one-sided tests, or TOST, procedure) is the correct tool when the actual research question is “are these two things practically the same,” rather than “is there any detectable difference” — a question a standard p value cannot answer, because a non-significant p value is evidence of absence only when the study was well powered to detect a meaningful difference. CASRAI covers a related clinical-trial design in equivalence trial.

A plain-language interpretation template

A p value can be reported correctly, in plain language, using a template close to this:

“If [the null hypothesis] were actually true, a result this extreme, or more extreme, would occur about [p × 100]% of the time across repeated studies. We observed p = [value], which [is / is not] below our pre-specified threshold of [α]. This [does / does not] tell us the size of the effect or the probability that [the null hypothesis] is true — see the effect size and confidence interval reported alongside it for that.”

Filling in that template forces the two disclosures a bare “p < .05, significant” statement omits: what, precisely, is being conditioned on, and what the number does and does not claim beyond that.

Frequently asked questions

What counts as a “small” p value?

There is no universal answer — only a threshold relative to a pre-specified α, conventionally 0.05 in many fields, but far stricter in fields that run many comparisons (see “The 0.05 threshold,” above). A p value should always be interpreted relative to the threshold and context specified in advance, not against an intuitive sense of what “counts” as small.

Can a p value ever be exactly zero?

No. A computed p value can round to .000 at limited decimal precision, but the true value is never exactly zero; standard reporting guidance is to write “p < .001” rather than “p = .000.” See CASRAI’s guide on how to report p values for the full formatting convention.

Does a non-significant p value prove there is no effect?

No. A non-significant result is consistent with “no effect,” but it is equally consistent with “a real effect the study was underpowered to detect.” Distinguishing the two requires looking at the study’s statistical power and the confidence interval around the effect estimate, not the p value alone.

Is a smaller p value always better evidence?

A smaller p value indicates the observed data would be more surprising under the null hypothesis, but “better evidence” also depends on the study’s design, sample size, whether the analysis was pre-specified, and whether multiple comparisons were corrected for. A very small p value from a small, uncorrected, exploratory analysis can be weaker evidence than a moderate p value from a large, preregistered, confirmatory study.

What is the difference between a p value and statistical significance?

A p value is the underlying number; “statistical significance” is a label applied when that number crosses a pre-specified threshold. See CASRAI’s comparison of statistical significance vs. clinical significance for how the significance label relates — and often fails to relate — to whether a finding actually matters.