Written and maintained by CASRAI Editorial Board
Last updated
Statistical fallacies are reasoning errors that make a statistical result look more convincing than it is. The Stanford Encyclopedia of Philosophy defines a fallacy in general as “an argument that seems to be better than it really is.” Most of the ones that damage research share a pattern: the numbers are correct, but the conclusion drawn from them does not follow. This guide names the common ones, works the arithmetic for the two that are easiest to demonstrate, and links to the detailed pages on this site.
What is a statistical fallacy?
A statistical fallacy is a form of statistical argument that seems correct but proves, on examination, not to be. The wording follows the textbook definition of a fallacy quoted by the Stanford Encyclopedia of Philosophy (Copi): “a form of argument that seems to be correct but which proves, upon examination, not to be so.” A fallacy is different from a calculation error. The calculation can be exact and the inference still wrong.
The table lists the fallacies covered below and where the detail lives.
| Fallacy | The faulty step | Where to read more |
|---|---|---|
| Simpson’s paradox | Reading a pooled association as if it held in every subgroup | Worked example below |
| Base-rate neglect | Ignoring how common a condition is when reading a test result | Worked example below |
| P-value misreadings | Treating p as the probability the hypothesis is true, or significance as importance | What is a p-value |
| Ecological fallacy | Inferring individual-level relationships from group-level data | Ecological study design and the ecological fallacy |
| Correlation taken as causation | Reading an association as proof that one variable produces the other | Correlation vs causation and spurious correlation |
| Gambler’s fallacy | Expecting independent trials to “balance out” | Law of large numbers |
Formula: the percentage arithmetic behind Simpson’s paradox
Simpson’s paradox needs no special formula, only a rate computed two ways. For any group, the rate is r = x / n, where x is the number of recoveries (or events) and n is the number of people in the group. The paradox appears when the rate comparison computed on the pooled data points one way and the same comparison computed within each subgroup points the other way. The Stanford Encyclopedia of Philosophy entry on Simpson’s paradox describes the general phenomenon as an association between two variables that “emerges, disappears or reverses” when a population is split into subpopulations.
Worked example: Simpson’s paradox with real numbers
The counts below are Table 1 of the Stanford Encyclopedia of Philosophy entry on Simpson’s paradox (a medical treatment, recoveries out of patients). The percentages are computed here as x / n.
| Group | Treatment | Control |
|---|---|---|
| Men | 8 / 13 = 61.5% | 4 / 7 = 57.1% |
| Women | 12 / 27 = 44.4% | 2 / 5 = 40.0% |
| All patients (8 + 12 and 13 + 27; 4 + 2 and 7 + 5) | 20 / 40 = 50.0% | 6 / 12 = 50.0% |
Treatment does better than control among men (61.5% against 57.1%) and among women (44.4% against 40.0%), yet the pooled rates are identical at 50.0%. The reason is composition: men recover more often than women in both arms, and men make up a smaller share of the treatment arm (13 of 40) than of the control arm (7 of 12). Pooling mixes the sex effect into the treatment comparison.
The same source discusses the UC Berkeley graduate admissions case: men were admitted more often overall, yet the authors were unable to detect a bias towards men in any individual department, because women applied more to departments with lower acceptance rates.
Interpretation: which number to trust
Neither the pooled nor the stratified figure is automatically correct. The choice depends on the causal story, which the data alone cannot supply. The table gives reading guidance for this example.
| What you see | What to ask |
|---|---|
| Pooled rates equal, subgroup rates differ | Is group membership (here, sex) unevenly spread across the arms? 13 of 40 versus 7 of 12 are men. |
| Direction changes between pooled and subgroup results | Is the subgrouping variable a confounder you should adjust for, or a consequence of the treatment you should not? |
| Subgroup counts are very small (for example 2 / 5) | A single patient moves the rate by 20 percentage points, so subgroup rates are unstable. |
For the base-rate fallacy the illustration is hypothetical, with assumed inputs chosen for easy arithmetic and not taken from any real test. Suppose a condition affects 1% of a group, and a test detects 90% of people who have it and correctly clears 90% of people who do not. Out of 1,000 people, 10 have the condition and 990 do not. The test flags 0.90 × 10 = 9 true cases and 0.10 × 990 = 99 healthy people. Of the 9 + 99 = 108 positive results, only 9 are real: 9 / 108 = 8.3%. A “90% accurate” test can still be wrong about most of its positives when the condition is rare.
P-value fallacies and the ASA statement
In March 2016 the American Statistical Association issued a statement on p-values (Wasserstein and Lazar, The American Statistician, doi 10.1080/00031305.2016.1154108), announced in an ASA press release dated 7 March 2016. Three of its six principles map directly onto common fallacies:
- Principle 2: “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.”
- Principle 3: “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.”
- Principle 5: “A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.”
The release also names p-hacking and data dredging, and the file-drawer effect, in which significant results are more likely to be published. Wasserstein is quoted in the release as saying: “The p-value was never intended to be a substitute for scientific reasoning.” See what is a p-value for the definition and the correct reading.
Assumptions and common mistakes
- Treating the pooled number as final. Check whether the pooled comparison hides unequal subgroup composition, as in the 13 / 40 versus 7 / 12 split above.
- Ignoring prevalence. A positive result means little until you know how many people tested actually have the condition; the 8.3% result above came from nothing more than a 1% base rate.
- Moving from groups to individuals. A relationship between regional averages does not tell you about the people inside each region. The unit of analysis has to match the unit the conclusion is about; see unit of analysis and the ecological fallacy.
- Expecting streaks to reverse. After five heads in a row, a fair coin still lands heads with probability 0.5 on the next flip; the chance of five heads in a row was 0.55 = 0.03125 before any flip happened, and it plays no part in the sixth. Averages settle through accumulation of trials, not correction of earlier ones: see the law of large numbers.
- Reading association as cause. Two variables that move together may share a third driver; see spurious correlation.
How to report it
Report the counts as well as the percentages, and report the pooled and the stratified results side by side so a reader can see any reversal. An example sentence for the data above: “Recovery was 20 of 40 (50.0%) with treatment and 6 of 12 (50.0%) with control. Within sex, recovery was 8 of 13 (61.5%) versus 4 of 7 (57.1%) among men and 12 of 27 (44.4%) versus 2 of 5 (40.0%) among women.” For a test result, report prevalence, sensitivity, specificity and the proportion of positives that are true positives, not accuracy alone. ASA principle 4 states that proper inference requires full reporting and transparency.
How to calculate it in Excel, R or Python
Each snippet reproduces the numbers above.
Excel. With recoveries in B2 and patients in C2 for a group, the rate is =B2/C2. For the base-rate example, with prevalence in A2 (0.01), sensitivity in B2 (0.9), specificity in C2 (0.9): =(A2*B2)/(A2*B2+(1-A2)*(1-C2)) returns 0.0833.
R.
x = c(men_t = 8, men_c = 4, women_t = 12, women_c = 2)
n = c(men_t = 13, men_c = 7, women_t = 27, women_c = 5)
round(100 * x / n, 1) # 61.5 57.1 44.4 40.0
c(sum(x[c(1,3)]) / sum(n[c(1,3)]), sum(x[c(2,4)]) / sum(n[c(2,4)])) # 0.5 0.5
prev = 0.01; sens = 0.9; spec = 0.9
prev * sens / (prev * sens + (1 - prev) * (1 - spec)) # 0.0833Python.
x = {"men_t": 8, "men_c": 4, "women_t": 12, "women_c": 2}
n = {"men_t": 13, "men_c": 7, "women_t": 27, "women_c": 5}
print({k: round(100 * x[k] / n[k], 1) for k in x})
print((x["men_t"] + x["women_t"]) / (n["men_t"] + n["women_t"]),
(x["men_c"] + x["women_c"]) / (n["men_c"] + n["women_c"])) # 0.5 0.5
prev, sens, spec = 0.01, 0.9, 0.9
print(prev * sens / (prev * sens + (1 - prev) * (1 - spec))) # 0.0833...FAQ
What are the most common statistical fallacies?
The ones in this guide: Simpson’s paradox, base-rate neglect, p-value misreadings, the ecological fallacy, mistaking correlation for causation, and the gambler’s fallacy.
What is the difference between a statistical fallacy and a statistical error?
An error is a mistake in computation or procedure. A fallacy is a flaw in the inference drawn from a result that may be computed correctly, such as a pooled rate that conceals a reversal in every subgroup.
Does a small p-value mean the hypothesis is probably true?
No. ASA principle 2 states that p-values do not measure the probability that the studied hypothesis is true.
Can a result be statistically significant and unimportant?
Yes. ASA principle 5 states that a p-value or statistical significance does not measure the size of an effect or the importance of a result.
Sources
- Stanford Encyclopedia of Philosophy, “Simpson’s Paradox,” https://plato.stanford.edu/entries/paradox-simpson/ (checked 2026-10-29).
- Stanford Encyclopedia of Philosophy, “Fallacies” (first published 2015, substantive revision 2024), https://plato.stanford.edu/entries/fallacies/ (checked 2026-10-29).
- American Statistical Association, press release “ASA releases statement on statistical significance and p-values,” 7 March 2016, https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf (checked 2026-10-29).
- Wasserstein, R. L., and Lazar, N. A. (2016), “The ASA Statement on p-Values: Context, Process, and Purpose,” The American Statistician, https://doi.org/10.1080/00031305.2016.1154108. Cited here through the ASA release above.
