Bonferroni Correction vs FDR: When to Use Each

A p-value below 0.05 from one test does not mean the same threshold is safe once you have run ten, fifty, or a thousand tests looking for the same kind of effect. Run 20 independent tests at α = 0.05 on data with no real effect at all, and the chance of at least one “significant” result by chance alone is not 5% — it is roughly 64%. That inflation is the multiple comparisons problem, and how you handle it depends on what kind of research question you are asking, not on which correction happens to be built into your software’s default output.

This page covers the two corrections researchers reach for most often — the Bonferroni correction and the Benjamini-Hochberg false discovery rate (FDR) procedure — what each one actually controls, how to apply them by hand, and the decision rule that determines which one belongs in your analysis.

The multiple comparisons problem

Every individual hypothesis test carries a fixed false-positive rate, conventionally α = 0.05. That is the probability of rejecting a true null hypothesis on that one test. When you run several tests on the same family of hypotheses — multiple outcome measures, multiple subgroups, multiple pairwise group comparisons, multiple candidate biomarkers — the probability that at least one test crosses the 0.05 threshold purely by chance climbs with every additional test.

For m independent tests each run at α = 0.05, the probability of at least one false positive (the family-wise error rate, or FWER, under the null) is 1 − (1 − 0.05)^m. At m = 5 that is about 23%; at m = 20 it is about 64%; at m = 100 it is effectively certain. A correction exists to bring that inflated error rate back under control — the question is which kind of control you actually need.

Bonferroni correction: the adjusted-alpha table

The Bonferroni correction, based on the Bonferroni inequalities named for Italian mathematician Carlo Emilio Bonferroni and popularised for multiple-comparisons testing by Olive Jean Dunn’s 1961 paper, is the simplest way to control the FWER: divide your target α by the number of comparisons, m, and require each individual test to clear that stricter threshold.

Adjusted α = α / m, for a family-wise target of α = 0.05:

Number of comparisons (m)	Bonferroni-adjusted α
2	0.0250
3	0.0167
5	0.0100
10	0.0050
20	0.0025
50	0.0010
100	0.0005

Equivalently, you can leave α alone and multiply each raw p-value by m (capping at 1.0) — the “Bonferroni-corrected p-value.” Both approaches produce the same reject/fail-to-reject decisions.

The correction’s main property is also its main limitation: it controls the FWER regardless of any dependency structure between the tests, which makes it valid in essentially any situation, but that generality comes at the cost of statistical power. As m grows, the adjusted threshold shrinks fast, and real effects get buried along with the false positives.

Worked example: interpreting a multiple-testing output

Illustrative worked example — the numbers below are constructed for this walkthrough, not drawn from a real study. Suppose a researcher tests the association between one exposure and ten candidate biomarkers in an exploratory dataset, producing ten raw p-values:

Biomarker	Rank	Raw p-value	BH critical value (i/m × Q, Q=0.05)	Bonferroni (α=0.005)	BH-FDR
A	1	0.001	0.005	Survives	Survives
B	2	0.004	0.010	Survives	Survives
C	3	0.008	0.015	Fails	Survives
D	4	0.012	0.020	Fails	Survives
E	5	0.020	0.025	Fails	Survives
F	6	0.031	0.030	Fails	Fails
G	7	0.045	0.035	Fails	Fails
H	8	0.060	0.040	Fails	Fails
I	9	0.210	0.045	Fails	Fails
J	10	0.480	0.050	Fails	Fails

The Bonferroni-adjusted threshold for m = 10 is α = 0.005: only biomarkers A and B clear it.

The Benjamini-Hochberg procedure (worked through below) finds the largest rank i where the raw p-value is still below its own critical value (i/m) × Q, then treats every test at or above that rank as a discovery. Here, rank 5 (p = 0.020) clears its critical value of 0.025, and rank 6 (p = 0.031) does not clear 0.030 — so biomarkers A through E are all flagged as discoveries.

The sentence a researcher would write up: “After Benjamini-Hochberg correction for the false discovery rate (Q = 0.05), five of the ten candidate biomarkers (A–E) remained significantly associated with the exposure; only the association with biomarker A also survived the more conservative Bonferroni correction for family-wise error (adjusted α = 0.005).” Note what the write-up does not say: it does not claim any of the five are “5% likely to be false individually” — the FDR is a property of the whole rejected set, not of each p-value in isolation.

Family-wise error rate vs. false discovery rate

These two error rates answer different questions, and confusing them is the most common misreading in this territory:

Family-wise error rate (FWER) — the probability of making at least one false-positive rejection anywhere in the family of tests. Bonferroni (and the related Holm-Bonferroni and Šidák procedures) controls this.
False discovery rate (FDR) — the expected proportion of false positives among all the tests you reject. The Benjamini-Hochberg procedure controls this. A 5% FDR does not mean any single result has a 5% chance of being wrong; it means that, on average, no more than 5% of everything you called “significant” is expected to be a false positive.

Controlling FWER is a stricter, more conservative standard than controlling FDR at the same nominal level — that is precisely why Bonferroni loses power faster as m grows.

The Benjamini-Hochberg FDR procedure, step by step

Choose a target FDR, Q (commonly 0.05 or 0.10 — state and justify the choice, the same way you would state α).
Sort the m raw p-values from smallest to largest, giving each a rank i (1 = smallest).
Calculate the critical value for each rank: (i / m) × Q.
Find the largest rank i where the raw p-value is still less than or equal to its critical value.
Reject the null hypothesis for that test and every test with a smaller p-value (i.e., every test ranked at or below it).

Software (R’s p.adjust(method = "BH"), Python’s statsmodels.stats.multitest.multipletests(method='fdr_bh'), and most stats packages) does this arithmetic for you and returns “BH-adjusted p-values” or “q-values” directly comparable to Q — but knowing the mechanics is what lets you sanity-check the output rather than trust it blindly.

When to correct with Bonferroni, and when FDR is the right tool

This is a decision about the cost of a false positive relative to the cost of a false negative in your specific study, not a rule about which procedure is “more rigorous” in the abstract:

Use Bonferroni (or Holm-Bonferroni) for a small number of pre-specified, confirmatory comparisons where a single false positive would be a serious problem — primary and key secondary endpoints in a confirmatory clinical trial, a small set of planned pairwise comparisons following a significant ANOVA, or any setting where regulators or reviewers expect strict FWER control.
Use Benjamini-Hochberg FDR for exploratory research with many simultaneous tests, where the goal is to generate a manageable, mostly-correct shortlist for follow-up rather than to certify each individual finding — genome-wide association studies, screening panels of dozens or hundreds of biomarkers, exploratory subgroup analyses, or any setting where Bonferroni’s power loss would mean missing most real effects.

Holm-Bonferroni is worth knowing as a middle ground: it is a step-down procedure that still guarantees FWER control (unlike FDR methods) but is uniformly more powerful than plain Bonferroni, because it tests the smallest p-value against the full α/m threshold and progressively relaxes the threshold for larger p-values. If you need strict FWER control and have more than a handful of comparisons, Holm-Bonferroni dominates plain Bonferroni with no downside.

Assumptions and when not to apply these corrections

Define the family before you see the results. Both corrections assume a pre-specified family of tests. Running a correction only over the tests that already looked promising, after seeing all the results, defeats the purpose and is a form of p-hacking.
Bonferroni is valid under any dependency structure between tests — positively correlated, negatively correlated, or independent — which is why it is trusted in high-stakes confirmatory settings. The standard Benjamini-Hochberg procedure assumes independence or positive regression dependence among the tests; under arbitrary or negative dependence, the more conservative Benjamini-Yekutieli variant is the correct choice.
A correction does not rescue an underpowered study. Adding more comparisons without adding more data raises the evidentiary bar every individual test must clear. A large, all-null result after correction can mean there is genuinely nothing there, or it can mean the study was never powered to detect the effect once the multiplicity penalty was applied — distinguishing the two requires a power calculation, not just a look at the corrected p-values.
At very large m (thousands of tests, as in genome-wide or omics screens), plain Bonferroni is frequently so conservative it has near-zero power to detect real effects, which is why FDR control and permutation-based approaches dominate that literature rather than being a stylistic preference.

Frequently asked questions

What is the multiple comparisons problem?

It is the inflation of the overall false-positive rate that occurs when multiple statistical tests are run on the same family of hypotheses. Each individual test keeps its own nominal α, but the probability that at least one test is significant by chance alone grows with the number of tests, unless a correction is applied.

How do I calculate a Bonferroni-corrected p-value?

Multiply each raw p-value by the number of comparisons, m (capping the result at 1.0), and compare it to your original α. Equivalently, divide α by m and compare each raw p-value to that stricter threshold — both give identical accept/reject decisions.

Is the Bonferroni correction too conservative?

For a large number of comparisons, often yes — that is the trade-off for guaranteeing family-wise error control under any dependency structure. Whether that trade-off is appropriate depends on the cost of a false positive in your specific study; it is not a universal flaw, and Holm-Bonferroni recovers some of the lost power without giving up FWER control.

What is the difference between Bonferroni and Benjamini-Hochberg?

Bonferroni controls the family-wise error rate: the probability of any false positive at all. Benjamini-Hochberg controls the false discovery rate: the expected proportion of false positives among the results you call significant. FDR control is less conservative and better suited to exploratory research with many simultaneous tests.

Do I need to correct for multiple comparisons in exploratory research?

Generally yes, but the right tool is usually FDR control rather than Bonferroni. Uncorrected multiple testing in an exploratory setting with many comparisons will reliably produce false “discoveries”; the FDR framework lets you control the expected error rate in that shortlist without discarding as much true signal as a strict FWER correction would.

What is Holm-Bonferroni?

A step-down modification of the Bonferroni correction that still guarantees family-wise error control but is more powerful. It sorts p-values from smallest to largest, tests the smallest against α/m, and progressively relaxes the threshold (α/(m−1), α/(m−2), and so on) for each successive comparison, stopping at the first test that fails to reject.

Bonferroni Correction and Multiple Comparisons: When to Correct and When FDR Is Better

Ask about Bonferroni Correction and Multiple Comparisons: When to Correct and When FDR Is Better