Written and maintained by CASRAI Editorial Board
Published · Last updated
On this page
The Bonferroni correction controls the chance of any false positive across a family of m tests by dividing the significance level by m (or multiplying each p-value by m). Holm and Šidák are slightly more powerful ways to control the same error rate; Benjamini-Hochberg controls the false discovery rate instead and suits exploratory screens.
A p-value below 0.05 from one test does not mean the same threshold is safe once you have run ten, fifty, or a thousand tests looking for the same kind of effect. Run 20 independent tests at α = 0.05 on data with no real effect at all, and the chance of at least one “significant” result by chance alone is not 5% — it is roughly 64%. That inflation is the multiple comparisons problem, and how you handle it depends on what kind of research question you are asking, not on which correction happens to be built into your software’s default output.
This page covers the two corrections researchers reach for most often — the Bonferroni correction and the Benjamini-Hochberg false discovery rate (FDR) procedure — what each one actually controls, how to apply them by hand, and the decision rule that determines which one belongs in your analysis.
The multiple comparisons problem
Every individual hypothesis test carries a fixed false-positive rate, conventionally α = 0.05. That is the probability of rejecting a true null hypothesis on that one test. When you run several tests on the same family of hypotheses — multiple outcome measures, multiple subgroups, multiple pairwise group comparisons, multiple candidate biomarkers — the probability that at least one test crosses the 0.05 threshold purely by chance climbs with every additional test.
For m independent tests each run at α = 0.05, the probability of at least one false positive (the family-wise error rate, or FWER, under the null) is 1 − (1 − 0.05)m. At m = 5 that is about 23%; at m = 20 it is about 64%; at m = 100 it is effectively certain. A correction exists to bring that inflated error rate back under control — the question is which kind of control you actually need.
Bonferroni correction: the adjusted-alpha table
The Bonferroni correction, based on the Bonferroni inequalities named for Italian mathematician Carlo Emilio Bonferroni and popularised for multiple-comparisons testing by Olive Jean Dunn’s 1961 paper, is the simplest way to control the FWER: divide your target α by the number of comparisons, m, and require each individual test to clear that stricter threshold.
Adjusted α = α / m, for a family-wise target of α = 0.05:
| Number of comparisons (m) | Bonferroni-adjusted α |
|---|---|
| 2 | 0.0250 |
| 3 | 0.0167 |
| 5 | 0.0100 |
| 10 | 0.0050 |
| 20 | 0.0025 |
| 50 | 0.0010 |
| 100 | 0.0005 |
Equivalently, you can leave α alone and multiply each raw p-value by m (capping at 1.0) — the “Bonferroni-corrected p-value.” Both approaches produce the same reject/fail-to-reject decisions.
The correction’s main property is also its main limitation: it controls the FWER regardless of any dependency structure between the tests, which makes it valid in essentially any situation, but that generality comes at the cost of statistical power. As m grows, the adjusted threshold shrinks fast, and real effects get buried along with the false positives.
Šidák correction and how fast uncorrected error grows
The Šidák correction is the exact companion of Bonferroni for independent tests. Instead of α/m, it uses the per-test threshold 1 − (1 − α)1/m, which is the value that makes the family-wise error rate exactly α when the m tests are independent. The statsmodels documentation lists both sidak and holm-sidak as methods, and the library returns the two per-test thresholds side by side as alphacSidak and alphacBonf. Šidák is never less powerful than Bonferroni, but its derivation assumes independence (or positive dependence), whereas Bonferroni makes no dependence assumption, and the difference is tiny in practice, as the table shows.
| Tests (m) | Uncorrected family-wise error, 1 − 0.95m | Bonferroni per-test α | Šidák per-test α |
|---|---|---|---|
| 1 | 0.0500 | 0.050000 | 0.050000 |
| 2 | 0.0975 | 0.025000 | 0.025321 |
| 3 | 0.1426 | 0.016667 | 0.016952 |
| 5 | 0.2262 | 0.010000 | 0.010206 |
| 10 | 0.4013 | 0.005000 | 0.005116 |
| 20 | 0.6415 | 0.002500 | 0.002561 |
| 50 | 0.9231 | 0.001000 | 0.001025 |
| 100 | 0.9941 | 0.000500 | 0.000513 |
| 1,000 | 1.0000 | 0.000050 | 0.000051 |
| 20,000 | 1.0000 | 0.0000025 | 0.0000026 |
All values are computed from the formulas above at α = 0.05 for independent tests. The Šidák threshold is always a little larger than the Bonferroni one, which is why the gain is described as marginal: at m = 20 the two thresholds differ by about 0.0001. Dependence between tests changes the true uncorrected error rate, so the first column is exact only for independent tests.
Worked example: interpreting a multiple-testing output
Illustrative worked example — the numbers below are constructed for this walkthrough, not drawn from a real study. Suppose a researcher tests the association between one exposure and ten candidate biomarkers in an exploratory dataset, producing ten raw p-values:
| Biomarker | Rank | Raw p-value | BH critical value (i/m × Q, Q=0.05) | Bonferroni (α=0.005) | BH-FDR |
|---|---|---|---|---|---|
| A | 1 | 0.001 | 0.005 | Survives | Survives |
| B | 2 | 0.004 | 0.010 | Survives | Survives |
| C | 3 | 0.008 | 0.015 | Fails | Survives |
| D | 4 | 0.012 | 0.020 | Fails | Survives |
| E | 5 | 0.020 | 0.025 | Fails | Survives |
| F | 6 | 0.031 | 0.030 | Fails | Fails |
| G | 7 | 0.045 | 0.035 | Fails | Fails |
| H | 8 | 0.060 | 0.040 | Fails | Fails |
| I | 9 | 0.210 | 0.045 | Fails | Fails |
| J | 10 | 0.480 | 0.050 | Fails | Fails |
The Bonferroni-adjusted threshold for m = 10 is α = 0.005: only biomarkers A and B clear it.
The Benjamini-Hochberg procedure (worked through below) finds the largest rank i where the raw p-value is still below its own critical value (i/m) × Q, then treats every test at or above that rank as a discovery. Here, rank 5 (p = 0.020) clears its critical value of 0.025, and rank 6 (p = 0.031) does not clear 0.030 — so biomarkers A through E are all flagged as discoveries.
The sentence a researcher would write up: “After Benjamini-Hochberg correction for the false discovery rate (Q = 0.05), five of the ten candidate biomarkers (A–E) remained significantly associated with the exposure; only the association with biomarker A also survived the more conservative Bonferroni correction for family-wise error (adjusted α = 0.005).” Note what the write-up does not say: it does not claim any of the five are “5% likely to be false individually” — the FDR is a property of the whole rejected set, not of each p-value in isolation.
Family-wise error rate vs. false discovery rate
These two error rates answer different questions, and confusing them is the most common misreading in this territory:
- Family-wise error rate (FWER) — the probability of making at least one false-positive rejection anywhere in the family of tests. Bonferroni (and the related Holm-Bonferroni and Šidák procedures) controls this.
- False discovery rate (FDR) — the expected proportion of false positives among all the tests you reject. The Benjamini-Hochberg procedure controls this. A 5% FDR does not mean any single result has a 5% chance of being wrong; it means that, on average, no more than 5% of everything you called “significant” is expected to be a false positive.
Controlling FWER is a stricter, more conservative standard than controlling FDR at the same nominal level — that is precisely why Bonferroni loses power faster as m grows.
The Benjamini-Hochberg FDR procedure, step by step
- Choose a target FDR, Q (commonly 0.05 or 0.10 — state and justify the choice, the same way you would state α).
- Sort the m raw p-values from smallest to largest, giving each a rank i (1 = smallest).
- Calculate the critical value for each rank:
(i / m) × Q. - Find the largest rank i where the raw p-value is still less than or equal to its critical value.
- Reject the null hypothesis for that test and every test with a smaller p-value (i.e., every test ranked at or below it).
Software (R’s p.adjust(method = "BH"), Python’s statsmodels.stats.multitest.multipletests(method='fdr_bh'), and most stats packages) does this arithmetic for you and returns “BH-adjusted p-values” or “q-values” directly comparable to Q — but knowing the mechanics is what lets you sanity-check the output rather than trust it blindly.
Holm-Bonferroni worked on the same ten p-values
Holm’s step-down procedure (Holm 1979, Scandinavian Journal of Statistics) sorts the ten p-values from the example above and compares the k-th smallest with α/(m − k + 1), stopping at the first failure. With α = 0.05 and m = 10 the thresholds are 0.0050, 0.00556, 0.00625, 0.00714, 0.00833 and so on.
- Biomarker A: 0.001 < 0.0050, reject.
- Biomarker B: 0.004 < 0.00556, reject.
- Biomarker C: 0.008 > 0.00625, fail, so stop. Every larger p-value is retained.
Holm therefore rejects A and B, the same two as plain Bonferroni, because no p-value sits in the narrow band where the relaxed threshold would matter. Holm can only reject more hypotheses than Bonferroni, never fewer; the R documentation for p.adjust states that Holm’s method dominates the unmodified Bonferroni correction and that both are valid under arbitrary assumptions about dependence, so there is no reason to prefer plain Bonferroni when software offers Holm. The adjusted p-values for the same data make the relationship visible:
| Biomarker | Raw p | Bonferroni adjusted | Holm adjusted | BH adjusted |
|---|---|---|---|---|
| A | 0.001 | 0.010 | 0.010 | 0.010 |
| B | 0.004 | 0.040 | 0.036 | 0.020 |
| C | 0.008 | 0.080 | 0.064 | 0.027 |
| D | 0.012 | 0.120 | 0.084 | 0.030 |
| E | 0.020 | 0.200 | 0.120 | 0.040 |
| F | 0.031 | 0.310 | 0.155 | 0.052 |
| G | 0.045 | 0.450 | 0.180 | 0.064 |
| H | 0.060 | 0.600 | 0.180 | 0.075 |
| I | 0.210 | 1.000 | 0.420 | 0.233 |
| J | 0.480 | 1.000 | 0.480 | 0.480 |
These are the figures R’s p.adjust and statsmodels’ multipletests return for the constructed example (computed here by hand from the definitions: Holm multiplies the k-th smallest p-value by m − k + 1 and enforces a non-decreasing sequence; BH multiplies by m/i and takes a running minimum from the largest rank down). Reading a column against α = 0.05: Bonferroni and Holm keep A and B; BH keeps A to E, since biomarker F’s BH-adjusted value of 0.052 is just above 0.05.
When to correct with Bonferroni, and when FDR is the right tool
This is a decision about the cost of a false positive relative to the cost of a false negative in your specific study, not a rule about which procedure is “more rigorous” in the abstract:
- Use Bonferroni (or Holm-Bonferroni) for a small number of pre-specified, confirmatory comparisons where a single false positive would be a serious problem — primary and key secondary endpoints in a confirmatory clinical trial, a small set of planned pairwise comparisons following a significant ANOVA, or any setting where regulators or reviewers expect strict FWER control.
- Use Benjamini-Hochberg FDR for exploratory research with many simultaneous tests, where the goal is to generate a manageable, mostly-correct shortlist for follow-up rather than to certify each individual finding — genome-wide association studies, screening panels of dozens or hundreds of biomarkers, exploratory subgroup analyses, or any setting where Bonferroni’s power loss would mean missing most real effects.
Holm-Bonferroni is worth knowing as a middle ground: it is a step-down procedure that still guarantees FWER control (unlike FDR methods) but is uniformly more powerful than plain Bonferroni, because it tests the smallest p-value against the full α/m threshold and progressively relaxes the threshold for larger p-values. If you need strict FWER control and have more than a handful of comparisons, Holm-Bonferroni dominates plain Bonferroni with no downside.
Where FDR control is used in practice: GWAS, RNA-seq, and fMRI
The decision rule above plays out differently across the three fields that run the most simultaneous hypothesis tests in research:
- Genome-wide association studies (GWAS). Primary discovery in a GWAS uses a Bonferroni-style threshold, not FDR — the conventional genome-wide significance cutoff of p < 5 × 10-8 is itself a Bonferroni correction sized to roughly one million independent common SNPs (see CASRAI’s genome-wide significance threshold guide for the full derivation). FDR control shows up downstream of that primary scan instead — in candidate-gene follow-up, gene-set enrichment testing, and expression quantitative trait loci (eQTL) studies, where the number of gene-variant pairs tested is large enough that a genome-wide Bonferroni threshold would bury almost all real signal, and the goal is a manageable, mostly-correct shortlist rather than a single certified hit.
- RNA-seq differential expression. This is the field where BH-FDR is the default, not an alternative. Standard differential-expression tools — DESeq2 and edgeR, both Bioconductor packages — report a per-gene Benjamini-Hochberg adjusted p-value (DESeq2 labels the column
padj) alongside the raw p-value, and papers conventionally filter on that adjusted value at a 5% or 10% FDR rather than on the raw p-value at all. With 10,000–20,000 genes tested per comparison, a Bonferroni threshold would be far too conservative to recover a usable gene list. See CASRAI’s RNA-seq experimental design and analysis guide for the pipeline this correction sits inside. - fMRI voxel-wise statistics. A whole-brain fMRI analysis tests a hypothesis at every voxel — commonly tens of thousands per scan — which is severe enough that Genovese, Lazar, and Nichols’ 2002 paper introducing FDR thresholding for functional neuroimaging (NeuroImage 15(4), 870–878, DOI 10.1006/nimg.2001.1037) became a standard reference in the field almost immediately. Neuroimaging analysis packages (SPM, FSL, AFNI) offer voxel-wise FDR alongside two competing correction families — random field theory (RFT)-based cluster correction and permutation-based cluster correction (e.g., FSL’s
randomise, AFNI’s3dClustSim) — and the three make different assumptions about the spatial smoothness of the data, so they are not interchangeable defaults.
Across all three, the q-value is the natural companion statistic to BH-FDR — it reports, per test, the smallest FDR at which that specific result would be called significant. See CASRAI’s q-values and the positive false discovery rate guide for how q-values relate to (and differ from) the BH-adjusted p-value covered above.
Choosing a method in R and Python
The method names differ between the two main environments. In R, p.adjust accepts holm, hochberg, hommel, bonferroni, BH (with the alias fdr) and BY. In Python, statsmodels.stats.multitest.multipletests accepts bonferroni, sidak, holm, holm-sidak, simes-hochberg, hommel, fdr_bh and fdr_by, plus a two-stage FDR variant. The data table below this page’s text sets the names, the error rate each targets and the dependence note each package documents side by side.
Two points are worth knowing. First, the R documentation describes BY as valid under the most general form of dependence, which is the conservative choice when tests may be negatively associated, while the statsmodels documentation notes fdr_bh for independent or non-negatively correlated tests and fdr_by for tests that may be negatively correlated. Second, for every method except the two-stage FDR variant, the adjusted p-values do not depend on the α you later compare them with, so you can report adjusted p-values and let readers apply their own threshold. For a task-oriented walkthrough, see Bonferroni correction in R.
Bonferroni after ANOVA, and how many tests belong in the family
A significant omnibus ANOVA does not by itself fix the multiplicity of follow-up pairwise tests. If you run all pairwise comparisons among k groups, the family has k(k − 1)/2 members (six for four groups, ten for five), and Bonferroni divides α by that number. A small set of contrasts planned before seeing the data carries a smaller family and therefore a smaller penalty. When all pairs are of interest, procedures built for that case such as Tukey’s HSD are often more powerful; see Bonferroni vs Tukey post-hoc correction and ANOVA (analysis of variance).
Deciding what counts as one family is a judgement made before analysis: tests addressing the same scientific claim, outcomes assessed on the same participants for the same decision, or subgroup analyses feeding one conclusion generally belong together. Silently dropping tests from the family, or splitting one family into several small ones so each looks small, undoes the control. This is the same discipline that separates a planned analysis from p-hacking.
Bonferroni-corrected confidence intervals
The Bonferroni idea also applies to interval estimates. To report m confidence intervals that hold simultaneously with overall confidence 1 − α, compute each interval at confidence level 1 − α/m, a construction associated with Dunn’s 1961 paper in the Journal of the American Statistical Association (56(293): 52–64). For m = 5 and α = 0.05, each interval is a 99% interval. The intervals are wider than ordinary 95% intervals, and that extra width is the visible price of the simultaneous guarantee. See confidence intervals explained for the basics.
Regulatory practice in confirmatory trials
For confirmatory clinical trials, multiplicity adjustment is an expectation rather than a choice. The US Food and Drug Administration’s guidance for industry Multiple Endpoints in Clinical Trials (final, October 2022) explains that assessing more endpoints raises the chance of a false conclusion that a drug is effective unless the multiplicity is adjusted, and it describes strategies for grouping and ordering endpoints so the overall error rate is controlled. A flat Bonferroni split across every endpoint is the simplest way to apply such control; testing endpoints in a pre-specified priority order is a common alternative that can preserve more power when the endpoints have a natural ranking. A statistical analysis plan should name the strategy before unblinding. See statistical power and sample size for how the number of tested hypotheses feeds into the sample you need.
Criticisms and limits of Bonferroni
Bonferroni’s critics make three consistent points. Moran (2003, Oikos 100(2): 403–405) argued against routine use of the sequential Bonferroni procedure in ecological studies, and Nakagawa (2004, Behavioral Ecology 15(6): 1044–1045) argued that the cost in statistical power and the resulting bias toward publishing only large effects outweigh the benefit in many fields. Both arguments rest on the same property: the correction is conservative when tests are positively correlated, because it treats correlated tests as if they were separate chances for error. The usual responses are to use Holm, to pre-specify a smaller family, to move to FDR control for screens, or to report effect sizes and confidence intervals so readers can judge magnitude rather than only a threshold crossing (see effect size).
How to report a multiple-comparison correction
- State the family: which tests were corrected together and why.
- Name the method and its software implementation, for example Holm via
p.adjustin R. - Give m, and the target level (α for FWER methods, Q for FDR methods).
- Report raw and adjusted p-values together, and say which one each statement refers to.
- For FDR results, describe the shortlist as a set with an expected false-discovery proportion, not as individually certified findings.
- Include effect sizes with intervals; see how to report p-values.
Assumptions and when not to apply these corrections
- Define the family before you see the results. Both corrections assume a pre-specified family of tests. Running a correction only over the tests that already looked promising, after seeing all the results, defeats the purpose and is a form of p-hacking.
- Bonferroni is valid under any dependency structure between tests — positively correlated, negatively correlated, or independent — which is why it is trusted in high-stakes confirmatory settings. The standard Benjamini-Hochberg procedure assumes independence or positive regression dependence among the tests; under arbitrary or negative dependence, the more conservative Benjamini-Yekutieli variant is the correct choice.
- A correction does not rescue an underpowered study. Adding more comparisons without adding more data raises the evidentiary bar every individual test must clear. A large, all-null result after correction can mean there is genuinely nothing there, or it can mean the study was never powered to detect the effect once the multiplicity penalty was applied — distinguishing the two requires a power calculation, not just a look at the corrected p-values.
- At very large m (thousands of tests, as in genome-wide or omics screens), plain Bonferroni is frequently so conservative it has near-zero power to detect real effects, which is why FDR control and permutation-based approaches dominate that literature rather than being a stylistic preference.
Frequently asked questions
What is the multiple comparisons problem?
It is the inflation of the overall false-positive rate that occurs when multiple statistical tests are run on the same family of hypotheses. Each individual test keeps its own nominal α, but the probability that at least one test is significant by chance alone grows with the number of tests, unless a correction is applied.
How do I calculate a Bonferroni-corrected p-value?
Multiply each raw p-value by the number of comparisons, m (capping the result at 1.0), and compare it to your original α. Equivalently, divide α by m and compare each raw p-value to that stricter threshold — both give identical accept/reject decisions.
Is the Bonferroni correction too conservative?
For a large number of comparisons, often yes — that is the trade-off for guaranteeing family-wise error control under any dependency structure. Whether that trade-off is appropriate depends on the cost of a false positive in your specific study; it is not a universal flaw, and Holm-Bonferroni recovers some of the lost power without giving up FWER control.
What is the difference between Bonferroni and Benjamini-Hochberg?
Bonferroni controls the family-wise error rate: the probability of any false positive at all. Benjamini-Hochberg controls the false discovery rate: the expected proportion of false positives among the results you call significant. FDR control is less conservative and better suited to exploratory research with many simultaneous tests.
Do I need to correct for multiple comparisons in exploratory research?
Generally yes, but the right tool is usually FDR control rather than Bonferroni. Uncorrected multiple testing in an exploratory setting with many comparisons will reliably produce false “discoveries”; the FDR framework lets you control the expected error rate in that shortlist without discarding as much true signal as a strict FWER correction would.
What is Holm-Bonferroni?
A step-down modification of the Bonferroni correction that still guarantees family-wise error control but is more powerful. It sorts p-values from smallest to largest, tests the smallest against α/m, and progressively relaxes the threshold (α/(m−1), α/(m−2), and so on) for each successive comparison, stopping at the first test that fails to reject.
What is the false discovery rate (FDR)?
The false discovery rate is the expected proportion of false positives among all the results a study calls “significant.” It is controlled by the Benjamini-Hochberg procedure (see the step-by-step method above), and it is a property of the whole rejected set of tests, not a per-result probability the way a p-value is often (mis)read.
Is FDR control used in GWAS, RNA-seq, or fMRI studies?
Differently in each. RNA-seq differential expression uses BH-FDR as the default correction (DESeq2 and edgeR both report a padj column). fMRI voxel-wise analysis uses FDR alongside cluster-based alternatives (random field theory, permutation testing). GWAS primary discovery uses a Bonferroni-style genome-wide significance threshold instead, with FDR control appearing downstream in follow-up and eQTL analysis. See the section above for the full breakdown.
What is the difference between Bonferroni and Šidák?
Bonferroni uses α/m; Šidák uses 1 − (1 − α)1/m, which is slightly larger and exact for independent tests. The gain is small (at m = 20 and α = 0.05, 0.002561 versus 0.0025), and Bonferroni needs no independence assumption.
Is Holm better than Bonferroni?
For controlling the family-wise error rate, yes: Holm is valid under the same arbitrary dependence and rejects at least everything Bonferroni rejects. The R documentation for p.adjust says Holm dominates the unmodified Bonferroni correction.
When should I not use Bonferroni?
Avoid flat Bonferroni when m is in the hundreds or thousands (genome-wide or omics screens), when tests are strongly positively correlated, or when a pre-specified testing sequence is available and pre-specified. In those cases Holm, FDR control or a structured testing sequence keeps more power.
Can I use Benjamini-Hochberg when my tests are correlated?
BH is justified for independent or positively dependent tests. For arbitrary dependence the Benjamini-Yekutieli variant (BY in R, fdr_by in statsmodels) is the conservative choice.
How many comparisons count in the family?
Every test addressing the same scientific claim that you ran or would have reported, fixed before the analysis. For all pairwise comparisons among k groups, it is k(k − 1)/2.
How do I report corrected p-values?
Name the method, the family and m, and give raw and adjusted p-values together, as set out in the reporting checklist above.
Related reading
- Q-values and the positive false discovery rate
- Genome-wide significance thresholds in GWAS
- RNA-seq: experimental design through analysis
- What is a p-value?
- How to report p-values
- Confidence intervals explained
- Effect size
- Confounding variable
- Chi-square test
- Chi-square test: a worked guide
- Correlation coefficient
- Normality of distribution
- Cronbach’s alpha
- Intraclass correlation coefficient (ICC)
- PRISMA and systematic review methodology
Sources
- R Core Team, Adjust P-values for Multiple Comparisons (stats::p.adjust), R documentation.
- statsmodels, statsmodels.stats.multitest.multipletests.
- U.S. Food and Drug Administration, Multiple Endpoints in Clinical Trials: Guidance for Industry (final, October 2022).
- Wikipedia, Bonferroni correction and its reference list (Dunn 1961; Moran 2003; Nakagawa 2004).
- Benjamini Y, Hochberg Y (1995). Controlling the false discovery rate. Journal of the Royal Statistical Society, Series B 57(1): 289–300. DOI 10.1111/j.2517-6161.1995.tb02031.x.
- Holm S (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6: 65–70.
- Genovese CR, Lazar NA, Nichols T (2002). Thresholding of statistical maps in functional neuroimaging using the false discovery rate. NeuroImage 15(4): 870–878. DOI 10.1006/nimg.2001.1037.
Data: Multiple-testing adjustment methods in R p.adjust and Python statsmodels: error rate controlled and dependence notes
Compares the multiple-testing adjustment methods exposed by R's stats::p.adjust and Python's statsmodels multipletests: the method name in each, whether it targets the family-wise error rate or the false discovery rate, and the dependence condition stated in each package's documentation. Compiled from the two documentation pages; cells say 'not stated' where the page gives no information.
| Method | R p.adjust name | statsmodels multipletests name | Error rate targeted | Dependence note in documentation |
|---|---|---|---|---|
| Bonferroni | bonferroni | bonferroni | FWER | R: holm is valid under arbitrary assumptions and dominates it; statsmodels: none stated |
| Holm (step-down) | holm | holm | FWER | R: valid under arbitrary assumptions; statsmodels: none stated |
| Sidak | not offered | sidak | FWER (not stated on page) | statsmodels: none stated |
| Holm-Sidak | not offered | holm-sidak | FWER (not stated on page) | statsmodels: none stated |
| Hochberg (step-up) | hochberg | simes-hochberg | FWER in R docs | R: independent or non-negatively associated; statsmodels: independent |
| Hommel | hommel | hommel | FWER in R docs | R: independent or non-negatively associated; statsmodels: non-negative |
| Benjamini-Hochberg | BH (alias fdr) | fdr_bh | FDR | R: not stated; statsmodels: non-negative |
| Benjamini-Yekutieli | BY | fdr_by | FDR | R: most general form of dependence; statsmodels: negative |
Sources
- R documentation: Adjust P-values for Multiple Comparisons (stats::p.adjust) (accessed 2026-11-17)
- statsmodels: statsmodels.stats.multitest.multipletests (accessed 2026-11-17)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







