Publication Bias

Publication bias is the systematic tendency for studies with statistically significant, positive, or novel results to be submitted for publication, accepted by journals, and published more quickly than studies with null, negative, or inconclusive results. Because the published literature is the primary evidence base researchers, systematic reviewers, and policymakers draw on, this selective visibility inflates the apparent size — and can even manufacture the apparent existence — of an effect wherever it goes undetected and uncorrected.

References

• Higgins JPT, Thomas J, et al. (eds). Cochrane Handbook for Systematic Reviews of Interventions, Chapter 13: Assessing risk of bias due to missing evidence in a meta-analysis.
• Egger M, Davey Smith G, Schneider M, Minder C. “Bias in meta-analysis detected by a simple, graphical test.” BMJ. 1997;315(7109):629–634.
• Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. “Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy.” New England Journal of Medicine. 2008;358(3):252–260.
• Page MJ, McKenzie JE, Bossuyt PM, et al. “The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.” BMJ. 2021;372:n71.
• Rosenthal R. “The file drawer problem and tolerance for null results.” Psychological Bulletin. 1979;86(3):638–641.

Why it matters for evidence synthesis

A systematic review or meta-analysis is only as reliable as the body of studies it can find and pool. If studies are missing from the published record specifically because of what they found, rather than at random, the pooled effect estimate is biased in a predictable direction — usually toward overstating a treatment’s benefit or an association’s strength. This is why publication bias sits inside the risk-of-bias assessment for every well-conducted synthesis, and why it is one of the domains formally downgraded under the GRADE approach to rating the certainty of evidence. A synthesis is not just summarizing what was found; it is implicitly summarizing what was allowed to be seen.

How it happens

Publication bias is rarely a single actor’s decision. It accumulates across several points in the pipeline from finished study to indexed literature:

• Researcher non-submission — investigators are demonstrably less motivated to write up and submit a null or messy result than a clean, significant one. Robert Rosenthal named this the “file drawer problem” in 1979: completed studies with unremarkable findings are set aside rather than submitted at all.
• Editorial and reviewer preference — positive, novel, or “interesting” findings are, all else equal, judged more publishable, which shapes what authors bother submitting in the first place as much as what gets accepted.
• Sponsor and funder suppression — a trial sponsor with a commercial or reputational stake in a particular outcome may simply decline to submit an unfavorable trial for publication, even though the study was completed and reported to a regulator.
• Publication delay (time-lag bias) — negative results, when they are eventually published, tend to appear later than positive ones, so any synthesis conducted before that lag closes will over-represent early, positive findings.

The Cochrane Handbook treats publication bias as one specific mechanism within a broader family it calls reporting bias — which also includes outcome reporting bias (a published study selectively reports only some of the outcomes it measured), language bias, citation bias, and time-lag bias. Publication bias specifically concerns whether an entire study enters the literature at all, based on its results; the others concern how a study that was published gets reported, cited, or found.

Detecting it: funnel plots and Egger’s test

Because publication bias operates on which studies a reviewer can see, it cannot be measured directly — only inferred from patterns in the studies that were found. The standard visual tool is the funnel plot: a scatter plot with each included study’s effect estimate on one axis and a measure of its precision (typically standard error or sample size) on the other. In a well-behaved set of studies with no bias, small, imprecise studies scatter widely around the pooled effect while large, precise studies cluster tightly near it, producing a symmetric, inverted-funnel shape. A gap or asymmetry in the plot — commonly a missing cluster of small studies on the unfavorable side — is consistent with publication bias, but the Cochrane Handbook is explicit that asymmetry is a generic signal of “small-study effects” with several possible causes, including genuine clinical heterogeneity between small and large trials, not proof of bias on its own.

Egger’s test (Egger, Davey Smith, Schneider & Minder, BMJ 1997) formalizes the same idea as a regression: it fits a weighted linear regression of each study’s standardized effect against its precision, and tests whether the regression intercept differs significantly from zero. A significant intercept is read as evidence of funnel-plot asymmetry. The test has well-documented limits — it has very low statistical power with fewer than roughly 10 studies and is not recommended below that threshold — and, like the visual funnel plot itself, it should be interpreted alongside domain knowledge about the studies involved rather than treated as a standalone verdict.

Documented real-world scale

The clearest documented illustration of publication bias comes from comparing a published literature against a complete, results-blind record of what was actually studied. Turner and colleagues compared FDA regulatory reviews of 74 antidepressant trials (which the FDA receives regardless of outcome) against the matching published literature. The FDA classified 38 of the 74 trials as positive and 36 as negative or questionable. Of the 38 positive trials, 37 were published. Of the 36 negative or questionable trials, only 3 were published as negative — 22 were never published at all, and 11 were published in a way that conveyed a positive outcome. The published literature alone suggested about 94% of trials were positive, against the FDA’s actual 51%, and pooling only the published trials inflated the apparent effect size by roughly 32% overall relative to pooling the complete FDA dataset.

Mitigations built into the modern publishing pipeline

None of these mitigations eliminates publication bias on its own, but together they are the standard toolkit a research-administration office or systematic reviewer relies on:

• Prospective trial registration. ICMJE has required prospective registration in a public registry (ClinicalTrials.gov or another WHO ICTRP primary registry) as a condition of considering a trial for publication since 2005. A registered-but-unpublished trial is discoverable, which lets a systematic reviewer identify what is missing from the published record rather than being limited to what was published. CASRAI’s guide on preregistering a study protocol and its pre-registration entry cover the registry-based mechanics.
• Registered Reports address the problem upstream of registration by having a journal peer-review and conditionally accept a study’s protocol before the results are known, so the editorial decision to publish is locked in independent of whether the findings turn out to be significant.
• Venues and policies that welcome null results. A negative-result paper reports a study whose outcome did not support the tested hypothesis; some journals and publisher policies (evaluating manuscripts on methodological soundness rather than the direction of the finding) explicitly counter the incentive that produces publication bias in the first place.
• Comprehensive search strategy in systematic reviews. The PRISMA 2020 reporting guideline requires reviewers to search trial registries and grey literature, not just indexed journal databases, specifically to reduce the impact of unpublished studies on a synthesis.
• Certainty downgrading. Under GRADE, a body of evidence with credible signs of publication bias (funnel-plot asymmetry, an unusually small evidence base dominated by industry-funded positive trials, etc.) has its overall certainty rating downgraded, separate from and in addition to any point-estimate correction.

Publication bias vs. a legitimate risk-of-bias exclusion

Publication bias is easy to conflate with ordinary study selection in a review, but they are not the same failure mode. Publication bias happens upstream, before a reviewer ever sees a study, because the study’s own results affected whether it was submitted or accepted anywhere. A reviewer excluding a study from a meta-analysis because it fails an explicit, pre-specified methodological quality criterion — for example, an invalid randomization method — is a normal, transparent part of systematic review methodology, applied identically regardless of whether that study’s result was positive or null. The first silently shrinks the evidence base by result; the second visibly and symmetrically screens it by method.

Frequently asked

Does a symmetric funnel plot rule out publication bias? No. Symmetry is reassuring but not conclusive, particularly with a small number of studies, where a funnel plot and Egger’s test both have limited power to detect real asymmetry.

Is publication bias the same thing as p-hacking? No. P-hacking (manipulating analysis choices within a single study until a result crosses a significance threshold) happens during analysis of one study; publication bias happens afterward, at the level of which completed studies enter the literature at all. The two can compound each other but are distinct problems with different fixes.

Can publication bias affect a whole field even if most individual studies are done well? Yes. Publication bias is a property of the literature as a body, not of any single study’s internal validity — a set of individually rigorous, honestly reported studies can still produce a badly biased summary picture if enough negative results among them were simply never published.