Sampling bias means the sample you collected systematically differs from the population you’re trying to describe — not by chance, but in a consistent direction. That distinction matters more than it sounds: sampling bias is a problem of validity, not precision. A larger biased sample doesn’t fix the problem; it just makes the wrong estimate more confident. This is the single most common misunderstanding about sampling bias, so it’s worth stating plainly before anything else: doubling your sample size tightens a confidence interval around the wrong number. It does nothing to correct the direction of the error.
What Sampling Bias Is (and Isn’t)
Random sampling error is the expected, unavoidable gap between a sample statistic and the true population value, caused purely by chance. It shrinks as sample size grows, and it’s symmetric — as likely to overestimate as underestimate. Sampling bias is different in kind, not just degree. It’s a systematic, directional distortion introduced by how the sample was selected, recruited, or retained, and it does not average out or shrink with more data. A poll of 100,000 people drawn from a frame that excludes a third of the population is not more accurate than a poll of 1,000 people drawn from a representative frame — it’s just precisely wrong.
This is why sample size and sampling quality are separate questions. Statistical power (see CASRAI’s guide to descriptive statistics and sample-size planning) governs how well you can detect a real effect once you have an unbiased sample. Sampling bias governs whether the number you’re estimating is even the right target. No amount of statistical power fixes a biased frame.
Types of Sampling Bias
Selection (Undercoverage) Bias
Selection bias, in its sampling-specific form, arises when the sampling frame — the actual list or mechanism used to reach potential respondents — omits or under-represents part of the target population. The frame, not just the selection procedure within it, is the point of failure.
The textbook case is the Literary Digest‘s 1936 U.S. presidential election poll. The magazine mailed roughly 10 million straw ballots, drawing its address list from telephone directories, automobile registrations, and its own subscriber and club-membership rolls. In 1936, in the middle of the Depression, owning a telephone or a car was itself correlated with income, and wealthier voters skewed Republican. The frame systematically under-covered lower-income, Democratic-leaning voters before a single ballot was returned. The poll predicted a landslide for Alf Landon; Franklin Roosevelt won in a landslide instead. (The episode is also a textbook non-response bias case, covered next — the two compounded each other.)
Modern equivalents: an online-only survey undercovers people without reliable internet access; a survey distributed through a professional association’s mailing list undercovers non-members; an app-based data collection method undercovers people who don’t own the required device. See CASRAI’s guide to data collection methods for how frame choice interacts with each collection mode.
Non-Response Bias
Non-response bias occurs when the people who respond to a survey or agree to participate differ systematically from those who don’t — on the very characteristics the study is trying to measure. This is worth stating carefully because response rate alone is often treated as the relevant number, when representativeness is what actually matters. A survey with a 90% response rate can still be badly biased if the missing 10% differ sharply from respondents on the outcome of interest; a survey with a 20% response rate can be usably unbiased if non-responders don’t differ from responders in any way that matters to the research question. Response rate is a useful red flag — low rates make bias more likely because there’s more room for a gap to open up — but it is not itself the thing to optimize.
In the Literary Digest case, this compounded the frame problem: political scientists have documented that Landon supporters returned their ballots at a measurably higher rate than Roosevelt supporters did within the same mailed sample, so even respondents drawn from the (already skewed) frame were not representative of everyone who received a ballot.
Self-Selection (Volunteer) Bias
Self-selection bias is a specific, common form of non-response bias where participants opt themselves into a study rather than being sampled and then choosing to respond. Open online surveys, ad-hoc web polls, opt-in consumer panels, and patient registries that enroll whoever chooses to enroll are all structurally exposed to this. People who volunteer for research tend to differ systematically from non-volunteers — more engaged, more affected by the topic, more available, sometimes more extreme in their views on it — and there is no sampling frame at all to compare them against, which makes self-selection bias harder to detect than frame-based selection bias. A patient registry for a rare condition, for instance, systematically over-represents patients who are motivated enough, well enough, and connected enough to a specialist center to enroll.
Survivorship Bias
Survivorship bias occurs when a sample is implicitly restricted to cases that “survived” some selection process, and the analysis draws conclusions without accounting for the cases that didn’t survive to be observed at all.
The canonical example comes from Abraham Wald’s work with the Statistical Research Group during the Second World War. The U.S. military had examined the damage patterns on bomber aircraft returning from missions and proposed reinforcing the areas most frequently hit — chiefly the wings and fuselage. Wald pointed out the flaw: the study only ever saw aircraft that survived to return. Planes hit in the areas that were rarely seen damaged on returning aircraft — the engines and cockpit — were the ones that didn’t make it back. He recommended reinforcing the areas that showed the least damage on survivors, not the most, because those were the hits that proved fatal. The military adopted the recommendation.
In research contexts, survivorship bias shows up whenever a dataset only contains cases that persisted long enough to be captured — companies that didn’t go bankrupt, patients who didn’t drop out, manuscripts that were eventually published. See CASRAI’s guide to causal analysis for how unmeasured selection processes like this distort causal inference more broadly, and the guide to spurious correlation for a related failure mode.
Healthy-Worker Bias
Healthy-worker bias is a specific form of selection bias common in occupational and epidemiological research: people who are employed, and especially people who remain employed in physically demanding jobs, tend to be healthier on average than the general population, because serious illness tends to remove people from the workforce before or during a study. Comparing an occupational cohort’s health outcomes to general-population rates without accounting for this can make a genuinely harmful workplace exposure look protective, or a neutral one look beneficial, simply because the comparison population includes people who were too unwell to be employed in the first place.
Referral (Berkson’s) Bias
Referral bias, also known as Berkson’s bias, arises when a sample is drawn from a clinical or referral setting — a specialist clinic, a tertiary hospital, a registry fed by referrals — and the pathway into that setting is itself associated with the exposures or outcomes under study. Patients referred to a specialist center often have more comorbidities, more severe disease, or different care-seeking behavior than patients with the same underlying condition managed elsewhere, which can create or mask an apparent association between an exposure and a disease that doesn’t hold in the broader population.
Prevalence-Incidence (Neyman) Bias
Prevalence-incidence bias, also called Neyman bias, occurs when a study samples existing (prevalent) cases of a condition rather than newly diagnosed (incident) cases, and the condition’s duration or severity is itself related to the exposure being studied. Sampling prevalent cases systematically under-represents people who died or recovered quickly after onset — cases that would have been captured by incident sampling — which can distort the apparent relationship between an exposure and the condition. CASRAI’s guide to prevalence vs. incidence covers the underlying measures this bias depends on in more detail.
Convenience and Snowball Sampling
Convenience sampling (recruiting whoever is easiest to reach) and snowball sampling (recruiting existing participants’ contacts) are legitimate, often necessary tools — particularly for hard-to-reach or hidden populations — but both trade representativeness for feasibility, and that trade-off has a real inferential cost: neither method supports a claim that the sample reflects a defined population, because neither has a known, quantifiable probability of selection. Findings from convenience or snowball samples describe the sample itself credibly; extending them to a wider population requires an explicit, justified argument, not an assumption. See CASRAI’s comparison of random vs. convenience sampling and the guide to snowball sampling for where each is defensible.
Attrition Bias
Attrition bias is sampling bias that develops over time in longitudinal studies: the sample that started out representative erodes as participants drop out, and drop-out is rarely random. Participants who leave a study often differ systematically from those who stay — sicker or healthier, more or less engaged, more or less mobile — so the final analytic sample can be biased even when the baseline sample wasn’t. Reporting the baseline sample alone, without disclosing attrition and comparing those lost to follow-up against those retained, hides this from readers.
How to Detect and Mitigate Sampling Bias
- Compare against known population benchmarks. Where independent, trustworthy population data exists (census figures, administrative records, prior high-quality surveys), compare your sample’s demographic and outcome distributions against it. Divergence on measured characteristics is a warning sign, even though it can’t rule out bias on unmeasured ones.
- Follow up with non-responders. A brief follow-up contact or a short alternate-mode survey of a random subsample of non-responders lets you check whether they differ from responders on key variables, rather than assuming a response rate alone is informative.
- Weight and post-stratify. Survey weights and post-stratification adjust the sample’s composition to match known population totals on variables like age, sex, or region. This can meaningfully reduce bias from known, measured imbalances — but it is not a general fix. Weighting corrects only for differences on variables you actually measured and included in the weighting scheme; it cannot correct for unmeasured differences between responders and non-responders, or between volunteers and non-volunteers, on the outcome itself.
- Run a sensitivity analysis. Re-estimate key results under a range of plausible assumptions about how the missing or excluded population might differ, to see how much the conclusion depends on assumptions you can’t verify.
- Pre-register the sampling plan. Specifying the sampling frame, recruitment strategy, and planned handling of non-response before data collection begins prevents the frame or inclusion criteria from being adjusted, consciously or not, in a way that happens to support a preferred result.
Reporting Sampling Bias
Transparent reporting doesn’t eliminate sampling bias, but it lets readers judge how much it matters for a given claim. At minimum, a methods section should describe: the sampling frame actually used (not just the target population); the recruitment route (how people were invited or enrolled); the response or participation rate, reported alongside a comparison of responders to non-responders wherever that comparison is possible; and, explicitly, who is likely missing or under-represented as a result. This last point is what determines generalisability — the extent to which findings from the sample can reasonably be extended to a broader population — and it’s a judgment call the reader should be equipped to make, not one the researcher should make silently on their behalf.
Frequently Asked Questions
Is sampling bias the same as sampling error?
No. Sampling error is the random, chance-driven gap between a sample estimate and the true population value; it shrinks as sample size increases and is not directional. Sampling bias is a systematic, directional distortion introduced by how the sample was selected or retained, and it does not shrink with a larger sample.
Can a larger sample size fix sampling bias?
No. A larger sample drawn through the same biased process produces a more precise — not more accurate — estimate of the wrong value. Fixing sampling bias requires changing the sampling frame, recruitment method, or retention process, not the sample size.
What’s the difference between selection bias and sampling bias?
Sampling bias is a specific type of selection bias that arises from the sampling process itself — the frame, recruitment, or response pattern. Selection bias is the broader category, which also includes bias introduced after sampling, for example through non-random allocation to treatment groups or non-random loss to follow-up in an already-sampled cohort.
Does a low response rate always mean a biased sample?
Not necessarily. A low response rate increases the risk of non-response bias because it widens the pool of people who might differ from respondents, but it’s only actually biasing if non-responders differ from responders on the variables the study measures. A high response rate reduces but does not eliminate this risk.
How is sampling bias related to sampling methods like stratified sampling?
Probability-based methods such as simple random sampling and stratified sampling are designed specifically to give every population member a known, non-zero chance of selection, which is what makes population-level inference statistically justified in the first place. Non-probability methods, including convenience and self-selected samples, don’t offer that guarantee, which is why they carry a structurally higher risk of sampling bias. See CASRAI’s dictionary entry on sampling methods for the full taxonomy.
