Non-Response Bias: How to Measure It and What to Report

Non-response bias is the difference between what your survey estimated and what it would have estimated if everyone you sampled had answered. It is not a property of your survey. It is a property of each statistic your survey produces — the same fielding can leave one estimate almost untouched and another badly wrong. That single fact governs how you measure it and how you write it up, and it is the reason the response rate on its own tells a reviewer almost nothing.

This page is procedures. It gives you three measurement designs — wave (continuum-of-resistance) analysis, a non-respondent follow-up sub-sample, and benchmark comparison against the sampling frame or a population register — the rule for choosing between them, the reporting conventions from AAPOR’s Standard Definitions, and the actual sentences a reviewer expects to find in your limitations paragraph.

Response rate is a weak proxy for non-response bias

The most common error in a methods section is treating the response rate as a bias measurement — “the response rate was 42%, which is a limitation” — and stopping there. A low response rate does not by itself establish bias, and a high one does not preclude it. What matters is whether non-response is related to the survey variable you are estimating.

This is not a soft methodological preference; it falls out of arithmetic. For a sample mean, the non-response bias of the respondent mean is exactly:

bias(y_r) = (1 - RR) × (y_r - y_m)

where RR is the response rate, y_r is the mean among respondents and y_m is the mean among non-respondents. The expression is a product of two terms. The response rate is one of them. If the second term — the respondent/non-respondent difference on that specific variable — is zero, the bias is zero no matter how low the response rate goes. If the second term is large, a high response rate shrinks the bias but does not remove it.

Robert Groves made this the central argument of “Nonresponse Rates and Nonresponse Bias in Household Surveys” (Public Opinion Quarterly 70(5): 646–675, 2006), whose abstract states plainly that non-response “can, but need not, induce nonresponse bias in survey estimates” and that there is no simple minimum response rate below which estimates are necessarily biased. Groves and Emilia Peytcheva then tested it empirically in “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis” (Public Opinion Quarterly 72(2): 167–189, 2008), which assembled 59 methodological studies designed to estimate the magnitude of non-response bias directly. The compilation found very little correlation between the non-response rate and the measured bias.

Cite both properly if you lean on this point. A reviewer who knows the literature will accept the argument immediately with the citation attached and will not accept it as an unsourced assertion.

A demonstration you can rerun

The following is real output from R 4.6.1, seeded so you can reproduce it. It builds a population with a true mean of 8, then draws three response patterns: a 40% response rate where non-response is unrelated to the survey variable, a 40% response rate where non-response depends on it, and an 85% response rate with that same dependence.

set.seed(4400)
N <- 20000
y <- rgamma(N, shape = 2, rate = 0.25)   # survey variable, true mean 8

# Response propensity with a target response rate and a given
# dependence on y (slope b). b = 0 means non-response is unrelated to y.
propensity <- function(b, target) {
  a <- uniroot(function(a) mean(plogis(a + b * y)) - target, c(-40, 40))$root
  plogis(a + b * y)
}
draw <- function(b, target) rbinom(N, 1, propensity(b, target))

rA <- draw( 0.00, 0.40)   # 40% response, unrelated to y
rB <- draw(-0.28, 0.40)   # 40% response, strongly related to y
rC <- draw(-0.28, 0.85)   # 85% response, same dependence

row <- function(lab, r) data.frame(
  scenario     = lab,
  resp_rate    = round(mean(r), 3),
  mean_resp    = round(mean(y[r == 1]), 2),
  mean_nonresp = round(mean(y[r == 0]), 2),
  true_mean    = round(mean(y), 2),
  bias         = round(mean(y[r == 1]) - mean(y), 2)
)
out <- rbind(row("A  40% RR, unrelated to y", rA),
             row("B  40% RR, related to y",   rB),
             row("C  85% RR, related to y",   rC))
print(out, row.names = FALSE)

                  scenario resp_rate mean_resp mean_nonresp true_mean  bias
 A  40% RR, unrelated to y     0.399      8.01         8.00         8  0.01
   B  40% RR, related to y     0.405      4.89        10.13         8 -3.12
   C  85% RR, related to y     0.851      6.71        15.36         8 -1.29

Scenario A has a response rate most reviewers would flag, and a bias of 0.01 — about a tenth of one percent of the true value. Scenario C has a response rate most reviewers would wave through, and a bias of −1.29 — 16% of the true value, more than a hundred times larger. Between A and C, the survey with the better response rate is the one with the serious bias problem.

The identity above reproduces each figure exactly from the reported components:

                  scenario bias_observed bias_identity
 A  40% RR, unrelated to y          0.01          0.01
   B  40% RR, related to y         -3.12         -3.12
   C  85% RR, related to y         -1.29         -1.29

Note what the identity requires: the mean among non-respondents. In a real survey you do not have it — that is precisely what non-response means. Every procedure below is an attempt to estimate that second term, or to bound it, from evidence you can actually obtain.

Procedure 1: wave analysis (the continuum of resistance)

Wave analysis uses level of effort as a proxy for reluctance. Its premise is that respondents recruited only after repeated contact attempts sit closer, on the survey variables, to the people who never responded at all than early respondents do. If your estimate drifts systematically as later, harder-won waves are added, that drift is evidence the non-respondents would have carried it further in the same direction.

The steps

1. Record effort at case level while you are still fielding. This is the step that gets skipped and cannot be recovered afterwards. For every completed case store the number of contact attempts, which reminder or wave produced it, the date of completion, and whether an incentive escalation or mode switch was involved. If you did not capture this, you cannot run a wave analysis at all.
2. Define waves before you look at the outcome. Use the fielding protocol, not the data: initial invitation, reminder 1, reminder 2, final push. Defining waves after inspecting the estimates invites the same selective analysis you are trying to rule out.
3. Compute your key estimates cumulatively by wave. For each target statistic, report the estimate using wave 1 only, waves 1–2, waves 1–3, and so on to the full sample, with confidence intervals at each step.
4. Test the trend, do not eyeball it. Regress the survey variable on wave number (or on the count of contact attempts) across respondents. The coefficient and its interval are what you report. A significant slope is evidence of a continuum; a flat slope with a tight interval is genuine reassurance and is worth reporting as such.
5. Extrapolate only if you are willing to defend the assumption. Projecting the wave trend past the last wave gives a rough estimate of the non-respondent mean. State the extrapolation explicitly and treat it as an indication of direction and rough magnitude, not a correction.
6. Report it per statistic. Run this for each headline estimate. It is normal and reportable for one estimate to show a strong wave trend and another none.

What it assumes, and when it misleads

The continuum-of-resistance model assumes late respondents genuinely resemble non-respondents. That assumption fails whenever non-response is driven by non-contact rather than refusal, because those are different populations reached by different mechanisms: people who are hard to reach (mobile, working irregular hours, institutionalised) are not the same group as people who are easy to reach and decline. Separate the two in your dispositions and analyse them separately where you can — AAPOR’s disposition codes make this separation possible, which is one of the practical reasons to use them.

It also fails when a late-stage design change did the recruiting. If your final wave introduced a larger incentive or a different mode, the people it brought in are a response to that change, not a deeper point on the same resistance continuum. Say so rather than pooling them.

A flat wave trend is meaningful but it is weaker evidence than it looks: it rules out one mechanism, not all of them. Do not write it up as “no non-response bias”.

Procedure 2: a non-respondent follow-up sub-sample

This is the strongest of the three, because it stops proxying and goes and measures. You draw a random sub-sample of your non-respondents and pursue them with substantially greater intensity — a shortened instrument, a different mode, a meaningful incentive, a named interviewer — and treat the responses you obtain as a direct estimate of the non-respondent mean. It is the classic double-sampling design for non-response.

The steps

1. Draw the sub-sample at random from non-respondents, and record the sampling fraction. Not the ones who look reachable, not the partial completes. A convenience sub-sample of non-respondents reproduces the original problem one level down and is worth nothing.
2. Cut the instrument down hard. Keep only the items you need to estimate bias in your headline statistics plus the demographics used in weighting. A follow-up that takes as long as the original will fail for the same reasons the original did. Five to ten items is a realistic target.
3. Change something material about the approach. A different mode, a substantially larger incentive, an interviewer-administered contact where the original was self-administered. Repeating the original approach harder mostly re-collects the original refusals.
4. Keep the item wording identical. The comparison is only valid if the follow-up items are the same items. Shorten the instrument by dropping questions, never by rewording them — a reworded item introduces a measurement difference you will not be able to separate from the non-response difference. See questionnaire design and survey question types for why small wording changes move answers.
5. Compute the difference and apply the identity. With the sub-sample mean as your estimate of y_m, the formula above gives an estimated bias with an interval. Report the estimated bias, not just the two means.
6. Report the follow-up’s own response rate. The sub-sample has non-respondents too. Its rate is part of the evidence and its own residual non-response is a residual limitation, not a solved problem.

Design notes

Budget for it at the design stage. A follow-up sub-sample is the one procedure here that cannot be added retrospectively once fielding has closed and the contact window has passed, and it is the one that most often gets cut when a project runs late. If your protocol has a realistic chance of a low response rate, reserving a slice of the fieldwork budget for a non-respondent follow-up is a better investment than spending the same money chasing marginal points of headline response rate.

Ethics approval must cover it. Re-contacting people who did not respond is a distinct activity from the original approach and reviewers will want it specified — including how many contacts constitute the limit and how a declination is recorded.

Procedure 3: benchmark comparison against the frame or a population register

Where you sampled from a frame that carries auxiliary variables — an employee list, a student record system, a patient register, a professional membership roll — you already hold data on your non-respondents. Compare respondents against the full issued sample on every frame variable available.

The steps

1. Inventory what the frame actually holds. Age, sex, grade or seniority, department, region, tenure, site. Include process variables the frame carries incidentally — a prior wave’s participation, service usage counts, contract type. These are often more predictive of response than demographics.
2. Compare respondents to the full issued sample, not to the non-respondents. Both comparisons are informative, but the respondents-versus-issued-sample comparison is the one that answers “does my achieved sample represent what I set out to survey”.
3. Report standardised differences, not just p-values. With a few thousand cases, trivial differences reach significance; with a few hundred, real ones do not. A standardised mean difference or a rate ratio, with an interval, tells a reader the size of the discrepancy. A p-value on a large sample tells them mostly about the sample size.
4. Where no frame exists, use an external population benchmark. A national statistical office release, a census table, an administrative register or an established reference survey covering the same population and period. Match the definition and the reference period exactly — an age band defined differently, or a benchmark two years stale, produces a difference that is an artefact of the comparison rather than a finding about your sample.
5. State the benchmark’s own provenance. Name the source, the release, the reference date, and any known limitations. A benchmark comparison is only as good as the benchmark, and a reviewer will check.

The trap

Benchmark agreement on demographics is the weakest reassurance of the three, and it is the one most often over-claimed. Demonstrating that your respondents match the population on age and sex establishes only that non-response was unrelated to age and sex. It says nothing about whether it was related to your outcome. If you are surveying burnout, matching on grade and region is not evidence about burnout unless grade and region predict burnout strongly — and if they do, say so and show it.

The same limit applies to weighting. Post-stratification, raking and non-response weighting adjust for the variables you put into them and for nothing else. They reduce bias to the extent that the weighting variables predict both response and the outcome; they do not remove non-response bias and they can inflate variance. Weighting is a mitigation you report and justify, not a claim that the problem has been handled. The same caution applies to propensity score methods: they are only as good as the covariates available to them.

Choosing between the three

Procedure	Use it when	Strength of evidence	Must be planned in advance?
Wave / continuum-of-resistance analysis	You have no frame variables and no budget for follow-up, but you logged contact effort	Indirect; assumption-dependent	Yes — effort must be recorded during fielding
Non-respondent follow-up sub-sample	The estimate is consequential and re-contact is feasible and approved	Strongest; measures the quantity directly	Yes — budget, protocol and ethics approval
Benchmark comparison	You sampled from a frame with auxiliary data, or a credible external benchmark exists	Moderate; limited to variables held	No — can be done post hoc

These are complements. The strongest write-up runs the benchmark comparison because it is nearly free, runs the wave analysis because the fielding data is already there, and reserves the follow-up sub-sample for the estimates that carry the paper’s conclusion. Where two procedures point the same way, say so — convergent evidence from methods with different assumptions is much stronger than either alone. Where they disagree, report the disagreement rather than the one you prefer.

Reporting the response rate: AAPOR RR1–RR6

“The response rate was 42%” is uninformative, because there are at least six defensible ways to compute it and they can differ by tens of percentage points on the same fieldwork. The standard reference is AAPOR’s Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. The current edition is the 10th, published in 2023, edited by Ned English; AAPOR’s requested citation is “The American Association for Public Opinion Research. 2023. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 10th edition. AAPOR.” Two supplements were issued afterwards: a report on estimating the eligibility status of cases with unknown eligibility (October 2025) and a separate Standard Definitions for Establishment Surveys (December 2025) — the 10th edition removed the establishment-survey section pending that update, so if you are surveying organisations rather than individuals, the December 2025 document is the one you need.

Two things distinguish the six response rates: whether partial interviews count as respondents, and how cases of unknown eligibility are handled — the latter through e, the estimated proportion of unknown-eligibility cases that are in fact eligible.

Rate	Partials counted as respondents?	Treatment of unknown eligibility	Notes from the 10th edition
RR1	No	All unknown cases included in the denominator	The minimum response rate; a lower bound
RR2	Yes	All unknown cases included in the denominator	RR1 with partials counted
RR3	No	Only the estimated eligible share, e, of unknown cases	“The most-common AAPOR response rate in reporting”
RR4	Yes	Only the estimated eligible share, e, of unknown cases	RR3 with partials counted
RR5	No	Assumes e = 0, or there are no unknown cases	Only valid when that assumption genuinely holds
RR6	Yes	Assumes e = 0, or there are no unknown cases	The maximum response rate

RR3 is the one to report by default. If you report RR5 or RR6, the 10th edition is explicit that they “are only appropriate when it is valid to assume that none of the unknown cases are eligible or when there are no unknown cases” — so you must say which of those two situations applies. Reporting RR6 without that justification is the single most common way a response rate gets quietly inflated.

Where you use e, the document is equally explicit about how it must be chosen: one must be guided by the best available scientific information on the eligible share among unknown cases, and “one must not select a proportion to boost the response rate.” The basis for the estimate must be explicitly stated and detailed, and may be given as a range from different estimation procedures.

The other three rate families, and why they matter to bias

Standard Definitions also specifies cooperation rates (COOP1–COOP4), refusal rates (REF1–REF3) and contact rates (CON1–CON3). These are not decoration. The response rate confounds two entirely different failures; the other three separate them:

• Contact rate — the proportion of cases where a responsible member of the sampled unit was reached at all. A low contact rate points at frame, coverage and fielding-effort problems.
• Refusal rate — the proportion who were reached and declined. A low contact rate with a high cooperation rate implies a different bias mechanism from a high contact rate with a high refusal rate, and calls for different mitigation.
• Cooperation rate — completions as a share of eligible units ever contacted. COOP1 and COOP2 differ by whether partials count; COOP3 and COOP4 additionally exclude those unable to be interviewed from the base as incapable of cooperating.

Reporting all four families lets a reader diagnose which mechanism is operating, which is exactly what the wave-analysis caveat above depends on. The 10th edition notes that refusal, non-contact and other rates sum to the non-response rate.

AAPOR’s Code of Minimal Disclosure requires researchers to provide “the response rates computed according to AAPOR Standard Definitions” and states that “at a minimum, a summary of disposition of sample cases should be provided so that response rates could be computed.” In practice that means publishing the full disposition table — completes, partials, refusals, non-contacts, others, ineligibles and unknowns — so a reader can recompute any of the six rates themselves. A disposition table is short, it is the single most useful methodological artefact you can supply, and journals increasingly expect it in supplementary material.

The limitations paragraph: what a reviewer expects to read

A non-response limitations paragraph is judged on four things: whether you reported a named rate, whether you produced evidence about the direction and magnitude of any bias, whether you kept that evidence attached to specific estimates rather than the survey as a whole, and whether you stated what remains unresolved. Here are the sentence shapes that do each job. Fill in your own numbers; the structure is what carries the credit.

1. The rate, named and computed to a standard

“The response rate was 38.4% (AAPOR RR3), computed with e = 0.62 estimated from the proportion of screened cases found eligible in the pilot. A full disposition table is provided in Supplementary Table S1. The contact rate was 71.2% (CON2) and the cooperation rate was 54.0% (COOP3), indicating that non-response arose predominantly from non-contact rather than refusal.”

2. The evidence you actually collected

“Respondents were compared with the full issued sample on all variables held in the sampling frame. Respondents did not differ materially by region (standardised difference 0.04) or contract type (0.06), but were older (standardised difference 0.31) and more likely to be in senior grades (0.28). Wave analysis showed no significant trend in the primary outcome across the four contact waves (b = 0.03 per wave, 95% CI −0.05 to 0.11).”

3. Direction and magnitude, tied to a specific estimate

“A random sub-sample of 200 non-respondents was re-approached with a five-item instrument, of whom 84 responded (42.0%). Their mean score on the primary outcome was 3.1 points higher than that of the main-survey respondents. Applying this difference to the observed non-response rate implies the reported prevalence understates the population value by approximately 1.9 percentage points. Estimates of secondary outcomes B and C showed no comparable difference.”

4. What is left unresolved

“Frame variables were limited to demographic and administrative characteristics; we could not assess whether non-response was related to the outcome itself, and agreement on demographics does not establish that it was not. Non-response weights were constructed from the frame variables above and reduce bias only to the extent that those variables predict both response and the outcome. Residual non-response bias in the primary estimate cannot be excluded.”

Wording to avoid

• “The response rate was low, which is a limitation.” This names the rate without naming a definition and offers no evidence about bias. It is the sentence this page exists to replace.
• “The response rate of 68% is above the threshold generally considered acceptable.” There is no such threshold, and Groves (2006) is the standard citation for why. Any figure you name here — 50%, 60%, 70% — is convention presented as a criterion.
• “Data were weighted to population totals, so the sample is representative.” Weighting adjusts for the weighting variables. Claiming representativeness from it overstates what it does.
• “Non-response bias was assessed and no significant differences were found.” On which variables, by which procedure, with what power to detect a difference? A non-significant test on a small comparison is not evidence of absence. Report the interval.
• “A large sample size mitigates non-response bias.” It does not. Bias does not shrink with n; only variance does. A larger biased sample produces a tighter interval around the wrong value, which is worse. The same logic appears in sampling bias and response bias.

For the surrounding structure of the section these sentences live in, see how to write a limitations section and the worked example.

Non-response, response and attrition bias are three different things

These three are routinely conflated, including by reviewers, and they call for entirely different fixes. Getting the distinction right in your methods section is itself a signal of competence.

	Who is affected	What goes wrong	Where the fix lives
Non-response bias	People invited who never took part	The achieved sample differs from the issued sample on something related to the outcome	Fielding effort, follow-up design, weighting — not the instrument
Response bias	People who did take part	Their answers are systematically distorted — acquiescence, extreme responding, social desirability	Instrument wording, item order, administration mode
Attrition bias	People who took part and then dropped out	Loss to follow-up is related to the outcome, so completers misrepresent the cohort	Retention design, CONSORT flow reporting, missingness-sensitivity analysis

A single study can have all three at once, and each needs its own evidence. Non-response is also one form of the broader family covered in selection bias, alongside self-selection and survivorship; where non-response arises from the frame rather than from willingness, it becomes sampling bias. Where the distortion runs through what people were willing to admit rather than whether they answered, see social desirability bias.

One boundary worth stating precisely: unit non-response (a sampled person supplies no data at all) is what this page addresses. Item non-response (a respondent completes the survey but skips particular questions) is a missing-data problem within an achieved sample and is handled by imputation and by reporting per-item completion, not by the three procedures above. Report the two separately; collapsing them into a single “response rate” hides both.

Frequently asked questions

What is an acceptable response rate?

There is no defensible universal threshold, and this is the settled position in the methodological literature rather than a contrarian one. Groves (2006) argues directly that no simple minimum response rate exists below which estimates are necessarily biased. Some funders and journals impose their own conventions, and you should meet them where they apply, but meeting a convention is a compliance fact, not evidence about bias. Report a named rate, then report evidence about bias separately.

Does a low response rate mean my survey is unusable?

No. It means you have to do more work to establish what your estimates are worth. A 30% response rate accompanied by a benchmark comparison, a wave analysis and a non-respondent follow-up is a stronger piece of evidence than a 70% response rate with no bias assessment at all. Reviewers respond well to this when it is done properly, because it is uncommon.

Can weighting fix non-response bias?

Partially and conditionally. Non-response weighting and post-stratification reduce bias to the degree that the weighting variables predict both the propensity to respond and the outcome, and they increase variance. They cannot adjust for a variable you did not measure, which means they cannot address non-response driven by the outcome itself. Report weighting as a mitigation with its variables named, never as a resolution.

Which AAPOR response rate should I report?

RR3 by default — the 10th edition describes it as the most common in reporting. Report RR1 alongside it if you want to show a conservative lower bound. Only use RR5 or RR6 if you can justify that no unknown-eligibility cases are eligible, or that there are none, and say which. Publishing the full disposition table lets readers compute any of the six themselves, which is the most robust option.

How large should a non-respondent follow-up sub-sample be?

Large enough to estimate the respondent/non-respondent difference with a usable interval on your primary outcome — which is an ordinary sample size calculation, powered on the difference you would consider material rather than on the outcome itself. Plan for the sub-sample’s own response rate to be well below your main survey’s, and size the issued sub-sample accordingly.

Is non-response bias relevant outside surveys?

Yes, wherever participation is voluntary and the invited set is known: clinical trial screening, registry enrolment, consent to record linkage, institutional audits. The mechanism is identical and the three procedures transfer. In longitudinal designs the same logic applied to dropout is attrition bias, and in trials the analytic response is intention-to-treat analysis.

Do I need to assess non-response bias if my sample is not probability-based?

The bias framework above assumes a known issued sample, so the response-rate arithmetic does not apply to a self-selected or opt-in sample — there is no denominator. AAPOR’s 10th edition is explicit that such samples “should be clearly identified as non-probability or self-selected samples.” That does not exempt you from the underlying question; it means the relevant threat is selection bias more broadly, and benchmark comparison against a population reference becomes your main available evidence. See also convenience sampling and quota sampling.

Related reading

• Research methods — the full cluster hub
• Survey research methods and research questionnaire — controlled-vocabulary definitions
• Questionnaire design and survey question types — instrument construction
• Likert scale survey design and analysis and the visual analogue scale — specific response formats
• Simple random sampling and systematic sampling — probability designs with a defined issued sample
• Generalizability and random assignment vs random sampling — what your achieved sample supports
• Minimal detectable change and effect size — deciding what difference is material before you power a follow-up