Survey Bias: Types, Examples and Fixes

Survey bias is any systematic error that pushes survey results away from the true value in one direction. It is different from random sampling error, which shrinks as the sample grows and has no preferred direction. A larger sample does not fix survey bias: it only makes a wrong answer more precise. The useful way to organise the many named biases is by the stage at which they enter: before people answer (who is asked), when they decide whether to answer (who responds), and while they answer (how they respond).

What is survey bias?

In general statistical terms, the bias of an estimate is the expected value of the estimate minus the true population value. A survey is biased when its method, not chance, makes the expected result differ from that true value. The American Association for Public Opinion Research (AAPOR) describes how a sampling frame, defined there as “a list of information that will allow you to contact potential respondents”, can leave out groups, how low response can mean “your survey estimates may be biased” because the survey is “missing some types of respondents entirely”, and how respondents may give “a socially desirable answer” (AAPOR best practices, checked 2026-10-29).

Bias before, during and after answering

Each named survey bias belongs to one of three stages. Knowing the stage tells you which fix can work, because a correction aimed at the wrong stage does nothing.

Stage	Question it answers	Typical biases	What addresses it
Who is asked	Does the list of people you can reach match the population?	Sampling bias, coverage error from the sampling frame	A better frame, or a different sampling method
Who answers	Do the people who respond differ from those who do not?	Non-response bias	Follow-up contact, and weighting as a partial adjustment
How they answer	Does the question or setting push answers in one direction?	Measurement bias, response bias, acquiescence, social desirability, leading questions, double-barrelled questions, question order, mode effects, recall bias	Question design and pretesting; weighting does not repair these

The row to remember is the last one. Weighting adjusts for who is in the sample. It cannot undo an answer that was shaped by the question itself.

Survey bias examples with real numbers

Pew Research Center, in its guide “Writing Survey Questions” (checked 2026-10-29), reports experiments in which only the question or its context changed. They are clear examples of the third stage.

Question order

Pew notes that “earlier questions can unintentionally provide context for the questions that follow.” In an October 2003 poll, 45% favored legal agreements giving same-sex couples the same rights as married couples when the question came after a same-sex marriage question, against 37% without that context. In a December 2008 poll, 88% were dissatisfied with the way things were going in the country when the question came right after one on George W. Bush’s job approval, against 78% without it. See the guide on question order effects.

Question wording

In January 2003, 68% favored military action in Iraq to end Saddam Hussein’s rule. When the question added possible U.S. casualties, 43% favored and 48% opposed. In 2005, 51% favored making it legal for doctors to give terminally ill patients the means to end their lives, against 44% when the wording used assisted suicide. See leading questions in surveys.

Social desirability

Pew states that respondents “understate alcohol and drug use, tax evasion and racial bias” and may overstate admirable behavior such as voting and charitable giving. One mitigation Pew describes is offering response options that make an infrequent behavior easier to admit. See social desirability bias.

Acquiescence

Pew reports that “less educated and less informed respondents have a greater tendency to agree with such statements”, and that the behavior is more pronounced with an interviewer present than in a self-administered survey. Its suggested alternative is to offer respondents a choice between alternative statements. See acquiescence bias.

Worked example: sizing an order effect

Take the October 2003 legal-agreements result. The effect of question order is the difference between the two versions.

With the marriage question first: 45%. Without it: 37%.
Difference = 45 – 37 = 8 percentage points.
The December 2008 country-direction result: 88 – 78 = 10 points.
The 2003 Iraq wording result: 68 – 43 = 25 points.

To judge whether a gap like this is larger than chance, you need each version’s sample size. As a purely hypothetical illustration, with sample sizes chosen only to show the arithmetic, suppose each version had n = 1,000 respondents with the proportions 0.45 and 0.37. The standard error of the difference is the square root of (0.45 × 0.55 / 1000 + 0.37 × 0.63 / 1000) = the square root of (0.0002475 + 0.0002331) = 0.0219. The 95% confidence interval is 0.08 ± 1.96 × 0.0219 = 0.037 to 0.123, or about 3.7 to 12.3 points. Under those assumed sizes the interval excludes zero, so the gap would not be explained by sampling error alone.

Interpretation: reading the numbers

What you see	What it points to
Two versions of a question differ by more than sampling error	The wording or order is changing answers; the question is a measurement problem, not a sampling one
Respondents differ from the population on known traits	Coverage or non-response bias; weighting may partly adjust for traits you can measure
Sensitive behavior reported at a low rate	Possible social desirability; Pew lists alcohol and drug use, tax evasion and racial bias as examples of understatement
Agreement is high on statements phrased as assertions	Possible acquiescence; test a version that offers a choice between alternative statements

Assumptions and common mistakes

Treating a bigger sample as a fix. A larger sample reduces random error only; bias stays.
Using weighting for a wording problem. Weighting changes who counts, not what they were asked.
Changing wording or mode between waves and calling it a trend. Pew’s questionnaire guidance notes that wording or mode changes can mimic a real change over time.
Translating word for word. Pew’s questionnaire guidance also says translations should be comparable at the level of meaning.
Reading the confidence interval as covering bias. A margin of error describes random error and says nothing about systematic error.

How to report it (APA example)

Report which biases you considered, what you did about each, and any split-ballot or order test you ran. Example wording using hypothetical figures: “Version A (n = 1,000) yielded 45% agreement and Version B (n = 1,000) yielded 37%, a difference of 8 percentage points, 95% CI [3.7, 12.3].” Reference for the source discussed above, in APA 7th format: Pew Research Center. (n.d.). Writing survey questions. https://www.pewresearch.org/writing-survey-questions/ (checked 2026-10-29)

How to calculate a difference between two survey versions

Each snippet computes the difference in proportions and its 95% confidence interval, using the hypothetical values above.

Excel

A1: 0.45   B1: 0.37   C1: 1000   D1: 1000
Difference:  =A1-B1
Std. error:  =SQRT(A1*(1-A1)/C1 + B1*(1-B1)/D1)
Lower 95%:   =(A1-B1) - 1.96*SQRT(A1*(1-A1)/C1 + B1*(1-B1)/D1)
Upper 95%:   =(A1-B1) + 1.96*SQRT(A1*(1-A1)/C1 + B1*(1-B1)/D1)

R

prop.test(x = c(450, 370), n = c(1000, 1000), correct = FALSE)

This reports the estimated proportions and a 95% interval for the difference. Because the continuity correction is turned off, it matches the hand calculation.

Python

from math import sqrt
p1, p2, n1, n2 = 0.45, 0.37, 1000, 1000
se = sqrt(p1*(1-p1)/n1 + p2*(1-p2)/n2)
d = p1 - p2
print(round(d, 3), round(d - 1.96*se, 3), round(d + 1.96*se, 3))  # 0.08 0.037 0.123

FAQ

What are examples of survey bias?

Leaving out people who cannot be reached by your sampling frame, low response from one group, a question order that primes later answers, leading wording, and respondents who give socially desirable answers. Pew’s wording experiments above show swings of 7 to 25 points from changes to the question alone.

Does a larger sample reduce survey bias?

No. It reduces random error. Systematic error from the frame, from non-response or from the questions stays the same.

Can weighting remove survey bias?

It can adjust for differences in who responded on traits you can measure. It does not correct answers that were shaped by wording, order or social desirability.