Written and maintained by CASRAI Editorial Board
Last updated
The main advantage of a survey is that it puts the same standardised questions to many people at a cost and speed that few other methods match. The main disadvantage is that the answers are only as good as the question wording, the people who actually respond and the way the survey is delivered. This guide ties each weakness to a named mechanism using Pew Research Center methods research, rather than listing pros and cons in the abstract.
What a survey is
A survey collects self-reported answers from a sample of people using a fixed set of questions, delivered by phone, online, by mail or in person. Because every respondent sees the same questions, answers can be counted and compared. A survey that measures a population at one point in time is a cross-sectional study. For sampling designs behind surveys, see quota sampling and multistage sampling.
Advantages of surveys
- Standardisation. Every respondent answers the same questions, so results are comparable across people and across waves.
- Scale. A well-run panel can reach large samples. Pew reports that participation among active members of its American Trends Panel is high, nearly 80% on average, and that 78% of invited panelists took part in a late-2018 wave.
- Quantifiable results. Closed questions produce data that can be tabulated and analysed statistically; see quantitative data examples.
- Reach into attitudes and behaviours. A survey can ask about opinions, intentions and experiences that cannot be observed directly.
- Repeatability. A fixed instrument can be fielded again, which supports trend measurement.
Disadvantages of surveys
Question wording changes the answer
Pew states that even small wording differences can substantially affect the answers people provide. A survey result is therefore a result for that wording, not for the topic in general.
Question order shapes later answers
Pew notes that earlier questions can unintentionally provide context for the questions that follow. Two patterns are described: a contrast effect, where the order produces greater differences in responses, and an assimilation effect, where responses become more similar because of their order.
Social desirability bias
On sensitive subjects people may give inaccurate answers. Pew gives the example that respondents understate alcohol and drug use, and says social desirability bias can be greater when an interviewer is present.
Coverage: not everyone can be reached
Pew observes that not everyone in the U.S. uses the internet, that households without internet are demographically quite different, and that nearly half of the panel members without internet access are aged 65 and older. An online-only survey can therefore under-represent exactly those people. Pew describes the under-representation of non-internet households as a continuing challenge.
Nonresponse: the people who answer differ from those who do not
Pew reports that some groups are overrepresented among respondents, naming registered voters, non-Hispanic whites and college graduates, and that incentives help harder-to-survey groups stay in a panel but do not completely solve the problem. Phone changes have caused phone survey response rates to decline.
Weighting repairs imbalance but costs precision
Weighting addresses demographic imbalances for the variables used in the weighting, and Pew states that it reduces the precision of the survey. Its reported design effect was 1.65 for a subsampled wave versus around 2.5 or higher for full-sample waves. A larger design effect means a given sample size behaves like a smaller one. Weighting cannot correct for variables that were not used in it.
Strengths and weaknesses by survey mode
Pew states that question wording and responses can vary based on the mode used to survey respondents, and that Pew found that trends on sensitive topics or with volunteered answers tended to show larger differences than other trends when moving from phone polls to the online panel.
| Mode | What Pew reports | Weakness that applies |
|---|---|---|
| Phone | Response rates have declined as phone practices changed (Pew ATP) | Nonresponse; interviewer presence can raise social desirability bias |
| Online panel | High participation among active panelists, nearly 80% on average (Pew ATP) | Coverage: households without internet are left out and skew older |
| Interviewer-administered (phone or in person) | Social desirability bias can be greater when an interviewer is present (Pew) | Under-reporting on sensitive topics |
| Switching modes between waves | Sensitive-topic and volunteered-answer trends tended to show larger differences than other trends when moving from phone to online (Pew) | Trend comparisons can reflect the mode change, not real change |
Reducing the weaknesses
- Keep the same wording and mode when a trend is the goal, since both affect answers.
- Consider whether an earlier question could colour a later one, and place sensitive items with the mode’s social desirability risk in mind.
- Recruit using methods that reach offline households where coverage matters; Pew’s address-based recruitment brought 94% of those who completed the screener into the panel, against roughly half for phone recruitment.
- Weight on the variables that matter and report the loss of precision it brings.
When not to use a survey
A survey is a poor fit when the topic is highly sensitive and the mode cannot protect candour, when the people of interest cannot be reached by the chosen mode, or when the question needs depth that a fixed set of items cannot capture. Respondent burden is a further limit; see survey fatigue. For building and fielding a questionnaire, see how to use Qualtrics.
FAQ
What is the biggest disadvantage of a survey?
No single one dominates. The most consequential are those that make the respondents unrepresentative (coverage and nonresponse) and those that make answers unreliable (wording, order and social desirability).
Are surveys reliable?
They can be, within limits. Results depend on wording, order, mode, who responds and how the data are weighted, so a result should be reported with those design choices.
Is a survey the same as a questionnaire?
Not exactly. The questionnaire is the instrument of questions; the survey is the whole method, including sampling, delivery and analysis.
Are online surveys better than phone surveys?
Each has a different weakness. Online surveys risk missing offline households; phone surveys face declining response rates and interviewer effects.
Sources
- Pew Research Center, Writing Survey Questions, checked 2026-10-29.
- Pew Research Center, Growing and improving Pew Research Center’s American Trends Panel (2019-02-27), checked 2026-10-29.
