Written and maintained by CASRAI Editorial Board
Published
On this page
A survey experiment is a randomized experiment run inside a survey: respondents are randomly assigned to two or more versions of a question, vignette, prompt or list, and the difference in their answers between versions estimates the effect of the manipulation. The design combines the causal leverage of random assignment with the reach of survey data collection, and its basic analysis is a difference in group means.
What is a survey experiment?
The EGAP Methods Guide defines survey experiments as experiments conducted within a survey. Respondents are randomly assigned to different conditions, so any systematic difference in their responses can be attributed to the treatment, while the remaining variation is noise. That separates a survey experiment from an ordinary survey, which measures what people say but does not vary what they are shown. It also separates it from survey research methods in general, where comparisons between groups are observational and confounded by who ends up in each group.
Mutz (2011) drew a further distinction for what she called population-based survey experiments: random assignment to conditions combined with a representative sample of the target population. Assignment supports the causal claim, and the sampling supports generalising it. A survey experiment on a convenience panel keeps the first property but not the second.
Types of survey experiment
The EGAP guide groups survey experiments by what the researcher varies and how the answer is read. Each type below is a experimental design with its own estimand, but all share one mechanism: random assignment inside the questionnaire.
List experiments (item count technique)
Control respondents see a list of non-sensitive items and report how many apply; treatment respondents see the same list plus one sensitive item and also report only the count. The difference in mean counts estimates the share of respondents for whom the sensitive item applies. Because nobody says which items apply, the design can reduce social desirability bias.
Endorsement and priming experiments
An endorsement experiment attaches a source (for example a named group) to a policy proposal in one condition and not another. A priming experiment exposes some respondents to a cue before the outcome question. In both, the difference in mean outcomes between conditions estimates the effect.
Conjoint and vignette experiments
A conjoint experiment shows respondents profiles whose attributes are randomly varied. Hainmueller et al. (2014), cited in the EGAP guide, varied nine attributes of immigrant profiles. Ripberger’s course materials list vignette, framing, information-provision, measurement (wording, order, response options) and contingent-valuation experiments as further types.
Randomized response
In the randomized response design described by EGAP (Blair et al. 2015), a die roll decides whether a respondent must answer truthfully or give a forced answer; in the die example a forced yes occurs with probability one-sixth. The researcher knows the randomization probabilities, so the prevalence of the sensitive trait can be recovered for the group even though no individual answer is revealing.
Estimator and formula for a two-condition survey experiment
For two conditions, the average treatment effect is estimated by the difference in group means:
τ̂ = X̄T − X̄C
Its standard error and a large-sample 95% confidence interval are:
SE = √( sT² / nT + sC² / nC ), CI = τ̂ ± 1.96 × SE
- τ̂ (tau-hat): the estimated treatment effect, in the units of the outcome (for a list experiment, item counts).
- X̄T, X̄C: the mean outcome in the treatment and control groups.
- sT², sC²: the sample variances of the outcome in each group.
- nT, nC: the number of respondents assigned to each group.
- 1.96: the standard normal critical value for 95% coverage; use a t critical value in small samples.
In a list experiment the same difference has a direct reading: the EGAP guide states that the difference in mean item counts between treatment and control estimates the share of respondents for whom the sensitive item applies, so τ̂ is itself a proportion.
Worked example: a list experiment, with the arithmetic
The EGAP guide reports the list experiment of Kuklinski et al. (1997): a mean count of 2.37 in the treatment group and 1.95 in the control group. The estimate is 2.37 − 1.95 = 0.42, or 42% of respondents. The guide does not report group sizes or standard deviations, so the standard error below uses illustrative figures that are assumed for demonstration only: 500 respondents per group, sT = 0.95 and sC = 0.85. They are not from the original study.
- Variance terms: 0.95² / 500 = 0.9025 / 500 = 0.001805, and 0.85² / 500 = 0.7225 / 500 = 0.001445.
- Sum: 0.001805 + 0.001445 = 0.003250.
- SE = √0.003250 ≈ 0.0570.
- z = 0.42 / 0.0570 ≈ 7.37.
- 95% CI = 0.42 ± 1.96 × 0.0570 = 0.42 ± 0.112, that is [0.31, 0.53].
Under these assumed inputs, the estimate of 42% would carry a 95% interval of roughly 31% to 53%. The width comes from the noise term: because the list-experiment outcome is a count with its own variance, it needs far larger samples than a direct yes/no question to reach the same precision.
Interpreting the result
The estimate describes a group, not a person: the EGAP guide notes that these designs estimate prevalence in a population or the average effect of a manipulation, not an individual’s answer. The table gives a reading guide for a difference-in-means result; it is editorial guidance built on the definitions above, not a rule from any one source.
| Result | Reading |
|---|---|
| τ̂ clearly away from zero, CI excludes zero | The manipulation shifted the average outcome (or, in a list experiment, the sensitive item applies to a non-zero share). |
| τ̂ near zero, wide CI | Not informative: the design may be underpowered or the treatment weak. EGAP lists weak treatments and inattention as limits. |
| τ̂ near zero, narrow CI | Evidence against a large effect for this treatment, in this survey setting. |
| List experiment with a negative τ̂ | A share cannot be negative, so check for sampling noise, a flawed list or a failed randomization before interpreting. |
Whether a survey effect carries over to the real world is a separate question. Fu and Li (arXiv:2405.06779) argue that a survey changes which considerations respondents attend to and how heavily they weight them, so survey effects can be larger than real-world effects, and in some cases context-dependent salience can flip the sign. Boas et al. (2019) and Barabas and Jerit (2010), both cited by EGAP, compare survey and field or real-world estimates.
Assumptions and common mistakes
- Random assignment must actually be random. Check covariate balance across conditions; the whole causal argument rests on it. Attrition that differs by arm breaks it (Ripberger).
- Treatments should map to the theoretical variable. EGAP advises avoiding confounds and keeping stimuli short and simple, because wording and order matter.
- Inattention and satisficing. Respondents who skim receive a weaker treatment than the one you designed; check how many pass attention screens and whether passing differs by condition.
- Floor and ceiling effects in list experiments. If a respondent would answer yes to every non-sensitive item (or none), agreeing with the sensitive item reveals the answer, undermining the privacy the design promises.
- Demand effects and spillover. Ripberger lists demand effects and, in within-subject designs, spillover between conditions. For the trade-off see within-subjects vs between-subjects designs.
- Treating a short-lived effect as lasting. Survey treatments typically measure immediate effects only.
- Reading the result as causal for the wrong population. Unrepresentative samples limit generalisation; see non-response bias. Randomization is not the same as the correlation vs causation problem for observational comparisons, but it only answers it inside the sample.
- Poor question construction. The stimulus is still a questionnaire item; see questionnaire design.
How to report a survey experiment (APA style)
Report the design, the conditions, how respondents were assigned, group sizes, group means and standard deviations, the effect estimate with its confidence interval, and the test. A template with illustrative numbers:
Respondents were randomly assigned to a control list (n = 500) or a treatment list with the sensitive item added (n = 500). The mean item count was higher in the treatment group (M = 2.37, SD = 0.95) than in the control group (M = 1.95, SD = 0.85), a difference of 0.42 (95% CI [0.31, 0.53]), Welch t = 7.37, p below .001.
Also state the sampling frame and mode, the exact wording of every condition, any exclusions and their rates by arm, and the pre-registration, if any. The sample sizes, standard deviations and test statistic here are illustrative, matching the worked example above.
How to calculate it in Excel, R or Python
Each snippet assumes one outcome column and one column marking treatment (1) or control (0).
Excel
With treatment outcomes in B2:B501 and control outcomes in C2:C501: difference =AVERAGE(B2:B501)-AVERAGE(C2:C501); standard error =SQRT(VAR.S(B2:B501)/COUNT(B2:B501)+VAR.S(C2:C501)/COUNT(C2:C501)); p-value =T.TEST(B2:B501,C2:C501,2,3) (two-tailed, unequal variances).
R
fit = t.test(count ~ treat, data = df); diff(rev(fit$estimate)); fit$conf.int returns the Welch test; diff(rev(fit$estimate)) gives treatment minus control when treat is coded 0/1. An equivalent is lm(count ~ treat, data = df) with robust standard errors.
Python
from scipy import stats; t = df[df.treat==1]['count']; c = df[df.treat==0]['count']; diff = t.mean() - c.mean(); se = (t.var(ddof=1)/len(t) + c.var(ddof=1)/len(c))**0.5; res = stats.ttest_ind(t, c, equal_var=False) gives the Welch test; the interval is diff ± 1.96*se.
Sources
- EGAP Methods Guide: 10 Things to Know About Survey Experiments, checked 2026-11-18
- Ripberger, J.: Survey Experiments (lecture slides), checked 2026-11-18
- Fu, Y. and Li, X.: A Formal Theory of Survey Experiment Generalizability: Attention and Salience (arXiv:2405.06779), checked 2026-11-18
- CASBS Stanford: Population-Based Survey Experiments (Mutz, Princeton University Press, 2011), checked 2026-11-18
Data: Survey experiment designs: what is varied and how the answer is read
Compares five survey experiment designs named in the EGAP Methods Guide: list, priming, endorsement, conjoint and randomized response. For each design it records an example study cited in the guide, what the researcher randomly varies, and how the effect or prevalence is estimated. Compiled from the guide's descriptions on 2026-11-18.
| Design | Example study in the guide | What is randomly varied | How the result is read |
|---|---|---|---|
| List experiment | Kuklinski et al. 1997 | Whether a sensitive item is added to a list of non-sensitive items | Difference in mean item counts = share for whom the sensitive item applies |
| Priming experiment | Brader et al. 2008 | Exposure to a cue before the outcome question | Difference in mean outcomes between conditions |
| Endorsement experiment | Nicholson 2012 | Whether a source endorses the proposal | Difference in mean outcomes between conditions |
| Conjoint experiment | Hainmueller et al. 2014 | Several attributes of a profile (nine in the cited immigrant study) | Respondents' choices or ratings of the varied profiles |
| Randomized response | Blair et al. 2015 | A die roll that forces a yes answer with probability one-sixth in the die example | Known randomization probabilities let the group-level prevalence be recovered |
Sources
- EGAP Methods Guide: 10 Things to Know About Survey Experiments (accessed 2026-11-18)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







