Attention Checks on Prolific: Policy, Design, and Rejection Rules

Prolific enforces a specific, written policy on how researchers may use attention and comprehension checks to screen participants — and that policy is stricter, and more precise, than the general “add an attention check” advice found in most survey-methodology guidance. A check that would be perfectly normal on a different panel (a hidden timer, a memory-recall item, a single failed check triggering rejection in a 20-minute study) can violate Prolific’s terms and expose a researcher to a reversed rejection, a participant appeal, or an account-level compliance flag. This guide walks through Prolific’s actual rejection rules, how to design a check that survives them, and — separately — what the published methodology literature says about how much these checks actually improve data quality once they are used.

Prolific’s official attention and comprehension check policy

Prolific distinguishes two categories of check, and treats them differently for rejection purposes.

Attention checks are permitted in two forms:

• Instructional manipulation checks (IMCs) — an explicit instruction embedded in an otherwise-normal-looking item (“select ‘Strongly Disagree’ for this question to show you are reading carefully”).
• Nonsensical items — a question with an objectively correct answer that requires no domain knowledge (e.g., a basic arithmetic or factual item).

Comprehension checks are a separate category, permitted only when checking genuine understanding of information the participant needs to complete the task (e.g., informed-consent comprehension, or task instructions with real consequences if misunderstood).

The rejection thresholds differ by study length:

• Studies 5 minutes or longer: a participant must fail at least two separate checks before a rejection is justified. A single failed check is not sufficient grounds for rejection in a study of this length.
• Studies under 5 minutes: a single failed attention check can justify rejection.
• Comprehension checks specifically cannot be used as rejection grounds at all. Prolific requires researchers to give participants at least two attempts to answer a comprehension check correctly; if a participant still fails after two attempts, the correct action is to return the submission (removing it from the dataset without penalizing the participant’s account), not to reject it.

Prolific also lists explicit prohibitions on check design:

• No hidden timers or speed-based screening used as an attention check.
• No memory-recall requirements built into an IMC.
• No “neutral”/ambiguous response option on a nonsensical item that a careful participant could reasonably select.
• No free-text response format for a comprehension check (the correct/incorrect answer must be unambiguous to score).
• Checks must be presented in normal, easily readable formatting — no deliberately small font, low contrast, or visual burial intended to trip up attentive participants.

Designing a check that survives Prolific’s fairness rule

Prolific’s underlying standard, stated directly in its policy, is that a check is unfair if there is any plausible, good-faith reading under which an attentive participant could give a different answer than the one the researcher intended. In practice this means:

• Write the correct answer so unambiguously that a reasonable disagreement is not possible — avoid checks built on a participant’s subjective agreement/disagreement with a statement unless every plausible reading points to one answer.
• Place the check somewhere a genuinely attentive participant would naturally encounter it while reading normally, not buried in fine print or dependent on the participant remembering an instruction given several screens earlier.
• If you use a comprehension check, write the underlying information plainly enough that a careful read is sufficient — the check should test whether they read it, not whether the material itself was confusing.
• Budget for the two-strikes rule on longer studies: a single unlucky misclick on one check should not disqualify an otherwise-attentive participant, and Prolific’s own threshold reflects that.

Researchers combining Prolific with other data-collection tools should also confirm the check design carries over correctly — see this site’s guide on Qualtrics vs REDCap for academic and clinical research for how attention-check logic behaves differently across survey platforms, and the broader guide to survey question types for how check-item format interacts with the rest of the instrument.

What the published evidence says about how well attention checks actually work

Separately from what Prolific’s policy permits, a real methodological question is how much a passed attention check actually tells you about response quality. The most direct empirical test of this, a Pew Research Center study published in 2020, examined 62,639 respondents across six samples (crowdsourced panels, opt-in panels, and address-based-recruited panels) and defined “bogus” respondents independently — people who reported living outside the intended country, gave nonsensical open-text answers, took the same survey multiple times, or straight-lined every question regardless of content.

The study then checked how many of those independently-identified bogus respondents were actually caught by the two most common quality checks:

• A standard trap/attention-check question (e.g., “select this specific answer to show you are paying attention”) — 84% of bogus respondents passed it anyway.
• Speed-based screening (flagging unusually fast completion times) — 87% of bogus respondents were not flagged as speeders.
• Combined, 76% of all bogus respondents passed both checks and would have remained in a dataset that used only these two screens.

Pew’s own conclusion is direct: a standard attention check does not catch the large majority of respondents actually giving low-quality, biasing data. This does not mean attention checks are worthless — they still catch some genuinely inattentive participants, and Prolific’s policy exists precisely because researchers rely on them — but it is evidence against treating “passed the attention check” as a strong proxy for “gave trustworthy data,” and against stacking harsh, borderline-unfair checks on the theory that more screening automatically means cleaner data.

Separately, industry survey-research reporting (Morning Consult, citing its own and peer research) has put the share of generally inattentive online survey respondents at roughly 30% of a typical sample — consistent, directionally, with why researchers reach for checks in the first place, even though the checks above catch only a minority of that group.

Commitment requests and other complements to attention checks

Because attention checks alone leave most low-quality response patterns uncaught, some data-quality research has moved toward asking participants to explicitly commit to attentiveness rather than (or in addition to) testing for it after the fact. Qualtrics has reported that adding a brief commitment request — asking participants to pledge thoughtful, honest responses before starting — reduced measured quality issues by more than half relative to a control group that received only standard attention checks. For a Prolific study, a commitment-style item is straightforward to add and, unlike a nonsensical-item check, carries no risk of tripping Prolific’s fairness rule, since it isn’t scored pass/fail at all.

Combining approaches — a fairly-designed IMC that satisfies Prolific’s policy, plus a commitment request at the start of the instrument, plus post-hoc data cleaning against the same “bogus respondent” indicators Pew used (geographic implausibility, straight-lining, duplicate submissions) — is a more defensible data-quality strategy than relying on a single hard-and-fast attention check to do all the screening work.

When rejection is (and isn’t) justified on Prolific

Bringing the policy and the evidence together into a practical checklist before rejecting a submission:

• Is the study 5 minutes or longer, and did the participant fail only one check? If so, rejection is not justified under Prolific’s policy regardless of how the check performed statistically.
• Was the failure on a comprehension check rather than an attention check? If so, the correct action is offering a second attempt, then a return if they fail again — never a rejection.
• Could an attentive, good-faith participant plausibly justify the answer given? If yes, the check itself may not meet Prolific’s fairness standard, and a rejection built on it is a strong candidate for a successful participant appeal.
• Was the check based on timing, memory recall, or a free-text comprehension answer? Any of these puts the check outside Prolific’s allowed design and rejecting on that basis is a policy violation, not a judgment call.

Frequently asked questions

Can I reject a Prolific participant for failing one attention check?
Only if the study is under 5 minutes long. For studies of 5 minutes or longer, Prolific requires at least two failed checks before a rejection is justified.

What’s the difference between an attention check and a comprehension check on Prolific?
An attention check (an IMC or a nonsensical item) tests whether a participant is reading carefully and can be grounds for rejection under the rules above. A comprehension check tests whether a participant understood task-critical information (like consent details) and can never be used as rejection grounds — a failed comprehension check, after two attempts, should be handled as a return instead.

Do attention checks actually improve data quality?
They catch some inattentive respondents, but the best available direct test (Pew Research Center, 2020) found that 76% of independently-identified low-quality respondents passed both a standard attention check and a speed-based screen. Attention checks are a useful, policy-required screen, not a comprehensive data-quality solution on their own.

What happens if a participant fails a comprehension check twice on Prolific?
The researcher should return the submission rather than reject it — this removes the response from the dataset without penalizing the participant’s account or approval rate.

For the ethics-review side of designing checks and other embedded study elements that participants may not fully anticipate, see this site’s guide on deception in research and debriefing requirements. For the measurement-theory background behind why researchers screen for inattentive responding in the first place, see reliability in research: what it means and how to assess it and test-retest reliability. For the wider toolkit this fits into, see the Research Tools & Software hub.