Examples
Worked examples
- Is an instance
29 teams given the same football refereeing dataset reaching effect-size estimates from 0.89 to 2.93.
- Is an instance
70 teams analysing the same fMRI dataset producing divergent activation maps.
Counter-examples
Looks similar, but isn't
- Not an instance
A single analyst's robustness check.
- Not an instance
A meta-analysis of separately-conducted studies.
Editorial commentary
Many-analysts studies (Silberzahn et al., 2018; Botvinik-Nezer et al., 2020; Breznau et al., 2022) demonstrate substantial heterogeneity in conclusions even when the data and question are held constant. They provide direct evidence on the magnitude of inferential variability driven by researcher degrees of freedom and have been used to argue both for and against the adequacy of any single published analysis.
References
- Silberzahn et al. (Advances in Methods and Practices in Psychological Science, 2018); Botvinik-Nezer et al., 'Variability in the analysis of a single neuroimaging dataset by many teams' (Nature, 2020).
Also known as
multi-analyst study · crowd analytics
Machine-readable encodings
Use in your systems
<role vocab="credit"
vocab-identifier="https://casrai.org/dictionary/"
vocab-term="Many-analysts study"
vocab-term-identifier="https://casrai.org/dictionary/term/many-analysts-study" />{
"@context": "https://schema.org",
"@type": "DefinedTerm",
"name": "Many-analysts study",
"identifier": "https://casrai.org/dictionary/term/many-analysts-study",
"description": "A study design in which a single dataset and research question are given to multiple independent analysts or teams who proceed without coordination, and the distribution of their conclusions is then compared.",
"inDefinedTermSet": "https://casrai.org/dictionary/domain/reproducibility-and-computational-research/",
"url": "https://casrai.org/dictionary/term/many-analysts-study",
"sameAs": [
"multi-analyst study",
"crowd analytics"
],
"license": "https://creativecommons.org/licenses/by/4.0/"
}
