Written and maintained by CASRAI Editorial Board
Last updated
STARD-AI (Standards for Reporting of Diagnostic Accuracy Studies–Artificial Intelligence) and QUADAS-AI (Quality Assessment of Diagnostic Accuracy Studies–Artificial Intelligence) are the international consensus frameworks developed to govern transparent reporting and risk-of-bias appraisal in studies evaluating artificial intelligence and machine learning diagnostic tests. Initiated by the STARD-AI Steering Group (led by Viknesh Sounderajah, Hutan Ashrafian, Ara Darzi, Patrick M. Bossuyt, and colleagues in Nature Medicine 2020/2021 and BMJ Open 2021), these instruments extend the foundational STARD 2015 30-item reporting checklist (Bossuyt et al., BMJ 2015) and the QUADAS-2 critical appraisal tool to address failure modes unique to deep learning, computer vision, and clinical diagnostic algorithms.
Within the broader ecosystem of STARD, STARD-AI, and TRIPOD+AI reporting guidelines, STARD-AI applies specifically when an AI system acts as a diagnostic index test—classifying the presence, stage, or absence of a target clinical condition at the time of testing—whereas TRIPOD+AI governs multivariable prediction models that estimate individualized probability weights or future prognostic risk.
How Do STARD 2015 and STARD-AI Structure Reporting for Diagnostic Accuracy Studies of Medical Tests and AI Algorithms?
The baseline STARD 2015 statement defines 30 essential reporting items organized across Title/Abstract, Introduction, Methods (Study Design, Participants, Test Methods, Analysis), Results (Participants, Test Results), and Discussion, accompanied by a mandatory participant flow diagram linking eligible patients, index test results, and reference standard outcomes.
While STARD 2015 works well for laboratory assays and conventional imaging modalities, diagnostic AI studies introduce black-box preprocessing, iterative hyperparameter tuning, and dataset shift. STARD-AI adapts and expands the STARD 2015 architecture across six AI-specific reporting dimensions:
| STARD 2015 Section | Classical STARD 2015 Requirement | STARD-AI Extension for Diagnostic AI Algorithms |
|---|---|---|
| Title, Abstract & Intended Use (Items 1–4) | Identify as a study of diagnostic accuracy; specify clinical role of index test (triage, add-on, replacement). | Explicitly identify the index test as an AI/ML intervention; specify exact position in the clinical pathway (autonomous triage, concurrent second reader, or post-hoc quality assurance). |
| Participants & Data Provenance (Items 5–9) | Eligibility criteria, setting, dates, consecutive vs. random sampling. | Detail dataset origin across training, tuning (validation), and external test cohorts; scanner/hardware manufacturers, acquisition protocols, inclusion/exclusion rules for poor-quality artifacts, and demographic/socioeconomic composition. |
| Index Test & Preprocessing (Items 10a, 12a, 13a) | Technical specifications of the index test; definition of positivity cutoffs; blinding to reference standard. | Report image/signal preprocessing (resampling, normalization, cropping), model architecture, version number, training-to-test dataset partitioning at the patient level, and how continuous probability outputs are thresholded into binary diagnostic calls. |
| Reference Standard & Ground Truth (Items 10b, 11, 13b) | Clinical rationale for reference standard; blinding of assessors to index test. | Specify how ground-truth labels were constructed: histopathology/laboratory gold standard vs. expert clinician consensus (number of annotators, specialty qualifications, inter-rater reliability kappa, and adjudication rules for disagreements). |
| Statistical Analysis & Calibration (Items 14–18) | Methods for estimating sensitivity, specificity, PPV, NPV, and likelihood ratios; handling of indeterminate results and missing data. | Report threshold-dependent metrics (sensitivity, specificity, F1) alongside threshold-independent discrimination (AUROC, AUPRC), calibration curves, subgroup fairness performance across age/sex/race/scanner vendor, and failure analysis of ungradable inputs. |
| Reproducibility & Governance (Items 28–30) | Protocol registration number, full protocol access, and funding sources. | Declare availability of code, trained model weights, test datasets, or containerized inference interfaces, and explain any regulatory or intellectual-property restrictions. |
How Does QUADAS-AI Evaluate Risk of Bias Across Patient Selection, Index Test, Reference Standard, and Flow/Timing in AI Diagnostic Studies?
Published by Sounderajah and colleagues in Nature Medicine (2021), QUADAS-AI extends the 4-domain structure of QUADAS-2 to provide systematic reviewers, regulatory bodies, and health technology assessors with signaling questions tailored to AI diagnostic test accuracy (DTA) studies. Each domain is evaluated for Risk of Bias, and the first three domains are also evaluated for Concerns Regarding Applicability:
- Domain 1 — Patient Selection (Data Selection & Provenance):
- Signaling Focus: Were patients enrolled consecutively or randomly from a representative clinical population? Did the study avoid artificial two-gate (case-control) enrichment where severe confirmed cases are compared against healthy controls (“spectrum bias”)?
- AI-Specific Appraisal: Evaluates whether the test dataset is genuinely external (collected at independent clinical sites or time periods) rather than an internal random split of a single open-access repository, and whether image/record exclusion rules (such as discarding motion-blurred scans) artificially inflate accuracy.
- Domain 2 — Index Test (AI Algorithm Execution & Thresholding):
- Signaling Focus: Was the AI model evaluated in a locked state without exposure to reference standard labels? If a positivity cutoff was used, was it pre-specified prior to evaluating the external test set?
- AI-Specific Appraisal: Screens for patient-level data leakage (where multiple images, patches, or encounters from the same patient appear in both training and test partitions) and hyperparameter tuning on the evaluation cohort.
- Domain 3 — Reference Standard (Ground Truth Quality & Incorporation Bias):
- Signaling Focus: Is the reference standard likely to classify the target condition correctly, and were reference standard annotators blinded to the AI index test output?
- AI-Specific Appraisal: Evaluates label noise (e.g., relying on automated NLP extraction of radiology reports rather than biopsy or multi-specialist adjudication as ground truth) and prevents incorporation bias where the AI output influences the final clinical consensus label.
- Domain 4 — Flow and Timing (Interval & Verification Completeness):
- Signaling Focus: Was there an appropriate time interval between the acquisition of the AI input data and the reference standard assessment? Did all patients receive the same reference standard?
- AI-Specific Appraisal: Identifies partial and differential verification bias in diagnostic accuracy studies—for example, when only patients flagged as high-risk by the AI algorithm undergo invasive confirmatory biopsy, while AI-negative patients are followed only by clinical observation—and penalizes studies that silently drop “ungradable” or out-of-distribution AI failures from the 2×2 contingency table.
Why Do Pre-Specified Positivity Cutoffs and Dataset-Splitting Leakage Matter in Diagnostic AI Evaluation?
Two methodological flaws account for the majority of inflated accuracy claims in published medical AI diagnostic studies:
1. Pre-Specified vs. Data-Driven Positivity Cutoffs
Most deep learning classifiers output a continuous probability score between 0.0 and 1.0. To generate a clinical 2×2 diagnostic contingency table (True Positives, False Positives, True Negatives, False Negatives) and calculate sensitivity and specificity, researchers must apply a positivity cutoff (operating threshold). When investigators select the cutoff post hoc on the evaluation test set itself—for example, choosing the point on the test ROC curve that maximizes Youden’s J index (Sensitivity + Specificity − 1)—random sampling variability in the test set is exploited, systematically overestimating both sensitivity and specificity (Leeflang et al., 2008). Both STARD-AI and QUADAS-AI require that operating thresholds (such as a high-sensitivity rule-out threshold or a high-specificity rule-in threshold) be locked on the validation/tuning split before running inference on the held-out external test cohort.
2. Patient-Level vs. Sample-Level Dataset-Splitting Leakage
In medical imaging, histopathology, and wearable telemetry, a single patient often contributes dozens of chest X-rays, thousands of whole-slide image patches, or multiple ICU admission episodes. If a dataset is split randomly at the image, slice, or encounter level rather than grouped strictly by unique patient identifier (PatientID), near-identical slices or longitudinal scans from the same individual fall into both the training set and the test set. The neural network memorizes patient-specific anatomical biometrics, implants, or scanner artifacts rather than pathology biomarkers, producing severe optimism bias that collapses when deployed on new patients.
Frequently Asked Questions
How do STARD 2015 and STARD-AI structure reporting for diagnostic accuracy studies of medical tests and AI algorithms?
STARD 2015 provides a 30-item checklist and participant flow diagram covering study design, consecutive participant recruitment, index test and reference standard blinding, and 2×2 diagnostic accuracy metrics. STARD-AI extends those 30 items to require explicit reporting of dataset provenance across training, tuning, and external test cohorts, image/signal preprocessing pipelines, ground-truth annotation consensus, patient-level data splitting, and subgroup fairness analysis.
How does QUADAS-AI evaluate risk of bias across Patient Selection, Index Test, Reference Standard, and Flow/Timing in AI diagnostic studies?
QUADAS-AI evaluates four domains: (1) Patient Selection checks for representative consecutive sampling and external test cohorts free of case-control spectrum bias; (2) Index Test checks for locked model weights, pre-specified positivity thresholds, and absence of training-test leakage; (3) Reference Standard checks for blinded, high-validity ground truth free of NLP label noise; and (4) Flow and Timing checks that all patients receive the same reference standard within a valid clinical interval and that ungradable AI inputs are reported transparently.
Why do pre-specified positivity cutoffs and dataset-splitting leakage matter in diagnostic AI evaluation?
Selecting a positivity cutoff post hoc on the test set capitalizes on chance fluctuations in the ROC curve and inflates sensitivity and specificity. Similarly, splitting datasets at the image or slice level rather than the patient level allows the AI model to memorize patient-specific anatomy across training and test sets, causing false optimism that fails in real-world clinical deployment.
