DECIDE-AI Reporting Guideline and Checklist: Early-Stage Live Clinical Evaluation of AI Decision Support Systems

Clinical AI investigators, biomedical informaticists, and journal editors searching for DECIDE-AI (and the DECIDE-AI checklist) use the international consensus guideline to design and report the first live clinical deployments of artificial intelligence decision support systems. Co-published in May 2022 in Nature Medicine (28, 924–933) and The BMJ (2022;377:e070904; PMC9117998) by Baptiste Vasey, Myura Nagendran, Bilal Mateen, Davidered McCulloch, Peter McCulloch, and the DECIDE-AI Steering Group, DECIDE-AI stands for Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence.

What Is the DECIDE-AI Reporting Guideline and Why Was It Created for Early-Stage Live Clinical AI Evaluation?

High retrospective accuracy in an offline (in silico) dataset frequently fails to translate into improved patient outcomes at the bedside. Once an algorithm is integrated into a live Electronic Health Record (EHR) or clinical diagnostic workflow, its real-world performance depends on clinician trust, user interface ergonomics, alert timing, automation bias, and local data drift.

Developed through a multi-round international modified Delphi process involving clinicians, human-factors engineers, regulators (including FDA and MHRA representatives), patient advocates, and methodologists, DECIDE-AI provides a minimum reporting checklist specifically for early-stage, small-scale, live clinical evaluations of AI-based clinical decision support systems (CDSS). Rather than treating an AI algorithm as a static pharmaceutical molecule, DECIDE-AI borrows from the surgical/device IDEAL framework (Stages 2a and 2b — Development and Exploration), recognizing that early live clinical deployments are iterative studies where both the user interface and the underlying model thresholds may require structured, protocolized refinement before locking the intervention for a definitive randomized controlled trial (RCT).

How Does DECIDE-AI Bridge the Translational Gap Between Preclinical In-Silico Validation (TRIPOD+AI) and Large RCTs (CONSORT-AI)?

Within the broader family of AI reporting guidelines (CONSORT-AI, TRIPOD+AI, DECIDE-AI), each EQUATOR Network checklist governs a specific phase of the clinical AI translational pipeline:

Translational Stage	EQUATOR Reporting Guideline	Study Design & Primary Evaluation Focus
Stage 1: Preclinical / Offline Development & Validation	TRIPOD+AI (27 items) & MINIMAR	Retrospective or silent prospective in silico model development and external validation; evaluates discrimination (AUROC), calibration, and subgroup fairness without influencing live patient care.
Stage 2: Early-Stage Live Clinical Evaluation (IDEAL 2a/2b)	DECIDE-AI (17 AI-specific + 10 generic items)	Small-scale prospective live deployment where clinicians interact with AI outputs during real patient care; evaluates human factors, usability, workflow integration, safety/errors, and iterative system modifications.
Stage 3: Definitive Comparative Trial Protocol	SPIRIT-AI (15 AI-specific protocol extensions)	Pre-trial protocol registration for randomized or interventional clinical trials evaluating an AI intervention.
Stage 4: Definitive Comparative Trial Results	CONSORT-AI (14 AI-specific trial extensions)	Large-scale randomized controlled trial (RCT) reporting comparative clinical efficacy and patient health outcomes against standard of care.

What Human Factors, Safety Monitoring, and Algorithm Modification Items Are Required on the DECIDE-AI Checklist?

The complete DECIDE-AI checklist comprises 27 items in total: 17 AI-specific reporting items (numbered I through XVII, subdivided into 28 specific sub-items) and 10 generic clinical study items (numbered 1 through 10). Across the Methods and Results sections, DECIDE-AI introduces four load-bearing reporting requirements that do not appear in offline machine learning guidelines:

1. Description of the AI System, Intended Use, and Integration (Items III–V)

• Item III (AI System Description): Identify the exact version of the AI system, input data modalities, preprocessing rules, output format (e.g., probability score, binary alert, highlighted bounding box, or natural language text), and whether the model runs locally or in the cloud.
• Item IV (Intended Use and User Group): Specify the target patient population, the clinical decision being supported, the intended user group (e.g., triage nurses, resident physicians, subspecialty attendings), and the timing of the AI recommendation within the care pathway.
• Item V (Implementation and Training): Document how the AI output is displayed in the EHR/device interface, what training or onboarding users received prior to study start, and what degree of autonomy/adherence was expected of clinicians.

2. Human Factors, Usability, and Ergonomics Engineering (Items IX & XIII)

• Item IX (Human Factors Methods): Describe the methodology and validated instruments used to evaluate usability, cognitive workload, and human–computer interaction (such as the System Usability Scale [SUS], NASA Task Load Index [NASA-TLX], think-aloud protocols, or time-motion workflow observations).
• Item XIII (Human Factors Results & Adherence): Report how often clinicians actually viewed, accepted, modified, or overrode the AI recommendation, the reasons for clinician overrides, changes in decision time, and any evidence of automation bias or alert fatigue.

3. Protocolized Modifications During the Live Study (Items VII & XII)

• Item VII (Modification Procedure): Unlike a fixed-protocol Phase III drug trial, early live AI pilots often discover an interface flaw or threshold miscalibration mid-study. Item VII requires investigators to state a priori the governance rules and triggers under which the AI algorithm, decision threshold, or user interface could be modified during the study.
• Item XII (Modifications Made): Report every modification actually implemented during the live evaluation, the exact date/cohort cutoff of the change, the rationale, and how clinical performance and user adherence shifted before versus after the modification.

4. Safety Monitoring, Errors, and Near-Miss Analysis (Items X & XIV)

• Item X (Safety and Error Assessment Methods): Detail the governance mechanism used to identify, classify, and mitigate algorithmic failures, software downtime, data pipeline errors, and clinical adverse events or near-misses during live use.
• Item XIV (Safety and Error Results): Report all observed technical malfunctions, out-of-distribution input errors, near-misses caught by human clinicians in the loop, and any patient harm attributable to the AI system or human–AI interaction.

Summary Table of the 17 AI-Specific DECIDE-AI Items (I–XVII)

Manuscript Section	AI-Specific Item #	Required DECIDE-AI Reporting Element
Title / Abstract	Item I	Identify the study as a clinical evaluation of an AI-based decision support system and summarize live performance, human factors, and safety.
Introduction	Item II	Define the clinical need, intended place in the care pathway, and prior preclinical (in silico) validation evidence.
Methods	Items III–X	Study design/ethics (generic 1–3); Participants and user groups (III); AI system inputs/outputs/version (IV); Implementation & user training (V); Outcomes & clinical performance metrics (VI); Planned modification governance (VII); Human-factors evaluation tools (IX); Safety and error monitoring plan (X).
Results	Items XI–XIV	Participant and user characteristics (XI); Modifications executed during the study and their impact (XII); Human–computer interaction, usability, and clinician override rates (XIII); Safety events, technical errors, and algorithmic failure modes (XIV).
Discussion	Items XV–XVII	Support for intended use and readiness for large-scale trials (XV); Safety and human-factors limitations (XVI); Reproducibility and code/model availability statement (XVII).