PROBAST Risk of Bias Tool: 4 Assessment Domains, 20 Signaling Questions & PROBAST+AI

PROBAST (Prediction model Risk Of Bias ASsessment Tool) is the international methodological reference standard used by systematic reviewers, biostatisticians, guideline developers, and health technology assessors to evaluate the risk of bias (ROB) and applicability of studies developing, validating, or updating multivariable diagnostic or prognostic clinical prediction models. Developed through a consensus process by Wolff, Moons, Riley, Whiting, Westwood, Collins, Reitsma, Kleijnen, and Mallett and published as twin core papers in Annals of Internal Medicine (2019;170:51–58 and 170:W1–W33), PROBAST complements transparent reporting guidelines such as TRIPOD+AI and the STARD and TRIPOD checklists when auditing clinical risk calculators like the ASCVD Risk Estimator or machine learning diagnostic models.

What Is PROBAST and What Are Its Four Assessment Domains?

In PROBAST, risk of bias occurs when shortcomings in a study’s design, conduct, or statistical analysis lead to systematically distorted estimates of a prediction model’s predictive performance (primarily calibration and discrimination). Unlike tools designed for randomized controlled trials (such as Cochrane RoB 2) or diagnostic accuracy studies of a single test (QUADAS-2), PROBAST evaluates multivariable models across four structured domains comprising 20 signaling questions:

PROBAST Domain	Signaling Questions	Risk of Bias Focus	Applicability Concern Rated?
Domain 1: Participants	2 questions (1.1–1.2)	Appropriate data sources (prospective cohort, RCT, registry vs non-nested case-control) and inclusion/exclusion criteria	Yes (High / Low / Unclear)
Domain 2: Predictors	3 questions (2.1–2.3)	Consistent predictor definition/measurement across participants, blinding to outcome data, and availability at intended time of model use	Yes (High / Low / Unclear)
Domain 3: Outcome	6 questions (3.1–3.6)	Pre-specified outcome determination, exclusion of predictors from outcome definition (incorporation bias), blinding to predictors, and appropriate time interval	Yes (High / Low / Unclear)
Domain 4: Analysis	9 questions (4.1–4.9)	Sample size/events, continuous predictor handling, missing data imputation, overfitting/optimism adjustment, calibration, and discrimination	No (Evaluated for Risk of Bias only)

Each signaling question is answered as Yes, Probably Yes, No, Probably No, or No Information. Domain-level risk of bias is rated as Low, High, or Unclear. Under PROBAST scoring rules, a study can receive an overall Low Risk of Bias rating only if all four domains are rated Low ROB; if even one domain is rated High ROB, the overall judgment is High ROB (unless a development model with high ROB in development was subsequently externally validated with low ROB).

How Do Signaling Questions in Domain 4 (Analysis) Detect Overfitting, Improper Missing-Data Handling, and Lack of Calibration?

Empirical meta-epidemiological reviews consistently show that Domain 4 (Analysis) is the most frequent source of High Risk of Bias in published clinical prediction models. Its nine signaling questions (4.1–4.9) target specific biostatistical pitfalls:

• 4.1 Reasonable Number of Participants with the Outcome: Evaluates whether the effective sample size (number of outcome events relative to candidate predictor parameters, historically benchmarked at Events Per Variable [EPV] ≥ 20 for model development or Riley et al. formal sample size criteria, and ≥ 100 events for external validation) prevents severe small-sample overfitting.
• 4.2 Continuous and Categorical Predictors Handled Appropriately: Flags arbitrary dichotomization or categorization of continuous physiological variables (such as age, blood pressure, or biomarkers) rather than modeling non-linear functional forms (e.g., restricted cubic splines or fractional polynomials).
• 4.3 All Enrolled Participants Included in the Analysis & 4.4 Missing Data Handled Appropriately: Penalizes complete-case analysis that discards participants with missing values and causes selection bias, favoring principled methods such as multiple imputation.
• 4.5 Selection of Predictors Based on Univariable Analysis Avoided: Flags pre-screening predictors via univariable p-value thresholds (e.g., p < 0.05) prior to multivariable modeling, which distorts predictor selection and inflates optimism.
• 4.6 Complexities in the Data Accounted For: Checks whether censoring, competing risks, or clustered/multi-level observations were modeled properly.
• 4.7 Relevant Model Performance Measures Evaluated Appropriately: Requires both calibration (preferably calibration plots, calibration-in-the-large, and calibration slope rather than relying solely on the Hosmer-Lemeshow test) and discrimination (C-statistic / AUROC).
• 4.8 Model Overfitting and Optimism Accounted For: Requires internal validation via bootstrapping or cross-validation (rather than a simple random split of a small dataset) coupled with shrinkage of regression coefficients.
• 4.9 Predictors and Assigned Weights Match the Reported Multivariable Analysis: Verifies that the final published equation, nomogram, or risk score uses the exact regression weights derived in the final model.

How Does PROBAST+AI Expand Risk-of-Bias Evaluation for Machine Learning and Clinical AI Prediction Models?

In March 2025, Moons, Damen, Kaul, Collins, et al. published PROBAST+AI in The BMJ (2025;388:e082505) to replace the original 2019 PROBAST tool and harmonize quality assessment with the TRIPOD+AI reporting statement across regression, machine learning, deep learning, and foundation-model clinical prediction studies. Key methodological updates in PROBAST+AI include:

1. Split into Model Development vs Model Evaluation Tracks: Clearly separates signaling questions for model development (including internal validation and hyperparameter tuning) from external model evaluation/validation.
2. Prevention of Patient-Level Data Leakage: Explicitly audits whether multiple encounters, imaging slices, or waveform segments from the same patient were strictly isolated across training, tuning, and test folds to prevent artificial AUROC inflation.
3. Algorithmic Fairness and Sociodemographic Subgroup Evaluation: Introduces signaling items examining sample representativeness, differential data missingness, and stratified calibration/discrimination across race, ethnicity, sex, and socioeconomic groups.
4. Preprocessing, Hyperparameter Tuning, and Architecture Transparency: Evaluates whether data rescaling, class-imbalance resampling (such as SMOTE applied improperly before splitting), feature engineering, and hyperparameter optimization were nested inside cross-validation loops.

Frequently Asked Questions

What is PROBAST (Prediction model Risk Of Bias ASsessment Tool) and what are its four assessment domains?

PROBAST is a structured methodological tool published in Annals of Internal Medicine (Wolff et al., 2019; Moons et al., 2019) for assessing the risk of bias (ROB) and applicability of diagnostic and prognostic prediction model studies across four domains: 1. Participants (2 signaling questions), 2. Predictors (3 signaling questions), 3. Outcome (6 signaling questions), and 4. Analysis (9 signaling questions).

How do signaling questions in Domain 4 (Analysis) detect overfitting, improper missing-data handling, and lack of calibration?

Domain 4 uses nine signaling questions (4.1 to 4.9) to audit adequate sample size and events per variable (EPV), preservation of continuous predictors without arbitrary categorization, multiple imputation rather than complete-case analysis for missing data, avoidance of univariable predictor screening, bootstrap or cross-validation optimism correction, and joint reporting of calibration (calibration plots/slopes) and discrimination.

How does PROBAST+AI expand risk-of-bias evaluation for machine learning and clinical AI prediction models?

Published by Moons et al. in The BMJ (2025;388:e082505), PROBAST+AI updates the original tool to evaluate regression, machine learning, and deep learning prediction models, adding signaling questions on dataset shifts, data leakage across patient-level train/tune/test splits, algorithmic fairness across sociodemographic subgroups, hyperparameter tuning transparency, and model updating.

Sources

• Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S; PROBAST Group. “PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies.” Annals of Internal Medicine. 2019;170(1):51–58. https://pubmed.ncbi.nlm.nih.gov/30596875/
• Moons KGM, Wolff RF, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S. “PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration.” Annals of Internal Medicine. 2019;170(1):W1–W33. https://pubmed.ncbi.nlm.nih.gov/30596876/
• Moons KGM, Damen JAA, Kaul T, Hooft L, Andaur Navarro C, Dhiman P, Beam AL, Van Calster B, Celi LA, Denaxas S, Ghassemi M, Heinze G, Kengne AP, Maier-Hein L, Liu X,Logullo P, McCradden MD, Oakden-Rayner L, Riley RD, Sidey-Gibbons C, Steyerberg EW, Wynants L, van Smeden M, Collins GS. “PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods.” BMJ. 2025;388:e082505. https://www.bmj.com/content/388/bmj-2024-082505