AI Governance in Healthcare: WHO Six Core Principles, 2024 LMM Guidance, ONC HTI-1, and Clinical Oversight Committees

AI governance in healthcare is the institutional, regulatory, and technical oversight framework through which health systems, clinical trial sponsors, and medical software developers evaluate, validate, deploy, and continuously monitor predictive machine learning and generative artificial intelligence models in clinical care and biomedical research. Anchored internationally by the World Health Organization (WHO) Ethics and Governance of Artificial Intelligence for Health (2021), the WHO Regulatory Considerations on Artificial Intelligence for Health (2023), and the WHO Guidance on Large Multi-Modal Models (LMMs) (2024)—and enforced in U.S. health IT through the ONC HTI-1 Final Rule (45 CFR § 170.315(b)(11)), the FDA Software as a Medical Device (SaMD) framework, and the NIST AI Risk Management Framework (AI RMF 1.0)—clinical AI governance prevents algorithmic harm, uncalibrated drift, and demographic bias before and after EHR integration.

What Are the Six Core Principles in the WHO Ethics and Governance of Artificial Intelligence for Health Framework?

Published in June 2021 following a two-year international consultation of bioethicists, clinicians, legal scholars, and health ministries, the WHO report Ethics and Governance of Artificial Intelligence for Health establishes six core consensus principles that govern the design, regulation, and clinical deployment of health AI globally:

1. 1. Protect Human Autonomy: In healthcare, human beings must remain in full command of medical decision-making and health systems. AI systems must never usurp clinician judgment or patient self-determination, must preserve valid informed consent, and must enforce strict data privacy and confidentiality controls over patient health records.
2. 2. Promote Human Well-Being, Human Safety, and the Public Interest: Medical AI technologies must satisfy rigorous pre-deployment regulatory standards for clinical safety, analytical accuracy, and real-world efficacy. Health systems and developers must implement continuous quality control and post-market surveillance to prevent patient harm or unintended adverse clinical outcomes.
3. 3. Ensure Transparency, Explainability, and Intelligibility: Sufficient documentation regarding model architecture, training cohort demographics, intended clinical context of use, and known failure modes must be published or disclosed before clinical deployment. Models must be intelligible and explainable to the healthcare professionals who rely on their outputs and to the patients affected by their predictions.
4. 4. Foster Responsibility and Accountability: Because an algorithm is not a moral or legal person, clear lines of human and institutional responsibility must be defined across the entire AI value chain (model developers, validating health systems, and deploying clinicians). Governance structures must provide effective mechanisms for independent audit, error investigation, and legal redress when patients suffer harm.
5. 5. Ensure Inclusiveness and Equity: AI for health must be designed and evaluated to ensure equitable performance across age, sex, gender, race, ethnicity, socioeconomic status, and geographic region—explicitly preventing the amplification of historical healthcare disparities or the deployment of high-income-country models that fail in low- and middle-income country (LMIC) clinical populations.
6. 6. Promote AI That Is Responsive and Sustainable: Health AI systems must be continuously evaluated in live clinical environments to verify whether they actually meet clinical and public health needs, must be adaptable or decommissioned when clinical guidelines or disease epidemiology shift, and must account for environmental sustainability (compute energy, carbon, and water footprints) and healthcare workforce impacts.

WHO 2023 Regulatory Considerations and 2024 Guidance on Large Multi-Modal Models (LMMs)

Building on the 2021 ethical baseline, the WHO issued two major operational expansions addressing regulatory oversight and foundation models:

WHO Regulatory Considerations on Artificial Intelligence for Health (October 2023)

Designed for national medical device regulators and hospital compliance officers, the 2023 WHO guidance organizes regulatory evaluation into six operational pillars:

• Transparency and Documentation: Complete traceability of model design, version history, and training/testing dataset provenance.
• Risk Management and Total Product Lifecycle (TPLC): Applying ISO 14971 risk management principles from initial data curation through post-market model updates.
• Intended Use and Clinical Validation: Requiring prospective or rigorous external clinical validation showing that analytical performance translates into safe clinical action in the target patient population.
• Data Quality and Representativeness: Auditing training and test sets for missingness, label bias (such as healthcare cost-proxy errors), and subgroup underrepresentation.
• Privacy and Data Protection: Enforcing statutory privacy safeguards (such as HIPAA, GDPR, and national health data laws) across training and inference pipelines.
• Engagement and Collaboration: Involving clinicians, patients, biostatisticians, and regulators throughout model development.

What Does the 2024 WHO Guidance on Large Multi-Modal Models (LMMs) Require for Clinical AI Deployment?

In January 2024, the WHO published Ethics and Governance of Artificial Intelligence for Health: Guidance on Large Multi-Modal Models, delivering more than 40 specific governance recommendations for generative foundation models capable of accepting and generating text, medical images, audio, and video across five healthcare domains: (1) diagnosis and clinical care, (2) patient-guided symptom investigation and self-management, (3) clerical and EHR administrative workflows (such as ambient scribing and coding), (4) medical and nursing education, and (5) scientific research and drug development.

Key clinical deployment mandates in the 2024 WHO LMM Guidance include:

• Benchmarking Beyond Standardized Medical Exams: Passing a multiple-choice medical licensing benchmark (such as USMLE-style questions) is not sufficient evidence for clinical deployment; LMMs require task-specific clinical evaluation in real healthcare workflows before patient use.
• Mitigating Confabulation (‘Hallucination’) and Automation Bias: Health systems must implement technical guardrails (such as retrieval-augmented generation [RAG] linked to verified clinical source sentences) and mandatory clinician human-in-the-loop sign-off so fabricated citations, invented lab values, or plausible-sounding diagnostic errors never enter the legal medical record unverified.
• Mandatory Patient Disclosure and Liability Allocation: Patients and clinicians must be transparently informed whenever they are interacting with or receiving outputs generated by an LMM, and governance frameworks must establish shared accountability across foundation model providers, downstream application integrators, and deploying healthcare institutions rather than shifting all liability onto bedside clinicians.
• Independent Third-Party Post-Release Audits: High-risk clinical LMM deployments require independent auditing of data protection, cybersecurity resilience (including prompt-injection and data-poisoning defenses), and demographic subgroup equity.

How Do Health Systems Structure an AI Governance Committee for Pre-Deployment Validation, Local Calibration, and Post-Market Monitoring?

Unlike a general corporate IT steering board (see Hospital AI Governance Committee vs. Corporate AI Governance Board and AI Governance Board vs. AI Ethics Committee vs. Audit Committee), a health system’s Clinical AI Governance Committee operates as an interdisciplinary clinical safety body co-led by the Chief Medical Information Officer (CMIO), Chief Health AI Officer (CAIO), Chief Quality/Patient Safety Officer, biostatisticians, clinical informaticists, IRB/privacy officers, and frontline nursing and physician representatives. Payer organizations maintain parallel structures governed by state insurance laws (see Insurer AI Governance: NAIC Model Bulletin vs. Colorado SB 21-169 vs. NY DFS).

In alignment with the ONC HTI-1 Final Rule (45 CFR § 170.315(b)(11))—which mandates Intervention Risk Management (IRM) practices and disclosure of 31 predictive Decision Support Intervention (DSI) source attributes—and the NIST AI RMF 1.0 (`GOVERN`, `MAP`, `MEASURE`, `MANAGE`), hospital AI governance executes a four-gate lifecycle:

Governance Gate	Required Technical & Clinical Verification	Key Metrics & Regulatory Artifacts Audited
Gate 1: Intake, Inventory & Risk Classification (MAP)	Register the tool in the centralized enterprise AI inventory; determine FDA regulatory status (Section 520(o)(1)(E) Non-Device CDS vs. regulated SaMD) and ONC HTI-1 Predictive DSI classification.	Model Card / HTI-1 31 source attributes, HIPAA BAA & data-use verification (confirming zero secondary training on patient PHI without authorization), and clinical owner assignment.
Gate 2: Retrospective Local Validation & Subgroup Calibration (MEASURE)	Evaluate the vendor or internal model on a retrospective local hospital EHR cohort before enabling any clinical alerts.	Discrimination (AUROC, AUPRC), local calibration (calibration-in-the-large, calibration slope, Brier score), and subgroup fairness audits across age, sex, race/ethnicity, and payer class.
Gate 3: Silent (‘Shadow-Mode’) Prospective Trial & Human Factors Testing (MEASURE / MANAGE)	Run the model silently in live EHR production (logging predictions in the background without displaying alerts to clinicians) to measure real-time latency, missing-data behavior, and alert fatigue.	Positive Predictive Value (PPV) at the chosen operating threshold, daily alert firing rate per clinician shift, and workflow integration plan preventing automation bias.
Gate 4: Phased Go-Live & Continuous Post-Market Drift Monitoring (MANAGE)	Authorize production deployment with scheduled re-accreditation (every 6–12 months) and automated statistical drift telemetry.	Population Stability Index (PSI) for input feature drift (covariate shift), rolling calibration slope monitoring, adverse event reporting channel, and kill-switch / rollback SOP.

Frequently Asked Questions

What are the six core principles in the WHO Ethics and Governance of Artificial Intelligence for Health framework?

The six core principles established in the 2021 World Health Organization (WHO) Ethics and Governance of Artificial Intelligence for Health framework are: (1) Protect autonomy; (2) Promote human well-being, human safety, and the public interest; (3) Ensure transparency, explainability, and intelligibility; (4) Foster responsibility and accountability; (5) Ensure inclusiveness and equity; and (6) Promote AI that is responsive and sustainable.

How do health systems structure an AI Governance Committee for pre-deployment validation, local calibration, and post-market monitoring?

Health systems establish a multidisciplinary Clinical AI Governance Committee (spanning the CMIO, Chief AI Officer, biostatistics, patient safety, nursing, and privacy/compliance) that enforces a four-gate lifecycle aligned with ONC HTI-1 Intervention Risk Management (45 CFR § 170.315(b)(11)) and NIST AI RMF 1.0: (1) central inventory registration and HTI-1 31-attribute review; (2) retrospective local validation testing discrimination, local calibration slope/intercept, and demographic subgroup fairness on local EHR data; (3) prospective silent (‘shadow-mode’) production evaluation of alert rates and positive predictive value; and (4) continuous post-market monitoring for covariate shift, calibration drift, and clinical safety incidents.

What does the 2024 WHO guidance on Large Multi-Modal Models (LMMs) require for clinical AI deployment?

The January 2024 WHO Guidance on Large Multi-Modal Models (LMMs) requires that generative AI tools undergo real-world clinical workflow validation rather than relying solely on medical licensing exam benchmarks, enforce human-in-the-loop verification and retrieval guardrails to prevent clinical confabulations (‘hallucinations’) and automation bias, transparently disclose to patients and clinicians whenever an LMM is used, allocate accountability across developers and deploying institutions, and undergo independent post-deployment audits for privacy, cybersecurity, and health equity.