Written and maintained by CASRAI Editorial Board
Last updated
Med-PaLM 2 is a medical domain-aligned large language model (LLM) developed by researchers at Google Research and DeepMind (introduced by Karan Singhal et al. in Towards Expert-Level Medical Question Answering with Large Language Models). Built upon the base PaLM 2 foundation model architecture and aligned through clinical instruction fine-tuning and ensemble refinement prompting, Med-PaLM 2 became the first language model to reach 86.5% accuracy on the United States Medical Licensing Examination (USMLE)-style multiple-choice dataset (MedQA), a 19.3-percentage-point improvement over the original Med-PaLM model (67.2%).
Beyond multiple-choice exam scores, Med-PaLM 2 established a widely adopted methodological template for evaluating clinical generative AI: pairing automated MultiMedQA medical LLM benchmarks with blinded, multi-axis physician and lay-user human evaluation rubrics that measure scientific consensus, omission of critical context, and potential clinical harm.
What Is Med-PaLM 2 and Who Developed It?
Developed by Google Research and DeepMind, Med-PaLM 2 was designed to address two core limitations of first-generation biomedical language models (such as encoder-only BERT variants or smaller domain-specific transformers like BioGPT): limited multi-step clinical reasoning across complex patient vignettes and fragile alignment with clinical safety norms during open-ended, long-form question answering.
Med-PaLM 2 combines three technical components:
- Base Foundation Model Scaling (PaLM 2): Compared with the original 540-billion-parameter PaLM base model, PaLM 2 incorporates a more compute-optimal transformer training regime, a multilingual and scientific pre-training corpus, and stronger baseline logical reasoning.
- Medical Instruction Fine-Tuning: The base model is aligned on curated long-form medical question-answering demonstrations written and reviewed by qualified clinicians across diverse clinical specialties and patient populations, teaching the model to structure clinical explanations, state differential considerations, and qualify uncertainty.
- Ensemble Refinement (ER) Prompting: For complex multiple-choice clinical vignettes, Med-PaLM 2 introduces ensemble refinement, a two-stage inference strategy that extends Chain-of-Thought (CoT) and self-consistency prompting:
- Stage 1 (Stochastic Reasoning Path Generation): Given a few-shot CoT prompt and a clinical question, the model samples multiple independent temperature-controlled reasoning chains and candidate answers.
- Stage 2 (Conditioned Synthesis): Rather than taking a simple majority vote over final answer letters (standard self-consistency), the model is prompted a second time with the original question plus all Stage 1 generated explanations concatenated in context, enabling it to weigh competing rationales and synthesize a refined explanation and final answer. Applying this second stage multiple times and taking a plurality vote yields the final prediction.
MultiMedQA Benchmark Performance: Med-PaLM vs. Med-PaLM 2
Singhal et al. evaluated Med-PaLM 2 across the MultiMedQA benchmark suite, which unifies multiple-choice medical licensing and research datasets with open-ended consumer health question datasets:
| MultiMedQA Dataset | Evaluation Task & Source | Original Med-PaLM Score | Med-PaLM 2 Score |
|---|---|---|---|
| MedQA (USMLE) | 4-option USMLE Step 1, Step 2 CK, and Step 3 clinical vignette questions (1,273 test items) | 67.2% | 86.5% (Ensemble Refinement) |
| MedMCQA | Indian medical entrance exam (AIIMS / NEET PG) 4-option questions (4,183 dev items) | 57.6% | 72.3% |
| PubMedQA | Biomedical literature yes/no/maybe reasoning over PubMed abstracts (500 test items) | 79.0% | 81.8% |
| MMLU Clinical Topics | Massive Multitask Language Understanding subsets (Clinical Knowledge, Medical Genetics, Anatomy, Professional Medicine, College Medicine, College Biology) | 80.3% – 89.0% across subsets | 88.3% – 95.0% across subsets |
The 9-Axis Physician Evaluation Rubric for Long-Form Clinical Answers
A central contribution of the Med-PaLM 2 study is demonstrating that high USMLE multiple-choice accuracy does not automatically guarantee safe long-form clinical communication. To evaluate open-ended responses on 1,066 consumer medical queries from MultiMedQA and 240 adversarial safety questions, Singhal et al. paired automated metrics with a blinded comparative evaluation by practicing physicians and lay raters across nine clinical quality and safety axes:
- Alignment with Scientific and Clinical Consensus: Whether the response reflects current peer-reviewed medical consensus rather than outdated or unsupported claims.
- Reading Comprehension, Knowledge Recall, and Clinical Reasoning: Three distinct diagnostic error dimensions tracking whether the model misunderstood the prompt, failed to retrieve relevant medical facts, or made a flawed logical inference.
- Inclusion of Inappropriate/Incorrect Content vs. Omission of Important Information: Distinguishing errors of commission (hallucinated or contraindicated statements) from errors of omission (failing to mention red-flag symptoms requiring urgent evaluation).
- Extent and Likelihood of Possible Harm: Stratifying whether acting on the response could lead to mild, moderate, or severe clinical harm or death, and the probability of that harm occurring.
- Bias and Health Equity: Assessing whether the response contains demographic stereotyping or fails to generalize across patient subpopulations.
In pairwise blinded comparisons between Med-PaLM 2 answers and answers written by generalist physicians without internet or LLM assistance, physician raters preferred Med-PaLM 2 responses on eight of the nine axes, while noting that further longitudinal real-world validation is required before autonomous clinical deployment.
Regulatory Boundaries and EHR Informatics Integration
What standards or regulatory requirements govern Med-PaLM 2, and how does it integrate with EHR and health informatics workflows? Med-PaLM 2 itself is a foundation model rather than an FDA-cleared medical device. When health systems or digital health developers integrate a clinical LLM into an Electronic Health Record (EHR) or clinical research pipeline, governance depends on the specific downstream workflow:
- FDA SaMD vs. Non-Device CDS Boundaries: General-purpose foundation models do not fit neatly into deterministic software validation rules (see Foundation Models vs. FDA’s SaMD Framework). If an LLM integration is intended to provide patient-specific diagnostic or treatment recommendations where a clinician cannot independently verify the underlying basis, it crosses into regulated Device CDS / Software as a Medical Device (SaMD). Conversely, administrative drafting tools (such as ambient clinical documentation scribes, discharge summary drafting, or literature retrieval with transparent primary citations) are structured so a qualified clinician reviews and edits every output prior to signing the legal medical record.
- Early-Stage Live Clinical Evaluation: Academic medical centers piloting clinical LLMs in live EHR environments follow the DECIDE-AI reporting guideline to track human factors, automation bias, prompt sensitivity, and clinician edit distance between the LLM draft and the signed clinical note.
Frequently Asked Questions
What is Med-PaLM 2 and how is it used in clinical research and healthcare?
Med-PaLM 2 is a medical domain-tuned large language model developed by Google Research and DeepMind (Singhal et al.) on top of the PaLM 2 foundation model. In clinical AI research and healthcare evaluation, it serves as a benchmark system for expert-level medical question answering (scoring 86.5% on USMLE-style MedQA questions), clinical reasoning via ensemble refinement, and multi-axis physician rubric evaluation of long-form clinical responses.
What standards or regulatory requirements govern Med-PaLM 2?
Med-PaLM 2 is a research and enterprise foundation model rather than a standalone FDA-cleared medical device. When health systems or software developers embed clinical LLMs into diagnostic or treatment-recommendation workflows, the resulting application is evaluated under the FDA Clinical Decision Support (CDS) guidance (FD&C Act Section 520(o)(1)(E)), FDA Software as a Medical Device (SaMD) premarket frameworks, HIPAA Security and Privacy Rule Business Associate requirements, and early-stage clinical evaluation reporting guidelines such as DECIDE-AI.
How does Med-PaLM 2 integrate with EHR and health informatics workflows?
In health informatics research and enterprise cloud pilots (such as Google Cloud Vertex AI healthcare workflows), clinical LLMs like Med-PaLM 2 operate behind HIPAA-compliant API boundaries where they ingest de-identified or FHIR-structured clinical notes to draft clinical summaries, synthesize literature evidence, or answer clinical queries under mandatory clinician-in-the-loop review.
