MedQA Benchmark Dataset: USMLE, MCMLE, and TWMLE Medical Question-Answering Architecture and Clinical AI Evaluation Limits

MedQA is a large-scale, multilingual, open-domain multiple-choice question-answering (QA) benchmark dataset derived from professional medical licensing examinations, introduced by Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits in 2021 (Applied Sciences, 11(14), 6421). Designed to evaluate clinical reading comprehension and multi-hop diagnostic reasoning beyond simple surface-level fact retrieval, MedQA compiles 61,097 questions across three languages: English from the United States Medical Licensing Examination (USMLE), Simplified Chinese from the Mainland China Medical Licensing Examination (MCMLE), and Traditional Chinese from the Taiwan Medical Licensing Examination (TWMLE). Its English USMLE subset serves as a foundational pillar of modern medical large language model (LLM) evaluation suites, including Google Research’s MultiMedQA (Med-PaLM 2) and comparative benchmark studies alongside HealthBench and BioGPT.

What Is MedQA? Dataset Structure and Multilingual Splits

Prior to MedQA, biomedical NLP benchmarks such as PubMedQA and BioASQ focused primarily on answering questions where the relevant PubMed abstract or gold context passage was explicitly provided to the model. In contrast, Jin et al. (2021) constructed MedQA as an open-domain clinical reasoning benchmark modeled on real-world clinical board examinations. A typical MedQA item presents a multi-sentence patient vignette—containing demographics, chief complaint, history of present illness, physical examination findings, and laboratory or imaging values—and asks the model to select the single best diagnosis, pharmacotherapy, pathophysiological mechanism, or next step in patient management.

To prevent data leakage and drop questions requiring visual inspection, Jin et al. deduplicated scraped question banks and filtered out any items containing images, tables, or graphs. The resulting 61,097 questions are partitioned using an 80% / 10% / 10% random split across training, development (validation), and test sets:

MedQA Subset	Source Examination	Language	Answer Options	Train Split	Dev Split	Test Split	Total Questions
MedQA-USMLE (Full)	United States Medical Licensing Examination (Step 1, Step 2 CK, Step 3)	English	4 or 5 options (A–E)	10,178	1,272	1,273	12,723
MedQA-MCMLE	Mainland China National Medical Licensing Examination	Simplified Chinese	5 options (A–E)	27,400	3,425	3,426	34,251
MedQA-TWMLE	Taiwan Medical Licensing Examination	Traditional Chinese	4 options (A–D)	11,298	1,412	1,413	14,123
Total MedQA Corpus	Combined Tri-Jurisdictional Corpus	EN / ZH-CN / ZH-TW	4 or 5 options	48,876	6,109	6,112	61,097

The 4-Option vs. 5-Option USMLE Sub-Tracks and Textbook Corpus

When interpreting published LLM leaderboard scores on “MedQA (USMLE),” clinical AI researchers must verify which sub-track of the English dataset was evaluated:

• 5-Option USMLE Track (Full 12,723-Question Split): In the raw USMLE preparation banks compiled by Jin et al., questions contain either 4 or 5 candidate choices (A through E). Standardizing the full dataset requires padding or evaluating up to 5 choices across the 1,273 test items.
• 4-Option USMLE Sub-Track (A–D Standardized): To establish uniform 4-choice comparison across all three jurisdictions (matching TWMLE’s native 4 options), Jin et al. also released a 4-option configuration where a fifth distractor option is removed when not the gold answer (or filtered). In the widely adopted MultiMedQA benchmark suite used for Med-PaLM 2, GPT-4, and open-weight clinical LLMs, evaluation is standardized on the 4-option MedQA-USMLE format comprising 10,178 training, 1,272 development, and 1,273 test questions.
• Accompanying Medical Textbook Retrieval Corpus: Alongside the question splits, Jin et al. curated an open-domain knowledge corpus consisting of 18 English medical textbooks (covering USMLE Step 1/2/3 disciplines such as internal medicine, pathology, pharmacology, pediatrics, surgery, and psychiatry, totaling 226,268 paragraphs) and 33 Simplified Chinese medical textbooks (totaling 210,035 paragraphs). Baseline retrieval-reader pipelines use BM25 or dense embeddings to retrieve top-k textbook passages before scoring candidate options.

How MedQA Is Used in Clinical Research and Medical LLM Benchmarking

In biomedical natural language processing and clinical AI assurance, MedQA serves three primary research functions:

1. Multi-Hop Clinical Reasoning Stress-Testing: Unlike single-fact recall tasks (for example, “What is the mechanism of action of lisinopril?”), over 90% of MedQA-USMLE questions require at least two reasoning hops: first inferring the underlying diagnosis from a patient’s symptoms and laboratory abnormalities, and second selecting the appropriate second-line treatment, contraindicated medication, or expected biopsy finding for that inferred condition.
2. Prompting and Inference Scaling Experiments: Researchers use the 1,273-item MedQA-USMLE test split to measure the marginal accuracy gains of domain-specific pretraining, Chain-of-Thought (CoT) reasoning, self-consistency sampling, ensemble refinement, and retrieval-augmented generation (RAG). While the initial 2021 BioBERT retrieval-reader baseline achieved 36.7% accuracy on the 4-option USMLE test split, frontier clinical models since 2023 (Med-PaLM 2 at 86.5%, GPT-4 Medprompt at >90%) have surpassed the approximate 60% human passing threshold.
3. Cross-Lingual Clinical Knowledge Transfer: Because MedQA provides parallel licensing structures across English (USMLE), Simplified Chinese (MCMLE), and Traditional Chinese (TWMLE), researchers evaluate whether clinical knowledge learned in English transfers to non-English medical terminologies and region-specific clinical practice guidelines (including Traditional Chinese Medicine items present in MCMLE).

Why High MedQA Accuracy Does Not Prove Clinical Safety

As frontier LLMs saturate MedQA above 90% accuracy, clinical safety researchers and regulatory bodies emphasize five structural limitations that prevent multiple-choice board scores from serving as a proxy for real-world clinical readiness:

• Closed-World Option Elimination vs. Open-Ended Differential Diagnosis: In a 4- or 5-option multiple-choice question, the true answer is guaranteed to be among options A–D/E, allowing models to exploit test-taking heuristics and distractor elimination. When the exact same MedQA clinical vignettes are converted into open-ended free-response prompts without answer choices, model accuracy drops substantially.
• Pre-Curated Vignettes vs. Sequential Information Gathering: MedQA vignettes hand the model a complete, noise-free summary of every pertinent positive and negative physical exam and laboratory finding. Real clinical practice requires sequential history-taking, deciding which diagnostic tests to order under cost and risk constraints, and reconciling contradictory or missing EHR documentation.
• Pretraining Data Contamination: Because USMLE sample items andMedQA JSON files have been publicly hosted on GitHub and Hugging Face since 2021, web-scale pretraining corpora risk memorizing exact vignettes or near-duplicate question stems, inflating zero-shot test performance.
• Absence of Calibration and Abstention Metrics: Standard MedQA leaderboard reporting tracks top-1 accuracy under forced choice, ignoring whether the model’s predicted probabilities are well-calibrated (calibration curve machine learning) or whether the model knows when to abstain and defer to a specialist.
• Zero Evaluation of Harm Severity: Scoring every incorrect choice as an equal 0 treats a minor administrative error identically to a fatal medication contraindication—a limitation addressed by rubric-based, clinician-graded benchmarks such as HealthBench and MultiMedQA long-form axes.

Frequently Asked Questions

How does MedQA differ from MedMCQA and PubMedQA?

MedQA contains 61,097 patient-vignette licensing questions from the US (USMLE), Mainland China (MCMLE), and Taiwan (TWMLE). MedMCQA (Pal et al., 2022) contains 193,155 shorter, often factual 4-option questions from Indian postgraduate medical entrance exams (AIIMS and NEET PG). PubMedQA (Jin et al., 2019) asks yes / no / maybe research questions grounded in the titles and abstracts of PubMed biomedical studies. For a side-by-side comparison across all four major benchmarks, see HealthBench and Medical LLM Benchmarks.

Where can researchers download the official MedQA dataset?

The official dataset and textbook corpora created by Jin et al. (2021) are publicly available through the MIT Clinical Decision Making Group repository (`github.com/jind11/MedQA`) as well as standardized Hugging Face Datasets distributions (`bigbio/med_qa` and `GBaker/MedQA-USMLE-4-options`).

What is the passing score on the USMLE compared to MedQA accuracy?

While the National Board of Medical Examiners (NBME) scores USMLE Step examinations on a three-digit scale rather than a fixed raw percentage, historical NBME guidance places the passing threshold at approximately 60% of scored items correct. However, actual USMLE exams include multimedia audio/video clips, imaging radiographs, and interactive computer-based case simulations (CCS on Step 3) that are excluded from the text-only MedQA benchmark.