Written and maintained by CASRAI Editorial Board
Last updated
PubMedBERT (officially distributed on Hugging Face under microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract and renamed microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract) is a foundational bidirectional transformer encoder introduced by Microsoft Research (Gu et al., ACM Transactions on Computing for Healthcare, 2021) for biomedical natural language processing (BioNLP). Prior to PubMedBERT, the prevailing paradigm in biomedical AI assumed that models should initialize from general-domain English BERT weights (trained on Wikipedia and BooksCorpus) and undergo continual pretraining on medical text. PubMedBERT overturned that assumption—demonstrating across the 13-dataset BLURB benchmark that domain-specific pretraining from scratch with a custom biomedical WordPiece vocabulary consistently outperforms mixed-domain continual pretraining.
Why Did Domain-Specific Pretraining from Scratch (PubMedBERT) Outperform Continual Pretraining (BioBERT) on BLURB?
To evaluate biomedical language models systematically, Gu et al. (2021) established BLURB (Biomedical Language Understanding and Reasoning Benchmark), a comprehensive evaluation suite aggregating 13 publicly available biomedical datasets across six core BioNLP tasks:
- Named Entity Recognition (NER — 5 datasets): BC5-chem, BC5-disease, NCBI-disease, BC2GM (gene mention), and JNLPBA.
- Evidence-Based Medical Information Extraction (PICO — 1 dataset): EBM-NLP (extracting Population, Intervention, Comparator, and Outcome spans from clinical trial abstracts).
- Biomedical Relation Extraction (3 datasets): ChemProt (chemical-protein interactions), DDI (drug-drug interactions), and GAD (gene-disease associations).
- Sentence Similarity (1 dataset): BIOSSES.
- Document Classification (1 dataset): HoC (Hallmarks of Cancer multi-label classification).
- Biomedical Question Answering (2 datasets): PubMedQA and BioASQ.
Why did PubMedBERT surpass BioBERT (Lee et al., 2020) and BlueBERT across these tasks? The decisive factor is in-domain WordPiece vocabulary construction:
- The Subword Fragmentation Problem in Continual Pretraining: Because BioBERT and ClinicalBERT initialize from standard BERT-base, they are permanently locked to BERT’s original 30,522-token WordPiece vocabulary derived from Wikipedia and fiction books. In that general vocabulary, specialized pharmacological and pathological terms are shattered into meaningless character fragments—for example,
naloxoneis split into["na", "##lo", "##x", "##one"], andacetyltransferaseorcardiomyopathyare broken into 4 to 6 generic subwords, forcing the transformer layers to expend capacity reconstructing basic medical words. - PubMedBERT’s Pure Biomedical Vocabulary and Corpus: PubMedBERT built its 30,522-token WordPiece vocabulary directly from 14 million PubMed abstracts (3.1 billion words, 21 GB of text)—and in its extended variant (`-abstract-fulltext`), combined PubMed abstracts with PubMed Central (PMC) full-text articles (16.8 billion words, 107 GB). Because common medical concepts like
lymphoma,naloxone, andbradycardiaexist as single whole tokens in PubMedBERT’s vocabulary, sequence lengths are shorter, attention heads attend directly to intact clinical concepts, and Whole-Word Masked Language Modeling (WWM-MLM) learns sharper biomedical representations.
How Do PubMedBERT, BioLinkBERT, ClinicalBERT, and GatorTron Differ in Corpus and Vocabulary?
Selecting the right encoder transformer for a biomedical or EHR pipeline requires matching the model’s pretraining corpus and vocabulary to whether your target text is published scientific literature or hospital clinical notes:
| Model Family | Pretraining Strategy & Architecture | Pretraining Corpus & Scale | Vocabulary Source | Optimal Downstream Use Case |
|---|---|---|---|---|
| BioBERT (Lee et al., 2020) |
Continual pretraining from general BERT-base (`110M`) | Wikipedia + BooksCorpus → +4.5B words PubMed abstracts + 13.5B words PMC full text | General English BERT (`30k` WordPiece) | Legacy baseline for literature NER and relation extraction |
| PubMedBERT / BiomedBERT (Gu et al., 2021) |
From-scratch biomedical pretraining (`110M` base / `340M` large) | 14M PubMed abstracts (`3.1B` words) ± PMC full-text articles (`16.8B` words) | 100% Biomedical (`PubMed` WordPiece) | Gold-standard encoder for PubMed/PMC NER, ChemProt/DDI relation extraction, PICO, and PubMedQA |
| BioLinkBERT (Yasunaga et al., 2022) |
From-scratch link-aware pretraining (`110M` base / `340M` large) | PubMed abstracts + citation graph hyperlinks (pairs cited documents in the same context window) | 100% Biomedical (`PubMed` WordPiece) | Multi-hop biomedical reasoning, QA (MedQA, PubMedQA), and cross-document relation inference |
| Bio_ClinicalBERT (Alsentzer et al., 2019) |
Continual pretraining initialized from BioBERT (`110M`) | BioBERT → +~2M clinical notes from MIMIC-III ICU database (`~0.5B` words) | General English BERT (`30k` WordPiece) | ICU/inpatient discharge summary classification, readmission prediction, and clinical shorthand |
| GatorTron (Yang et al., 2022 — UF Health / NVIDIA) |
From-scratch scaled clinical Megatron-BERT (`345M`, `3.9B`, and `8.9B` parameters) | >90B words total: `82B` words UF Health de-identified clinical notes + `6B` PubMed + `2.5B` Wiki + `0.5B` MIMIC-III | Custom Clinical + Biomedical Byte-Level BPE / WordPiece | Enterprise EHR phenotyping, SDOH extraction, clinical NER, and medical relation extraction at scale |
When Should Clinical Researchers Use Encoder-Only Biomedical Transformers vs. Generative LLMs?
Even with the rise of autoregressive decoder LLMs (such as BioGPT, Med-PaLM, and GPT-4), encoder-only models like PubMedBERT, BioLinkBERT, and GatorTron-base remain the production workhorses of biomedical informatics for three engineering reasons:
- Bidirectional Token-Level Precision for Sequence Labeling (NER & PICO): Autoregressive causal decoders apply a causal attention mask (each token only attends to preceding tokens) and generate free text that must be parsed back into character offsets. Bidirectional encoders attend to both left and right context simultaneously and output calibrated per-token logits (`BIO` tags), consistently outperforming zero- and few-shot generative LLMs on strict span-boundary F1 benchmarks like BC5CDR, NCBI-Disease, and EBM-NLP.
- Orders-of-Magnitude Higher Throughput and Lower Latency for Million-Note Cohorts: Fine-tuning `PubMedBERT-base` (`110M` parameters) on a single GPU allows a hospital research team to process 10 million EHR progress notes or all 36 million PubMed abstracts in hours at negligible compute cost and zero external API PHI exposure, whereas running a 70B+ parameter generative LLM across 10 million notes is orders of magnitude slower and more expensive.
- Dense Biomedical Embeddings for RAG and Vector Search: Variants fine-tuned with contrastive learning on PubMed query-article pairs (such as `PubMedBERT-base-embeddings` and `MedCPT`) serve as the standard dense retrieval encoders inside clinical Retrieval-Augmented Generation (RAG) architectures, retrieving relevant PubMed evidence passages that a downstream generative LLM then synthesizes.
Frequently Asked Questions
Why did domain-specific pretraining from scratch (PubMedBERT) outperform continual pretraining (BioBERT) on BLURB?
Continual pretraining models like BioBERT inherit standard BERT’s general-English WordPiece vocabulary (built from Wikipedia and books), which fragments specialized biomedical terms like naloxone or cardiomyopathy into multiple generic subwords. By pretraining from scratch on 14 million PubMed abstracts (3.1 billion words), PubMedBERT constructs a dedicated biomedical vocabulary that keeps domain terms intact as whole tokens, yielding superior accuracy across all six task categories of the 13-dataset BLURB benchmark.
How do PubMedBERT, BioLinkBERT, ClinicalBERT, and GatorTron differ in corpus and vocabulary?
PubMedBERT and BioLinkBERT are trained from scratch on PubMed literature with a pure biomedical vocabulary, with BioLinkBERT additionally incorporating citation links between papers during pretraining. Bio_ClinicalBERT adapts BioBERT via continual pretraining on MIMIC-III ICU notes (retaining general BERT’s vocabulary), whereas GatorTron scales from-scratch pretraining to 345M–8.9B parameters across more than 90 billion words (including 82 billion words of real-world UF Health EHR clinical notes) with a custom clinical vocabulary.
When should clinical researchers use encoder-only biomedical transformers vs. generative LLMs?
Researchers should use encoder-only models (PubMedBERT, BioLinkBERT, GatorTron-base) for high-throughput token-level extraction (NER, PICO extraction, relation classification), multi-label document classification, and dense vector embedding retrieval across millions of abstracts or EHR notes. Generative decoder LLMs are preferred for open-ended clinical summarization, ambient SOAP note drafting, and conversational question answering.
