Skip to main content
v2026.11,986 entries · CC-BY 4.0

scispaCy Biomedical NLP: Core Pipelines, Specialized NER Models, UMLS EntityLinker, and medspaCy Workflows

A technical reference to AllenAI’s scispaCy Python library for biomedical and clinical natural language processing, covering core and NER models, Schwartz-Hearst abbreviation detection, UMLS/RxNorm/HPO EntityLinker, and medspaCy integration.

Written and maintained by CASRAI Editorial Board

Published · Last updated

On this page

scispaCy is an open-source Python library developed by the Allen Institute for Artificial Intelligence (AllenAI / AI2) (Neumann et al., Proceedings of the 18th BioNLP Workshop, ACL 2019) that adapts the industrial-strength spaCy framework for biomedical, scientific, and clinical natural language processing (cNLP). Because general-domain NLP pipelines trained on web news and Wikipedia fail when encountering complex biochemical nomenclature, parenthetical gene symbols, and clinical dosing syntax, scispaCy biomedical NLP provides domain-specific tokenization rules, dependency parsers trained on biomedical treebanks, specialized Named Entity Recognition (NER) models, and high-speed ontology linking to the Unified Medical Language System (UMLS).

What Is scispaCy and Which Biomedical NER Models Does It Provide?

Built directly on spaCy‘s Cython architecture, scispaCy modifies the base tokenizer to prevent splitting hyphenated chemical formulas, slash-separated gene/protein complexes (e.g., IL-6/STAT3), and numeric dosage units. Its dependency parser and part-of-speech (POS) tagger are trained on the GENIA 1.0 biomedical corpus converted to Universal Dependencies and aligned with OntoNotes 5.0, while biomedical entity mention detection is trained on the MedMentions corpus. AllenAI distributes eight pre-trained installable Python packages divided into two functional groups:

Model Package Name Category & Word Vectors / Transformer Training Corpus Extracted Entity Labels
en_core_sci_sm Core Pipeline (compact ~100k vocabulary, no word vectors) GENIA 1.0 + OntoNotes 5.0 + MedMentions Generic ENTITY (all biomedical entity spans prior to ontology typing)
en_core_sci_md Core Pipeline (~360k vocab + 50k 200-dim word2vec vectors trained on PubMed + MIMIC-III) GENIA 1.0 + OntoNotes 5.0 + MedMentions Generic ENTITY
en_core_sci_lg Core Pipeline (~785k vocab + 600k 200-dim word2vec vectors trained on PubMed + MIMIC-III) GENIA 1.0 + OntoNotes 5.0 + MedMentions Generic ENTITY
en_core_sci_scibert Transformer Core Pipeline powered by SciBERT (allenai/scibert_scivocab_uncased, Beltagy et al. 2019) GENIA 1.0 + OntoNotes 5.0 + MedMentions Generic ENTITY (highest dependency parsing and span boundary F1)
en_ner_bc5cdr_md Specialized NER Pipeline BioCreative V Chemical-Disease Relation (BC5CDR) corpus DISEASE, CHEMICAL
en_ner_craft_md Specialized NER Pipeline Colorado Richly Annotated Full Text (CRAFT) corpus GGP (Gene/Protein), SO, TAXON, CHEBI, GO, CL (Cell Type)
en_ner_jnlpba_md Specialized NER Pipeline JNLPBA molecular biology corpus DNA, RNA, PROTEIN, CELL_TYPE, CELL_LINE
en_ner_bionlp13cg_md Specialized NER Pipeline BioNLP 2013 Cancer Genetics shared task corpus 16 fine-grained oncology labels including CANCER, ORGAN, TISSUE, CELL, GENE_OR_GENE_PRODUCT, SIMPLE_CHEMICAL, AMINO_ACID

Notice the architectural distinction: the four core pipelines (`en_core_sci_*`) intentionally label all detected biomedical spans simply as `ENTITY` so downstream entity linkers can assign granular UMLS Semantic Types (TUIs), whereas the four `en_ner_*` models classify spans directly into corpus-specific biomedical categories (such as `DISEASE` and `CHEMICAL` in `en_ner_bc5cdr_md`).

How Does the scispaCy EntityLinker Map Text Spans to UMLS, MeSH, RxNorm, and HPO?

In biomedical informatics and EHR phenotyping, extracting raw text strings (such as ”T2DM”, ”NIDDM”, and ”adult-onset diabetes”) is insufficient without normalizing them to a canonical ontology concept. The scispaCy UMLS linker (`scispacy_linker`) and companion pipeline components accomplish this in two deterministic stages:

  1. Local Acronym Expansion via `AbbreviationDetector`: Clinical abstracts and progress notes frequently define local abbreviations on first mention—e.g., ”idiopathic pulmonary fibrosis (IPF)”. Before running entity linking, adding `nlp.add_pipe(“abbreviation_detector”)` executes the Schwartz-Hearst algorithm (Schwartz & Hearst, PSB 2003), binding every subsequent short-form token (`ent._.long_form`) to its expanded full phrase so ambiguous 3-letter acronyms resolve accurately.
  2. Character 3-Gram TF-IDF + Approximate Nearest Neighbor (`nmslib`) Search: When `nlp.add_pipe(“scispacy_linker”, config={“resolve_abbreviations”: True, “linker_name”: “umls”})` is appended to a pipeline, scispaCy converts each detected entity span (or its expanded Schwartz-Hearst long form) into a character 3-gram TF-IDF vector and queries a pre-built HNSW (Hierarchical Navigable Small World) graph index over the target knowledge base. It populates `ent._.kb_ents` with ranked `(concept_id, cosine_similarity_score)` tuples along with canonical concept definitions and UMLS Semantic Types (TUIs).

Out of the box, `EntityLinker` supports five curated biomedical knowledge bases via the `linker_name` parameter:

  • umls (Unified Medical Language System): Links mentions across ~3 million UMLS Concept Unique Identifiers (CUIs) derived from level 0, 1, 2, and 9 source vocabularies including SNOMED CT.
  • mesh (Medical Subject Headings): Normalizes literature spans to NLM MeSH descriptor IDs for systematic review and PubMed indexing.
  • rxnorm (RxNorm): Maps medication brand names, generic ingredients, and clinical drugs to NLM RxNorm Concept Unique Identifiers (RXCUIs).
  • go (Gene Ontology): Links biological processes, molecular functions, and cellular components to GO terms.
  • hpo (Human Phenotype Ontology): Normalizes rare disease and clinical genetics phenotypic abnormalities to `HP:` identifiers for whole-exome/genome diagnostic pipelines.

How Do scispaCy and medspaCy Work Together for Clinical Note Extraction?

While scispaCy excels at biomedical syntactic parsing, NER span detection (`en_ner_bc5cdr_md`), and UMLS CUI normalization, its training corpora consist primarily of scientific abstracts (PubMed/PMC) rather than hospital EHR progress notes. Consequently, scispaCy alone does not know whether a detected `DISEASE` span in a clinical note was negated (”denies chest pain”), historical (”history of appendectomy in 2012”), or experienced by a relative (”mother had breast cancer”).

In production hospital informatics—offering a lightweight Python alternative to Java-based Apache cTAKES—engineers combine scispaCy and medspaCy inside a single unified `spaCy` pipeline:

import spacy
import scispacy
from scispacy.abbreviation import AbbreviationDetector
from scispacy.linking import EntityLinker
import medspacy

# 1. Load scispaCy BC5CDR clinical/chemical NER model
nlp = spacy.load("en_ner_bc5cdr_md")

# 2. Add scispaCy Schwartz-Hearst abbreviation expansion & UMLS CUI linker
nlp.add_pipe("abbreviation_detector")
nlp.add_pipe("scispacy_linker", config={"resolve_abbreviations": True, "linker_name": "umls", "threshold": 0.80})

# 3. Add medspaCy clinical sectionizer and ConText assertion modifier
nlp.add_pipe("medspacy_sectionizer")
nlp.add_pipe("medspacy_context")

In this hybrid architecture, scispaCy identifies the `DISEASE` and `CHEMICAL` spans and resolves each mention to its exact UMLS `CUI` (`ent._.kb_ents`), while medspaCy identifies which EHR note section the entity appeared in (`ent._.section_category`, such as `family_history` or `past_medical_history`) and applies the `ConText` algorithm to set boolean assertion flags (`ent._.is_negated`, `ent._.is_uncertain`, `ent._.is_historical`, `ent._.is_family`), ensuring only affirmed active patient conditions are written to downstream clinical registries.

Frequently Asked Questions

What is scispaCy and which biomedical NER models does it provide?

scispaCy is an open-source Python library created by the Allen Institute for AI (AllenAI) that extends spaCy for biomedical and scientific natural language processing. It provides four general biomedical pipelines (en_core_sci_sm, en_core_sci_md, en_core_sci_lg, and transformer-based en_core_sci_scibert) that detect generic biomedical entity spans, plus four specialized NER models: en_ner_bc5cdr_md (diseases and chemicals), en_ner_craft_md, en_ner_jnlpba_md, and en_ner_bionlp13cg_md (16 cancer genetics entity types).

How does the scispaCy EntityLinker map text spans to UMLS, MeSH, RxNorm, and HPO?

After optionally expanding local acronyms using the Schwartz-Hearst AbbreviationDetector, the scispaCy EntityLinker encodes each entity span into a character 3-gram TF-IDF vector and performs an approximate nearest-neighbor search (`nmslib`) against pre-indexed concept aliases and definitions in UMLS (`umls`), Medical Subject Headings (`mesh`), RxNorm (`rxnorm`), Gene Ontology (`go`), or the Human Phenotype Ontology (`hpo`), returning ranked concept IDs and similarity scores in ent._.kb_ents.

How do scispaCy and MedSpaCy work together for clinical note extraction?

Because both libraries are native spaCy pipeline extensions, developers load a scispaCy model (such as en_ner_bc5cdr_md or en_core_sci_scibert) with scispacy_linker to perform biomedical tokenization, NER, and UMLS CUI normalization, and then append medspacy_sectionizer and medspacy_context to the same pipeline to classify clinical note sections and filter out negated (`is_negated`), uncertain, historical, or family-history mentions.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about scispaCy Biomedical NLP: Core Pipelines, Specialized NER Models, UMLS EntityLinker, and medspaCy Workflows

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →