Written and maintained by CASRAI Editorial Board
Last updated
Machine learning researchers, clinical data stewards, and AI assurance auditors implementing Datasheets for Datasets use a standardized seven-stage documentation protocol to record the provenance, demographic composition, consent boundaries, and prohibited uses of training and evaluation datasets. First proposed in 2018 and codified in Communications of the ACM (2021, Vol. 64, No. 12, pp. 86–92; DOI: 10.1145/3458723) by Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford, a datasheet for datasets functions like an electronic component operating specification—preventing downstream model failures caused by undocumented sampling bias, label noise, or distributional mismatch.
What Are Datasheets for Datasets?
In electronics engineering, every integrated circuit, sensor, or resistor ships with a standardized datasheet detailing its operating characteristics, test conditions, tolerances, and failure thresholds. By contrast, machine learning benchmarks and clinical research corpora were historically distributed as raw archive files or table dumps without standardized metadata explaining why the data were collected, who annotated the labels, or which subpopulations were excluded.
Datasheets for Datasets addresses this gap by requiring dataset creators to answer a structured set of questions spanning the full lifecycle of a dataset. Together with Model Cards and System Cards (which document trained models rather than the underlying data), datasheets form the foundational training data provenance layer recommended by the NIST AI Risk Management Framework (AI RMF 1.0).
The 7 Core Sections of a Datasheet for Datasets
The Gebru et al. (2021) specification organizes dataset documentation into seven lifecycle sections, each targeting a distinct stage of dataset creation, curation, release, and long-term stewardship:
| Section | Lifecycle Stage | Core Governance & Provenance Questions Answered |
|---|---|---|
| 1. Motivation | Project Conception & Funding | For what purpose was the dataset created? Who created the dataset (institution, research team) and who funded its creation (grant numbers, industry sponsors)? |
| 2. Composition | Instance & Cohort Characterization | What do the instances represent? How many instances exist in total and per split? Are there missing values, label errors, confidential/PHI attributes, or demographic subpopulations (age, sex, race/ethnicity, geography)? |
| 3. Collection Process | Acquisition, Sampling & Ethics | How was data acquired (direct clinical trial, EHR extraction, sensor telemetry, web scrape)? What was the sampling frame and timeframe? Who collected or crowd-annotated the data, how were they compensated, and what IRB/ethics review and informed consent governed collection? |
| 4. Preprocessing / Cleaning / Labeling | Transformation & Quality Control | What filtering, imputation, de-identification, tokenization, or normalization was applied? Is the raw uncleaned data preserved? What software and inter-rater reliability rules governed human or automated labeling? |
| 5. Uses | Task Scope & Safety Guardrails | What tasks has the dataset already been used for? What other tasks is it suitable for? Crucially, for what tasks or clinical/social contexts should the dataset not be used? |
| 6. Distribution | Licensing, Access & Export Control | Will the dataset be distributed to third parties, and how (DOI, repository, Data Use Agreement)? Under what copyright license, credentialed access tier, or export control regime is it released? |
| 7. Maintenance | Stewardship, Errata & Versioning | Who hosts and maintains the dataset? How are errata, retractions, or subject opt-out/deletion requests handled? How will older versions be archived when updates are published? |
Operational Checklist for Research & Clinical AI Teams
When preparing a dataset for public repository deposit, peer-reviewed publication, or regulatory submission (such as an FDA Software as a Medical Device [SaMD] premarket package), research data stewards can verify completeness against the following item-by-item checklist:
Section 1–2 Checklist: Motivation and Composition
- Explicit Purpose and Funding Disclosure: State the exact gap the dataset was built to fill, name all contributing laboratories or clinical sites, and list specific grant award identifiers or commercial sponsors.
- Instance Schema and Split Accounting: Define the unit of observation (e.g., individual patient encounter, 12-lead ECG waveform, histopathology whole-slide image patch), total count, and exact partition logic (ensuring patient-level grouping so multiple encounters from the same patient never leak across training, validation, and test splits).
- Subpopulation and Sensitive Attribute Audit: Report joint distributions across protected demographic strata (age, biological sex, race/ethnicity, socioeconomic indicators, and clinical site/scanner model) and explicitly state whether direct or quasi-identifiers remain present.
Section 3–4 Checklist: Collection, Ethics, and Preprocessing
- Sampling Frame and Non-Response Bias: Document inclusion and exclusion criteria, temporal collection dates, and systematically excluded populations (for example, patients lacking portal access or sensors failing on darker skin tones).
- IRB Oversight, Consent, and Annotator Compensation: Record the Institutional Review Board (IRB) approval or exemption determination, participant consent mechanism (or HIPAA waiver/de-identification pathway), and the qualifications and compensation rates of clinical adjudicators or crowdworkers.
- Reproducible Preprocessing Pipeline: Specify every outlier removal rule, missing-data imputation method, de-identification algorithm (e.g., Safe Harbor date shifting or DICOM header scrubbing), and link to the exact versioned preprocessing code.
Section 5–7 Checklist: Uses, Distribution, and Maintenance
- Out-of-Scope and Prohibited Uses: Explicitly list clinical settings, patient cohorts, or autonomous decision contexts where deploying a model trained on the dataset is unsafe or unvalidated.
- Persistent Identifier and Access Terms: Assign a persistent DOI, specify the exact license or Data Use Agreement (DUA) required (such as PhysioNet credentialed health data access), and document how downstream researchers can submit errata or verify dataset checksums across versions.
How Datasheets for Datasets Compare to Model Cards and Data Nutrition Labels
In AI assurance workflows, three complementary documentation standards operate across different layers of the machine learning stack:
- Datasheets for Datasets (Gebru et al., 2021): Comprehensive narrative and tabular provenance artifact covering the 7 lifecycle stages of a dataset (from motivation and collection to maintenance).
- Dataset Nutrition Labels (Holland et al., 2018): Modular, visual “nutrition facts” summary panels designed for rapid at-a-glance inspection of dataset summary statistics, variable distributions, and correlation alerts prior to model training.
- Model Cards (Mitchell et al., 2019): Standardized documentation for a trained machine learning model, detailing model architecture, intended clinical or operational use, evaluation metrics disaggregated across demographic groups, and ethical considerations.
