Data provenance is the documented history of a research data object: where it came from, what happened to it, and who or what acted on it, from collection or generation through every transformation to its current state. A provenance record is what lets someone other than the original researcher — a peer reviewer, a re-user, an auditor, or the researcher’s own future self — trust and verify a dataset without having to take its accuracy on faith.
This guide covers the operational definition, the standards used to encode provenance in a machine-readable way, how provenance differs from related concepts like data lineage and metadata, and practical approaches to capturing it across a research project’s lifecycle.
What Is Data Provenance? An Operational Definition
A dataset has documented provenance when its record answers, at minimum, four questions:
- What entity exists? The specific data object — a raw instrument output, a cleaned dataset, a derived statistic, a figure.
- What activity produced or changed it? Collection, measurement, transformation, aggregation, analysis, or any other process step.
- What agent was responsible? The person, organization, instrument, or software that carried out the activity.
- What did the activity use or derive from? The upstream entities (raw data, reference datasets, code, parameters) consumed to produce this entity.
This entity–activity–agent structure is not CASRAI’s invention — it is the core model of the W3C PROV Ontology (PROV-O), a W3C Recommendation since April 30, 2013, produced by the W3C Provenance Working Group, together with its underlying conceptual model, PROV-DM. PROV-O defines Entity (“a physical, digital, conceptual, or other kind of thing with some fixed aspects”), Activity (“something that occurs over a period of time and acts upon or with entities”), and Agent (“something that bears some form of responsibility for an activity”), connected by properties such as wasGeneratedBy, used, wasAssociatedWith, and wasDerivedFrom. Whether or not a given research group encodes provenance in formal PROV-O triples, this is the underlying question set any provenance record — a lab notebook entry, a README, a workflow-engine log, a Git commit history — is implicitly trying to answer.
Worked example
A gene-expression dataset with documented provenance would record: the raw sequencing reads (entity) generated by a specific sequencer run (activity) operated by a named technician using a named instrument (agents) on a given date; the quality-filtered reads (a new entity) produced by a named QC pipeline version (activity, an agent in software form) that used the raw reads and a specific reference genome build (upstream entities) as inputs; and the final normalized expression matrix (entity) derived from the filtered reads via a named analysis script and parameter set.
Counter-example
A dataset accompanied only by a filename like final_data_v3.csv and no record of what “v2” contained, what changed, who made the change, or what upstream files it was derived from does not have documented provenance — even if it happens to be accurate. Provenance is about the record being available and verifiable, not about the underlying data being correct.
Data Provenance vs. Data Lineage vs. Metadata
These three terms are often used loosely and interchangeably, but they answer different questions:
- Data provenance is the full history of an object — origin, custody, and every transformation, typically including the “who” and “why,” not just the “what.”
- Data lineage is a narrower, usually pipeline-oriented term (more common in data engineering than research contexts) describing the flow of data through a system — which tables, jobs, or transformations touched a field, largely for the purpose of impact analysis and debugging. Lineage is often a subset of what a full provenance record captures.
- Metadata is the broader category of “data about data” — provenance is one type of metadata (descriptive of history and process), alongside other types like descriptive metadata (title, subject, keywords), structural metadata (how a dataset’s parts relate), and administrative metadata (rights, format, preservation actions). Every provenance record is metadata; not every metadata field is provenance.
For research data specifically, provenance also has a narrower AI-specific cousin worth distinguishing from the general concept: when the object in question is a training corpus used to build a machine learning model, see CASRAI’s Training Data Provenance and AI Provenance entries, and the guide on AI training data provenance, copyright, and TDM exceptions for research — those cover copyright/text-and-data-mining questions specific to AI training data that don’t apply to provenance in the general research-data sense covered here.
Why Data Provenance Matters in Research
Provenance documentation does concrete work at several points in the research lifecycle:
- Reproducibility and verification. A result can only be independently checked if the chain from raw data to reported finding is traceable. This is the same underlying concern addressed by CASRAI’s guide on writing a data availability statement for reproducibility — a data availability statement tells a reader where the data is; provenance documentation tells them how it got to its current state.
- Trust and quality assessment. The W3C Provenance Working Group’s own framing is that provenance is “information about entities, activities, and people involved in producing a piece of data or thing, which can be used to form assessments about its quality, reliability, or trustworthiness.” A dataset with no traceable history offers a re-user nothing to assess.
- FAIR data compliance. The FAIR Guiding Principles (Findable, Accessible, Interoperable, Reusable) explicitly require rich provenance as part of “R1” reusability — a FAIR dataset should be “associated with detailed provenance.” Documenting provenance is a direct, practical step toward FAIR compliance, not a separate exercise.
- Error tracing and correction. When a downstream result turns out to be wrong, a documented provenance chain lets an investigator localize which upstream step introduced the error, rather than re-deriving the entire analysis from scratch.
- Audit and compliance obligations. Funder data management and sharing requirements, journal data-availability policies, and institutional research-integrity investigations all depend on being able to reconstruct how a dataset was produced.
Standards and Vocabularies for Encoding Provenance
Several established, machine-readable standards exist so that provenance can be shared, queried, and validated across systems rather than living only in prose:
- W3C PROV-O / PROV-DM — the general-purpose W3C Recommendation for representing provenance as Entity/Activity/Agent graphs, expressible as RDF/OWL. Domain-agnostic; widely used as the underlying model other, more specific provenance schemas build on.
- RO-Crate — a lightweight packaging format that bundles a research dataset (or workflow, or software) together with a JSON-LD metadata file built on schema.org, designed specifically to capture “who and what created or contributed to a dataset and individual files within it,” including full workflow-run provenance (inputs, outputs, computational environment, researchers involved). Increasingly used across bioinformatics, digital humanities, and regulatory science as a practical, human- and machine-readable way to ship provenance alongside the data itself rather than as a separate document.
- Workflow-engine provenance logs — tools such as Common Workflow Language (CWL) runners, Nextflow, Galaxy, and Snakemake record execution provenance (exact commands, container/environment versions, parameter values, input/output checksums) automatically as a byproduct of running a computational pipeline, which can then be exported to PROV or RO-Crate.
- Version control history — a Git (or similar) commit log is an informal but genuine provenance record for code and, when used with data-versioning tools (DVC, git-annex, or a repository’s own versioning), for data files: each commit is effectively an Activity node with a named Agent, a timestamp, and a diff against the prior Entity state.
Provenance and fixity are complementary, not the same thing: a checksum verifies a file has not changed since a known point, while provenance explains the sequence of changes that produced the file in the first place. See CASRAI’s guide on checksum verification and fixity checking for archived research data for the fixity side of that picture.
What to Capture in a Research Provenance Record
In practice, a usable provenance record — whether kept as structured metadata, a workflow log, or careful documentation in a data management plan — should capture:
- Origin — how and when the data was first collected or generated, by whom, and with what instrument, protocol, or survey design.
- Custody chain — every hand-off between people, systems, or repositories the data passed through.
- Transformations — each processing, cleaning, filtering, aggregation, or derivation step, with enough detail (software name and version, parameter values, code or script reference) that the step is reproducible in principle.
- Inputs consumed — any reference datasets, calibration files, or external resources a transformation depended on.
- Timestamps — when each step occurred, not just when the final dataset was published.
- Responsible agents — named individuals, and where relevant the software/instrument, associated with each activity — this is also where provenance and CRediT contributor roles intersect: a well-kept provenance record can substantiate who actually performed which activity on a dataset.
Practical Ways to Capture Provenance Across the Data Lifecycle
- Plan for it in the data management plan. A Data Management Plan (DMP) should specify, before data collection begins, how provenance will be tracked — not left as an afterthought once analysis is underway.
- Use electronic lab notebooks (ELNs) or workflow managers instead of relying on memory or scattered README files — both generate a timestamped, attributable record as a side effect of normal use.
- Version data and code together so a given result can be tied to an exact code state, not just an approximate description of “the analysis we ran.”
- Package data with its provenance using a format like RO-Crate at the point of deposit, rather than leaving provenance as separate, easily-lost documentation.
- Preserve intermediate files, or at minimum the exact steps and parameters needed to regenerate them, rather than only the final derived dataset.
- Record provenance at deposit, not retroactively. Reconstructing provenance after the fact is far less reliable than capturing it as work happens; repositories certified under frameworks like CoreTrustSeal generally expect provenance and integrity information to accompany a deposit, not be added later.
Frequently Asked Questions
Is data provenance the same as a data management plan?
No. A Data Management Plan is a prospective document describing how data will be handled across a project’s lifecycle, including how provenance will be tracked. Provenance itself is the retrospective (and ideally contemporaneous) record of what actually happened to a specific dataset.
What is the difference between data provenance and data lineage?
Provenance is the broader concept — the full origin-to-current-state history of a data object, typically including who was responsible and why a step was taken. Lineage is usually used more narrowly, especially in data-engineering contexts, to describe the technical flow of data through a system’s pipelines and transformations. In research data contexts the two terms overlap heavily and are often used interchangeably, but “provenance” is the more established term in the standards literature (W3C PROV, RO-Crate).
What standard should I use to document provenance?
For most research teams, the practical starting point is not writing raw PROV-O triples by hand but adopting a tool or packaging format that generates PROV-compatible or RO-Crate-compatible provenance automatically — a workflow manager, an ELN, or a repository’s own deposit process. W3C PROV-O/PROV-DM is the underlying general model; RO-Crate is a more approachable, JSON-LD-based way to package that information alongside the actual data files.
Does FAIR data require documented provenance?
Yes, in substance if not always by that exact word. The FAIR Guiding Principles’ reusability facet (R1) calls for data to be released with a clear and accessible data usage license, associated with detailed provenance, and to meet domain-relevant community standards — provenance is one of the explicit criteria under “Reusable,” not an optional extra.
Who is responsible for documenting provenance on a research project?
In practice it is shared: the researcher or technician performing a given activity is best placed to record it at the time, the project’s data management plan should specify what is tracked and how, and increasingly the tools used for collection and analysis (ELNs, workflow managers, versioned repositories) automate part of the record so it does not depend entirely on manual note-taking.
