Checksum Verification and Fixity Checking for Archived Research Data

Fixity checking is the practice of proving, mathematically, that a digital file retrieved from a repository today is byte-for-byte identical to the file that was deposited. It is one of the least visible parts of research data archiving and one of the most load-bearing: a dataset can sit untouched for a decade and still be corrupted by a failing disk sector, a bad transfer, or a botched migration, with nothing about the file’s name, size, or folder location giving any hint that it happened. Checksum verification is the specific technical mechanism that makes fixity checking possible.

What fixity and checksum verification actually are

Fixity, in the digital-preservation sense, is the assurance that a digital object has remained unchanged — “fixed” — since a known-good point in time. It applies to any bitstream: a dataset file, a metadata record, a disk image.

A checksum (or hash) is how that assurance is actually produced. A cryptographic hash algorithm — commonly MD5, SHA-1, or SHA-256 — takes a file’s contents as input and outputs a short, fixed-length string (the “digest” or “fingerprint”), regardless of the file’s original size. The same input always produces the same output, and changing even a single bit of the file produces a completely different digest. That property is what makes the check useful: it can’t reliably tell you where in a file a change occurred, but it can tell you, with extremely high confidence, whether anything changed at all.

The verification workflow itself is simple and is the same regardless of which algorithm is used:

1. Compute a checksum for the file at the point it is considered trustworthy — typically at deposit/ingest, before it enters long-term storage.
2. Store that checksum alongside the file’s metadata as the reference value.
3. At a later point (on a schedule, or before/after any action taken on the file), recompute the checksum from the file as it currently exists.
4. Compare the new value against the stored reference value. A match confirms the file is unchanged; a mismatch confirms it has been altered or corrupted and needs to be replaced from a good copy or investigated.

This is distinct from a Data Management Plan‘s broader preservation commitment, which describes an intention to keep data available and usable over time. Fixity checking is one of the concrete technical mechanisms that makes such a commitment verifiable rather than aspirational.

Why repositories require it

Storage media degrades. Hard drives develop bad sectors, tape degrades, network transfers occasionally drop or duplicate bytes, and format migrations carried out over years of a repository’s operation introduce their own risk of silent error. None of this reliably announces itself — a corrupted file usually still opens, still has the right file size, and still has the right name. Fixity checking is the mechanism that catches “bit rot” and unauthorized or accidental modification before a researcher downloads a dataset that looks fine but silently isn’t the dataset that was originally deposited.

This is why fixity/checksum verification isn’t just good internal practice — it’s a documented requirement in the trust frameworks that govern CoreTrustSeal-certified and other trustworthy repositories:

• CoreTrustSeal‘s published requirements explicitly cover this ground twice: Requirement R7 (Data Integrity and Authenticity) requires that a repository guarantee data integrity and be able to detect unintentional or unauthorized changes, and Requirement R14 (Storage and Integrity) requires documented processes for data and metadata storage integrity — checksum verification against stored reference values is the standard way repositories satisfy both.
• The NDSA Levels of Digital Preservation matrix, maintained by the National Digital Stewardship Alliance, treats “File Fixity and Data Integrity” as one of its five core functional areas (alongside storage/geographic location, information security, metadata, and file format), with each of its four progressive levels requiring more rigorous and more frequent checksum practice than the last — from generating a checksum on ingest at the most basic level, through periodic, proactive fixity checking of content at rest at the higher levels.
• Higher tiers of the certification stack — the nestor Seal (extended-level certification) and ISO 16363 (formal-level, externally audited) — build on the same expectation with progressively more rigorous audit evidence, not a different technical requirement.

CASRAI’s guide to choosing an open data repository covers these certification tiers and what “certified” signals about a repository’s overall trustworthiness. This page covers the specific technical practice — fixity checking — that sits underneath that certification claim; a repository that can’t demonstrate real checksum verification in practice has nothing behind the certification badge.

When fixity checks should happen

Digital preservation guidance — including the Digital Preservation Coalition’s Digital Preservation Handbook and the National Digital Stewardship Alliance’s fixity guidance — converges on three points in a file’s lifecycle where a check adds real value:

• On ingest — verifying that the file received by the repository matches the checksum the depositor generated before transfer, confirming nothing was altered or dropped in transit.
• At rest — periodic re-checks of files already in storage (annually is a commonly cited baseline for tape, more frequently for spinning disk, though the right interval depends on the storage medium and the repository’s risk tolerance), to catch degradation that happens with no read or write event to trigger it.
• On retrieval — verifying the file delivered to a researcher matches the reference checksum, so a corrupted download is caught before it’s used rather than discovered later when results don’t reproduce.

Common tools and practical approaches

Fixity checking doesn’t require specialized software — the core operation is a standard cryptographic hash computation available on every major operating system — but a few tools and conventions have become the practical default in research-data and archival contexts:

• Command-line hash utilities — md5sum, sha256sum, and similar tools ship with most Unix-like systems and are frequently used for one-off or scripted checksum generation and comparison. md5deep/hashdeep extend this to recursive, bulk checking across directory trees.
• BagIt (RFC 8493) — a file-packaging convention, developed out of a Library of Congress/California Digital Library collaboration and standardized as IETF RFC 8493, that bundles a set of files (a “bag”) together with a manifest file listing every payload file’s path alongside a checksum for it. Repositories and archives use BagIt specifically so that fixity information travels with a data transfer rather than being tracked separately, and so a recipient can validate an entire deposit — not just individual files — in one operation.
• Dedicated fixity-checking tools — purpose-built tools such as AVP’s Fixity application automate scheduled, repeated checksum verification across a managed storage location and alert on mismatches, rather than requiring someone to re-run a command-line check manually.
• PREMIS metadata — the PREMIS (Preservation Metadata: Implementation Strategies) data dictionary provides a standard structure for recording an object’s fixity events — which algorithm was used, when, and the resulting value — as part of a file’s preservation metadata, rather than leaving that record ad hoc.

For a repository selecting or auditing tooling, the choice of algorithm matters less than doing the check consistently and recording the result. That said, the practical trade-off among the common algorithms is real:

• MD5 is fast to compute and widely supported, and remains adequate for detecting accidental damage such as storage corruption or a failed transfer.
• SHA-1 and, increasingly, SHA-256 are used where the checksum also needs to carry evidentiary weight — resistance to deliberate, engineered file manipulation, not just accidental change — since MD5 and SHA-1 both have known cryptographic weaknesses against a deliberate attacker with enough computing resources, even though those weaknesses are largely irrelevant to the accidental-corruption case fixity checking is usually guarding against.

Frequently asked questions

Does a checksum mismatch tell you what changed in the file?

No. A hash mismatch is a reliable signal that something changed, but the hash itself does not localize the change to a specific byte range or explain the cause. Diagnosing the cause (storage failure, transfer error, or an actual unauthorized edit) is a separate investigation; the checksum’s job is only to flag that the file no longer matches its trusted reference state.

Is fixity checking the same thing as backup?

No, and confusing the two is a real risk. A backup protects against losing a file; fixity checking detects that a file (backed up or not) has silently changed. A repository can faithfully back up a corrupted file for years without fixity checking ever being run to notice the corruption. The two practices are complementary — fixity checking tells you a copy is bad, backups (ideally in more than one location) are what let you replace it with a good one.

Do researchers depositing data need to generate their own checksums before uploading?

Practice varies by repository, but generating a checksum before transfer and providing it alongside the deposit is considered good practice by digital-preservation guidance (including the Library of Congress’s), since it lets the receiving repository verify the file arrived intact rather than trusting the transfer implicitly. Many repository deposit workflows and packaging tools such as BagIt automate this step so depositors don’t have to run a separate command manually.

Does CoreTrustSeal certification mean a repository’s fixity checking has been independently tested?

CoreTrustSeal is a peer-reviewed self-assessment, not an external audit — a repository documents its fixity/integrity processes (under Requirements R7 and R14) and community reviewers assess the self-assessment’s plausibility. Repositories seeking independently audited verification of these same practices, rather than self-assessment, pursue the higher formal-level tier (ISO 16363), which does involve an accredited external auditor. See CASRAI’s CoreTrustSeal and extended-level certification dictionary entries for how the three certification tiers relate.

Related CASRAI resources

• How to Choose an Open Data Repository — the broader trust-framework and certification-tier picture this page’s technical detail sits underneath.
• CoreTrustSeal — the core-level trustworthy-repository certification whose requirements include documented data integrity practice.
• Extended-level certification (trustworthy repository) — the nestor Seal/DIN 31644 tier above CoreTrustSeal.
• Preservation commitment (in DMP) — how a Data Management Plan’s stated preservation intent connects to concrete practices like this one.
• Data Stewardship and the Data Curator Role in Research — the operational role that typically owns fixity monitoring inside a repository or institution.
• The Research Data Lifecycle — where fixity checking fits within the archiving and long-term-retention stage of the broader lifecycle.