CASRAI Dictionary

Tag: ddi metadata standard

  • FAIR Data Point: Making Data Machine-Actionable

    A FAIR data point is a lightweight metadata server that exposes structured, standardised descriptions of a dataset — its identifier, creator, licence and access route — through a REST API, so software (not just people) can discover and assess it automatically. Under the GO FAIR implementation network, FAIR Data Points are the working infrastructure that turns the FAIR principles from a policy statement into a queryable service.

    In formal terms, a FAIR Data Point (FDP) is a metadata repository that follows the DCAT2 vocabulary and organises records in a fixed hierarchy — repository, catalogue, dataset, distribution — using Linked Data Platform containers, as set out in the peer-reviewed FDP specification (da Silva Santos et al., 2023, Data Intelligence, MIT Press).

    What Is a FAIR Data Point?

    A FAIR Data Point separates metadata from data. It does not host the dataset itself; it hosts a machine-readable description of the dataset, reachable at a stable HTTP endpoint. This separation is what makes the metadata queryable independently of wherever the underlying files are actually stored.

    The reference specification defines four nested layers, each exposed as its own resource:

    • Repository — the top-level FDP instance, describing the organisation or project running it
    • Catalogue — a themed grouping of related datasets
    • Dataset — the described research object, with identifier, creator, licence and rights statement
    • Distribution — the concrete access point (a download URL, an API, a query service)

    Every layer is exposed via a REST API and encoded as RDF using the DCAT2 vocabulary, which is why an FDP can be crawled and indexed by external harvesters without bespoke integration work per institution.

    How Does the GO FAIR Initiative Use FAIR Data Points?

    GO FAIR is a grassroots, community-run implementation network for the FAIR principles, not a standards body with formal ownership of a single specification. It organises its work around three self-described pillars — GO CHANGE (policy and culture), GO TRAIN (skills) and GO BUILD (technical infrastructure) — coordinated through the GO FAIR Foundation.

    FAIR Data Points sit inside the GO BUILD pillar. GO FAIR pairs FDPs with FAIR Implementation Profiles (FIPs): a documented set of choices a specific research community makes about identifiers, vocabularies, access protocols and licensing terms. The FIP tells an FDP deployment which controlled vocabularies to use at the dataset and distribution layers, so that metadata from two unrelated institutions in the same domain remains interoperable rather than merely similar.

    The combined goal is what GO FAIR calls the “Internet of FAIR Data & Services” — a distributed network of FDPs that automated agents can traverse to locate relevant data without a central index. A working example already in production is the European Joint Programme on Rare Diseases (EJP RD) Virtual Platform, whose index runs on a federated network of FDPs contributed by member registries across Europe, funded through the EU Horizon research programme.

    FAIR Data Point vs Machine-Actionable DMP: What Is the Difference?

    The two are frequently conflated because both are described as “machine-actionable,” but they describe different objects at different points in the research lifecycle. A machine-actionable Data Management Plan (maDMP) — built on the Research Data Alliance’s DMP Common Standard and served by tools such as DMPTool or DMPonline — describes intentions: what data a project will produce, where it will deposit it and under what licence. An FDP describes an already-deployed dataset that a machine can query right now.

    Aspect FAIR Data Point Machine-Actionable DMP
    Lifecycle stage Post-deposit, dataset already exists Pre-project, data not yet produced
    Governing spec GO FAIR / FDP specification (DCAT2, LDP) RDA DMP Common Standard
    Query interface REST API over a live metadata service JSON export or plan-management tool API
    Granularity Per dataset / per distribution Per project or funding award
    Typical operator Data repository or institutional archive Institution, funder, or research office

    Confusing the two leads institutions to procure the wrong tool: an maDMP platform will not make a finished dataset crawlable, and an FDP deployment will not help a project plan its future data management obligations.

    How Is FAIRness Measured? The F-UJI Evaluator

    F-UJI is an automated FAIR assessment tool developed under the Horizon 2020 FAIRsFAIR project. It scores a dataset’s exposed metadata — including metadata served by an FDP — against a fixed set of maturity indicators grouped under the four FAIR facets, returning a numeric FAIRness score rather than a binary pass/fail.

    F-UJI can only evaluate what is machine-visible: it checks whether a licence, persistent identifier or access protocol is declared in the metadata, not whether the underlying data file is actually reusable in practice. This is precisely why the metadata layer an FDP provides matters — a well-structured FDP deployment is what allows a tool like F-UJI to detect FAIRness signals automatically, while a plain data-download page with no structured metadata will score poorly regardless of how well-organised the actual dataset is.

    Where Does DDI Fit Into the FAIR Data Point Stack?

    The Data Documentation Initiative (DDI) is an XML/RDF metadata standard maintained by the DDI Alliance for describing social, behavioural and economic science data at the variable level — survey questions, coding frames, sampling design. DCAT2, the vocabulary an FDP uses by default, describes a dataset at the catalogue-entry level; it was never designed to capture variable-level detail.

    A research community whose FAIR Implementation Profile specifies DDI alongside DCAT2 gets both: FDP-level crawlability for discovery, and DDI-level granularity for reuse. Social-science archives affiliated with the Consortium of European Social Science Data Archives (CESSDA) and the UK Data Service already publish DDI metadata; wiring that metadata into an FDP endpoint is a genuine interoperability gain rather than duplicated effort.

    Frequently Asked Questions

    What is a FAIR data point?

    A FAIR Data Point is a metadata repository that exposes a dataset’s identifier, licence, creator and access route through a REST API, structured according to the DCAT2 vocabulary. It publishes metadata about data, not the data itself, so automated tools can discover and evaluate the dataset without human involvement.

    What does FAIR data mean?

    FAIR data meets the 2016 principles of Findability, Accessibility, Interoperability and Reusability, first formally published by Wilkinson et al. in Scientific Data. The principles apply to metadata as much as to the underlying files, which is why machine-readable metadata infrastructure, such as an FDP, is required to satisfy them at scale.

    What are the four pillars of the FAIR data principles?

    The four pillars are Findable (a persistent identifier and rich metadata exist), Accessible (metadata is retrievable via an open protocol, even if the data itself is restricted), Interoperable (metadata uses a shared, formal vocabulary such as DCAT2), and Reusable (a clear licence and provenance are attached).

    What This Means for Data Stewards and Developers

    Deploying a FAIR Data Point is an infrastructure decision, not a documentation exercise. In practice it requires three steps: agreeing a FAIR Implementation Profile with the relevant research community, mapping local repository metadata onto DCAT2 at the dataset and distribution layers, and registering the resulting endpoint so external harvesters and tools such as F-UJI can find it.

    • Pair persistent dataset identifiers from DataCite with the FDP’s dataset layer so citation and discovery metadata stay consistent
    • Use ROR identifiers for the institutional agent fields rather than free-text organisation names
    • Treat the FDP as complementary to, not a replacement for, an maDMP — one documents intent, the other serves the finished product

    Funders are moving in this direction: the UNESCO Recommendation on Open Science (2021) names FAIR data as a foundational pillar, and Horizon Europe grant conditions increasingly expect data to be discoverable by machines, not just listed in a repository catalogue. For institutions building research-data infrastructure now, a standards-conformant FAIR Data Point is a defensible way to demonstrate machine-actionability rather than assert it in a data management plan.

    For related definitions and terminology, see the CASRAI dictionary and the research administration pillar.

  • DDI Metadata Standard: FAIR Data Checklist for Survey Archives

    The DDI metadata standard (Data Documentation Initiative) is an international, XML-based specification for documenting surveys, censuses, and other social, behavioural, and economic science microdata at both the study and variable level. It is the metadata backbone that most social science data archives use to make survey data findable, accessible, interoperable, and reusable (FAIR) — turning a raw data file plus a PDF codebook into a machine-readable, citable, cataloguable research object.

    DDI is not a government mandate or a funder requirement; it is a community-maintained documentation standard. The DDI Alliance, an international collaboration established in 2003, maintains the specification and its schemas. This guide explains what the standard covers, who uses it, how it maps onto the FAIR principles, and the practical steps a repository or research team needs to adopt it.

    What is the DDI metadata standard?

    The Data Documentation Initiative is a metadata standard for describing the full lifecycle of a research data collection: study design, sampling, data collection, processing, variables, and access conditions. It was built specifically for social, behavioural, and economic sciences data — surveys, censuses, panel studies, and administrative microdata — rather than as a general-purpose schema.

    Records are encoded in Extensible Markup Language (XML), which makes them machine-readable and harvestable. A DDI catalogue record typically documents three layers: the study description (bibliographic citation, scope, geography, time period, methodology), the data file description (format, structure, missing-data conventions, weighting), and the variable description (question text, value labels, codes). This granularity is what separates DDI from simpler discovery schemas such as Dublin Core, which describe a resource but not its internal variable structure.

    Who maintains DDI and which archives use it?

    The DDI Alliance, an international collaboration of research institutions, statistical agencies, and data archives established in 2003, develops and maintains the specification. DDI is listed as a recognised research-data metadata standard in the Research Data Alliance Metadata Standards Catalog (entry m13), which documents its scope, schemas, and adoption.

    According to the UK Data Service, DDI “is used by most social science data archives in the world” to structure catalogue records, and it forms the basis of the discovery metadata behind its own collection. The Inter-university Consortium for Political and Social Research (ICPSR) at the University of Michigan and the members of CESSDA, the Consortium of European Social Science Data Archives, likewise build their cataloguing infrastructure on DDI, harvesting records via the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) so aggregators can index them without direct database access.

    How does DDI support the FAIR data principles?

    The FAIR Guiding Principles — findable, accessible, interoperable, reusable — were formalised for the research community in 2016. DDI operationalises each principle for survey and social science data specifically, rather than leaving them as abstract goals.

    • Findable: structured study-level metadata (title, creators, keywords, abstract, coverage) makes records indexable by catalogues and search engines, and DDI records are commonly assigned persistent identifiers, including DOIs registered through DataCite.
    • Accessible: standardised access-condition fields tell a would-be reuser exactly how to request or download the data, and harvesting via OAI-PMH gives repositories a predictable retrieval protocol.
    • Interoperable: a shared XML vocabulary and controlled thesauri — the European Language Social Science Thesaurus (ELSST), maintained by CESSDA, is one widely used example — let metadata move between archives and languages without semantic drift.
    • Reusable: variable-level documentation (question wording, value labels, derivation logic) and provenance information are what actually let a second researcher re-run or extend an analysis, which is the point FAIR exists to serve.

    DDI-Codebook vs DDI-Lifecycle vs DDI-CDI: which do you need?

    DDI is not a single schema. Three variants serve different documentation depths, and choosing the wrong one is the most common early adoption mistake.

    Variant Best for Documents Status
    DDI-Codebook (DDI-C) A single finished dataset Study, file, and variable description for one deposit Simpler, widely used legacy format
    DDI-Lifecycle (DDI-L) Longitudinal or multi-wave studies The full research lifecycle: concept, instrument, collection, processing, archiving, reuse Comprehensive, versioned in the 3.x series
    DDI-CDI (Cross-Domain Integration) Integrating structured data across statistical and research domains Model-driven descriptions that link datasets, variables, and classifications across systems Developed jointly by the DDI Alliance and the SDMX community

    A single-wave survey deposited once needs only DDI-Codebook. A cohort study revisited over years — the kind of resource the UK Data Service and ICPSR both hold in volume — needs DDI-Lifecycle to capture instrument changes between waves. DDI-CDI is aimed at repositories that need to align microdata with aggregate statistics (for example, linking a survey to official statistics published under SDMX), which is an emerging rather than default requirement.

    A practical checklist for adopting DDI

    Repositories and research teams introducing DDI documentation for the first time should work through these steps in order:

    1. Identify your lifecycle stage. A one-off dataset needs DDI-Codebook; a repeated or panel study needs DDI-Lifecycle.
    2. Model metadata before ingest, not after. Capture study description, sampling, collection dates, and variable labels/codes at deposit time using a structured deposit form, as the UK Data Service does, rather than reverse-engineering them from a finished file.
    3. Use a DDI-aware authoring tool (for example Colectica or Nesstar-derived CESSDA tooling) instead of hand-writing XML, which is error-prone at scale.
    4. Register a persistent identifier. Crosswalk core fields to the DataCite metadata schema so the dataset gets a citable DOI alongside its DDI record.
    5. Adopt a controlled vocabulary such as ELSST for subject keywords to keep records interoperable across languages and archives.
    6. Enable OAI-PMH harvesting so catalogue aggregators and search services can index the record without bespoke integration work.
    7. Validate against peer practice — check the record structure against the RDA Metadata Standards Catalog entry and against comparable ICPSR or CESSDA holdings before publishing.

    Answer-first Q&A

    What is the metadata standard DDI?

    DDI (Data Documentation Initiative) is an international metadata standard for documenting socioeconomic surveys, censuses, and microdata. It is maintained by the DDI Alliance, encoded in XML, and used by most social science data archives worldwide to capture study, file, and variable-level documentation in one structured record.

    What is the best metadata standard for survey data?

    For general resource discovery, Dublin Core (ISO 15836) is the simplest and most widely implemented option. For social science survey and microdata specifically, DDI is the domain standard, because it documents variables and methodology in a depth Dublin Core does not attempt.

    How does DDI support the FAIR data principles?

    DDI supports FAIR by pairing structured, machine-readable metadata with persistent identifiers for findability, standardised access fields for accessibility, a shared XML vocabulary and thesauri for interoperability, and variable-level provenance for reusability — the depth needed to re-run a secondary analysis.

    What is the difference between DDI-Codebook and DDI-Lifecycle?

    DDI-Codebook documents a single finished dataset. DDI-Lifecycle documents the entire research process — instrument design, fieldwork, processing, and archiving — across multiple waves, making it the correct choice for longitudinal and panel studies rather than one-off deposits.

    What this means for research data repositories

    Funder and journal data-sharing policies increasingly ask for FAIR-compliant deposits, but “FAIR” is a set of principles, not a file format. DDI is one of the few domain standards that translates those principles into a concrete, testable schema for survey and social science data — which is why it underpins the cataloguing infrastructure at the UK Data Service, ICPSR, and CESSDA member archives rather than being a niche archival choice.

    Institutions building or upgrading a research data repository for social science holdings should treat DDI-Lifecycle adoption, ELSST keywording, and DataCite DOI registration as a single connected workflow rather than three separate projects. Repositories that skip variable-level documentation still get a catalogue entry, but they do not get reuse — and reuse, not deposit, is the actual measure of FAIR success. Institutional research administration and data management guidance should reference DDI explicitly wherever survey or microdata deposit is in scope.