The Earth System Grid Federation (ESGF) is the international, federated data infrastructure through which the climate modelling community distributes and provides access to output from the Coupled Model Intercomparison Project (CMIP) — the coordinated set of standardized climate model experiments run by modelling centers around the world and organized under the World Climate Research Programme (WCRP). ESGF is not a single website or database. It is a network of independently operated, interoperating data and index nodes hosted by national laboratories, universities, and research computing centers on several continents, which together let a researcher search across, and download from, the full body of CMIP model output regardless of which participating institution originally produced or hosts a given dataset.
For research data managers and administrators supporting climate scientists, ESGF matters for three practical reasons: it is effectively the only route to primary CMIP model output, its holdings underpin the Intergovernmental Panel on Climate Change (IPCC) assessment reports, and it presents genuinely unusual data management demands — multi-petabyte volumes, distributed replication, and citation practices — that differ from a typical institutional data repository.
What ESGF is and why it exists
ESGF was created to solve a specific coordination problem: CMIP experiments involve dozens of independent modelling centers, each running large global climate simulations and producing model output on the order of terabytes to petabytes per model. No single institution has the funding, storage, or mandate to host all of it centrally. ESGF’s solution is federation — each contributing center (or a partner data center acting on its behalf) publishes its own model output to its own ESGF node, and the nodes exchange metadata about their holdings so the whole federation can be searched as if it were one archive. A user does not need to know, or care, which physical node in Germany, the UK, Australia, or the United States actually stores a given file; they search once and are directed to a node that holds (or replicates) it.
ESGF is operated as a collaboration among government agencies, national laboratories, and research computing centers rather than by a single owner. Contributing and hosting organizations include the U.S. Department of Energy’s Program for Climate Model Diagnosis and Intercomparison (PCMDI) at Lawrence Livermore National Laboratory, NASA, and international partners such as the German Climate Computing Centre (DKRZ), the UK’s Centre for Environmental Data Analysis (CEDA), and Australia’s National Computational Infrastructure (NCI), among others. This distributed governance mirrors the distributed architecture: no single node going offline should, in principle, take the whole federation down, since actively-replicated datasets remain reachable through other nodes.
ESGF’s role in CMIP and IPCC assessment reports
CMIP is organized in phases (CMIP5, CMIP6, and the CMIP7 cycle now underway), each defined by a common experimental protocol that lets modelling centers’ simulations be directly compared against one another. ESGF is the designated distribution channel for CMIP output: the CMIP data request specifies what each contributing model should produce, and ESGF is where that output is actually published, indexed, and made downloadable once produced. Because CMIP participation and ESGF publication are effectively coupled, ESGF’s holdings represent the standardized, comparable, multi-model ensemble that distinguishes CMIP data from a single institution’s own climate model runs.
This matters beyond climate modelling circles because CMIP output, distributed through ESGF, is the primary data source the IPCC’s Working Group I assessments draw on for projections of future climate under different emissions scenarios, and for evaluating how well current-generation models reproduce observed climate. CMIP5 fed into IPCC’s Fifth Assessment Report (AR5); CMIP6 fed into the Sixth Assessment Report (AR6). The scale of that dependence is reflected in the data volumes involved: CMIP5 output totals more than 5 petabytes, CMIP6 exceeds 20 petabytes, and CMIP7 — with higher-resolution simulations and larger ensembles — is projected to exceed 100 petabytes. For a research administrator, the practical implication is that any project working with CMIP-derived projections, whether for downstream climate-impact research, agricultural modelling, or policy analysis, is almost certainly drawing on data that passed through ESGF at some point in its provenance chain, even if the researcher accessed a repackaged or subset version through another platform.
How researchers access ESGF data
Researchers do not need an account with a single central authority to browse ESGF holdings, but downloading generally requires registering for federated access credentials (an OpenID-based identity recognized across ESGF nodes) so that node-to-node authentication and, where relevant, group-based access controls work consistently across the federation. The practical access routes are:
- Web search portals. ESGF’s current primary search interface (Metagrid, which has been superseding the older CoG portal) lets a user filter CMIP holdings by facets such as MIP era (CMIP5/CMIP6/CMIP7), activity, institution ID, source (model) ID, experiment ID, variant label, table ID, variable, and grid label, then browse or download matching datasets.
- Bulk download tools. For anything beyond a handful of files, most users generate a Wget script from the search interface to pull an entire matched dataset, or use a dedicated download client such as Synda, built specifically for large, resumable ESGF transfers.
- Globus. For very large transfers, several ESGF nodes support Globus-managed transfer, which handles the reliability and throughput problems that plain HTTP/Wget downloads run into at multi-terabyte scale. (See CASRAI’s guide to Globus data transfer for how the underlying GridFTP-based transfer service works more generally in research computing.)
- Programmatic and cloud-native access. ESGF exposes OPeNDAP endpoints for subsetting data without a full download, and CMIP6 output has been made available through a Pangeo-coordinated cloud mirror on major cloud object storage, letting researchers using Python tools such as xarray and intake-esm query and load only the variables and time slices they need rather than downloading whole files. Several national data centers — including CEDA/JASMIN in the UK, NCAR’s GLADE in the US, and DKRZ in Germany — also provide direct, in-place access to CMIP holdings alongside their own compute infrastructure, avoiding a download step entirely for users with accounts there.
- Errata and version tracking. Because CMIP datasets are occasionally corrected or reissued, ESGF maintains an Errata service that lets users check whether a specific dataset version has known issues, retractions, or superseding versions before relying on it — a step worth building into any data management plan that cites specific CMIP dataset versions.
A next-generation rebuild of the federation’s core infrastructure, referred to as ESGF2, has been under active development to modernize node software, authentication, and search as data volumes scale toward the CMIP7-era projections described above; researchers should expect access mechanics (portal names, auth flow specifics) to continue evolving and should consult ESGF’s own current documentation rather than treating any single tool name as permanent.
Data management considerations for researchers and institutions using ESGF
- Cite the specific dataset version, not just “CMIP6 data.” Because ESGF datasets are versioned and can be revised, reproducible research depends on recording the exact version identifier (and, where available, the dataset’s persistent identifier) used in an analysis — the same discipline CASRAI’s guide on how to cite data recommends generally, applied to a case where re-running an analysis against a later, corrected version can change results.
- Plan storage and transfer capacity realistically. A single CMIP6 experiment’s output for one model can run into hundreds of gigabytes to terabytes; institutions supporting climate researchers should budget local storage, network egress, and compute-adjacent staging space accordingly rather than assuming ESGF downloads behave like a typical repository deposit.
- Treat ESGF as a source, not a long-term dependency for downstream work. Researchers building derived products or analyses from CMIP data should still document their own provenance and retention plan for derived datasets — consistent with a standard data management plan — rather than relying on being able to re-fetch the identical inputs from ESGF indefinitely, since node availability and hosting arrangements can change.
- ESGF holdings generally meet FAIR expectations at the data level, but access still requires federation credentials. CMIP data is openly available in the sense that it is not paywalled, but “findable” and “accessible” (per the FAIR Data Principles) still assume a user going through ESGF’s registration and node-navigation process; institutions supporting new climate researchers should budget onboarding time for this rather than assuming access is as frictionless as a plain public URL.
Frequently asked questions
Is ESGF the same thing as CMIP?
No. CMIP is the coordinated climate modelling experiment protocol and the multi-model intercomparison project itself, organized under the World Climate Research Programme. ESGF is the data infrastructure through which CMIP’s output is distributed. CMIP defines what gets simulated and how experiments are designed to be comparable; ESGF is where the resulting data is published, searched, and downloaded.
Do I need an account to download CMIP data from ESGF?
Generally yes. While metadata and search are broadly browsable, downloading data typically requires registering for ESGF federated access credentials, which are then recognized across participating nodes rather than requiring separate accounts at each institution.
Why does ESGF matter for IPCC assessment reports?
The IPCC’s Working Group I assessments rely heavily on CMIP model output — CMIP5 for the Fifth Assessment Report and CMIP6 for the Sixth Assessment Report — for climate projections under different scenarios and for evaluating model performance against observations. ESGF is the channel through which that underlying model output is made available to the broader research community, including the researchers who author IPCC assessment chapters and the wider community that scrutinizes and builds on their analyses.
How large is the CMIP data distributed through ESGF?
CMIP5 output totals more than 5 petabytes; CMIP6 exceeds 20 petabytes; CMIP7, with higher-resolution simulations and larger ensembles, is projected to exceed 100 petabytes. These figures reflect the aggregate federation-wide holdings across all contributing modelling centers, not a single dataset.
What is the difference between an ESGF data node and an index node?
A data node actually hosts files (either original data published by a modelling center, or a replicated copy of another node’s data). An index node catalogs metadata about datasets — potentially including datasets physically hosted elsewhere in the federation — so that a search against one index node can surface results from across the network. Some nodes combine both roles.
Related CASRAI resources
- Globus Data Transfer — how large-scale research data transfer works, relevant to moving multi-terabyte CMIP downloads.
- NOAA NCEI: Accessing the National Centers for Environmental Information’s Climate and Ocean Data — a complementary observational (rather than model-output) climate and ocean data source.
- FAIR Data Principles — the findability/accessibility/interoperability/reusability framework relevant to evaluating ESGF’s own data practices.
- Data Management Plan (DMP) — foundational definition, relevant to planning storage and provenance for CMIP-derived analyses.
- How to Cite Data: Formats and the Data Citation Principles — applies directly to citing specific ESGF/CMIP dataset versions.
- Types of Research Data: A Taxonomy for Data Management Planning — where large-scale simulation output like CMIP model data fits within a broader research data taxonomy.
