Bulk FHIR (Flat FHIR): Asynchronous $export Operation, NDJSON Format, and Backend OAuth 2.0

Bulk FHIR (formally the HL7 FHIR Bulk Data Access Implementation Guide, also known as Flat FHIR) is a standardized, asynchronous application programming interface (API) specification for exporting large, population-level clinical datasets from an electronic health record (EHR) or FHIR server in Newline Delimited JSON (NDJSON) format. Mandated for certified U.S. health IT systems under ONC regulation 45 CFR § 170.315(g)(10), Bulk FHIR enables health systems, payers, and clinical researchers to extract records for thousands or millions of patients without overloading transactional clinical databases.

What Is Bulk FHIR (Flat FHIR)?

Standard synchronous HL7 FHIR RESTful APIs (such as GET /Patient/[id] or paginated FHIR Bundle searchsets) are optimized for interactive, single-patient workflows. When an analytics pipeline, value-based care registry, or clinical research repository attempts to extract clinical histories for a cohort of 100,000 patients using synchronous REST calls, millions of paginated HTTP requests quickly exhaust API rate limits and degrade EHR responsiveness.

The HL7 Bulk Data Access Implementation Guide (hl7.org/fhir/uv/bulkdata) solves this bottleneck through three architectural design choices:

• Three Export Scopes: Supports system-level export of all data on a server (GET [fhir base]/$export), all patients in a defined patient directory (GET [fhir base]/Patient/$export), or a specific research cohort/attributed payer roster defined by a FHIR Group resource (GET [fhir base]/Group/[id]/$export).
• Asynchronous Batch Execution: Decouples request initiation from file generation so the EHR can schedule heavy database queries against read replicas or background worker pools.
• SMART Backend Services Authorization: Authenticates autonomous server-to-server pipelines using OAuth 2.0 client_credentials grants signed with asymmetric JSON Web Tokens (RS384 or ES384) and system/*.read scopes (ONC § 170.315(g)(10)).

How Does the FHIR Asynchronous $export Operation Work?

A Bulk FHIR data extraction follows a four-stage asynchronous HTTP state machine across platforms such as Epic on FHIR, Oracle Health, and AWS HealthLake:

Stage	HTTP Request / Headers	Server Response	Protocol Mechanics
1. Kick-Off Request	GET /Group/[id]/$export?_type=Patient,Observation&_since=2026-01-01T00:00:00Z Accept: application/fhir+json Prefer: respond-async	202 Accepted Content-Location: https://ehr.example.org/jobs/98412	Client specifies optional parameters (_type, _since, _typeFilter). Server validates OAuth 2.0 token and queues background job.
2. Status Polling	GET https://ehr.example.org/jobs/98412 Authorization: Bearer [token]	While running: 202 Accepted with X-Progress: "45% complete" and Retry-After: 120	Client polls the Content-Location URL using exponential backoff or the server’s Retry-After interval (or sends DELETE to cancel).
3. Completion Manifest	GET https://ehr.example.org/jobs/98412	When finished: 200 OK returning a JSON completion manifest	The JSON manifest lists transactionTime, requiresAccessToken, an output array of NDJSON file URLs grouped by FHIR resource type, and an error array of OperationOutcome NDJSON files.
4. File Download	GET https://storage.example.org/export/Observation-001.ndjson Accept: application/fhir+ndjson	200 OK streaming application/fhir+ndjson	Client downloads one or more partitioned NDJSON files per resource type in parallel.

Why Is NDJSON Used for Bulk FHIR Clinical Data Exports?

In standard synchronous FHIR exchanges, multiple resources are wrapped inside a single root JSON object (a Bundle resource containing an entry[] array). When a file reaches several gigabytes in size, standard JSON parsers must either load the entire document tree into RAM to validate matching brackets or rely on complex SAX-style tokenizers, and a single malformed byte invalidates the entire file.

Bulk FHIR mandates Newline Delimited JSON (NDJSON) with the MIME type application/fhir+ndjson (hence the nickname “Flat FHIR”) for four engineering reasons:

1. Constant-Memory Line-by-Line Streaming: Every line in an NDJSON file is a complete, self-contained FHIR resource serialized without embedded newline (n or r) characters, allowing ingestion pipelines to read and parse multi-terabyte files one line at a time in O(1) memory.
2. Homogeneous Resource Partitioning: Each output NDJSON file contains exactly one FHIR resource type (for example, Patient-1.ndjson, Condition-1.ndjson, Observation-1.ndjson), simplifying direct bulk loading into columnar cloud data warehouses and lakehouses (Snowflake, BigQuery, Databricks, Apache Iceberg).
3. Fault Isolation and Appendability: Writers can append new FHIR resources directly to the end of an output stream without rewriting closing JSON array tags, and a parsing error on line 50,000 does not corrupt the remaining 99,999 lines.
4. Parallel Chunking: Large tables can be split by the server into multiple numbered NDJSON files of manageable size (typically 50 MB to 500 MB per file) listed in the completion manifest’s output[] array.

Frequently Asked Questions

What is Bulk FHIR (Flat FHIR)?

Bulk FHIR (Flat FHIR) is the HL7 FHIR Bulk Data Access standard that defines an asynchronous API operation ($export) and Newline Delimited JSON (NDJSON) file format for extracting population-level clinical datasets for thousands of patients at once.

How does the FHIR asynchronous $export operation work?

A client sends a GET request to /$export, /Patient/$export, or /Group/[id]/$export with the header Prefer: respond-async; the server returns HTTP 202 Accepted with a Content-Location polling URL, and once background processing finishes, the polling endpoint returns HTTP 200 OK with a JSON manifest containing download links to the generated NDJSON files.

Why is NDJSON used for Bulk FHIR clinical data exports?

NDJSON places one complete JSON FHIR resource on each line without an enclosing Bundle wrapper, enabling analytics and clinical research pipelines to stream, partition, and load multi-gigabyte exports line by line in constant memory without parsing a massive monolithic JSON document.