When One Bundle Stops Being Enough
FHIR's REST API is built around one patient at a time. Search for a patient's conditions, read an encounter, post an observation — each interaction is small, synchronous, and returns a Bundle. That model is a good fit for a clinician looking at a chart and a terrible fit for a question like "give me every Observation for the 40,000 patients in this quality measure population."
Bulk Data Access — informally "Flat FHIR" — is the specification written for that second question. It is an HL7 implementation guide, published separately from the core specification and built on the FHIR Asynchronous Request Pattern. It defines a $export operation that kicks off a long-running job, a polling protocol to track it, a JSON manifest describing the results, and an output format that is deliberately not a Bundle: newline-delimited JSON, one resource per line.
This guide walks through the whole flow — the kick-off, the parameters worth knowing, the polling states, the manifest fields, and the practical business of handling files that may be very large. Once you have NDJSON on disk you can open it directly in the FHIR Bundle Explorer, which auto-detects NDJSON alongside Bundles and bare resources.
Three Roles, One Flow
The implementation guide separates the provider side into three components, and knowing which is which explains several otherwise-odd details. There is a FHIR Authorization Server that issues access tokens; a FHIR Resource Server that accepts the kick-off request and provides job status and the completion manifest; and an Output File Server that serves the actual data files. The file server may be part of the FHIR server or entirely separate — very often it is object storage with signed URLs. On the other side is the Bulk Data Client, which requests tokens and downloads files.
Authorization is expected to follow the SMART Backend Services Authorization profile: a client registers, then presents a signed token request and receives a short-lived access token. This is machine-to-machine authorization with no user in the loop, which is the correct model for a nightly analytics extract. The guide requires TLS 1.2 or later for every exchange described.
The Kick-Off Request
An export begins with a single request to one of three endpoints, which differ only in scope:
[fhir base]/Patient/$export— all patients the client is authorized to see.[fhir base]/Group/[id]/$export— the members of a defined Group. How Groups are defined is left to each implementation; a payer roster imported into an EHR is a typical example.[fhir base]/$export— a system-level export, including data not associated with any patient. This supports use cases such as backing up a server or extracting terminology.
A server SHALL support GET for these and MAY support POST supplying parameters in a Parameters resource. Two headers matter. Accept specifies the format of the optional OperationOutcome in the kick-off response; currently only application/fhir+json is supported. Prefer should carry respond-async, telling the server to process asynchronously — if omitted, the server may return an error or may proceed as though it had been supplied.
Newer versions of the guide add an optional second Prefer value, separate-export-status. Without it, the HTTP status of a status request reflects the export job. With it, the status request returns its own HTTP status and the job's status appears in an X-Export-Status header. This exists because intermediaries and client libraries frequently mangle a 202 that is really about a job rather than about the request.
Parameters Worth Knowing
The kick-off accepts a set of query parameters, of which a handful do most of the work in practice:
| Parameter | What it does |
|---|---|
_outputFormat | Format of the generated files. Defaults to application/fhir+ndjson. Servers SHALL support NDJSON and SHALL also accept the abbreviations application/ndjson and ndjson. |
_since | A FHIR instant. Include resources whose state changed after this time — typically compared against Resource.meta.lastUpdated. The basis of incremental extracts. |
_type | Comma-delimited resource types to restrict the export to. Omit it and the server returns everything within the client's authorization. |
_typeFilter | A FHIR search query applied per resource type, e.g. MedicationRequest?status=active. May be repeated; multiple filters for the same type are OR'd together. |
_elements | Marked experimental. Asks the server to omit unlisted non-mandatory elements. Servers SHOULD tag the results as SUBSETTED. |
includeAssociatedData | Marked experimental. Controls inclusion of associated Provenance resources via values such as LatestProvenanceResources. |
Two cautions from the guide are worth internalising. First, support for _typeFilter is optional, and clients SHOULD be robust to servers that ignore it — never assume filtering happened. Second, use _typeFilter rather than _since when you want to filter by clinical dates: _since works on the resource modification time, which may bear no relationship to when the clinical event occurred. An encounter from 2019 edited last week has a recent meta.lastUpdated.
Servers that cannot support a requested parameter SHOULD return an error with an OperationOutcome so the client can resubmit without it. A Prefer: handling=lenient header asks the server to process the request anyway rather than failing.
The Asynchronous Handshake
A successful kick-off returns 202 Accepted with a Content-Location header holding the absolute URL of a status endpoint — the polling location. The body may optionally carry an OperationOutcome. Everything that follows happens against that polling URL.
The client then polls with GET [polling content location], using an Accept header of application/json. Three outcomes are possible:
- In progress — 202 Accepted, optionally with an
X-Progressheader carrying a free-text status under 100 characters ("50% complete", "in progress") and aRetry-Afterheader. - Error — a 4xx or 5xx status, with an OperationOutcome in the body under
application/fhir+json. - Complete — 200 OK,
Content-Type: application/json, anExpiresheader indicating when the files stop being available, and the output manifest in the body.
Polling etiquette is specified, not merely suggested. Clients SHOULD use exponential backoff. Servers SHOULD supply Retry-After as either a delay in seconds or an HTTP date, and clients SHOULD honour it. A server that sees a client polling too frequently SHOULD respond 429 Too Many Requests with a Retry-After, and if it persists MAY terminate the session. A tight polling loop is not just impolite; it can get your export cancelled.
A client may also send DELETE [polling content location] to cancel an in-flight export, or to signal after downloading that the server may clean up the files. The server responds 202 Accepted, and subsequent requests to the polling location SHALL return 404 with an OperationOutcome.
One subtlety about partial failure: even if some requested resources cannot be exported, the overall operation MAY still succeed. In that case the server uses a 200 status and populates the manifest's error array with files describing what went wrong. Where to draw the line between partial success and outright failure is left to the implementer, so a 200 does not by itself mean everything you asked for is present.
The Output Manifest
The manifest is a plain JSON object — not a FHIR resource — and it is the map to your data. Its fields:
transactionTime(required) — the server's time when the query was run. The response should include no resources modified after this instant, and SHALL include matching resources modified up to and including it. This is the value you retain and feed back as_sinceon your next incremental run.request(required) — the full URL of the original kick-off request.requiresAccessToken(required) — whether downloading the files needs the same authorization as$exportitself. It istruewhen both servers use OAuth 2.0 bearer tokens, and MAY befalsefor file servers using other schemes, such as signed object-storage URLs. When it isfalse, a client SHALL NOT send its access token to those URLs.output(required) — an array of file items, one per generated file, each with aurl, atypenaming the FHIR resource type in the file, and an optionalcountof resources. If nothing matched, the server SHOULD return an empty array.deleted(optional) — files listing resources deleted since the_sincetimestamp. Each line is a transaction Bundle whose entries carryrequest.methodofDELETEand arequest.url. Resources appearing here SHALL NOT also appear inoutput.error(required) — files of OperationOutcome resources describing errors, warnings, and informational messages. An empty array when there is nothing to report.extension(optional) — a reserved object for server-specific extras, such as a decryption key for encrypted output.
The deleted array is the piece most first implementations miss. Without processing it, an incremental pipeline accumulates records that no longer exist upstream — patients merged, observations entered in error — and your warehouse slowly diverges from the source of truth in a way that is very hard to detect after the fact.
Downloading the Files
Files are fetched with GET against the URLs in the manifest, within the window given by the Expires header. If requiresAccessToken is true, the request must carry a valid access token. The Accept header is optional and defaults to application/fhir+ndjson; a successful response carries a Content-Type matching the delivered format, which for NDJSON SHALL be application/fhir+ndjson.
Compression is worth doing. Clients SHOULD send Accept-Encoding including gzip, and servers SHALL provide files uncompressed, gzipped, or in another format from that header, signalling the choice via Content-Encoding. FHIR JSON is extremely repetitive — the same element names on every line — so gzip typically achieves a large reduction, and on a multi-gigabyte export that is the difference between a transfer that finishes and one that times out.
Exported data includes only the most recent version of each resource unless the client explicitly asks otherwise in a way the server supports. If your use case needs history, bulk export is the wrong instrument.
Why NDJSON Instead of a Bundle?
This is the design decision that defines the whole specification, and it comes down to memory.
A Bundle is a single JSON document. To read the last entry you must have parsed the whole thing, which means a conventional parser holds the entire export in memory at once. That is perfectly reasonable for a page of search results and impossible for a hundred million observations. Worse, it is all-or-nothing: one malformed byte anywhere in the document and the entire parse fails, taking every valid resource with it.
NDJSON inverts both properties. Each line is a complete, independent JSON document — one FHIR resource, no wrapper, no commas between records. A consumer reads a line, parses it, processes it, discards it, and moves on, at constant memory regardless of file size. A corrupt line costs you that line. Files can be split and processed in parallel by simply cutting on newlines, which is why every large-scale data tool from jq to Spark to BigQuery ingests NDJSON natively. And appending is trivial, so a server can stream output as it produces it.
The trade-off is real and worth stating: NDJSON has no envelope. There is no total, no paging links, no Bundle.type, and critically no fullUrl per entry. All that context moves out into the manifest and the file naming. It also means references resolve differently — with no fullUrl, a relative reference like Patient/123 must be interpreted against the exporting server's base rather than against per-entry metadata. If that distinction is unfamiliar, How FHIR References Work covers the resolution rules in full, and What Is a FHIR Bundle? covers the envelope that NDJSON deliberately discards.
NDJSON itself is not a FHIR invention. It is a simple community convention — each line is a valid JSON value, lines are separated by \n, and the text is UTF-8 — which FHIR adopted with its own media type, application/fhir+ndjson, rather than reinventing.
Handling Large Exports in Practice
A few habits separate pipelines that survive contact with a real health system from ones that do not.
- Never load a whole file into memory. Stream by line. The format was chosen precisely to make this possible; using it with a whole-file JSON parser throws away its only advantage.
- Scope the request. Use
_typeto ask only for what you need. Exporting everything and discarding most of it wastes hours of server time and your bandwidth, and Bulk Export is explicitly a resource-intensive operation that servers may throttle. - Retain
transactionTime. It is the correct watermark for the next incremental run. Do not use your own clock — clock skew and long-running jobs will create gaps. - Process the
deletedarray. Otherwise your copy drifts permanently out of sync. - Download within the
Expireswindow, and re-fetch the manifest if links have expired — servers may issue fresh URLs and a new expiry. - Treat files as immutable once retrieved. The guide states that output files SHALL NOT be altered once included in a returned manifest, which makes them safe to checksum and cache.
- Sample before you build. Take the first few hundred lines of one file and inspect the actual shape of the data. Which elements are populated, which codings are used, how references are written — these vary enormously between implementations and are far cheaper to discover now than after the transform is written.
Inspecting a Sample Without Uploading PHI
That last habit is where a local explorer earns its place. The FHIR Bundle Explorer auto-detects NDJSON, so a slice of an export file can be pasted straight in with no conversion. It reports a count per resource type, builds a reference map, and flattens any chosen resource type into a table you can export as CSV — with repeated values either joined into one cell with a semicolon or expanded to one row per repeat, so nothing is silently dropped. Paths are shown without array indices, so name[0].given[0] and name[1].given[0] both land in a single name.given column.
It is bounded on purpose: 25 MB of input and 20,000 resources, and when a cap is reached it reports exactly how many resources were not parsed rather than quietly truncating. A bad line in NDJSON becomes a reported issue on that line number instead of discarding the other 19,999 good ones. References are resolved only against what you pasted — never by fetching anything from the network, because export files are patient data. Everything stays in your browser. It is an exploration tool, not a validator; for conformance checking, use the FHIR Resource Validator.
Conclusion
Bulk Data Export exists because population-scale data movement has different constraints from chart-level access. The flow is consistent and worth memorising: authorize via SMART Backend Services, kick off $export at the system, patient, or group level with Prefer: respond-async, take the polling URL from Content-Location, poll with backoff until 200 OK, read the manifest, and download the files it lists — honouring requiresAccessToken, processing deleted, and retaining transactionTime for next time.
The output is NDJSON rather than a Bundle for one reason that outweighs all others: streaming. One resource per line means constant memory, parallel processing, and graceful degradation on a corrupt record — properties no single-document format can offer. Understand what the envelope gave you and where that context moved to, sample your files locally in the FHIR Bundle Explorer before you build the transform, and a bulk export becomes a routine batch job rather than a research project.