Direct comparison
HDF5 vs Zarr vs Parquet Compared
HDF5, Zarr, and Parquet compared: N-D arrays vs columnar data, local vs cloud storage, and single- vs parallel-write workloads.
Written and maintained by CASRAI Editorial Board
Last updated
Ask CASRAI · free to try
Ask about HDF5 vs Zarr vs Parquet Compared
Ask your first 2 questions free below. Subscribers get 150 a day for $29 a month.
An AI assistant specialized in research administration. It cites the sources behind every answer, labels web answers and says when it can't answer.
Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.
Works on this site and inside Claude, Cursor and the AI tools you already use.
Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays free to read, with no account and no card.
How do HDF5, Zarr, Parquet compare side by side?
The table below compares HDF5, Zarr, Parquet across 12 procurement-relevant dimensions, from data model through weakest fit.
Side-by-side comparison
| Dimension | HDF5 | Zarr | Parquet |
|---|---|---|---|
| Data model | Hierarchical, self-describing container: groups (like folders) holding typed N-dimensional datasets and attributes anywhere in the tree | Chunked N-dimensional array format with an optional group hierarchy; no single-file container | Columnar tabular format: rows grouped into row groups, stored column-by-column within each |
| Primary data shape | Large dense N-D numeric arrays and heterogeneous, deeply nested scientific data | Large N-D numeric arrays needing parallel or cloud-streamed read/write | Tabular / dataframe-shaped data -- event logs, transaction records, feature tables |
| Physical storage | Single binary file by default (B-tree indexed internally); can span family/external files | Each chunk is its own object/key -- directory of files locally, or native S3/GCS/Azure Blob objects | A file, or a set of files across a partitioned directory, commonly stored directly in object storage |
| Chunking | Native per-dataset chunked storage with a configurable chunk cache | Core design primitive -- array split into regular chunks up front, each independently compressed | Row groups plus per-column pages act as the internal chunking unit; enables column/row-group pruning |
| Compression | Built-in filters (gzip/deflate, szip) plus third-party filter plugins (e.g. Blosc) via the HDF5 filter pipeline | Pluggable codecs via numcodecs (Blosc, Zstandard, LZ4, gzip), configured per array | Per-column compression and encoding (Snappy, gzip, Zstandard, dictionary/run-length encoding) |
| Best storage target | Local or networked POSIX filesystem; cloud object storage is workable but awkward | Built for cloud/object storage from the ground up; equally usable on a local filesystem | Cloud object storage and distributed filesystems -- the standard data-lake format |
| Parallel / concurrent writes | SWMR mode (added in 1.10, 2016) allows one writer with concurrent readers; true multi-writer needs Parallel HDF5 (MPI-IO) into one shared file | Different processes can write different chunks of the same array concurrently with no shared file lock | Parallel workers typically write separate files/partitions rather than one shared file, sidestepping single-file contention |
| Concurrency / corruption risk | Uncoordinated concurrent writes to a plain, non-SWMR file risk corrupting it | Low, since each chunk is an independent object; two writers to the identical chunk still need external coordination | Low, since files are treated as immutable once written and any number of readers can read concurrently |
| Typical ecosystem | h5py, PyTables, native C/Fortran/Java/MATLAB APIs; NetCDF4 and MATLAB .mat v7.3 are built on it | zarr-python, Xarray, Dask, TensorStore | Apache Arrow, Spark, DuckDB, Pandas, Polars, Impala, cuDF |
| Random access pattern | Efficient hyperslab (sub-array) reads out of a large chunked dataset without loading the whole file | Reads only chunks intersecting the requested slice, over range requests against object storage | Efficient column pruning and row-group skipping via predicate pushdown; not built for N-D slicing |
| Maturity / origin | HDF Group; format dates to 1998, successor to HDF4 -- one of the most mature scientific-data formats | Community format from the mid-2010s scientific Python/bio-imaging world; v3 core spec ratified May 2023 | Originated 2013 at Twitter/Cloudera, drawing on Google’s Dremel paper; now an Apache top-level project |
| Weakest fit | Many workers writing the same array concurrently, or storing it as many small cloud objects | A single small array that fits in memory, where per-chunk object overhead outweighs the parallel-I/O benefit | A large dense N-dimensional numeric array (imaging volume, climate grid) -- loses native multi-D slicing |
Common questions
Common questions about HDF5 vs Zarr vs Parquet
Which format should I use for large N-dimensional scientific arrays?
+
HDF5 or Zarr, not Parquet. Choose HDF5 for a single-writer workflow on local or networked POSIX storage where mature tooling matters. Choose Zarr when the array needs parallel writers or lives in cloud object storage.
Can I use Parquet for array data?
+
Only by flattening the array into columns, which loses the native multi-dimensional chunking and slicing that HDF5 and Zarr provide. Parquet is built for tabular data, not dense N-dimensional arrays.
Is Zarr a replacement for HDF5?
+
Not a strict replacement -- the two overlap in what they can store, but Zarr specifically targets parallel, cloud-native workloads, while HDF5’s strength is a mature, self-describing single-file format for single-writer local workflows.
Which of the three is safest for concurrent writes?
+
Zarr (chunk-per-object) and Parquet (partition-per-file) both avoid the single shared-file contention that makes concurrent HDF5 writes fragile without SWMR mode or a Parallel HDF5 (MPI-IO) build.
Can I convert between these formats?
+
Yes. Xarray can read and write both HDF5- and Zarr-backed arrays through a common interface, and Arrow-based tooling reads and writes Parquet from most dataframe libraries.
Going deeper
