h5py: Read and Write HDF5 Files in Python

h5py is the Python library for reading and writing HDF5 files. If you analyse array-shaped research data in Python (instrument output, simulation results, image stacks), this guide shows you how to install it, create and read a file, store units and provenance next to the data, and avoid the common mistakes. Current release at the time of checking: 3.16.0 (PyPI, released 2026-03-06; BSD-3-Clause licence; requires Python 3.10 or later).

What h5py is used for in research

An HDF5 file is a container for two kinds of objects: datasets, which are array-like collections of data, and groups, which are folder-like containers that hold datasets and other groups. The h5py documentation sums up the model in one rule: groups work like Python dictionaries, and datasets work like NumPy arrays. Objects are arranged in a POSIX-style hierarchy with / separators, so a dataset can be addressed as /runs/run_01/temperature.

That structure suits research data for three reasons the documentation itself points to: you can read a slice of a large dataset without loading the whole file, you can compress and chunk datasets, and you can store metadata right next to the data it describes, as attributes. For a comparison of HDF5 with other array and columnar formats, see HDF5 vs Zarr vs Parquet.

Install h5py

Use the official package index or conda. The prebuilt wheels bundle HDF5, so no separate HDF5 install is needed.

pip install h5pyconda install h5py

To build from source against your own HDF5 (for example, an MPI-enabled build), the build documentation says to use pip install --no-binary=h5py h5py. Source builds need HDF5 1.10.4 or later with headers available; odd-numbered minor versions such as 1.13 are experimental. Most prebuilt packages are not built with MPI support.

Step by step: create, read and annotate an HDF5 file

The steps below follow the h5py quick start and the dataset and file reference pages.

1. Create a file. Open it in a with block so it is closed automatically:

   import h5pyimport numpy as npwith h5py.File("mytestfile.hdf5", "w") as f:    dset = f.create_dataset("mydataset", (100,), dtype="i")

2. Write data by slicing, exactly as with a NumPy array: dset[...] = np.arange(100). A partial slice such as dset[0:100:10] reads or writes only those elements.
3. Organise with groups. grp = f.create_group("subgroup") then grp.create_dataset("another_dataset", (50,), dtype="f"). Giving a full path, as in f.create_dataset("subgroup2/dataset_three", (10,), dtype="i"), creates the intermediate groups for you.
4. Attach attributes. dset.attrs["temperature"] = 99.5; test with "temperature" in dset.attrs, which returns True.
5. Reopen and read. f = h5py.File("mytestfile.hdf5", "r"), then list(f.keys()) lists the members, and dset = f["mydataset"] returns a dataset object (not yet an array). dset.shape and dset.dtype report its shape and data type; dset[...] loads the values.
6. Walk the whole file. Iterating over a group yields only its direct members; to visit every object, use the group methods visit() and visititems(), for example f.visit(print) to print each full path.

Compression, chunking and resizing

Pass chunks=(100, 100) (or chunks=True to let h5py choose) to create_dataset. Chunking is switched on automatically when you request compression or set maxshape. The documentation suggests chunk sizes between roughly 10 KiB and 1 MiB, larger for larger datasets. A chunk is read or written as a unit, so match the chunk shape to how you will slice the data.

Filter	What the h5py documentation says
compression="gzip"	Available in every HDF5 installation; good compression, moderate speed. compression_opts is an integer from 0 to 9 (default 4).
compression="lzf"	Ships with h5py; low to moderate compression, very fast; no options.

For a dataset that grows, set maxshape=(None, 10); a one-dimensional unlimited dataset needs maxshape=(None,). Resizing a dataset to a smaller shape discards the data outside the new shape.

Output and how to read it

h5py returns Python objects rather than printed output. A File and a Group behave like dictionaries (keys(), indexing by name); a Dataset reports shape, dtype and name (the full path, such as /mydataset) without reading the data. In the quick-start example, a dataset created with dtype="i" reports dtype('int32'). Values are only read into memory when you slice: dset[...] loads everything, dset[0:10] loads ten elements.

Record units and provenance as attributes

Because attributes live beside the data, they are a natural place for units, instrument settings, software versions and the processing step that produced the array. A consistent attribute convention supports reuse and the Findable/Accessible/Interoperable/Reusable aims of the FAIR data principles; h5py itself does not enforce any convention, so decide on one for your group and document it.

Troubleshooting

My data disappeared after re-running a script

Mode "w" creates the file and truncates it if it already exists. The file modes are: "r" read-only, file must exist (the default since h5py 3.0); "r+" read/write, file must exist; "w" create, truncate if exists; "w-" or "x" create, fail if exists; "a" read/write if it exists, create otherwise. Use "w-" when you must never overwrite, and "r+" or "a" to add to an existing file.

Errors after leaving the with block

Closing a file (by File.close() or by leaving the with block) makes objects obtained from it unusable. Read the values you need (dset[...]) before the file closes.

File-locking errors, for example on network drives

h5py.File accepts a locking argument (True, False, "best-effort" or None for HDF5 defaults); it needs HDF5 1.12.1 or later (or 1.10.7 or later in the 1.10 series). The HDF5_USE_FILE_LOCKING environment variable can override the parameter.

Installation or build problems

Try the prebuilt wheel first (pip install h5py). If you build from source, confirm that HDF5 1.10.4 or later and its headers are installed, and avoid experimental odd-numbered HDF5 minor versions.

Reading a file written by a newer or older HDF5

The libver argument of File sets the minimum and maximum file-format version. The default is "earliest" (best-effort backward compatibility); "latest" uses the newest structures without regard for compatibility with older HDF5 releases.

Free alternatives

h5py is one way to work with HDF5 and HDF5 is one way to store arrays. Other options in the research Python ecosystem are listed below; follow the links for the formats CASRAI covers in depth.

Option	What it is	Where to read more
PyTables	A separate Python package for working with HDF5 files	PyTables documentation (pytables.org)
NetCDF	A self-describing multidimensional data format with its own conventions, common in climate and earth science	NetCDF guide, how to open NetCDF files
Zarr and Parquet	Alternative storage formats for chunked arrays and tabular data	HDF5 vs Zarr vs Parquet

If your data is tabular rather than array-shaped, you may only need a DataFrame; see how to create a DataFrame in Python.

FAQ

What is h5py?

A Python library, distributed on PyPI under the BSD-3-Clause licence, that reads and writes HDF5 files through both a high-level interface (groups as dictionaries, datasets as NumPy-like arrays) and a low-level interface to the HDF5 C library.

Do I need to install HDF5 separately?

Not with the prebuilt wheels from pip install h5py, which bundle HDF5. You need your own HDF5 (1.10.4 or later) only when building from source.

How do I read only part of a large dataset?

Slice the dataset object, for example dset[0:100:10]. Only the requested elements are read. Chunk the dataset to match your access pattern.

Does h5py open files read-only by default?

Yes, since h5py 3.0 the default mode is "r".

Sources

• h5py documentation, Quick Start Guide: docs.h5py.org/en/stable/quick.html (read 2026-11-08)
• h5py documentation, File objects: docs.h5py.org/en/stable/high/file.html (read 2026-11-08)
• h5py documentation, Datasets: docs.h5py.org/en/stable/high/dataset.html (read 2026-11-08)
• h5py documentation, Installation: docs.h5py.org/en/stable/build.html (read 2026-11-08)
• h5py on PyPI: pypi.org/project/h5py (version 3.16.0 read 2026-11-08)