Skip to main content
v2026.11,986 entries · CC-BY 4.0

D4RL: Install, Load Datasets and Read Scores

D4RL workflows for researchers: load legacy datasets, handle episode boundaries and interpret normalized scores alongside the Minari migration.

Written and maintained by CASRAI Editorial Board

Published

On this page

D4RL is a benchmark of fixed datasets and simulated tasks for offline reinforcement learning. Researchers use it to compare learning algorithms under a shared data setting. For reproducible results, keep the legacy environment ID, dataset version, transition preprocessing and score scale together: Farama now directs ongoing development to Minari and the Gymnasium ecosystem.

What D4RL is used for in research

D4RL stands for Datasets for Deep Data-Driven Reinforcement Learning. Its research task is learning a policy from previously collected experience. The training data are fixed; collecting more experience during training changes that offline setting. The original Berkeley project explanation distinguishes this from off-policy learning with an ongoing stream of environment interactions.

The benchmark deliberately includes difficult data properties. A narrow state-action distribution leaves many decisions poorly represented. Demonstrations may depend on information or memory unavailable to the learner. Undirected trajectories can contain useful pieces of a solution without a complete successful demonstration. Suboptimal behavior provides a different challenge from imitating an expert. These distinctions explain why a dataset’s provenance matters alongside its size.

The original task families include Maze2D navigation, AntMaze locomotion and robotic manipulation. Maze2D asks a ball to navigate a maze; AntMaze adds quadruped control. Those tasks probe different difficulties, so a result on one family should retain its task identity. For the wider field, see artificial intelligence research areas.

D4RL supplies benchmark environments and experience, rather than one learning algorithm. Loading the dataset does not train a policy or produce a performance result. Treat the benchmark configuration as part of a reproducible AI experiment: report the exact task, data and transformation that produced the learner’s inputs.

How to install or access D4RL

Use the Farama Foundation repository for the legacy package. Its current README recommends moving to Minari for offline datasets and Gymnasium, MiniGrid or Gymnasium-Robotics for environments. Preserving an older experiment and beginning a new project therefore call for different installation decisions.

Install the legacy research stack

The repository documents this installation sequence:

git clone https://github.com/Farama-Foundation/d4rl.git
cd d4rl
pip install -e .

Check the package’s setup.py before choosing a Python environment. It declares Python versions at least 3.7 and below 3.11, and Gym versions below 0.24.0. Dependencies include mujoco_py, PyBullet, h5py and dm_control. These declared bounds describe the legacy package; they do not establish that every permitted combination installs successfully on every operating system.

For a reproduction, record the repository revision and the dependency versions alongside the experiment. An unpinned installation from a moving branch does not identify the exact software used in an earlier paper. Keep this legacy stack separate from a project using the newer environment API so that resolving one project’s dependencies does not silently redefine the other.

Access the maintained dataset interface

Minari’s official Basic Usage documentation gives the following commands. The dataset is the documentation’s example, rather than an asserted replacement for every legacy task:

pip install minari
minari list remote
minari download D4RL/door/human-v2
minari list local

The remote listing identifies available datasets; the local listing identifies downloads available to load. A Minari dataset name and a legacy Gym environment ID belong to different interfaces. Copy the identifier for the selected interface exactly, including its version, and record a migration as a change to the experiment rather than silently relabelling it.

How to load a D4RL dataset step by step

The legacy workflow creates a registered Gym environment, loads its associated data, and then chooses the representation expected by the learning code. The following sequence follows the repository’s Maze2D example; it requires the corresponding legacy dependencies.

  1. Register and create the task. Import gym and d4rl before creating maze2d-umaze-v1. The README notes that importing a task submodule may also be necessary. Preserve the full ID in the experiment record.
  2. Load the original arrays. Call env.get_dataset(). This is the data-loading operation; resetting the simulator or sampling an action is not a substitute for loading the offline experience.
  3. Inspect the representation. Review observations, actions, rewards and episode boundary fields before connecting the data to a learner. Keep the raw arrays available when assessing a later transformation.
  4. Choose transition preprocessing. Use d4rl.qlearning_dataset(env) when the learner expects the helper’s transition dictionary, including next_observations. Record its timeout handling with the model configuration.
import gym
import d4rl

env = gym.make('maze2d-umaze-v1')
dataset = env.get_dataset()
print(dataset['observations'])

transitions = d4rl.qlearning_dataset(env)

The documented automatic download directory is ~/.d4rl/datasets. The implementation reads D4RL_DATASET_DIR when configuring that path, so set a custom location before importing the package. Alternatively, get_dataset accepts a local HDF5 file through its h5path parameter. Its loader uses that supplied file rather than selecting a download URL.

Keep the association between the local file and environment explicit. The loader checks required arrays and their dimensions, but matching shapes alone cannot establish that a file is the intended dataset version. For inspecting the underlying storage separately, see reading HDF5 files with h5py.

Load an episode dataset through Minari

After downloading its example dataset, Minari’s documented interface loads an object rather than the legacy dictionary:

import minari

dataset = minari.load_dataset('D4RL/door/human-v2')
print('Observation space:', dataset.observation_space)
print('Action space:', dataset.action_space)
print('Total episodes:', dataset.total_episodes)
print('Total steps:', dataset.total_steps)

Minari exposes episode sampling and iteration. Its Basic Usage guide leaves replay-buffer construction to the user, so a successful load does not imply that a training library accepts the resulting object directly. Identify whether the learner consumes episodes or individual transitions before adapting its input pipeline.

Sampling also has a specific meaning here. Minari’s documented sample_episodes method samples episodes without repeating one within a single call, while an episode can reappear in a later call. set_seed controls that sampling generator. iterate_episodes provides deterministic iteration and accepts selected episode indices. Choose the operation that matches the experiment, rather than assuming repeated samples form disjoint partitions.

For an explicit partition, the Basic Usage documentation provides split_dataset with split sizes and a seed. Its sizes count episodes: nonnegative sizes are permitted, zero produces an empty split, and a total smaller than the dataset leaves episodes out. A negative size or a total exceeding the available episode count raises ValueError. Record the chosen sizes and seed when sharing a split. A reproducible partition describes which experience the learner received; it does not, by itself, establish the evaluation performance of the resulting policy.

How to read the output

The raw D4RL output describes recorded experience. observations and actions have one leading sample dimension plus their respective feature dimensions. rewards contains a reward per sample. terminals marks task termination; timeouts identifies an episode reaching its length limit. Optional task information is separate from those core arrays.

Read episode boundaries before counting transitions

The Q-learning helper is a transformation, not a second copy of the same arrays. Its implementation iterates through the first N − 1 positions and constructs each next observation from the following observation. With terminate_on_end=False, the default, it skips transitions marked as timeouts. When a timeout field is absent, it infers the boundary from the environment’s maximum episode length.

Consequently, a smaller transition count than the raw sample count can reflect documented preprocessing. Check the helper’s settings before diagnosing data loss. Its return dictionary contains observations, actions, next_observations, rewards and terminals; it does not preserve every optional raw field.

For example, a learner that needs a task-specific info array cannot assume that array survives qlearning_dataset. Retain it deliberately in the data preparation design. Likewise, do not treat a timeout as evidence of task failure merely because it ends a recorded episode: the raw format distinguishes those two boundary signals.

Distinguish returns from normalized scores

The README defines the input to env.get_normalized_score(returns) as an episode’s undiscounted total reward. The implementation computes a ratio using the environment’s lower and upper reference anchors:

normalized score = (episode return − lower reference) / (upper reference − lower reference).

The function does not multiply by 100 or clip the result. A table expressed on a scale of 100 therefore needs an explicitly stated multiplication. A value above 1 means the supplied return exceeded the upper reference under this formula; it does not mean that the function returned an invalid percentage.

Illustrative arithmetic example: maze2d-umaze-v1 has reference values 23.85 and 161.86 in infos.py. Suppose an episode had a fictional return of 92.855. Its reference span is 161.86 − 23.85 = 138.01; its excess above the lower anchor is 69.005. Dividing 69.005 by 138.01 gives 0.5. Multiplying that ratio by 100 gives 50. These are two reporting scales for the same calculation, not two measured algorithm results.

The downloadable data table below compiles the reference anchors and calculated spans for three Maze2D IDs. Use the row matching the exact task. Those constants are normalization inputs, not confidence limits, dataset reward ranges or estimates of a new policy’s performance.

Troubleshooting D4RL

Diagnose the operation that failed: dependency installation, environment registration, file loading, transformation or normalization. Each has a different contract in the repository. A successful import by itself does not prove that every environment family is available.

  • The environment is not registered. Confirm that d4rl was imported before gym.make and inspect any import warning for the required family. The package catches several environment-family ImportErrors and prints warnings. Hiding those messages with D4RL_SUPPRESS_IMPORT_ERROR does not install a missing dependency.
  • The Python environment conflicts with installation. Compare the interpreter and Gym versions with setup.py’s declared bounds. Avoid treating a modern Gymnasium example as a verified invocation of the older Gym interface.
  • A download or local load fails. Check the dataset directory and the selected file. The download implementation reuses an existing cache file; a file’s presence alone is not evidence of the intended contents. The loader checks observations, actions, rewards and terminals, then checks relevant array shapes.
  • The transition count changes. Compare the raw dataset with qlearning_dataset’s N − 1 iteration and default timeout filtering. Preserve those choices when reproducing a published training pipeline.
  • Scores differ by a factor of 100. Check whether one report multiplies the normalized ratio by 100. Also check that both calculations use the same task’s reference anchors and an episode return rather than a training minibatch’s rewards.

When normalization raises “Reference score not provided for env”, the implementation has no usable reference pair. That is different from a low score. Do not invent a denominator or borrow constants from a similarly named environment to force a number.

Free alternatives to D4RL

Farama’s open-source successors separate offline data handling from simulation environments. The table distinguishes their roles; an environment library alone does not supply an equivalent fixed benchmark dataset.

Project Documented role Relationship to a D4RL workflow
Minari Offline dataset API, hosting interface and episode access The maintained dataset direction identified by D4RL; use its dataset identifiers and loading API.
Gymnasium-Robotics Robotics simulation environments A destination for D4RL environments; distinguish an environment change from the selection of offline training data.
MiniGrid A maintained reinforcement-learning environment project Named in D4RL’s environment migration notice; it fills an environment role rather than replacing the dataset API.

Farama’s 2023 Minari announcement explains the move through common dataset interfaces, environment recovery and reproducibility metadata. Its current Basic Usage documentation also supports recovering an evaluation environment with eval_env=True; if no separate evaluation specification exists, it falls back to the collection environment. Make that distinction explicit when comparing evaluations.

For a separate research problem involving games and multiple agents, CASRAI’s OpenSpiel research software guide covers another environment and algorithm toolkit. It is adjacent research software, rather than a substitute dataset for reproducing a D4RL score. Keep such a change visible in the benchmark design.

Sources

Data: D4RL Maze2D normalization anchors and calculated spans

Original compilation for three exact Maze2D task IDs: lower and upper reference values from the official D4RL infos.py, with the reference span calculated as upper minus lower following OfflineEnv normalization. These constants are score anchors, not measured algorithm results or statistical bounds.

D4RL Maze2D normalization anchors and calculated spans
Legacy environment IDLower reference (return)Upper reference (return)Calculated reference span (return)
maze2d-umaze-v123.85161.86138.01
maze2d-medium-v113.13277.39264.26
maze2d-large-v16.7273.99267.29

Sources

Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about D4RL: Install, Load Datasets and Read Scores

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →