Skip to main content
v2026.11,986 entries · CC-BY 4.0

CleanRL: Single-File Deep RL Implementations, Install and Run PPO

CleanRL puts each deep RL algorithm in one standalone file. Install it, run PPO on CartPole, read the output and set up a reproducible experiment.

Written and maintained by CASRAI Editorial Board

Published · Last updated

On this page

CleanRL is an open-source (MIT-licensed) collection of deep reinforcement learning algorithms in which every detail of one algorithm variant lives in a single standalone Python file. You run that file from the command line, read it from top to bottom, and edit it directly. It is built for researchers who want to understand or modify an algorithm such as PPO or DQN, not for people who want to import an algorithm as a building block.

Ask CASRAI · free to tryQuestion about this guide? Ask CASRAI answers with linked sources. 2 free questions a day, no account.

Ask your first question freeSubscribe

What CleanRL is used for in research

The CleanRL documentation describes the design in one sentence: every detail about an algorithm variant is put into a single standalone file. The project’s GitHub page gives a concrete example: the ppo_atari.py file is about 340 lines and contains all of the implementation details for PPO on Atari. The stated purposes, in both the documentation and the JMLR paper that introduces the project, are understanding the details of deep reinforcement learning algorithms and prototyping new features quickly.

The trade-off is explicit in the docs. CleanRL is not a modular library and is not meant to be imported. Code is duplicated across files, which the maintainers accept in return for readability and easier debugging: when a result looks wrong, the whole algorithm is in front of you in one file rather than spread across a class hierarchy. The documentation covers online reinforcement learning only; offline reinforcement learning is handled by a separate project, CORL.

The algorithms listed in the documentation are PPO (including Atari, transformer-XL and RND variants), DQN, C51, Rainbow, DDPG, TD3, SAC, PPG, RPO, QDagger and PQN, with JAX variants for some of them. For a background on the family PPO belongs to, see the CASRAI guide to policy gradient methods.

Two research-practice features set it apart from a bare reference implementation. First, experiment tracking is built in: metrics go to TensorBoard, runs are seeded, gameplay video can be captured, and logging to Weights and Biases is available. Second, the project publishes benchmarks of at least seven algorithms across more than 34 games (the Open RL Benchmark), with the underlying runs accessible through Weights and Biases reports. That lets you compare your own run against recorded reference curves. Both points feed the reproducibility checklist further down this page.

How to install or access it

CleanRL is installed from its official repository at github.com/vwxyzjn/cleanrl, following the installation page of its documentation (read 2026-11-14). The page states a supported Python range of at least 3.7.1 and below 3.11, and names the uv package manager (version 0.7.19 or later) as the recommended route.

With uv, clone the repository and install from inside it:

  1. git clone https://github.com/vwxyzjn/cleanrl.git
  2. cd cleanrl
  3. uv pip install .

With pip, install the base requirements file instead: pip install -r requirements/requirements.txt. The repository provides optional requirement files for specific environment families: Atari, MuJoCo, mujoco_py, Procgen, EnvPool, PettingZoo, JAX, documentation tools and cloud tooling. With uv the equivalent optional extras are named .[atari], .[mujoco], .[dm_control], .[procgen], .[envpool], .[pettingzoo], .[jax], .[optuna], .[docs] and .[cloud]. Install only the extras for the environments your experiment uses; the base install is enough for the CartPole example below.

Step-by-step: train PPO on CartPole

This is the first run shown on the basic-usage page of the documentation (read 2026-11-14). It uses the standalone ppo.py file, which implements PPO for a classic control task.

  1. Open a terminal in the cloned cleanrl directory, with the environment from the install step active.
  2. Run the training script with a fixed seed and a small budget: uv run python cleanrl/ppo.py --seed 1 --env-id CartPole-v0 --total-timesteps 50000
  3. Let it finish. Metrics are written to TensorBoard format in a runs/ folder by default.
  4. Start TensorBoard on that folder: tensorboard --logdir runs
  5. Optionally repeat the run with --capture_video to save gameplay video to videos/{run_name}.

To see every option the script accepts, run python cleanrl/ppo.py --help. Because each algorithm is a single file, the same pattern applies to the others: change the script name and environment ID, and check that file’s own --help output, since flags differ between algorithms.

The flags the PPO script lists include the experiment and logging options (--exp-name, --env-id, --seed, --total-timesteps, --torch-deterministic, --cuda, --track, --wandb-project-name, --wandb-entity, --capture_video) and the algorithm hyperparameters (--learning-rate, --num-envs, --num-steps, --num-minibatches, --update-epochs, --gamma, --gae-lambda, --anneal-lr, --norm-adv, --clip-coef, --clip-vloss, --ent-coef, --vf-coef, --max-grad-norm, --target-kl). The data table on this page groups them by role.

Output and how to read it

A default run produces two kinds of output. The first is the runs/ directory, which holds the metrics TensorBoard displays as curves over training time. The second, only if you pass --capture_video, is the videos/{run_name} folder with recordings of the agent playing. Adding --track sends the same metrics to Weights and Biases, using the project and entity names you supply with --wandb-project-name and --wandb-entity.

The useful reading habit is comparative, not absolute. A single learning curve from one seed says little about an algorithm, because reinforcement learning results vary from seed to seed. Run the same configuration with several seeds, keep the run_name values distinct, and compare curves in TensorBoard or in a Weights and Biases workspace. For a standard Atari or control task you can also set your curves next to the recorded benchmark runs in the Open RL Benchmark instead of trusting a lone number.

Reproducibility checklist for a CleanRL experiment

Everything below comes directly from options and resources the project documents. It is a practical way to make a CleanRL result checkable by someone else; see the CASRAI entry on reproducibility and the comparison of reproducibility and replicability for the underlying definitions.

  • Seed every run. Pass --seed explicitly and report it. The documentation lists seeding among its tracking features.
  • Decide on determinism. The --torch-deterministic flag is exposed by the PPO script; state whether you used it and which device you ran on (--cuda).
  • Log to a shareable tracker. Use --track so that the full configuration and curves exist outside your machine.
  • Record the exact command and the code version. Since each algorithm is one file, the commit of the repository you cloned fully identifies the algorithm you ran; note it, along with any edits you made to the file.
  • Cite the software. The citation given by the project is: Huang, Dossa, Ye, Braga, Chakraborty, Mehta and Araujo, “CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms”, Journal of Machine Learning Research 23(274):1-18, 2022.
  • Compare against the benchmark where one exists. The Open RL Benchmark covers at least seven algorithms and more than 34 games.

Troubleshooting

Installation fails or imports break on a recent Python

The installation page states Python at least 3.7.1 and below 3.11. If your default interpreter is newer, create a virtual environment with a supported version before installing. The page also names uv 0.7.19 or later; if the documentation and the repository disagree on a version, the repository’s own requirement files are the current record of what the code expects.

“no kernel image is available” on a newer GPU

The documentation explains that the default PyTorch wheel targets CUDA 10.2, and that newer GPUs (it names the Ampere generation) can fail with a “no kernel image is available” error. Its fix is to reinstall PyTorch from a CUDA 11.3 wheel index: uv pip install "torch==1.12.1" --upgrade --extra-index-url https://download.pytorch.org/whl/cu113.

You wanted to call CleanRL from your own code

That is outside its design. The documentation states that CleanRL is not meant to be imported. If you need an algorithm as an importable component, use a modular library; the next section compares the two approaches.

You need an offline reinforcement learning method

The CleanRL documentation covers online algorithms only and points to a separate project, CORL, for offline reinforcement learning.

Free alternatives

CleanRL is one of several open-source options. The table compares it with Stable-Baselines3 using only statements from each project’s own documentation, read 2026-11-14. For a task-by-task walkthrough of the other library, see the CASRAI guide to Stable-Baselines3 install, train and evaluate.

Question CleanRL Stable-Baselines3
How the project describes itself High-quality single-file implementations of deep reinforcement learning algorithms with research-friendly features A set of reliable implementations of reinforcement learning algorithms in PyTorch
Code structure One standalone file per algorithm variant; not a modular library and not meant to be imported Presented as a set of algorithm implementations; see the CASRAI guide for usage
Algorithms named on the docs home page PPO, DQN, C51, Rainbow, DDPG, TD3, SAC, PPG, RPO, QDagger, PQN (plus variants) A2C, DDPG, DQN, HER, PPO, SAC, TD3
Offline reinforcement learning Not covered; the separate CORL project handles it Not listed among the algorithms on the docs home page
Best matched to Reading and modifying one algorithm’s full details; prototyping features Using ready-made algorithm implementations

The last row summarises the maintainers’ own framing for CleanRL (understanding details and prototyping) rather than ranking the two tools. The Open RL Benchmark that CleanRL publishes also lists Stable-Baselines3, openai/baselines and jaxrl among its contributors, so results from several libraries can sit side by side.

Frequently asked questions

What is CleanRL?

CleanRL is an open-source, MIT-licensed set of deep reinforcement learning implementations in which each algorithm variant is a single standalone Python file. It is described in a 2022 JMLR paper and aims to make algorithm details easy to read, debug and modify.

Can I import CleanRL like a normal Python library?

No. The documentation says it is not a modular library and not meant to be imported. You copy or run the relevant file and edit it.

Which algorithms does CleanRL include?

The documentation lists PPO (with Atari, transformer-XL and RND variants), DQN, C51, Rainbow, DDPG, TD3, SAC, PPG, RPO, QDagger and PQN, with JAX variants for some.

How do I run my first experiment?

Install from the repository, then run uv run python cleanrl/ppo.py --seed 1 --env-id CartPole-v0 --total-timesteps 50000 and view the results with tensorboard --logdir runs.

Does CleanRL support offline reinforcement learning?

Not in its own documentation, which covers online algorithms and refers offline work to the separate CORL project.

How do I cite CleanRL?

Cite the 2022 JMLR paper by Huang and colleagues, volume 23, issue 274, pages 1-18, and report the repository commit you ran.

Sources

  • CleanRL documentation, overview: https://docs.cleanrl.dev/ (checked 2026-11-14)
  • CleanRL documentation, installation: https://docs.cleanrl.dev/get-started/installation/ (checked 2026-11-14)
  • CleanRL documentation, basic usage: https://docs.cleanrl.dev/get-started/basic-usage/ (checked 2026-11-14)
  • CleanRL repository: https://github.com/vwxyzjn/cleanrl (checked 2026-11-14)
  • Huang et al., “CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms”, JMLR 23(274):1-18, 2022: https://jmlr.org/papers/v23/21-1342.html (checked 2026-11-14)
  • Stable-Baselines3 documentation: https://stable-baselines3.readthedocs.io/en/master/ (checked 2026-11-14)

Data: CleanRL ppo.py command-line flags grouped by role

The command-line flags listed by python cleanrl/ppo.py --help in the CleanRL basic-usage documentation, grouped by the role each plays in an experiment (logging, reproducibility, rollout, optimisation, PPO objective). Grouping and one-line descriptions compiled from the documentation's flag list and standard PPO terminology.

CleanRL ppo.py command-line flags grouped by role
FlagRoleWhat it controls
--exp-nameLoggingExperiment name used to label a run
--env-idEnvironmentEnvironment to train on (example: CartPole-v0)
--seedReproducibilityRandom seed for the run
--total-timestepsBudgetTotal training steps
--torch-deterministicReproducibilityPyTorch determinism option
--cudaHardwareGPU use
--trackLoggingWeights and Biases tracking
--capture_videoLoggingSaves gameplay video to videos/{run_name}
--learning-rateOptimisationLearning rate
--num-envsRolloutNumber of parallel environments
--num-stepsRolloutSteps collected per environment per rollout
--num-minibatchesOptimisationMinibatches per update
--update-epochsOptimisationEpochs per update
--gammaReturnDiscount factor
--gae-lambdaReturnGeneralised advantage estimation lambda
--anneal-lrOptimisationLearning-rate annealing option
--norm-advOptimisationAdvantage normalisation option
--clip-coefPPO objectiveSurrogate clipping coefficient
--clip-vlossPPO objectiveValue-loss clipping option
--ent-coefPPO objectiveEntropy bonus coefficient
--vf-coefPPO objectiveValue-function loss coefficient
--max-grad-normOptimisationGradient-norm clipping limit
--target-klPPO objectiveKL-divergence target for early stopping of updates

Sources

Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about CleanRL: Single-File Deep RL Implementations, Install and Run PPO

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →