Written and maintained by CASRAI Editorial Board
Published · Last updated
On this page
CleanRL is an open-source (MIT-licensed) collection of deep reinforcement learning algorithms in which every detail of one algorithm variant lives in a single standalone Python file. You run that file from the command line, read it from top to bottom, and edit it directly. It is built for researchers who want to understand or modify an algorithm such as PPO or DQN, not for people who want to import an algorithm as a building block.
Ask CASRAI · free to tryQuestion about this guide? Ask CASRAI answers with linked sources. 2 free questions a day, no account.
What CleanRL is used for in research
The CleanRL documentation describes the design in one sentence: every detail about an algorithm variant is put into a single standalone file. The project’s GitHub page gives a concrete example: the ppo_atari.py file is about 340 lines and contains all of the implementation details for PPO on Atari. The stated purposes, in both the documentation and the JMLR paper that introduces the project, are understanding the details of deep reinforcement learning algorithms and prototyping new features quickly.
The trade-off is explicit in the docs. CleanRL is not a modular library and is not meant to be imported. Code is duplicated across files, which the maintainers accept in return for readability and easier debugging: when a result looks wrong, the whole algorithm is in front of you in one file rather than spread across a class hierarchy. The documentation covers online reinforcement learning only; offline reinforcement learning is handled by a separate project, CORL.
The algorithms listed in the documentation are PPO (including Atari, transformer-XL and RND variants), DQN, C51, Rainbow, DDPG, TD3, SAC, PPG, RPO, QDagger and PQN, with JAX variants for some of them. For a background on the family PPO belongs to, see the CASRAI guide to policy gradient methods.
Two research-practice features set it apart from a bare reference implementation. First, experiment tracking is built in: metrics go to TensorBoard, runs are seeded, gameplay video can be captured, and logging to Weights and Biases is available. Second, the project publishes benchmarks of at least seven algorithms across more than 34 games (the Open RL Benchmark), with the underlying runs accessible through Weights and Biases reports. That lets you compare your own run against recorded reference curves. Both points feed the reproducibility checklist further down this page.
How to install or access it
CleanRL is installed from its official repository at github.com/vwxyzjn/cleanrl, following the installation page of its documentation (read 2026-11-14). The page states a supported Python range of at least 3.7.1 and below 3.11, and names the uv package manager (version 0.7.19 or later) as the recommended route.
With uv, clone the repository and install from inside it:
git clone https://github.com/vwxyzjn/cleanrl.gitcd cleanrluv pip install .
With pip, install the base requirements file instead: pip install -r requirements/requirements.txt. The repository provides optional requirement files for specific environment families: Atari, MuJoCo, mujoco_py, Procgen, EnvPool, PettingZoo, JAX, documentation tools and cloud tooling. With uv the equivalent optional extras are named .[atari], .[mujoco], .[dm_control], .[procgen], .[envpool], .[pettingzoo], .[jax], .[optuna], .[docs] and .[cloud]. Install only the extras for the environments your experiment uses; the base install is enough for the CartPole example below.
Step-by-step: train PPO on CartPole
This is the first run shown on the basic-usage page of the documentation (read 2026-11-14). It uses the standalone ppo.py file, which implements PPO for a classic control task.
- Open a terminal in the cloned
cleanrldirectory, with the environment from the install step active. - Run the training script with a fixed seed and a small budget:
uv run python cleanrl/ppo.py --seed 1 --env-id CartPole-v0 --total-timesteps 50000 - Let it finish. Metrics are written to TensorBoard format in a
runs/folder by default. - Start TensorBoard on that folder:
tensorboard --logdir runs - Optionally repeat the run with
--capture_videoto save gameplay video tovideos/{run_name}.
To see every option the script accepts, run python cleanrl/ppo.py --help. Because each algorithm is a single file, the same pattern applies to the others: change the script name and environment ID, and check that file’s own --help output, since flags differ between algorithms.
The flags the PPO script lists include the experiment and logging options (--exp-name, --env-id, --seed, --total-timesteps, --torch-deterministic, --cuda, --track, --wandb-project-name, --wandb-entity, --capture_video) and the algorithm hyperparameters (--learning-rate, --num-envs, --num-steps, --num-minibatches, --update-epochs, --gamma, --gae-lambda, --anneal-lr, --norm-adv, --clip-coef, --clip-vloss, --ent-coef, --vf-coef, --max-grad-norm, --target-kl). The data table on this page groups them by role.
Output and how to read it
A default run produces two kinds of output. The first is the runs/ directory, which holds the metrics TensorBoard displays as curves over training time. The second, only if you pass --capture_video, is the videos/{run_name} folder with recordings of the agent playing. Adding --track sends the same metrics to Weights and Biases, using the project and entity names you supply with --wandb-project-name and --wandb-entity.
The useful reading habit is comparative, not absolute. A single learning curve from one seed says little about an algorithm, because reinforcement learning results vary from seed to seed. Run the same configuration with several seeds, keep the run_name values distinct, and compare curves in TensorBoard or in a Weights and Biases workspace. For a standard Atari or control task you can also set your curves next to the recorded benchmark runs in the Open RL Benchmark instead of trusting a lone number.
Reproducibility checklist for a CleanRL experiment
Everything below comes directly from options and resources the project documents. It is a practical way to make a CleanRL result checkable by someone else; see the CASRAI entry on reproducibility and the comparison of reproducibility and replicability for the underlying definitions.
- Seed every run. Pass
--seedexplicitly and report it. The documentation lists seeding among its tracking features. - Decide on determinism. The
--torch-deterministicflag is exposed by the PPO script; state whether you used it and which device you ran on (--cuda). - Log to a shareable tracker. Use
--trackso that the full configuration and curves exist outside your machine. - Record the exact command and the code version. Since each algorithm is one file, the commit of the repository you cloned fully identifies the algorithm you ran; note it, along with any edits you made to the file.
- Cite the software. The citation given by the project is: Huang, Dossa, Ye, Braga, Chakraborty, Mehta and Araujo, “CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms”, Journal of Machine Learning Research 23(274):1-18, 2022.
- Compare against the benchmark where one exists. The Open RL Benchmark covers at least seven algorithms and more than 34 games.
Troubleshooting
Installation fails or imports break on a recent Python
The installation page states Python at least 3.7.1 and below 3.11. If your default interpreter is newer, create a virtual environment with a supported version before installing. The page also names uv 0.7.19 or later; if the documentation and the repository disagree on a version, the repository’s own requirement files are the current record of what the code expects.
“no kernel image is available” on a newer GPU
The documentation explains that the default PyTorch wheel targets CUDA 10.2, and that newer GPUs (it names the Ampere generation) can fail with a “no kernel image is available” error. Its fix is to reinstall PyTorch from a CUDA 11.3 wheel index: uv pip install "torch==1.12.1" --upgrade --extra-index-url https://download.pytorch.org/whl/cu113.
You wanted to call CleanRL from your own code
That is outside its design. The documentation states that CleanRL is not meant to be imported. If you need an algorithm as an importable component, use a modular library; the next section compares the two approaches.
You need an offline reinforcement learning method
The CleanRL documentation covers online algorithms only and points to a separate project, CORL, for offline reinforcement learning.
Free alternatives
CleanRL is one of several open-source options. The table compares it with Stable-Baselines3 using only statements from each project’s own documentation, read 2026-11-14. For a task-by-task walkthrough of the other library, see the CASRAI guide to Stable-Baselines3 install, train and evaluate.
| Question | CleanRL | Stable-Baselines3 |
|---|---|---|
| How the project describes itself | High-quality single-file implementations of deep reinforcement learning algorithms with research-friendly features | A set of reliable implementations of reinforcement learning algorithms in PyTorch |
| Code structure | One standalone file per algorithm variant; not a modular library and not meant to be imported | Presented as a set of algorithm implementations; see the CASRAI guide for usage |
| Algorithms named on the docs home page | PPO, DQN, C51, Rainbow, DDPG, TD3, SAC, PPG, RPO, QDagger, PQN (plus variants) | A2C, DDPG, DQN, HER, PPO, SAC, TD3 |
| Offline reinforcement learning | Not covered; the separate CORL project handles it | Not listed among the algorithms on the docs home page |
| Best matched to | Reading and modifying one algorithm’s full details; prototyping features | Using ready-made algorithm implementations |
The last row summarises the maintainers’ own framing for CleanRL (understanding details and prototyping) rather than ranking the two tools. The Open RL Benchmark that CleanRL publishes also lists Stable-Baselines3, openai/baselines and jaxrl among its contributors, so results from several libraries can sit side by side.
Frequently asked questions
What is CleanRL?
CleanRL is an open-source, MIT-licensed set of deep reinforcement learning implementations in which each algorithm variant is a single standalone Python file. It is described in a 2022 JMLR paper and aims to make algorithm details easy to read, debug and modify.
Can I import CleanRL like a normal Python library?
No. The documentation says it is not a modular library and not meant to be imported. You copy or run the relevant file and edit it.
Which algorithms does CleanRL include?
The documentation lists PPO (with Atari, transformer-XL and RND variants), DQN, C51, Rainbow, DDPG, TD3, SAC, PPG, RPO, QDagger and PQN, with JAX variants for some.
How do I run my first experiment?
Install from the repository, then run uv run python cleanrl/ppo.py --seed 1 --env-id CartPole-v0 --total-timesteps 50000 and view the results with tensorboard --logdir runs.
Does CleanRL support offline reinforcement learning?
Not in its own documentation, which covers online algorithms and refers offline work to the separate CORL project.
How do I cite CleanRL?
Cite the 2022 JMLR paper by Huang and colleagues, volume 23, issue 274, pages 1-18, and report the repository commit you ran.
Sources
- CleanRL documentation, overview: https://docs.cleanrl.dev/ (checked 2026-11-14)
- CleanRL documentation, installation: https://docs.cleanrl.dev/get-started/installation/ (checked 2026-11-14)
- CleanRL documentation, basic usage: https://docs.cleanrl.dev/get-started/basic-usage/ (checked 2026-11-14)
- CleanRL repository: https://github.com/vwxyzjn/cleanrl (checked 2026-11-14)
- Huang et al., “CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms”, JMLR 23(274):1-18, 2022: https://jmlr.org/papers/v23/21-1342.html (checked 2026-11-14)
- Stable-Baselines3 documentation: https://stable-baselines3.readthedocs.io/en/master/ (checked 2026-11-14)
Data: CleanRL ppo.py command-line flags grouped by role
The command-line flags listed by python cleanrl/ppo.py --help in the CleanRL basic-usage documentation, grouped by the role each plays in an experiment (logging, reproducibility, rollout, optimisation, PPO objective). Grouping and one-line descriptions compiled from the documentation's flag list and standard PPO terminology.
| Flag | Role | What it controls |
|---|---|---|
| --exp-name | Logging | Experiment name used to label a run |
| --env-id | Environment | Environment to train on (example: CartPole-v0) |
| --seed | Reproducibility | Random seed for the run |
| --total-timesteps | Budget | Total training steps |
| --torch-deterministic | Reproducibility | PyTorch determinism option |
| --cuda | Hardware | GPU use |
| --track | Logging | Weights and Biases tracking |
| --capture_video | Logging | Saves gameplay video to videos/{run_name} |
| --learning-rate | Optimisation | Learning rate |
| --num-envs | Rollout | Number of parallel environments |
| --num-steps | Rollout | Steps collected per environment per rollout |
| --num-minibatches | Optimisation | Minibatches per update |
| --update-epochs | Optimisation | Epochs per update |
| --gamma | Return | Discount factor |
| --gae-lambda | Return | Generalised advantage estimation lambda |
| --anneal-lr | Optimisation | Learning-rate annealing option |
| --norm-adv | Optimisation | Advantage normalisation option |
| --clip-coef | PPO objective | Surrogate clipping coefficient |
| --clip-vloss | PPO objective | Value-loss clipping option |
| --ent-coef | PPO objective | Entropy bonus coefficient |
| --vf-coef | PPO objective | Value-function loss coefficient |
| --max-grad-norm | Optimisation | Gradient-norm clipping limit |
| --target-kl | PPO objective | KL-divergence target for early stopping of updates |
Sources
- CleanRL documentation: basic usage (accessed 2026-11-14)
- CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms (JMLR 2022) (accessed 2026-11-14)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







