Stable-Baselines3: Install, Train, Evaluate

Stable-Baselines3 (SB3) is an open-source PyTorch library of reinforcement learning (RL) algorithm implementations, released under the MIT license. Researchers use it to train agents such as A2C, PPO, DQN, DDPG, TD3, SAC and HER on environments that follow the Gymnasium interface, with one consistent API across algorithms. This guide covers the install route from the official documentation, a train-and-evaluate workflow, what a saved model contains (which matters for reproducibility), and how to read the evaluation output.

Reader task: after reading you should be able to install SB3 from the official source, train and evaluate a first agent, save it, and record the versions a methods section needs.

What Stable-Baselines3 is used for in research

SB3 provides ready-made implementations of standard deep RL algorithms so that a lab can compare a new environment, reward design or hyperparameter setting against a known baseline instead of writing the algorithm from scratch. The project README describes it as a set of reliable implementations of reinforcement learning algorithms in PyTorch, and as the successor to the earlier Stable Baselines. Its authors report in the journal paper that the implementations were benchmarked against reference codebases and that unit tests cover 95% of the code (Raffin et al., 2021).

For the wider field, see what machine learning is and how it is used in research and what artificial intelligence is. For a robotics-oriented simulation workflow, see the Isaac Lab guide.

Which algorithms are in which package

Package	Algorithms listed in the README	Backend
Stable-Baselines3 (core)	A2C, PPO, DQN, DDPG, TD3, SAC, HER (ARS is listed separately under other algorithms)	PyTorch
SB3-Contrib	TRPO, Maskable PPO, RecurrentPPO, QR-DQN, TQC	PyTorch
SBX	CrossQ, DroQ, SAC-N, SimBa	JAX

Source: the DLR-RM/stable-baselines3 README, read 2026-11-08. Algorithm lists change between releases, so check the README for your installed version.

Install Stable-Baselines3 (official source)

The current documentation states that Stable-Baselines3 requires Python 3.10+ and PyTorch 2.8 or newer. On Windows, the documentation recommends a conda-style (miniforge) environment with Python 3.10 or newer.

Create and activate a Python 3.10+ environment.
Install the library with the optional extras (TensorBoard, OpenCV and ale-py): pip install 'stable-baselines3[extra]'. In zsh, keep the quotes.
Or install the core only: pip install stable-baselines3.

On a headless machine or in Docker, the documentation advises installing opencv-python-headless instead of the standard OpenCV build. For the development version, use pip install git+https://github.com/DLR-RM/stable-baselines3; for contributing, clone the repository and run pip install -e '.[docs,tests,extra]'.

Docker gotcha: the official images stablebaselines/stable-baselines3 (GPU) and stablebaselines/stable-baselines3-cpu contain the dependencies but not the SB3 package itself; they are intended for development, so you still install the package inside the container.

Train and evaluate a PPO or A2C agent

The quickstart in the documentation trains an A2C agent on CartPole. The steps below follow it, with the evaluation step from the evaluation helper page.

Create the environment: env = gym.make("CartPole-v1") after import gymnasium as gym.
Create the model: from stable_baselines3 import A2C, then model = A2C("MlpPolicy", env, verbose=1). Setting verbose=1 prints training progress.
Train: model.learn(total_timesteps=10_000).
Run the trained policy: get the vectorised environment with vec_env = model.get_env(), call obs = vec_env.reset(), then in a loop call action, _states = model.predict(obs, deterministic=True) and obs, rewards, dones, info = vec_env.step(action). A vectorised environment resets automatically when an episode ends.
Save: model.save("a2c_cartpole") (the documentation examples use the same call form, for instance model.save("dqn_lunar")). Load with the class, not an instance: A2C.load("a2c_cartpole", env=vec_env). The documentation warns that load is not in-place.

A shorter form from the quickstart is A2C("MlpPolicy", "CartPole-v1").learn(10_000). To log to TensorBoard, pass tensorboard_log, for example A2C("MlpPolicy", "CartPole-v1", verbose=1, tensorboard_log="./a2c_cartpole_tensorboard/"), then run tensorboard --logdir ./a2c_cartpole_tensorboard/. PPO is used the same way: import PPO in place of A2C.

Output and how to read it

Evaluating with evaluate_policy

evaluate_policy(model, env, n_eval_episodes=10, deterministic=True, render=False, callback=None, reward_threshold=None, return_episode_rewards=False, warn=True) returns the mean and standard deviation of per-episode reward. With return_episode_rewards=True it instead returns lists of per-episode returns and lengths.

from stable_baselines3.common.monitor import Monitor
from stable_baselines3.common.evaluation import evaluate_policy

eval_env = Monitor(gym.make("CartPole-v1"))
mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10)

Pitfall: the documentation says to wrap the environment in a Monitor wrapper before other wrappers when you want original (unmodified) rewards. Without it, rewards and episode lengths are counted from env.step, so any wrapper that alters reward or episode length changes the reported numbers. Also note that deterministic=True is the default, so evaluation uses the policy’s deterministic action rather than sampling.

Training logs and TensorBoard

With a Monitor wrapper in place, TensorBoard shows episode reward alongside model losses and algorithm-specific parameters. Metric names such as rollout/ep_len_mean and train/value_loss appear in the documentation’s TensorBoard examples; the Logger section of the documentation explains the logger’s output names.

Reproducibility: what a saved model contains

A saved SB3 model is a zip archive. According to the save-format page it holds a data file (class parameters as JSON), *.optimizer.pth, policy.pth, pytorch_variables.pth, a _stable_baselines3_version file and system_info.txt. Objects that cannot be serialised to JSON are stored with cloudpickle, which the documentation says can cause incompatibilities between Python versions.

For a methods section, record the Python, PyTorch and Stable-Baselines3 versions, the algorithm and policy class, total_timesteps, the evaluation settings (n_eval_episodes, deterministic) and whether a Monitor wrapper was used. To cite the library, the README gives: Raffin, Hill, Gleave, Kanervisto, Ernestus and Dormann, “Stable-Baselines3: Reliable Reinforcement Learning Implementations”, Journal of Machine Learning Research 22(268):1-8, 2021 (jmlr.org/papers/v22/20-1364.html).

Troubleshooting

Import or install errors: check that the interpreter is Python 3.10+ and PyTorch is 2.8 or newer, as the install page requires.
zsh: no matches found for stable-baselines3[extra]: keep the single quotes around the package spec.
OpenCV errors on a server or in Docker: install opencv-python-headless.
Docker image has no stable_baselines3 module: the official images do not include the package; install it inside the container.
Model fails to load: call PPO.load("ppo_saved", print_system_info=True) to compare the saved and current system information; the custom_objects argument lets you skip objects that cannot be restored. Prefer the same Python version for saving and loading because of cloudpickle.
Evaluation rewards look wrong: wrap the evaluation environment in Monitor first (see above).