Written and maintained by CASRAI Editorial Board
Published · Last updated
On this page
A policy gradient is a way to train a reinforcement-learning agent by directly adjusting the parameters of its policy, the function that chooses actions, in the direction that increases expected return. Instead of first learning a value for every state and then deriving behaviour from it, a policy gradient method treats the expected return J(πθ) as an objective and performs gradient ascent on it: θk+1 = θk + α ∇θJ(πθ) evaluated at θk. This guide explains the idea, then walks through training a policy-gradient agent (PPO) with the open-source Stable-Baselines3 library, and shows how to read the results without misreading the training loss.
Ask CASRAI · free to tryQuestion about this guide? Ask CASRAI answers with linked sources. 2 free questions a day, no account.
What policy gradient is used for in research
Policy gradient methods are used when an agent must learn a sequence of decisions from reward signals: control and robotics tasks, game-playing agents, and the reinforcement-learning stage of language-model alignment (see RLHF). In the foundational 1999 paper by Sutton, McAllester, Singh and Mansour, the policy is represented by its own function approximator whose parameters are updated along the gradient of expected reward; REINFORCE and actor-critic methods are examples of this family, and the gradient can be written in a form that is estimable from experience using an approximate action-value or advantage function.
The core result in plain terms
OpenAI’s Spinning Up introduction to policy optimization derives the estimator with the log-derivative trick. The gradient of the policy’s expected return turns into an expectation over sampled trajectories of the sum, over time steps, of ∇θ log πθ(at|st) multiplied by a weight. In the simplest version the weight is the whole trajectory return R(τ). The environment’s own dynamics drop out of the expression because they do not depend on θ, so the gradient can be estimated from sampled experience alone: ĝ = (1/|D|) Στ∈D Σt ∇θ log πθ(at|st) R(τ), where D is a set of collected trajectories.
Choosing the weight: the part that separates the variants
The expected gradient has the same form for several different weights Φt, but the variance of the estimate does not. Spinning Up lists the following valid choices; the table below is the practical map between the maths and the settings you will meet in software.
| Weight Φt | What it is | Role stated in the source |
|---|---|---|
| R(τ) | Return of the whole trajectory | The simplest policy gradient; every action in a trajectory is reinforced by the full return |
| Reward-to-go | Sum of rewards from step t onward | Only rewards after an action can reinforce it; lowers variance, so fewer trajectories are needed |
| Reward-to-go minus baseline b(st) | Reward-to-go with a state-only baseline subtracted | Leaves the expected gradient unchanged and reduces variance; a common choice is the on-policy value function Vπ(st), approximated by a network Vφ fit by squared error to the reward-to-go |
| Qπ(st,at) | On-policy action-value function | Same expectation, different variance |
| Aπ(st,at) = Qπ − Vπ | Advantage function | Same expectation, different variance; the weight used by advantage-based actor-critic methods |
The reason a baseline is allowed at all is the expected grad-log-prob (EGLP) lemma: the expectation of ∇θ log Pθ(x) over x drawn from Pθ is zero, so subtracting any function of the state alone does not bias the gradient.
How to install or access it
The reference implementation used here is Stable-Baselines3 (SB3), installed from its official documentation. According to the SB3 install page (read 2026-11-12), it requires Python 3.10 or newer and PyTorch 2.8 or newer. Two install commands are documented:
pip install 'stable-baselines3[extra]'installs SB3 with the extras: TensorBoard, OpenCV and ale-py.pip install stable-baselines3installs the library without those extras.
The SB3 PPO page notes that PPO is meant to be run primarily on the CPU when the policy does not use a convolutional network, so no GPU is needed for the CartPole example below.
Step-by-step: train a policy-gradient agent with PPO
Proximal Policy Optimization (PPO, arXiv:1707.06347) is a policy gradient method; SB3’s overview describes it as combining ideas from A2C (multiple workers) and TRPO. The numbered steps below follow the example on the SB3 PPO page.
- Import the pieces.
import gymnasium as gym,from stable_baselines3 import PPO, andfrom stable_baselines3.common.env_util import make_vec_env. - Create a vectorised environment.
vec_env = make_vec_env("CartPole-v1", n_envs=4)runs four copies of the CartPole task in parallel. - Create the model.
model = PPO("MlpPolicy", vec_env, verbose=1). The string"MlpPolicy"selects a fully connected network for the policy and value function. Verbosity 1 prints info messages such as the device and wrappers used; 0 prints nothing and 2 adds debug messages. - Train.
model.learn(total_timesteps=25000). The docs define total_timesteps as the total number of samples (environment steps) to train on, and note it is a lower bound. - Save and reload.
model.save("ppo_cartpole")writes the model;model = PPO.load("ppo_cartpole")restores it. - (Optional) log for TensorBoard. Pass
tensorboard_logto the constructor, for examplePPO("MlpPolicy", vec_env, verbose=1, tensorboard_log="./ppo_cartpole_tb/"). The docs describe this argument as the log location for TensorBoard, and if it is None no logging is done.learnalso acceptstb_log_nameto name the run.
Defaults that map onto the weight table
SB3 PPO ships with learning_rate 0.0003, n_steps 2048, batch_size 64, n_epochs 10, gamma 0.99, gae_lambda 0.95, clip_range 0.2, normalize_advantage True, ent_coef 0.0, vf_coef 0.5 and max_grad_norm 0.5. Three of these connect directly to the maths above: gamma is the discount factor; gae_lambda controls the advantage estimate (the A2C page states that a value of 1 is equivalent to the classic advantage); and normalize_advantage rescales the advantage before the update. The data table at the end of this page compares PPO and A2C defaults side by side.
Supported action spaces for PPO are Discrete, Box, MultiDiscrete and MultiBinary. Dict is not supported for actions.
Output and how to read it
The most common mistake when moving from supervised learning to policy gradients is reading the training loss as a score. Spinning Up is explicit: the policy-gradient “loss” is a surrogate, the negative mean of log π(a|s) times the weight. It is not a loss function in the supervised-learning sense, its value does not measure performance, and minimizing it on a given batch does not guarantee higher return. It is only a device that makes automatic differentiation produce the policy gradient for data collected by the current policy.
So judge an agent by the return it achieves in the environment, not by the loss curve. With verbose=1 SB3 prints info messages; for curves over time, set tensorboard_log as in step 6 and open the run in TensorBoard (installed with the [extra] option). Evaluate a saved model by loading it and running it in the environment, then comparing episode returns across several random seeds, since policy gradient estimates are noisy by construction; see computational reproducibility and reproducible AI experiment for how to document the seeds, versions and settings behind a reported result.
Troubleshooting
- Install fails or imports break. Check the documented requirements first: Python 3.10 or newer and PyTorch 2.8 or newer.
- Error about an unsupported action space. PPO and A2C accept Discrete, Box, MultiDiscrete and MultiBinary actions; a Dict action space is not supported.
- No output while training. verbose=0 prints nothing. Also, passing a custom logger with
set_loggeroverwrites thetensorboard_logandverboseconstructor settings, per the SB3 docs. - Training stopped at a slightly different step count than requested. total_timesteps is documented as a lower bound, not an exact count.
- The loss goes down but the agent does not improve. This is expected behaviour of a surrogate objective, as described above; measure episode return instead.
Alternatives and sibling implementations
| Option | What it is | Source |
|---|---|---|
| A2C in Stable-Baselines3 | A synchronous, deterministic variant of Asynchronous Advantage Actor Critic (A3C); same library, install and make_vec_env pattern; A2C("MlpPolicy", vec_env, verbose=1) |
SB3 A2C page |
| PPO in Stable-Baselines3 | The method used in the steps above; clipped-objective policy gradient | SB3 PPO page |
| OpenAI Spinning Up | Educational documentation of policy optimization, including the derivations summarised in this guide | Spinning Up, Part 3 |
The SB3 A2C page reports a PyBullet comparison (2M steps, 6 seeds) in which PPO with gSDE scored higher than A2C with gSDE on most environments, for example Walker2D at 2019 ± 64 versus 839 ± 56. That is a benchmark report on specific tasks, not a general ranking.
FAQ
What is the policy gradient in one sentence?
It is the gradient of a policy’s expected return with respect to the policy parameters, estimated from sampled trajectories and used to improve the policy by gradient ascent.
Is REINFORCE the same as policy gradient?
REINFORCE is one policy gradient algorithm, the simplest estimator with the trajectory return as the weight. Sutton et al. (1999) name REINFORCE and actor-critic methods as examples of the policy gradient family.
Why subtract a baseline?
Any state-only baseline leaves the expected gradient unchanged because of the EGLP lemma, while a good baseline such as the value function reduces the variance of the estimate.
Is PPO a policy gradient method?
Yes. PPO is a policy gradient method whose objective in SB3 is controlled by a clip range (default 0.2) that limits how far each update moves the policy.
Does policy gradient need a value function?
No. The simplest form uses only sampled returns. Value functions enter as baselines or critics to reduce variance, which is what actor-critic methods such as A2C and PPO do.
Sources
- OpenAI Spinning Up, “Part 3: Intro to Policy Optimization”, spinningup.openai.com (read 2026-11-12).
- Sutton, McAllester, Singh and Mansour (1999), “Policy Gradient Methods for Reinforcement Learning with Function Approximation”, NIPS 12, papers.nips.cc (abstract, read 2026-11-12).
- Stable-Baselines3 documentation: install, PPO and A2C (read 2026-11-12).
- Schulman et al., Proximal Policy Optimization Algorithms, arXiv:1707.06347.
Data: Stable-Baselines3 default settings: PPO versus A2C
Side-by-side default hyperparameters and action-space support for the PPO and A2C policy gradient implementations in Stable-Baselines3, compiled from the library's own algorithm documentation pages.
Period covered: 2026.
| Setting | PPO default | A2C default |
|---|---|---|
| learning_rate | 0.0003 | 0.0007 |
| n_steps | 2048 | 5 |
| gamma (discount factor) | 0.99 | 0.99 |
| gae_lambda | 0.95 | 1.0 |
| normalize_advantage | True | False |
| Supported action spaces | Discrete, Box, MultiDiscrete, MultiBinary | Discrete, Box, MultiDiscrete, MultiBinary |
| Dict action space | Not supported | Not supported |
Sources
- Stable-Baselines3 PPO documentation (accessed 2026-11-12)
- Stable-Baselines3 A2C documentation (accessed 2026-11-12)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







