Written and maintained by CASRAI Editorial Board
Published · Last updated
On this page
Multi-agent reinforcement learning (MARL) is reinforcement learning in which several agents act in one shared environment, each choosing actions from its own observations and each receiving its own reward. Single-agent tools such as Stable-Baselines3 assume one learner; MARL tools add what that assumption leaves out: a roster of agents that can change during an episode, dictionaries keyed by agent name, and a mapping from agents to the policies that control them. This guide shows how to run a first multi-agent environment with the PettingZoo library, how the same ideas are named in Ray RLlib, and what to check before you report MARL results.
What multi-agent reinforcement learning is used for in research
Researchers use MARL when the problem has more than one decision-maker whose choices affect the others: cooperative teams, competitive games, or mixed settings. The documentation of the tools below lists the kinds of tasks they target. PettingZoo states that its Agent Environment Cycle (AEC) API can support any game multi-agent RL can consider, and RLlib’s multi-agent documentation uses Rock-Paper-Scissors (simultaneous moves) and Tic-Tac-Toe (turn-based moves) as its worked examples. PyMARL is built around the StarCraft Multi-Agent Challenge (SMAC), and BenchMARL’s benchmark experiments run on VMAS tasks.
MARL sits next to, but is not the same as, policy-gradient methods for a single agent, imitation learning (which learns from demonstrations rather than rewards) and robot learning. Game-theoretic language such as simultaneous and turn-based play, covered in What is game theory?, is the usual way to describe the interaction structure.
Two training layouts you will see named
The BenchMARL paper describes CTDE (centralised training, decentralised execution) as a decentralised actor paired with a centralised critic, and notes that MAPPO and IPPO can be used either as DTDE (decentralised training, decentralised execution) or with centralised critics in CTDE. Always state which layout your experiment used, because the same algorithm name can describe both.
How to install or access it
PettingZoo installs with pip. The official basic-usage page gives three commands:
pip install pettingzooinstalls the base library.pip install 'pettingzoo[atari]'installs the dependencies for one environment family (Atari here).pip install 'pettingzoo[all]'installs the dependencies for every family.
The documentation states that PettingZoo is supported and maintained for Python 3.10, 3.11, 3.12, 3.13 and 3.14 on Linux and macOS, and that Windows pull requests are accepted but Windows is not officially supported. Install RLlib from the Ray documentation at docs.ray.io, PyMARL from its GitHub repository (Apache-2.0 licence) and BenchMARL from the project page linked in its paper; the Sources section lists each address.
Step-by-step: run a parallel multi-agent environment with PettingZoo
PettingZoo offers two APIs. The AEC API has agents act one after another through agent_iter(), last() and step(action). The Parallel API has all agents act at once, which matches simultaneous-move problems. The steps below use the Parallel API as documented on the PettingZoo Parallel API page.
- Create the environment. Import the factory and request the parallel version of a Pistonball environment:
from pettingzoo import make, thenpenv = make("parallel", "butterfly/pistonball-v6"). Use"aec"for the Agent Environment Cycle API and"parallel"for the Parallel API. Butterfly environments need their family dependencies, installed with thepipextras shown above. - Check that a parallel version exists. Not every environment provides one. The documentation shows that
"classic/chess-v6" in parallel_registryis False, because chess is AEC-only. Test the registry before assuming. - Reset with a seed.
observations, infos = parallel_env.reset(seed=42)returns two dictionaries keyed by agent name. - Loop while agents remain. The documented loop is:
observations, infos = parallel_env.reset(seed=42) while parallel_env.agents: actions = {agent: parallel_env.action_space(agent).sample() for agent in parallel_env.agents} observations, rewards, terminations, truncations, infos = parallel_env.step(actions) parallel_env.close()The random
sample()call is the place where your learned policy goes. - Close the environment with
parallel_env.close()when the loop ends.
For a turn-based game, the AEC loop in the same documentation iterates for agent in env.agent_iter():, calls observation, reward, termination, truncation, info = env.last(), passes None as the action when the agent is terminated or truncated, and otherwise passes an action to env.step(action).
Output and how to read it
Every return value in the Parallel API is a dictionary whose keys are agent names. reset() returns observations and infos. step(actions) takes one dictionary of actions and returns five dictionaries: observations, rewards, terminations, truncations and infos. A reward therefore belongs to one agent, not to the team, so a cooperative result needs an explicit aggregation rule that you state yourself.
Two attributes matter when you store results. agents is the current list of live agents and can change during an episode, which is why the loop above runs while parallel_env.agents. possible_agents is the fixed list of every agent that could appear, it matches the keys of the observation and action spaces, and max_num_agents gives its length. Allocate result arrays from possible_agents, not from agents.
The same ideas in RLlib
RLlib’s MultiAgentEnv is gymnasium-like: reset() and step() return dictionaries keyed by agent ID, and the environment exposes possible_agents, agents, observation_spaces and action_spaces. Two RLlib behaviours have no direct PettingZoo equivalent. The agent IDs present in the returned observation dictionary decide which agents act next, and the special "__all__" key ends the episode for every agent at once. RLlib also separates agents from policies: each agent maps to exactly one policy, one policy may control several agents, and a policy_mapping_fn does the routing. The data table below sets the two vocabularies side by side. RLlib ships a PettingZoo wrapper, for example PettingZooEnv(pistonball_v6.env(num_floors=...)), so a PettingZoo environment can be trained there; re-open the RLlib page for the full configuration before copying code.
Reporting MARL results so they can be reproduced
Three cautions come directly from the tools’ own documentation and papers.
- Simulator version. PyMARL’s README notes that SMAC results depend on the StarCraft II version (SMAC itself used SC2.4.6.2.69232) and are not always comparable across versions. Record the version.
- Shared evaluation setup. The BenchMARL paper treats a benchmark as a collection of experiments in which hyperparameters are shared where possible, and its initial release ran nine algorithms on three VMAS tasks. It cites an earlier comparison (Papoudakis et al., 2020, arXiv 2006.07869) of independent learning, centralised policy gradient and value decomposition on cooperative tasks, and observes that MARL lacks commonly used evaluation tasks. A single new environment is weak evidence on its own.
- Layout label. State CTDE or DTDE (see above) next to every algorithm name.
Troubleshooting
- The parallel version is missing. The environment is AEC-only. Use the AEC loop instead of forcing a parallel wrapper.
- Dependency errors for a family. Install that family’s extras, for example
pip install 'pettingzoo[atari]', or'pettingzoo[all]'. - Problems on Windows. PettingZoo’s documentation says Windows is not officially supported; use Linux or macOS for reported results.
- Dictionary keys do not match. Spaces are keyed by
possible_agents, whileagentslists only the live agents, so build each step’s action dictionary fromagents, as the documented loop does. - PyMARL will not install. Its README steers users to the Docker route (
docker/build.sh, theninstall_sc2.sh) and discourages installing fromrequirements.txt. A run looks likepython3 src/main.py --config=qmix --env-config=sc2 with env_args.map_name=2s3z, with results written to the Results folder. - An RLlib episode never ends. Return the
"__all__"key to end it for all agents.
Alternative tools
The table lists four open documentation sources and what each one documents. It describes each tool as stated in its own documentation, not a ranking.
| Tool | What its documentation covers | Where to start |
|---|---|---|
| PettingZoo | Environment API (AEC and Parallel) for multi-agent problems | Basic usage and Parallel API pages |
| Ray RLlib | Multi-agent training: policy mapping, agent groups, per-policy training and statistics | Multi-agent environments page |
| PyMARL | PyTorch implementations of QMIX, COMA, VDN, IQL and QTRAN on SMAC (Apache-2.0) | GitHub README |
| BenchMARL | Standardised benchmarking across algorithms, models and environments, built on TorchRL | arXiv 2312.01472 and the project documentation |
RLlib’s documentation also describes with_agent_groups(), which groups agents into one logical agent (needed for Q-Mix style algorithms), policies_to_train, which restricts which policies are updated, and per-policy reporting of statistics. The BenchMARL paper positions MARLlib as an RLlib-based predecessor and notes that RLlib lacks vectorised-environment support for benchmarking.
Frequently asked questions
What is multi-agent reinforcement learning?
It is reinforcement learning with several agents interacting in one environment, each with its own observations, actions and reward. Environments return dictionaries keyed by agent name rather than single values.
What is the difference between PettingZoo’s AEC and Parallel APIs?
AEC has agents act in turn, while Parallel has all agents act at the same step. PettingZoo documents AEC as able to support any game multi-agent RL can consider, but not every environment has a Parallel version.
What do CTDE and DTDE mean?
CTDE is centralised training with decentralised execution, described in the BenchMARL paper as a decentralised actor with a centralised critic. DTDE trains the agents in a decentralised way as well.
Can one policy control several agents?
Yes. In RLlib each agent maps to exactly one policy, but one policy may control several agents, with policy_mapping_fn routing between them.
Why are SMAC results not always comparable?
PyMARL’s README states that results depend on the StarCraft II version used, so results from different versions are not always comparable.
Sources
- PettingZoo, Basic usage: https://pettingzoo.farama.org/content/basic_usage/ (read 2026-11-14)
- PettingZoo, Parallel API: https://pettingzoo.farama.org/api/parallel/ (read 2026-11-14)
- Ray RLlib, Multi-agent environments: https://docs.ray.io/en/latest/rllib/multi-agent-envs.html (read 2026-11-14)
- PyMARL (oxwhirl), GitHub README: https://github.com/oxwhirl/pymarl (read 2026-11-14)
- BenchMARL: Benchmarking Multi-Agent Reinforcement Learning, arXiv 2312.01472: https://arxiv.org/abs/2312.01472 (read 2026-11-14)
- Papoudakis et al., Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks, arXiv 2006.07869, as cited in the BenchMARL paper
Data: PettingZoo and RLlib multi-agent API vocabulary compared
Compares how PettingZoo's Parallel API and Ray RLlib's MultiAgentEnv name the same multi-agent reinforcement learning concepts, compiled from each project's current official documentation pages.
| Concept | PettingZoo (Parallel API) | RLlib (MultiAgentEnv) |
|---|---|---|
| Reset return | observations, infos: dictionaries keyed by agent name | Dictionaries keyed by agent ID |
| Step input | One dictionary of actions keyed by agent name | Dictionary of actions keyed by agent ID |
| Step return | Five dictionaries: observations, rewards, terminations, truncations, infos | Dictionaries keyed by agent ID |
| Agent roster | agents (current, can change in an episode); possible_agents (fixed, matches space keys) | possible_agents, agents, observation_spaces, action_spaces |
| Who acts next | All agents in agents act each step | Agent IDs present in the returned observation dictionary |
| Episode end | Loop ends when agents is empty | The special key __all__ ends the episode for all agents |
| Agent-to-policy link | Not part of the environment API | Each agent maps to one policy; one policy may control several agents via policy_mapping_fn |
Sources
- PettingZoo Parallel API (accessed 2026-11-14)
- Ray RLlib Multi-agent environments (accessed 2026-11-14)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







