Skip to main content
v2026.11,986 entries · CC-BY 4.0

Multi-Agent Reinforcement Learning: PettingZoo and RLlib

Run a first multi-agent reinforcement learning environment in PettingZoo, map it to RLlib terms, and know what to record for reproducible MARL results.

Written and maintained by CASRAI Editorial Board

Published · Last updated

On this page

Multi-agent reinforcement learning (MARL) is reinforcement learning in which several agents act in one shared environment, each choosing actions from its own observations and each receiving its own reward. Single-agent tools such as Stable-Baselines3 assume one learner; MARL tools add what that assumption leaves out: a roster of agents that can change during an episode, dictionaries keyed by agent name, and a mapping from agents to the policies that control them. This guide shows how to run a first multi-agent environment with the PettingZoo library, how the same ideas are named in Ray RLlib, and what to check before you report MARL results.

What multi-agent reinforcement learning is used for in research

Researchers use MARL when the problem has more than one decision-maker whose choices affect the others: cooperative teams, competitive games, or mixed settings. The documentation of the tools below lists the kinds of tasks they target. PettingZoo states that its Agent Environment Cycle (AEC) API can support any game multi-agent RL can consider, and RLlib’s multi-agent documentation uses Rock-Paper-Scissors (simultaneous moves) and Tic-Tac-Toe (turn-based moves) as its worked examples. PyMARL is built around the StarCraft Multi-Agent Challenge (SMAC), and BenchMARL’s benchmark experiments run on VMAS tasks.

MARL sits next to, but is not the same as, policy-gradient methods for a single agent, imitation learning (which learns from demonstrations rather than rewards) and robot learning. Game-theoretic language such as simultaneous and turn-based play, covered in What is game theory?, is the usual way to describe the interaction structure.

Two training layouts you will see named

The BenchMARL paper describes CTDE (centralised training, decentralised execution) as a decentralised actor paired with a centralised critic, and notes that MAPPO and IPPO can be used either as DTDE (decentralised training, decentralised execution) or with centralised critics in CTDE. Always state which layout your experiment used, because the same algorithm name can describe both.

How to install or access it

PettingZoo installs with pip. The official basic-usage page gives three commands:

  • pip install pettingzoo installs the base library.
  • pip install 'pettingzoo[atari]' installs the dependencies for one environment family (Atari here).
  • pip install 'pettingzoo[all]' installs the dependencies for every family.

The documentation states that PettingZoo is supported and maintained for Python 3.10, 3.11, 3.12, 3.13 and 3.14 on Linux and macOS, and that Windows pull requests are accepted but Windows is not officially supported. Install RLlib from the Ray documentation at docs.ray.io, PyMARL from its GitHub repository (Apache-2.0 licence) and BenchMARL from the project page linked in its paper; the Sources section lists each address.

Step-by-step: run a parallel multi-agent environment with PettingZoo

PettingZoo offers two APIs. The AEC API has agents act one after another through agent_iter(), last() and step(action). The Parallel API has all agents act at once, which matches simultaneous-move problems. The steps below use the Parallel API as documented on the PettingZoo Parallel API page.

  1. Create the environment. Import the factory and request the parallel version of a Pistonball environment: from pettingzoo import make, then penv = make("parallel", "butterfly/pistonball-v6"). Use "aec" for the Agent Environment Cycle API and "parallel" for the Parallel API. Butterfly environments need their family dependencies, installed with the pip extras shown above.
  2. Check that a parallel version exists. Not every environment provides one. The documentation shows that "classic/chess-v6" in parallel_registry is False, because chess is AEC-only. Test the registry before assuming.
  3. Reset with a seed. observations, infos = parallel_env.reset(seed=42) returns two dictionaries keyed by agent name.
  4. Loop while agents remain. The documented loop is:
    observations, infos = parallel_env.reset(seed=42)
    while parallel_env.agents:
        actions = {agent: parallel_env.action_space(agent).sample() for agent in parallel_env.agents}
        observations, rewards, terminations, truncations, infos = parallel_env.step(actions)
    parallel_env.close()

    The random sample() call is the place where your learned policy goes.

  5. Close the environment with parallel_env.close() when the loop ends.

For a turn-based game, the AEC loop in the same documentation iterates for agent in env.agent_iter():, calls observation, reward, termination, truncation, info = env.last(), passes None as the action when the agent is terminated or truncated, and otherwise passes an action to env.step(action).

Output and how to read it

Every return value in the Parallel API is a dictionary whose keys are agent names. reset() returns observations and infos. step(actions) takes one dictionary of actions and returns five dictionaries: observations, rewards, terminations, truncations and infos. A reward therefore belongs to one agent, not to the team, so a cooperative result needs an explicit aggregation rule that you state yourself.

Two attributes matter when you store results. agents is the current list of live agents and can change during an episode, which is why the loop above runs while parallel_env.agents. possible_agents is the fixed list of every agent that could appear, it matches the keys of the observation and action spaces, and max_num_agents gives its length. Allocate result arrays from possible_agents, not from agents.

The same ideas in RLlib

RLlib’s MultiAgentEnv is gymnasium-like: reset() and step() return dictionaries keyed by agent ID, and the environment exposes possible_agents, agents, observation_spaces and action_spaces. Two RLlib behaviours have no direct PettingZoo equivalent. The agent IDs present in the returned observation dictionary decide which agents act next, and the special "__all__" key ends the episode for every agent at once. RLlib also separates agents from policies: each agent maps to exactly one policy, one policy may control several agents, and a policy_mapping_fn does the routing. The data table below sets the two vocabularies side by side. RLlib ships a PettingZoo wrapper, for example PettingZooEnv(pistonball_v6.env(num_floors=...)), so a PettingZoo environment can be trained there; re-open the RLlib page for the full configuration before copying code.

Reporting MARL results so they can be reproduced

Three cautions come directly from the tools’ own documentation and papers.

  • Simulator version. PyMARL’s README notes that SMAC results depend on the StarCraft II version (SMAC itself used SC2.4.6.2.69232) and are not always comparable across versions. Record the version.
  • Shared evaluation setup. The BenchMARL paper treats a benchmark as a collection of experiments in which hyperparameters are shared where possible, and its initial release ran nine algorithms on three VMAS tasks. It cites an earlier comparison (Papoudakis et al., 2020, arXiv 2006.07869) of independent learning, centralised policy gradient and value decomposition on cooperative tasks, and observes that MARL lacks commonly used evaluation tasks. A single new environment is weak evidence on its own.
  • Layout label. State CTDE or DTDE (see above) next to every algorithm name.

Troubleshooting

  • The parallel version is missing. The environment is AEC-only. Use the AEC loop instead of forcing a parallel wrapper.
  • Dependency errors for a family. Install that family’s extras, for example pip install 'pettingzoo[atari]', or 'pettingzoo[all]'.
  • Problems on Windows. PettingZoo’s documentation says Windows is not officially supported; use Linux or macOS for reported results.
  • Dictionary keys do not match. Spaces are keyed by possible_agents, while agents lists only the live agents, so build each step’s action dictionary from agents, as the documented loop does.
  • PyMARL will not install. Its README steers users to the Docker route (docker/build.sh, then install_sc2.sh) and discourages installing from requirements.txt. A run looks like python3 src/main.py --config=qmix --env-config=sc2 with env_args.map_name=2s3z, with results written to the Results folder.
  • An RLlib episode never ends. Return the "__all__" key to end it for all agents.

Alternative tools

The table lists four open documentation sources and what each one documents. It describes each tool as stated in its own documentation, not a ranking.

Tool What its documentation covers Where to start
PettingZoo Environment API (AEC and Parallel) for multi-agent problems Basic usage and Parallel API pages
Ray RLlib Multi-agent training: policy mapping, agent groups, per-policy training and statistics Multi-agent environments page
PyMARL PyTorch implementations of QMIX, COMA, VDN, IQL and QTRAN on SMAC (Apache-2.0) GitHub README
BenchMARL Standardised benchmarking across algorithms, models and environments, built on TorchRL arXiv 2312.01472 and the project documentation

RLlib’s documentation also describes with_agent_groups(), which groups agents into one logical agent (needed for Q-Mix style algorithms), policies_to_train, which restricts which policies are updated, and per-policy reporting of statistics. The BenchMARL paper positions MARLlib as an RLlib-based predecessor and notes that RLlib lacks vectorised-environment support for benchmarking.

Frequently asked questions

What is multi-agent reinforcement learning?

It is reinforcement learning with several agents interacting in one environment, each with its own observations, actions and reward. Environments return dictionaries keyed by agent name rather than single values.

What is the difference between PettingZoo’s AEC and Parallel APIs?

AEC has agents act in turn, while Parallel has all agents act at the same step. PettingZoo documents AEC as able to support any game multi-agent RL can consider, but not every environment has a Parallel version.

What do CTDE and DTDE mean?

CTDE is centralised training with decentralised execution, described in the BenchMARL paper as a decentralised actor with a centralised critic. DTDE trains the agents in a decentralised way as well.

Can one policy control several agents?

Yes. In RLlib each agent maps to exactly one policy, but one policy may control several agents, with policy_mapping_fn routing between them.

Why are SMAC results not always comparable?

PyMARL’s README states that results depend on the StarCraft II version used, so results from different versions are not always comparable.

Sources

  • PettingZoo, Basic usage: https://pettingzoo.farama.org/content/basic_usage/ (read 2026-11-14)
  • PettingZoo, Parallel API: https://pettingzoo.farama.org/api/parallel/ (read 2026-11-14)
  • Ray RLlib, Multi-agent environments: https://docs.ray.io/en/latest/rllib/multi-agent-envs.html (read 2026-11-14)
  • PyMARL (oxwhirl), GitHub README: https://github.com/oxwhirl/pymarl (read 2026-11-14)
  • BenchMARL: Benchmarking Multi-Agent Reinforcement Learning, arXiv 2312.01472: https://arxiv.org/abs/2312.01472 (read 2026-11-14)
  • Papoudakis et al., Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks, arXiv 2006.07869, as cited in the BenchMARL paper

Data: PettingZoo and RLlib multi-agent API vocabulary compared

Compares how PettingZoo's Parallel API and Ray RLlib's MultiAgentEnv name the same multi-agent reinforcement learning concepts, compiled from each project's current official documentation pages.

PettingZoo and RLlib multi-agent API vocabulary compared
ConceptPettingZoo (Parallel API)RLlib (MultiAgentEnv)
Reset returnobservations, infos: dictionaries keyed by agent nameDictionaries keyed by agent ID
Step inputOne dictionary of actions keyed by agent nameDictionary of actions keyed by agent ID
Step returnFive dictionaries: observations, rewards, terminations, truncations, infosDictionaries keyed by agent ID
Agent rosteragents (current, can change in an episode); possible_agents (fixed, matches space keys)possible_agents, agents, observation_spaces, action_spaces
Who acts nextAll agents in agents act each stepAgent IDs present in the returned observation dictionary
Episode endLoop ends when agents is emptyThe special key __all__ ends the episode for all agents
Agent-to-policy linkNot part of the environment APIEach agent maps to one policy; one policy may control several agents via policy_mapping_fn

Sources

Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about Multi-Agent Reinforcement Learning: PettingZoo and RLlib

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →