Skip to main content
v2026.11,986 entries · CC-BY 4.0

OpenSpiel: Multi-Agent Reinforcement Learning and Game Theory Guide

Master OpenSpiel for multi-agent reinforcement learning and game-theoretic research. Covers installation, state APIs, CFR solvers, and benchmark metrics.

Written and maintained by CASRAI Editorial Board

Published · Last updated

On this page

OpenSpiel is an open-source research framework developed by Google DeepMind and academic collaborators to support multi-agent reinforcement learning, decision-making, and computational game theory. Built with a high-performance C++ core and comprehensive Python bindings, OpenSpiel enables researchers to implement, simulate, and benchmark algorithms across perfect- and imperfect-information games, sequential and simultaneous interactions, and cooperative or competitive payoff structures. By providing standardized implementations of classical equilibrium solvers alongside modern deep reinforcement learning architectures, the platform establishes rigorous evaluation baselines across hundreds of game environments.

What it is used for in research

Multi-agent systems present computational challenges that single-agent reinforcement learning environments cannot capture. In games involving multiple interacting entities, the environment is inherently non-stationary from the perspective of any single learner: as other agents adjust their policies, the transition dynamics and expected rewards shift dynamically. OpenSpiel addresses this complexity by formalizing games through mathematical game theory, providing explicit representations of extensive-form games, normal-form games, and Markov games.

In academic research, OpenSpiel serves three primary roles across computer science, economics, and operations research:

  • Equilibrium Computation and Regret Minimization: Researchers investigate how algorithmic strategies converge toward theoretical solution concepts, such as Nash equilibria, correlated equilibria, and coarse correlated equilibria. OpenSpiel includes canonical algorithms such as Counterfactual Regret Minimization (CFR), CFR+, and Monte Carlo CFR (MCCFR), which compute exact or approximate equilibria in imperfect-information games like poker.
  • Multi-Agent Reinforcement Learning (MARL) Benchmarking: Modern deep RL methods, including Deep Q-Networks (DQN), Advantage Actor-Critic (A2C), and Neural Fictitious Self-Play (NFSP), require standardized simulation environments to ensure reproducibility. Rather than building custom environment abstractions for each experimental study, researchers utilize OpenSpiel to evaluate agents against established baselines. Researchers often compare these formulations with single-agent benchmarks in our Stable-Baselines3 reinforcement learning guide and minimal implementations in our CleanRL single-file implementations overview.
  • Game-Theoretic Evaluation Metrics: Assessing agent performance in multi-agent environments cannot rely solely on accumulated scalar rewards, because an agent’s empirical win rate depends directly on the opponent pool. OpenSpiel provides mathematical metrics such as Exploitability, NashConv, and alpha-rank, which quantify the absolute distance between an agent’s policy profile and an unexploitable equilibrium strategy. Understanding these metrics bridges empirical machine learning with the formal foundations of game theory.

Furthermore, OpenSpiel bridges the gap between classic matrix-based game solving and contemporary deep learning architectures. By exposing observations, information states, and action abstractions as vectorized tensors, researchers can integrate neural networks developed in PyTorch, JAX, or TensorFlow directly into multi-agent game loops.

How to install or access it

OpenSpiel is distributed under an Apache 2.0 open-source license. Researchers can install OpenSpiel via pre-compiled Python binary wheels or by compiling directly from the official Google DeepMind GitHub repository. System support targets Linux (Ubuntu 20.04 LTS or newer recommended) and macOS (x86_64 and Apple Silicon architectures). Windows users access OpenSpiel either through Windows Subsystem for Linux (WSL2) or via containerized Docker environments.

Method A: Pre-Compiled Binary Installation via Pip

For Python-focused experimentation on supported x86_64 and aarch64 Linux and macOS systems, pre-compiled wheels provide the quickest setup. A dedicated virtual environment is strongly recommended to isolate dependencies:

python3 -m venv openspiel-env
source openspiel-env/bin/activate
pip install --upgrade pip setuptools wheel
pip install open_spiel

Method B: Full Developer Installation from Source

Compiling from source is required when modifying underlying C++ engine routines, introducing custom games, running complete C++ test suites, or implementing low-level algorithms. The build system requires CMake (version 3.17 or higher), a C++20-compliant compiler (Clang 10+ or GCC 9.2+), and Python development headers.

Step 1: Clone the official Google DeepMind repository from GitHub:

git clone https://github.com/google-deepmind/open_spiel.git
cd open_spiel

Step 2: Install required system packages and third-party dependencies using OpenSpiel’s automated system installation script:

./open_spiel/scripts/install.sh

This script installs essential build tools, downloads required external dependencies (including Abseil-CPP and pybind11), and prepares the local Python environment.

Step 3: Create a separate build directory and invoke CMake with the designated C++ compiler:

mkdir build
cd build
cmake -DCMAKE_CXX_COMPILER=clang++ ../open_spiel
make -j$(nproc)

Step 4: Configure environment variables so Python can locate the compiled pyspiel bindings and shared libraries:

export PYTHONPATH=$PYTHONPATH:/path/to/open_spiel
export PYTHONPATH=$PYTHONPATH:/path/to/open_spiel/build
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/open_spiel/build

Step 5: Verify the installation by running the default test suite:

ctest -j$(nproc) --output-on-failure

Step-by-step task

The following walkthrough guides researchers through solving Kuhn Poker using tabular Counterfactual Regret Minimization (CFR) and measuring its convergence toward a Nash equilibrium. Kuhn Poker is an imperfect-information sequential game featuring a three-card deck (Jack, Queen, King), ante betting, and one round of betting. The task executes in Python using the official OpenSpiel API.

  1. Initialize Environment and Virtual Shell: Activate your Python environment and verify that OpenSpiel imports without library linkage warnings:
    python3 -c "import pyspiel; print(pyspiel.__file__)"
  2. Instantiate the Game Environment: Load the game configuration using pyspiel.load_game. Games accept optional dictionary parameters to modify parameters such as player counts, board dimensions, or deck sizes:
    import pyspiel
    game = pyspiel.load_game("kuhn_poker")
    print("Game Name:", game.get_type().short_name)
    print("Players:", game.num_players())
    print("Max Game Length:", game.max_game_length())
  3. Inspect the Initial State and Trajectory: Generate the root state of the game tree using game.new_initial_state(). In games with initial stochasticity (dealing cards), the initial node is a chance node where state.is_chance_node() returns true:
    state = game.new_initial_state()
    print("Is Chance Node:", state.is_chance_node())
    print("Chance Outcomes:", state.chance_outcomes())
    
    # Deal a card to Player 0 and Player 1
    state.apply_action(0)  # Chance deals card 0 (Jack) to Player 0
    state.apply_action(1)  # Chance deals card 1 (Queen) to Player 1
    
    print("Current Player:", state.current_player())
    print("Legal Actions:", state.legal_actions())
    print("Information State Player 0:", state.information_state_string(0))
  4. Configure the CFR Solver: Import the CFR algorithm module from open_spiel.python.algorithms. Initialize the CFRPlusSolver, an accelerated variant of Counterfactual Regret Minimization that sets negative cumulative regrets to zero at each step:
    from open_spiel.python.algorithms import cfr
    from open_spiel.python.algorithms import exploitability
    
    solver = cfr.CFRPlusSolver(game)
  5. Execute Training Iterations and Evaluate Exploitability: Train the solver across 1,000 iterations. In two-player zero-sum extensive-form games, convergence to an epsilon-Nash equilibrium is evaluated via exploitability, which computes the sum of the amounts that a best-responding adversary could exploit from each player’s policy:
    num_iterations = 1000
    for iteration in range(num_iterations):
        solver.evaluate_and_update_policy()
        if (iteration + 1) % 200 == 0:
            current_policy = solver.average_policy()
            conv = exploitability.exploitability(game, current_policy)
            print(f"Iteration {iteration + 1:4d} | Exploitability: {conv:.6f}")
  6. Extract the Converged Average Strategy: Extract the final average policy and inspect the specific behavioral probabilities for an information state:
    final_policy = solver.average_policy()
    final_exploitability = exploitability.exploitability(game, final_policy)
    print(f"Final Nash Approximation Exploitability: {final_exploitability:.6e}")
    
    # Inspect the action distribution for Player 0 holding a Jack (information state '0')
    state_jack = game.new_initial_state()
    state_jack.apply_action(0)
    state_jack.apply_action(1)
    action_probs = final_policy.action_probabilities(state_jack)
    print("Action Probabilities for Player 0 (Pass vs Bet):", action_probs)

Researchers training continuous or parameterized strategies often connect these foundations with policy gradient methods, comparing model-free optimization against tabular regret bounds.

Output and how to read it

Executing game solving and reinforcement learning scripts in OpenSpiel yields structured analytical outputs. Interpreting these logs correctly ensures that experimental assertions align with theoretical bounds.

1. Exploitability and NashConv Logs

During CFR or Fictitious Play execution, the terminal produces numeric values reflecting policy convergence:

Iteration  200 | Exploitability: 0.052184
Iteration  400 | Exploitability: 0.024109
Iteration  600 | Exploitability: 0.015732
Iteration  800 | Exploitability: 0.011648
Iteration 1000 | Exploitability: 0.009121
Final Nash Approximation Exploitability: 9.121345e-03

To read these values:

  • Exploitability: In a two-player zero-sum game, exploitability measures the expected value that a worst-case opponent can gain against the current policy profile beyond the game value. An exploitability of 0.0 represents an exact Nash equilibrium. Values below 0.01 in small games denote strong convergence.
  • NashConv: In games with more than two players or general-sum payoffs, exploitability is generalized to NashConv, defined as the sum over all players of the incentives to deviate unilaterally to a best response: NashConv = sum_i (u_i(BR_i(sigma_-i), sigma_-i) - u_i(sigma)). At a Nash equilibrium, NashConv equals zero.
  • Current vs. Average Policy: Standard CFR updates instantaneous regrets, which fluctuate across iterations. The theoretical convergence guarantees apply strictly to the cumulative average policy (solver.average_policy()), not the last-iterate policy. Evaluating exploitability on instantaneous policies produces erratic, non-convergent curves.

2. Information States vs. Observation Tensors

When interfacing OpenSpiel with neural networks, methods output distinct state representations:

  • state.information_state_string(player): Returns a human-readable text string encoding the sequence of all private observations and public actions visible to that player since the start of the game. For example, in Kuhn Poker, 0pb denotes that Player 0 holds a Jack (card 0), Player 0 passed, and Player 1 bet.
  • state.information_state_tensor(player): Returns a flattened one-dimensional vector representation of the information state suitable for input into multi-layer perceptrons or recurrent neural networks.
  • state.observation_tensor(player): Returns an instantaneous snapshot of the current visible board or state, omitting historical actions. Markovian algorithms rely on observation tensors, whereas non-Markovian imperfect-information algorithms require full information states.

When applying human preference alignments, as detailed in our guide to reinforcement learning from human feedback, information state strings allow annotators to inspect complete trajectory histories rather than instantaneous state slices.

Troubleshooting

When compiling, configuring, or running OpenSpiel in research workflows, researchers frequently encounter specific technical errors. The following diagnostics address documented operational issues:

1. ImportError: No module named ‘pyspiel’

Symptom: Attempting to import pyspiel in Python raises an ImportError or ModuleNotFoundError despite building the C++ codebase.

Cause: Python cannot locate the compiled dynamic shared object (e.g., pyspiel.so or pyspiel.cpython-*.so) because environment search paths have not been exported.

Solution: Ensure your shell environment defines both the repository root and the CMake build folder in PYTHONPATH, as well as the dynamic linker path:

export PYTHONPATH="${PYTHONPATH}:/path/to/open_spiel:/path/to/open_spiel/build"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/path/to/open_spiel/build"

If running within Jupyter notebooks, set sys.path.append('/path/to/open_spiel/build') at the top of the notebook before importing OpenSpiel modules.

2. CMake C++ Standard Compilation Failures

Symptom: CMake reports errors regarding missing C++20 language features, standard library types, or compiler incompatibilities during make.

Cause: Default system compilers on older Linux installations (such as GCC versions below 9.2 or Clang versions below 10) lack complete C++20 support required by recent OpenSpiel commits.

Solution: Install a modern compiler package and instruct CMake to prioritize it explicitly:

sudo apt-get install clang-12
cmake -DCMAKE_CXX_COMPILER=clang++-12 ../open_spiel

3. Segmentation Fault or Bad Action in State Traversal

Symptom: The simulation halts with SpielException: apply_action called with illegal action or an unhandled C++ exception.

Cause: Applying an action integer without checking state.legal_actions(), or attempting to execute a player action on a chance node.

Solution: Implement defensive branching in game loops. Check state.is_chance_node() first, sample from state.chance_outcomes(), and confirm that candidate player actions exist in state.legal_actions() before invoking state.apply_action().

4. Memory Exhaustion During Tabular CFR on Large Games

Symptom: The Python process is terminated by the operating system (OOM kill) during solver initialization or early iterations.

Cause: Tabular CFR maintains regret and average strategy tables for every information state in the game tree. In games like Leduc Hold’em or Heads-Up Limit Texas Hold’em, the number of information sets exceeds hundreds of millions, requiring gigabytes of memory.

Solution: Shift from exact tabular CFR to sample-based or function-approximation algorithms. Utilize Monte Carlo CFR (MCCFR) with external sampling, or train parameterized policies via Deep CFR (open_spiel.python.algorithms.deep_cfr) to bound memory consumption.

Free alternatives

While OpenSpiel provides an extensive library of game environments and game-theoretic solvers, other open-source frameworks specialize in distributed scaling, standardized Gymnasium-style APIs, or analytical economic computation. The comparative table below summarizes key architectural alternatives.

Framework Primary Focus Language Core Algorithm Implementations Multi-Agent Structure
OpenSpiel Academic game theory, CFR, and multi-agent RL research C++20 with Python bindings Extensive (CFR, NFSP, DQN, AlphaZero, tabular solvers) Extensive-form, simultaneous, and matrix games
PettingZoo Standardized Python API for multi-agent environments Pure Python Environment only (connects to CleanRL, SB3, RLlib) AEC (Agent-Environment-Cycle) and Parallel APIs
Ray RLlib Distributed, production-scale multi-agent reinforcement learning Python with Ray distributed backend Extensive deep MARL (MAPPO, QMIX, MADDPG, PPO) Vectorized multi-agent environments
Gambit Analytical and symbolic computation of Nash equilibria C++ with Python and GUI wrappers Exact solvers (support enumeration, Lemke-Howson, linear programming) Strategic normal-form and extensive-form games
Hanabi Learning Environment Specialized cooperative imperfect-information benchmark C++ with Python bindings Benchmark environment (integrated into OpenSpiel) Cooperative dec-POMDP card gameplay

Researchers can integrate OpenSpiel environments directly into PettingZoo-compliant training pipelines using the open-source Farama Foundation Shimmy compatibility wrapper.

To examine the mathematical and structural properties of the game environments packaged directly inside OpenSpiel, consult the comparative data table below.

Frequently asked questions

Does OpenSpiel support Python or only C++?

OpenSpiel provides native support for both C++ and Python. The underlying core game simulations and high-performance graph traversals are written in C++20 for speed, while comprehensive bindings are exposed via pybind11. Researchers can implement full training loops, algorithm prototypes, and data analysis entirely in Python without writing C++ code.

How does OpenSpiel differ from OpenAI Gym and Farama Gymnasium?

Gymnasium is specifically designed for single-agent Markov Decision Processes (MDPs), where a solitary agent interacts with an environment via a unified step function. OpenSpiel is built from the ground up for multi-agent game theory, supporting multiple players, alternating turns, simultaneous moves, imperfect information sets, stochastic chance nodes, and non-scalar game-theoretic evaluation metrics.

Can OpenSpiel model simultaneous-move games?

Yes. OpenSpiel natively supports simultaneous-move games, such as matrix games (e.g., Prisoner’s Dilemma or Rock-Paper-Scissors) and spatial grid games. In simultaneous states, the game engine collects actions from all eligible players concurrently via a joint action vector before advancing the environment state.

How is convergence to a Nash equilibrium measured in OpenSpiel?

In two-player zero-sum games, OpenSpiel evaluates convergence using exploitability, which computes the maximum expected payoff an optimal adversary could achieve against the current policy profile. In multi-player or general-sum games, it provides NashConv, measuring the sum of all individual players’ incentives to unilaterally deviate to an unconstrained best-response strategy.

Does OpenSpiel run natively on Windows?

OpenSpiel does not officially provide native Windows binary builds or MSVC project files. Windows researchers run OpenSpiel using Windows Subsystem for Linux (WSL2 with Ubuntu) or via official Docker containers, which provide complete compatibility with the C++ build chain and Linux Python wheels.

Can deep learning frameworks like PyTorch or JAX interface with OpenSpiel?

Yes. OpenSpiel includes observation and information state tensor conversion methods (such as state.observation_tensor() and state.information_state_tensor()) that output flat or multi-dimensional numerical vectors. These arrays convert directly into PyTorch tensors, JAX arrays, or NumPy arrays for neural network training.

Sources

  • Google DeepMind. “OpenSpiel: A Framework for Reinforcement Learning in Games.” GitHub repository, 2026-11-16. https://github.com/google-deepmind/open_spiel
  • Google DeepMind. “OpenSpiel Documentation.” Read the Docs, accessed 2026-11-16. https://openspiel.readthedocs.io/
  • Lanctot, Marc, Edward Lockhart, Jean-Baptiste Lespiau, Julien Pérolat, Jithendra K. Subramanian, et al. “OpenSpiel: A Framework for Reinforcement Learning in Games.” arXiv:1908.09453, 2019, accessed 2026-11-16. https://arxiv.org/abs/1908.09453
  • Farama Foundation. “PettingZoo Documentation.” Accessed 2026-11-16. https://pettingzoo.farama.org/
  • Farama Foundation. “Shimmy: Multi-Agent Environment Wrappers.” GitHub repository, accessed 2026-11-16. https://github.com/Farama-Foundation/Shimmy

Data: OpenSpiel Standard Game Environments and Structural Properties

Comparative dataset of canonical game environments implemented in Google DeepMind's OpenSpiel framework, detailing player dynamics, information availability, payoff structure, approximate game tree state space, and standard evaluation metrics compiled from official documentation and the foundational OpenSpiel specification.

Period covered: 2019/2026. Coverage: Global.

OpenSpiel Standard Game Environments and Structural Properties
EnvironmentInformation TypePlayer Count (players)Turn DynamicsPayoff StructureGame Tree State Size (log10 states)Primary Evaluation Metric
Kuhn PokerImperfect2SequentialZero-sum2Exploitability
Leduc Hold'emImperfect2SequentialZero-sum6Exploitability
Tic-Tac-ToePerfect2SequentialZero-sum4Minimax / Optimal Policy
Connect FourPerfect2SequentialZero-sum14AlphaZero / MCTS Win Rate
BreakthroughPerfect2SequentialZero-sum25AlphaZero / Elo Rating
Go (19x19)Perfect2SequentialZero-sum170AlphaZero / Elo Rating
Matrix Prisoner's DilemmaImperfect2SimultaneousGeneral-sum1Social Welfare / Defection Rate
HanabiImperfect5SequentialCooperative13Average Cooperative Score

Sources

Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about OpenSpiel: Multi-Agent Reinforcement Learning and Game Theory Guide

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →