• Below are the benchmark results of AlphaZero and MuZero on three board games: TicTacToe, Connect4, Gomoku.

• Below are the benchmark results of MuZero, MuZero w/ SSL , EfficientZero and Sampled EfficientZero on three discrete action space games in Atari.

• Below are the benchmark results of Sampled EfficientZero with Factored/Gaussian policy representation on three classic continuous action space games: Pendulum-v1, LunarLanderContinuous-v2, BipedalWalker-v3 and two MuJoCo continuous action space games: Hopper-v3, Walker2d-v3.

"Factored Policy" indicates that the agent learns a policy network that outputs a categorical distribution. After manual discretization, the dimensions of the action space for the five environments are 11, 49 (7^2), 256 (4^4), 64 (4^3), and 4096 (4^6), respectively. On the other hand, "Gaussian Policy" refers to the agent learning a policy network that directly outputs parameters (mu and sigma) for a Gaussian distribution.

• Below are the benchmark results of GumbelMuZero and MuZero (under different simulation cost) on four environments: PongNoFrameskip-v4, MsPacmanNoFrameskip-v4, Gomoku, and LunarLanderContinuous-v2.

• Below are the benchmark results of StochasticMuZero and MuZero on 2048 environment with varying levels of chance (num_chances=2 and 5).

• Below are the benchmark results of various MCTS exploration mechanisms of MuZero w/ SSL in the MiniGrid environment.

• AlphaZero
• MuZero
• EfficientZero
• SampledMuZero
• GumbelMuZero
• StochasticMuZero
• NotationTable

• MCTS
• AlphaZero
• MuZero
• EfficientZero
• SampledMuZero
• GumbelMuZero

LightZero Implemented series

• 2018 Science AlphaZero: A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
• 2019 MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• 2021 EfficientZero: Mastering Atari Games with Limited Data
• 2021 Sampled MuZero: Learning and Planning in Complex Action Spaces
• 2022 Stochastic MuZero: Planning in Stochastic Environments with A Learned Model
• 2022 Gumbel MuZero: Policy Improvement by Planning with Gumbel

AlphaGo series

• 2015 Nature AlphaGo Mastering the game of Go with deep neural networks and tree search
• 2017 Nature AlphaGo Zero Mastering the game of Go without human knowledge
• 2019 ELF OpenGo: An Analysis and Open Reimplementation of AlphaZero
  • Code
• 2023 Student of Games: A unified learning algorithm for both perfect and imperfect information games

MuZero series

• 2022 Online and Offline Reinforcement Learning by Planning with a Learned Model
• 2021 Vector Quantized Models for Planning
• 2021 Muesli: Combining Improvements in Policy Optimization.

MCTS Analysis

• 2020 Monte-Carlo Tree Search as Regularized Policy Optimization
• 2021 Self-Consistent Models and Values
• 2022 Adversarial Policies Beat Professional-Level Go AIs
• 2022 PNAS Acquisition of Chess Knowledge in AlphaZero.

MCTS Application

• 2023 Symbolic Physics Learner: Discovering governing equations via Monte Carlo tree search
• 2022 Nature Discovering faster matrix multiplication algorithms with reinforcement learning
  • Code
• 2022 MuZero with Self-competition for Rate Control in VP9 Video Compression
• 2021 DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning
• 2019 Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving

ICML

• Scalable Safe Policy Improvement via Monte Carlo Tree Search 2023
  • Alberto Castellini, Federico Bianchi, Edoardo Zorzi, Thiago D. Simão, Alessandro Farinelli, Matthijs T. J. Spaan
  • Key: safe policy improvement online using a MCTS based strategy, Safe Policy Improvement with Baseline Bootstrapping
  • ExpEnv: Gridworld and SysAdmin
• Efficient Learning for AlphaZero via Path Consistency 2022
  • Dengwei Zhao, Shikui Tu, Lei Xu
  • Key: limited amount of self-plays, path consistency (PC) optimality
  • ExpEnv: Go, Othello, Gomoku
• Visualizing MuZero Models 2021
  • Joery A. de Vries, Ken S. Voskuil, Thomas M. Moerland, Aske Plaat
  • Key: visualizing the value equivalent dynamics model, action trajectories diverge, two regularization techniques
  • ExpEnv: CartPole and MountainCar.
• Convex Regularization in Monte-Carlo Tree Search 2021
  • Tuan Dam, Carlo D'Eramo, Jan Peters, Joni Pajarinen
  • Key: entropy-regularization backup operators, regret analysis, Tsallis etropy,
  • ExpEnv: synthetic tree, Atari
• Information Particle Filter Tree: An Online Algorithm for POMDPs with Belief-Based Rewards on Continuous Domains 2020
  • Johannes Fischer, Ömer Sahin Tas
  • Key: Continuous POMDP, Particle Filter Tree, information-based reward shaping, Information Gathering.
  • ExpEnv: POMDPs.jl framework
  • Code
• Retro*: Learning Retrosynthetic Planning with Neural Guided A* Search 2020
  • Binghong Chen, Chengtao Li, Hanjun Dai, Le Song
  • Key: chemical retrosynthetic planning, neural-based A*-like algorithm, ANDOR tree
  • ExpEnv: USPTO datasets
  • Code

ICLR

• The Update Equivalence Framework for Decision-Time Planning 2024
  • Samuel Sokota, Gabriele Farina, David J Wu, Hengyuan Hu, Kevin A. Wang, J Zico Kolter, Noam Brown
  • Key: imperfect-information games, search, decision-time planning, update equivalence
  • ExpEnv: Hanabi, 3x3 Abrupt Dark Hex and Phantom Tic-Tac-Toe
• Efficient Multi-agent Reinforcement Learning by Planning 2024
  • Qihan Liu, Jianing Ye, Xiaoteng Ma, Jun Yang, Bin Liang, Chongjie Zhang
  • Key: multi-agent reinforcement learning, planning, multi-agent MCTS
  • ExpEnv: SMAC, LunarLander, MuJoCo, and Google Research Football
• Become a Proficient Player with Limited Data through Watching Pure Videos 2023
  • Weirui Ye, Yunsheng Zhang, Pieter Abbeel, Yang Gao
  • Key: pre-training from action-free videos, forward-inverse cycle consistency (FICC) objective based on vector quantization, pre-training phase, fine-tuning phase.
  • ExpEnv: Atari
• Policy-Based Self-Competition for Planning Problems 2023
  • Jonathan Pirnay, Quirin Göttl, Jakob Burger, Dominik Gerhard Grimm
  • Key: self-competition, find strong trajectories by planning against possible strategies of its past self.
  • ExpEnv: Traveling Salesman Problem and the Job-Shop Scheduling Problem.
• Explaining Temporal Graph Models through an Explorer-Navigator Framework 2023
  • Wenwen Xia, Mincai Lai, Caihua Shan, Yao Zhang, Xinnan Dai, Xiang Li, Dongsheng Li
  • Key: Temporal GNN Explainer, an explorer to find the event subsets with MCTS, a navigator that learns the correlations between events and helps reduce the search space.
  • ExpEnv: Wikipedia and Reddit, Synthetic datasets
• SpeedyZero: Mastering Atari with Limited Data and Time 2023
  • Yixuan Mei, Jiaxuan Gao, Weirui Ye, Shaohuai Liu, Yang Gao, Yi Wu
  • Key: distributed RL system, Priority Refresh, Clipped LARS
  • ExpEnv: Atari
• Efficient Offline Policy Optimization with a Learned Model 2023
  • Zichen Liu, Siyi Li, Wee Sun Lee, Shuicheng YAN, Zhongwen Xu
  • Key: Regularized One-Step Model-based algorithm for Offline-RL
  • ExpEnv: Atari，BSuite
  • Code
• Enabling Arbitrary Translation Objectives with Adaptive Tree Search 2022
  • Wang Ling, Wojciech Stokowiec, Domenic Donato, Chris Dyer, Lei Yu, Laurent Sartran, Austin Matthews
  • Key: adaptive tree search, translation models, autoregressive models,
  • ExpEnv: Chinese–English and Pashto–English tasks from WMT2020, German–English from WMT2014
• What's Wrong with Deep Learning in Tree Search for Combinatorial Optimization 2022
  • Maximili1an Böther, Otto Kißig, Martin Taraz, Sarel Cohen, Karen Seidel, Tobias Friedrich
  • Key: combinatorial optimization, open-source benchmark suite for the NP-hard maximum independent set problem, an in-depth analysis of the popular guided tree search algorithm, compare the tree search implementations to other solvers
  • ExpEnv: NP-hard MAXIMUM INDEPENDENT SET.
  • Code
• Monte-Carlo Planning and Learning with Language Action Value Estimates 2021
  • Youngsoo Jang, Seokin Seo, Jongmin Lee, Kee-Eung Kim
  • Key: Monte-Carlo tree search with language-driven exploration, locally optimistic language value estimates.
  • ExpEnv: Interactive Fiction (IF) games
• Practical Massively Parallel Monte-Carlo Tree Search Applied to Molecular Design 2021
  • Xiufeng Yang, Tanuj Kr Aasawat, Kazuki Yoshizoe
  • Key: massively parallel Monte-Carlo Tree Search, molecular design, Hash-driven parallel search,
  • ExpEnv: octanol-water partition coefficient (logP) penalized by the synthetic accessibility (SA) and large Ring Penalty score.
• Watch the Unobserved: A Simple Approach to Parallelizing Monte Carlo Tree Search 2020
  • Anji Liu, Jianshu Chen, Mingze Yu, Yu Zhai, Xuewen Zhou, Ji Liu
  • Key: parallel Monte-Carlo Tree Search, partition the tree into sub-trees efficiently, compare the observation ratio of each processor.
  • ExpEnv: speedup and performance comparison on JOY-CITY game, average episode return on atari game
  • Code
• Learning to Plan in High Dimensions via Neural Exploration-Exploitation Trees 2020
  • Binghong Chen, Bo Dai, Qinjie Lin, Guo Ye, Han Liu, Le Song
  • Key: meta path planning algorithm, exploits a novel neural architecture which can learn promising search directions from problem structures.
  • ExpEnv: a 2d workspace with a 2 DoF (degrees of freedom) point robot, a 3 DoF stick robot and a 5 DoF snake robot

NeurIPS

• LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios 2023
  • Yazhe Niu, Yuan Pu, Zhenjie Yang, Xueyan Li, Tong Zhou, Jiyuan Ren, Shuai Hu, Hongsheng Li, Yu Liu
  • Key: the first unified benchmark for deploying MCTS/MuZero in general sequential decision scenarios.
  • ExpEnv: ClassicControl, Box2D, Atari, MuJoCo, GoBigger, MiniGrid, TicTacToe, ConnectFour, Gomoku, 2048, etc.
• Large Language Models as Commonsense Knowledge for Large-Scale Task Planning 2023
  • Zirui Zhao, Wee Sun Lee, David Hsu
  • Key: world model (LLM) and the LLM-induced policy can be combined in MCTS, to scale up task planning.
  • ExpEnv: multiplication, travel planning, object rearrangement
• Monte Carlo Tree Search with Boltzmann Exploration 2023
  • Michael Painter, Mohamed Baioumy, Nick Hawes, Bruno Lacerda
  • Key: Boltzmann exploration with MCTS, optimal actions for the maximum entropy objective do not necessarily correspond to optimal actions for the original objective, two improved algorithms.
  • ExpEnv: the Frozen Lake environment, the Sailing Problem, Go
• Generalized Weighted Path Consistency for Mastering Atari Games 2023
  • Dengwei Zhao, Shikui Tu, Lei Xu
  • Key: Generalized Weighted Path Consistency, A weighting mechanism.
  • ExpEnv: Atari
• Accelerating Monte Carlo Tree Search with Probability Tree State Abstraction 2023
  • Yangqing Fu, Ming Sun, Buqing Nie, Yue Gao
  • Key: probability tree state abstraction, transitivity and aggregation error bound
  • ExpEnv: Atari, CartPole, LunarLander, Gomoku
• Spending Thinking Time Wisely: Accelerating MCTS with Virtual Expansions 2022
  • Weirui Ye, Pieter Abbeel, Yang Gao
  • Key: trade off computation versus performancem, virtual expansions, spend thinking time adaptively.
  • ExpEnv: Atari, 9x9 Go
• Planning for Sample Efficient Imitation Learning 2022
  • Zhao-Heng Yin, Weirui Ye, Qifeng Chen, Yang Gao
  • Key: Behavioral Cloning，Adversarial Imitation Learning (AIL)，MCTS-based RL.
  • ExpEnv: DeepMind Control Suite
  • Code
• Evaluation Beyond Task Performance: Analyzing Concepts in AlphaZero in Hex 2022
  • Charles Lovering, Jessica Zosa Forde, George Konidaris, Ellie Pavlick, Michael L. Littman
  • Key: AlphaZero’s internal representations, model probing and behavioral tests, how these concepts are captured in the network.
  • ExpEnv: Hex
• Are AlphaZero-like Agents Robust to Adversarial Perturbations? 2022
  • Li-Cheng Lan, Huan Zhang, Ti-Rong Wu, Meng-Yu Tsai, I-Chen Wu, 4 Cho-Jui Hsieh
  • Key: adversarial states, first adversarial attack on Go AIs.
  • ExpEnv: Go
• Monte Carlo Tree Descent for Black-Box Optimization 2022
  • Yaoguang Zhai, Sicun Gao
  • Key: Black-Box Optimization, how to further integrate samplebased descent for faster optimization.
  • ExpEnv: synthetic functions for nonlinear optimization, reinforcement learning problems in MuJoCo locomotion environments, and optimization problems in Neural Architecture Search (NAS).
• Monte Carlo Tree Search based Variable Selection for High Dimensional Bayesian Optimization 2022
  • Lei Song∗ , Ke Xue∗ , Xiaobin Huang, Chao Qian
  • Key: a low-dimensional subspace via MCTS, optimizes in the subspace with any Bayesian optimization algorithm.
  • ExpEnv: NAS-bench problems and MuJoCo locomotion
• Monte Carlo Tree Search With Iteratively Refining State Abstractions 2021
  • Samuel Sokota, Caleb Ho, Zaheen Ahmad, J. Zico Kolter
  • Key: stochastic environments, Progressive widening, abstraction refining
  • ExpEnv: Blackjack, Trap, five by five Go.
• Deep Synoptic Monte Carlo Planning in Reconnaissance Blind Chess 2021
  • Gregory Clark
  • Key: imperfect information, belief state with an unweighted particle filter, a novel stochastic abstraction of information states.
  • ExpEnv: reconnaissance blind chess
• POLY-HOOT: Monte-Carlo Planning in Continuous Space MDPs with Non-Asymptotic Analysis 2020
  • Weichao Mao, Kaiqing Zhang, Qiaomin Xie, Tamer Ba¸sar
  • Key: continuous state-action spaces, Hierarchical Optimistic Optimization.
  • ExpEnv: CartPole, Inverted Pendulum, Swing-up, and LunarLander.
• Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search 2020
  • Linnan Wang, Rodrigo Fonseca, Yuandong Tian
  • Key: learns the partition of the search space using a few samples, a nonlinear decision boundary and learns a local model to pick good candidates.
  • ExpEnv: MuJoCo locomotion tasks, Small-scale Benchmarks,
• Mix and Match: An Optimistic Tree-Search Approach for Learning Models from Mixture Distributions 2020
  • Matthew Faw, Rajat Sen, Karthikeyan Shanmugam, Constantine Caramanis, Sanjay Shakkottai
  • Key: covariate shift problem, Mix&Match combines stochastic gradient descent (SGD) with optimistic tree search and model re-use (evolving partially trained models with samples from different mixture distributions)
  • Code

Other Conference or Journal

• Learning to Stop: Dynamic Simulation Monte-Carlo Tree Search AAAI 2021.
• On Monte Carlo Tree Search and Reinforcement Learning Journal of Artificial Intelligence Research 2017.
• Sample-Efficient Neural Architecture Search by Learning Actions for Monte Carlo Tree Search IEEE Transactions on Pattern Analysis and Machine Intelligence 2022.