DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning

Published in

International Conference on Machine Learning(2021)

External Links:

Generate Graph

TL;DR

Through building DouZero, it is shown that classic Monte-Carlo methods can be made to deliver strong results in a hard domain with a complex action space, and the code and an online demo are released with the hope that this insight could motivate future work.

Abstract

Games are abstractions of the real world, where artificial agents learn to compete and cooperate with other agents. While significant achievements have been made in various perfect- and imperfect-information games, DouDizhu (a.k.a. Fighting the Landlord), a three-player card game, is still unsolved. DouDizhu is a very challenging domain with competition, collaboration, imperfect information, large state space, and particularly a massive set of possible actions where the legal actions vary significantly from turn to turn. Unfortunately, modern reinforcement learning algorithms mainly focus on simple and small action spaces, and not surprisingly, are shown not to make satisfactory progress in DouDizhu. In this work, we propose a conceptually simple yet effective DouDizhu AI system, namely DouZero, which enhances traditional Monte-Carlo methods with deep neural networks, action encoding, and parallel actors. Starting from scratch in a single server with four GPUs, DouZero outperformed all the existing DouDizhu AI programs in days of training and was ranked the first in the Botzone leaderboard among 344 AI agents. Through building DouZero, we show that classic Monte-Carlo methods can be made to deliver strong results in a hard domain with a complex action space. The code and an online demo are released at this https URL with the hope that this insight could motivate future work.

Authors

Ji Liu

7 papers

D. Zha

5 papers

Xia Hu

5 papers

References60 items

Simplifying Deep Reinforcement Learning via Self-Supervision

Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments

OpenHoldem: An Open Toolkit for Large-Scale Imperfect-Information Game Research

Towards Playing Full MOBA Games with Deep Reinforcement Learning

Human-Level Performance in No-Press Diplomacy via Equilibrium Search

DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning

Published in

International Conference on Machine Learning(2021)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Ji Liu

7 papers

D. Zha

5 papers

Xia Hu

5 papers

References60 items

Simplifying Deep Reinforcement Learning via Self-Supervision

Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments

OpenHoldem: An Open Toolkit for Large-Scale Imperfect-Information Game Research

Towards Playing Full MOBA Games with Deep Reinforcement Learning

Human-Level Performance in No-Press Diplomacy via Equilibrium Search

Sheng Zhang

1 papers

Xiangru Lian

1 papers

PettingZoo: Gym for Multi-Agent Reinforcement Learning

Combining Deep Reinforcement Learning and Search for Imperfect-Information Games

Revisiting Fundamentals of Experience Replay

Dual Policy Distillation

Suphx: Mastering Mahjong with Deep Reinforcement Learning

Mastering Complex Control in MOBA Games with Deep Reinforcement Learning

Dota 2 with Large Scale Deep Reinforcement Learning

Improving Policies via Search in Cooperative Partially Observable Games

Mastering Atari, Go, chess and shogi by planning with a learned model

Grandmaster level in StarCraft II using multi-agent reinforcement learning

RLCard: A Toolkit for Reinforcement Learning in Card Games

TorchBeast: A PyTorch Platform for Distributed RL

OpenSpiel: A Framework for Reinforcement Learning in Games

DeltaDou: Expert-level Doudizhu AI through Self-play

Superhuman AI for multiplayer poker

Experience Replay Optimization

Combinational Q-Learning for Dou Di Zhu

A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play

Deep Counterfactual Regret Minimization

Generative Adversarial Self-Imitation Learning

Solving Imperfect-Information Games via Discounted Regret Minimization

Botzone: an online multi-agent competitive platform for AI education

Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning

Simple random search provides a competitive approach to reinforcement learning

Modeling Others using Oneself in Multi-Agent Reinforcement Learning

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Superhuman AI for heads-up no-limit poker: Libratus beats top professionals

A Deeper Look at Experience Replay

A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning

Proximal Policy Optimization Algorithms

DeepStack: Expert-level artificial intelligence in heads-up no-limit poker

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Deep Reinforcement Learning from Self-Play in Imperfect-Information Games

Asynchronous Methods for Deep Reinforcement Learning

Mastering the game of Go with deep neural networks and tree search

Deep Reinforcement Learning in Large Discrete Action Spaces

Deep Residual Learning for Image Recognition

Deep Reinforcement Learning with Double Q-Learning

Fictitious Self-Play in Extensive-Form Games

Human-level control through deep reinforcement learning

Heads-up limit hold’em poker is solved

Generalized Sampling and Variance in Counterfactual Regret Minimization

Double Q-learning

Algorithms for Reinforcement Learning

Monte Carlo Sampling for Regret Minimization in Extensive Games

Regret Minimization in Games with Incomplete Information

Cooperative Multi-Agent Learning: The State of the Art

Self-improving reactive agents based on reinforcement learning, planning and teaching

Seed rl: Scalable and efficient deep-rl with accelerated central inference

Mastering DouDizhu with Self-Play learning

Comparing Value-Function Estimation Algorithms in Undiscounted Problems

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Issues in Using Function Approximation for Reinforcement Learning

Reinforcement Learning: An Introduction

Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence Accelerating Best Response Calculation in Large Extensive Games

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

International Conference on Machine Learning

Type

conference

URL

https://icml.cc/

Alternate Names

ICML
Int Conf Mach Learn