Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi (2023-08-20T00:00:00.000000Z)

TL;DR

This work formally defined a framework based on a popular cooperative multi-agent game called Hanabi to evaluate the adaptability of MARL methods and defined a new metric called adaptation regret that measures the agent's ability to efficiently adapt and improve its coordination performance when paired with some held-out pool of partners on top of its ZSC performance.

Abstract

Cooperative Multi-agent Reinforcement Learning (MARL) algorithms with Zero-Shot Coordination (ZSC) have gained significant attention in recent years. ZSC refers to the ability of agents to coordinate zero-shot (without additional interaction experience) with independently trained agents. While ZSC is crucial for cooperative MARL agents, it might not be possible for complex tasks and changing environments. Agents also need to adapt and improve their performance with minimal interaction with other agents. In this work, we show empirically that state-of-the-art ZSC algorithms have poor performance when paired with agents trained with different learning methods, and they require millions of interaction samples to adapt to these new partners. To investigate this issue, we formally defined a framework based on a popular cooperative multi-agent game called Hanabi to evaluate the adaptability of MARL methods. In particular, we created a diverse set of pre-trained agents and defined a new metric called adaptation regret that measures the agent's ability to efficiently adapt and improve its coordination performance when paired with some held-out pool of partners on top of its ZSC performance. After evaluating several SOTA algorithms using our framework, our experiments reveal that naive Independent Q-Learning (IQL) agents in most cases adapt as quickly as the SOTA ZSC algorithm Off-Belief Learning (OBL). This finding raises an interesting research question: How to design MARL algorithms with high ZSC performance and capability of fast adaptation to unseen partners. As a first step, we studied the role of different hyper-parameters and design choices on the adaptability of current MARL algorithms. Our experiments show that two categories of hyper-parameters controlling the training data diversity and optimization process have a significant impact on the adaptability of Hanabi agents.

References37 items

K-level Reasoning for Zero-Shot Coordination in Hanabi

On-the-fly Strategy Adaptation for ad-hoc Agent Coordination

Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination

Deep Reinforcement Learning at the Edge of the Statistical Precipice

Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi

A New Formalism, Method and Open Issues for Zero-Shot Coordination

Trajectory Diversity for Zero-Shot Coordination

Off-Belief Learning

Continuous Coordination As a Realistic Scenario for Lifelong Learning

The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning

Too Many Cooks: Bayesian Inference for Coordinating Multi-Agent Collaboration

"Other-Play" for Zero-Shot Coordination

Google Research Football: A Novel Reinforcement Learning Environment

Diverse Agents for Ad-Hoc Cooperation in Hanabi

The StarCraft Multi-Agent Challenge

The Hanabi Challenge: A New Frontier for AI Research

Recurrent Experience Replay in Distributed Reinforcement Learning

Learning Existing Social Conventions via Observationally Augmented Self-Play

Distributed Prioritized Experience Replay

Prosocial learning agents solve generalized Stag Hunts better than selfish ones

Value-Decomposition Networks For Cooperative Multi-Agent Learning

Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

Emergence of Grounded Compositional Language in Multi-Agent Populations

A Concise Introduction to Decentralized POMDPs

Learning Multiagent Communication with Backpropagation

Dueling Network Architectures for Deep Reinforcement Learning

Prioritized Experience Replay

Deep Reinforcement Learning with Double Q-Learning

Illuminating search spaces by mapping elites

Human-level control through deep reinforcement learning

A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning

Ad Hoc Autonomous Agent Teams: Collaboration without Pre-Coordination

Coordination and Adaptation in Impromptu Teams

Taming Decentralized POMDPs: Towards Efficient Policy Computation for Multiagent Settings

The Complexity of Decentralized Control of Markov Decision Processes

Making friends on the fly: Cooperating with new teammates

Multi Agent Reinforcement Learning Independent vs Cooperative Agents