Multi-Agent Collaboration via Reward Attribution Decomposition (2020-10-16T00:00:00.000000Z)

TL;DR

CollaQ is proposed, a Collaborative Q-learning that achieves state-of-the-art performance in the StarCraft multi-agent challenge and supports ad hoc team play and outperforms previous SoTA by over 30%.

Abstract

Recent advances in multi-agent reinforcement learning (MARL) have achieved super-human performance in games like Quake 3 and Dota 2. Unfortunately, these techniques require orders-of-magnitude more training rounds than humans and don't generalize to new agent configurations even on the same game. In this work, we propose Collaborative Q-learning (CollaQ) that achieves state-of-the-art performance in the StarCraft multi-agent challenge and supports ad hoc team play. We first formulate multi-agent collaboration as a joint optimization on reward assignment and show that each agent has an approximately optimal policy that decomposes into two parts: one part that only relies on the agent's own state, and the other part that is related to states of nearby agents. Following this novel finding, CollaQ decomposes the Q-function of each agent into a self term and an interactive term, with a Multi-Agent Reward Attribution (MARA) loss that regularizes the training. CollaQ is evaluated on various StarCraft maps and shows that it outperforms existing state-of-the-art techniques (i.e., QMIX, QTRAN, and VDN) by improving the win rate by 40% with the same number of samples. In the more challenging ad hoc team play setting (i.e., reweight/add/remove units without re-training or finetuning), CollaQ outperforms previous SoTA by over 30%.

Authors

K. Keutzer

18 papers

Joseph E. Gonzalez

10 papers

Xiaolong Wang

9 papers

TL;DR

Abstract

Authors

References47 items

Generating and Adapting to Diverse Ad-Hoc Cooperation Agents in Hanabi

Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning

"Other-Play" for Zero-Shot Coordination

Dota 2 with Large Scale Deep Reinforcement Learning

Grandmaster level in StarCraft II using multi-agent reinforcement learning

A Structured Prediction Approach for Generalization in Cooperative Multi-Agent Reinforcement Learning

Efficient Bimanual Manipulation Using Learned Task Schemas

Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games

Emergent Tool Use From Multi-Agent Autocurricula

Robust Multi-Agent Reinforcement Learning via Minimax Deep Deterministic Policy Gradient

Superhuman AI for multiplayer poker

Learning to Interactively Learn and Assist

Finding Friend and Foe in Multi-Agent Games

QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning

The StarCraft Multi-Agent Challenge

Benchmarks for reinforcement learning in mixed-autonomy traffic

Actor-Attention-Critic for Multi-Agent Reinforcement Learning

M^3RL: Mind-aware Multi-agent Management Reinforcement Learning

CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning

Zero Shot Transfer Learning for Robot Soccer

Human-level performance in 3D multiplayer games with population-based reinforcement learning

QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning

Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments

Proximal Policy Optimization Algorithms

Value-Decomposition Networks For Cooperative Multi-Agent Learning

Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

Counterfactual Multi-Agent Policy Gradients

Emergence of Grounded Compositional Language in Multi-Agent Populations

Multi-agent Reinforcement Learning in Sequential Social Dilemmas

Opponent Modeling in Deep Reinforcement Learning

Learning Multiagent Communication with Backpropagation

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Cooperating with Unknown Teammates in Complex Domains: A Robot Soccer Case Study of Ad Hoc Teamwork

Playing Atari with Deep Reinforcement Learning

Cooperating with a markovian ad hoc teammate

Learning Teammate Models for Ad Hoc Teamwork

Empirical evaluation of ad hoc teamwork in the pursuit domain

Ad Hoc Autonomous Agent Teams: Collaboration without Pre-Coordination

A Comprehensive Survey of Multiagent Reinforcement Learning

Coordination and Adaptation in Impromptu Teams

Markov Games as a Framework for Multi-Agent Reinforcement Learning

Credit Assignment For Collective Multiagent RL With Global Rewards

Half Field Offense: An Environment for Multiagent Learning and Ad Hoc Teamwork

Under Review as a Conference Paper at Iclr 2017 Delving into Transferable Adversarial Ex- Amples and Black-box Attacks

Multi Agent Reinforcement Learning Independent vs Cooperative Agents

Temporal credit assignment in reinforcement learning

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names